No, we know the count with 4 decks remaining. We leave and there are two decks remaining.
Most people would say keep the RC, and just assume the 2 decks that were dealt are basically unseen cards. So, e.g.:
With 4 decks left, TC +4, RC + 16.
Come back with two decks remaining, assume RC +16, and assume two decks that were dealt are unseen - so RC is still +16, TC is still +4, and still are at 4 decks.
I would argue, since its valid to assume the high card are evenly distributed, when we come back to the 2 remaining decks, can assume TC is +4, RC is +8, and work under the assumption of 2 decks.