Heat transfer by radiation complicates things. We lose the majority of our heat that way, and we'd lose a lot more if every cubic inch of the spaces we inhabit weren't flooded with thermal radiation from the objects that surround us.
You can absolutely judge the temperature of a volume, vacuum or not, by its radiation content at any given moment.
(it's an edit)