Yeah, it would be a problem. Especially if the others assume that they're just joking about living there.
Gwen would have joined them to chat and to watch TV with them, whereas Tosh would either have flushed and run away until Ianto asked her to join them, or she just wouldn't have noticed they were there, bless her. (Yeah, I ran a scenario with each of them. This was my favourite.)
Gxxx