I don't know how much I buy into that. Every Monday for the past 5 years I've come into an office that loudly talks about the episode from the night before. Do I notice the conversations because I watch the show? Maybe....but I feel I'd definitely hear a lot in passing too.
Plus Instagram and Facebook pages not even associated with GOT post all about it now.
You have to understand that people outside the bubble have absolutely no clue what you are on about in the slightest. They know nothing at all. You can make the most blatant reference right to their face and they'll just look at you blankly. Nothing you can possibly say will have any meaning to them at all. There is not even the tiniest bit of understanding to extrapolate from to make anything you'll say make sense without someone deliberately giving them the context to let them understand.
While you'll hear references in passing a lot, you won't be able to place them at all. You won't even be able to connect them together as being conversations about the same thing. If you don't watch then you don't know any key phrases for your brain to latch on to and start paying attention. It's similar to how you can be in a crowed room, but you'll here absolutely nothing until someone says your name and you can suddenly pick out that person's voice.
It's the same on social media, the posts are just more crap that you are uninterested and you ignore it the same way you ignore baby photos, inspirational quotes, and rants from that strange uncle.
While I imaging being in an office is annoying, I don't think it's a big deal. Even if you can't avoid people talking about it, if you are not interested you're not going to pay any real attention or remember any of it the next day. Think about a conversation a girlfriend has had about handbags. You probably have no clue what was said at all, even if you were the other half of the conversation(!).