It's literally personal taste. That's like saying someone who doesn't like seafood is wrong.
You either didn't really read what he said or you're misconstruing what he said to fit whatever argument you're trying to make here.
If he had said "I don't care about women's sports", he mightve gotten a bit of pushback but it would've went away. But that's not what he said, is it?
He's said multiple times that nobody cares about women's sports, people only pretend to, then challenges people to name players. This is, as Poke said, LITERALLY incorrect since viewership is smashing records right now. So no, your comparison about seafood doesn't work at all. It's actually really silly.
I personally could not care less about women's sports. I barely have enough time to follow men's sports. If someone were to call me a terrible person and try to get me fired for saying that, THEN your seafood analogy would work. You do see the difference, right?