I think I understand what you're saying, but ultimately I disagree.
If we call out only blatant racism, then frankly, racists can use subtext to intimidate the minorities they hate, simply because it's not overt. As it is, there are racists who use the term Chinese virus to perpetuate their attitudes and there are people, like K17, who use it for, lack of a better term, preference. Have a Chinese person hear it, and they will still feel targeted/intimidated all the same, with good reason. Sure, K17 has never said "I hate Chinese people" but if a term is used in a casual attack, no matter what, it brings people up on the defense. And it's AVOIDABLE.
I'm white and male as can be. I've been fortunate in my life where I haven't had to endure racism, sexism, age-ism, able-ism, or any other -ism. I'm someone who won't suffer from the "as long as it's not overt, it's not a problem" mentality. It still perpetuates this cycle of mistreatment and cynicism, and when the affected parties get frustrated, it reinforces the attitude of "this group of people is a problem"
I'm trying not to have a sociology lecture here, but as RJ pointed out, he has Middle Eastern descent. I'm sure he can provide you multiple times in his life he's had subliminal racism thrown his way. If you have a female partner, I'm sure she can tell you about sexism she's endured, even if it's just making a little bit less money than a male doing the same job.
Granted, just because something is unfair doesn't mean it needs to be labeled as discriminatory, nor should non-PC people be called as racists - but if you use a word or gesture, knowing the context of it (whether or not YOU feel it), you need to be aware of how people perceive it. Whether or not you care about the perception is up to you.