1. By that "logic" every rule is garbage to someone.
2. Life isn't fair.
3. We're ... what, about halfway through the 2nd period? Then on average, there's been another 9 hand passes by the offense that have occurred, that the officials didn't call. Can you point any of them out?
Doesn't matter. As soon as "in the referee's discretion, it had an impact on the play" gets written into the rule book, it's yet another thing for people to bitch and whine about and we're left grasping for what a given referee thinks and why they interpret a play one way while a different referee interprets the same thing a different way, and ...
This way, it's a black-and-white rule. Period.