I hear this all the time, and maybe it's a joke, but the two sports are dramatically different in the way that contact occurs.
Blocking is illegal in rugby, and blocking is responsible for a lot of football injuries: head injuries, accidental or intentional chop blocks, and so on.
The forward pass is illegal in rugby, which means that the receiver hit over the middle is non-existent, and there's no quarterback standing in the pocket waiting to be crushed.
In fact, moving forward of the line of the ball itself is mostly illegal for offensive players, so players tend to be moving much more often in similar directions than in opposite directions.
The scrum is basically non-collision; when the referee puts the ball in, everyone engages in a big shoving mass according to very specific rules of engagement.
If I were to compare a sport to American football from a crazy danger perspective, that sport would be Aussie rules football. Those dudes hit each other at full speed from opposite directions all the time. Contact at full speed, including blindside contact, is absolutely routine. It's the one sport that has similar, if not worse, CTE rates as American football.
My guess is that if you remove pads from American football players, you end up with a game that's more like Aussie footy, which is to say, a different game that's just as dangerous.