There's more to golf than just hitting the ball far. Shaping it, hitting off different lies etc all subject the body to different stress. During his prime, nobody hit the ball farther than Tiger. He could've modified his swing to take less pressure off his legs and reduce the torque on his back but I'm not in the business of questioning a legendary athlete. Daly, who had an incredibly brief period of success is a red herring people bring up to discredit golf, but he was never close to being one of the top golfers in the world.
Look at the general fitness of golfers today. Upper body wise, the top golfers are far more ripped than hockey players with similar body fat percentages.