Talent was more condensed because of the small number of teams and you were playing stars like Bobby Hull, Gordie Howe, Geoffrion, Beliveau, Bathgate, etc on a nightly basis basically. Plante's Hart trophy beat Bobby Hull who had the 3rd season in NHL history with 50 goals so their level of play wasn't just "good for a goalie", they were competing with all-time greats. Johnny Bower, Bill Durnan and Frank Brimsek (all top 15 goalie according to theHoH section) all had careers that overlapped with them. Basically, imagine the NHL today if you only had 6 teams and all the top Canadiens/Americans were forced to compete for 120 roster spots. Expansion tends to actually dilute the talent when you look at how scoring increased during the 70s and 93 for example. This is about legacy and all time rankings, you judge players relative to their peers at the time. They have just as much of an argument as Brodeur.
While Brodeur's career was overlapped with Roy and Hasek, he didn't win his first Vezina until Hasek missed a season and Roy was in his last season (Roy actually had a better sv% that season btw). You can argue the quality of competition for Brodeur was even lesser than Hall, Sawchuk and Plante when he won his vezinas. And even so, he was actually outperformed by players like Turco, Luongo and Kiprusoff in some of his Vezina seasons when you take wins out of the equation.