I'm not sure Varly got shit on. I think people gave up on him due to, as you said, his health. I don't think anyone said or believed he sucked though, it was just that "IT" sucked he couldn't stay healthy and thus be relied upon.
For me, I always viewed him the same way I viewed Frankie. I don't blame them for not having good health, but the fact that they didn't, hurt the Avs. Still, they both gave everything they had while playing for us and I'm not sure how anyone can't be satisfied with that.
For Starters, Varly is currently ranked #2 on my all-time Avs list behind Roy:
Roy > Varly > Darcy > Theo > Anderson > not going any further¹
Backups go:
Budaj² > Frankie² > Giggy² > Abby > Hamburgler > not going any further¹
¹No way in hell I'm ready to insert Blackwood or Wedgewood into these categories yet... but signs are promissing they will be pushing people down and out.
²Budaj vs GIggy vs Frankie is something I haven't really thought out in my mind. I think "my" two top spots for backups are Peter and Pavel , althouh not putting Giggy in there is so damn hard. Anyways my thoughts... Budaj played 242 games for us and was always an ultimate team and locker-room guy. Frankie, also a complete team guy who also just happened to sweep the Connor McDavid-Leon Draisaitl led Edmonton Oilers in the playoffs after our starter was injured. Giggy though... I think his voice inside the locker-room probably means more for the legacy of the organization this his performance on the ice... which was also admirable.