Baseball, the sport itself, is huge in Florida and way bigger than hockey, with a huge portion of Big Leaguers coming from there. Also College Baseball is huge down there. UF, FSU, UM are some of the top baseball schoosl in the country with lots of success, and those schools and teams have obviously existed for much longer than Rays/Marlins.
Quite simply, the Rays and Marlins are still relatively 'new' teams compared to toher MLB teams and haven't done enough to build large fanbases.
Everyone knows Tropicana Field, the stadium itself and it's location are not ideal, but it doesn't come close to fully explaining the piss-poor attendance imo.
I think being in the AL East with Boston and NYY hurts too. Transplants who would otherwise latch on to the Rays as a 2nd/alternate team, can't because their big division rivals with their primary team.
Never quite having a Mike Trout/Aaron Juge/Albert Pujols to latch onto as the face of the franchise kinda hurts too