The best advice is to do what your heart tells you, but to also have a backup plan in case things don't go your way. I was told this advice by someone very high up in the NHL (see below for the name).
I've always dreamed of working in sports and, a couple years ago, got accepted into a masters program in the US for sports management. I didn't end up going (too bloody expensive), so I put it on hold... but in the background I've been working in marketing and making my career that way. Sports was always my dream regardless.
My contact gave me some sage advice - the business side of sports is incredibly difficult to break into - people 'fight like wounded tigers' to hold on to their jobs. It's also a lot about 'who you know', not 'what you know'. You should always pursue your dreams, but realize sports is a bit of an odd industry in that people think it's glamorous, and then find out often that it really isn't once they're in. If you do pursue sports management and you don't have an athletic background with established contacts, perhaps go the MBA or law route; at least, with degrees like these, you can always find work elsewhere if the sports dream doesn't work out (which is a very realistic possibility). He specifically mentioned law as something that would be very beneficial.
My contact is Brian Burke.