Since you like creatures of the sea check out Ripley's Aquarium, its right next to the CN Tower.
Not sure if its your first time, but people like to go see the view from the CN Tower, check out a Jays game, go to the aquarium (they are all right next to each other). HHOF is a block east away from the CN tower.
Maybe you want to see Casaloma or the ROM(its a 10 min bus ride/train ride north) away from the CN tower.
If you are coming from Midland, might want to pack it all in at the same time so you don't have to make the drive everyday to downtown. Avoid rushhour (6:30am-9:30am) and (4pm-7pm); its a waste of your time so plan around it. My suggestion is to park for the day and walk/travel around the city for the closest events you are going to.
Plenty of good restaurants all over the city, but a nice and expensive area to get great food/culture is the 'distillery district'(reservations needed) which is a streetcar ride of 10 minutes east of the CN Tower. A cheaper place is St. Lawrence Market, great food at fair prices; this is located a couple blocks east away from the HHOF. Eaton Center is a 10 minute walk north of the HHOF. Around Eaton Center is old city hall(Toronto sign from the Pan AM games), and Dundas Square.
Center Island, Canada's Wonderland, the Zoo and Niagara Falls are separate day events for the family. Wonderland/Niagara/Zoo is not close to downtown FYI; Wonderland is about a 30 minute drive north of Toronto and Niagara is an hour drive west of Toronto and the Zoo is an 45 minute drive east of Toronto.
If you want to blow all your money/time do everything, but pick a few if you have the time/money)..
Suggested order of importance and time needed:
Day 1: Downtown: HHOF 2hours, Aquarium 4 hours, CN Tower 2 hours, Jays game 4 hours, ROM 4 hours, Casaloma 2 hours, Eaton Center 2 hours, City Hall 10 min, Dundas Square 10 min (This may need to be covered over a couple of days when you have kids but I would only suggest the first 5 events any ways)
Day 2: Niagara Falls(day event)
Day 3: Center Island(day event)
Day 4: Wonderland(day event)
Day 5: Zoo(day event)