Great clubhouse guy, it takes a special kind of leader to juggle personalities like he did in the biggest market in the US for as long as he did. But not necessarily known as a tactician, he trusted his horses as you say, and he had some great horses. If there's one ingredient that that 2004 Yankee team desperately missed it was Andy Petitte, the consummate stopper in any playoff series they fell behind in.
Honestly though, by the end of the regular season that year, the Red Sox were the better team. It just took them until after the deadline to find themselves. Deeper bullpen, deeper lineup, specialized bench. Sox batters 1-9 were threats to hit, Yankees had a monster heart of the lineup but at the back of it they were relying on Ruben Sierra, Tony Clark, and Miguel Cairo. They were even using Kenny Lofton as a DH some of those games for inexplicable reasons. They didn't have Giambi that year due to injuries, they acquired Olerud but he broke his foot in game 3, and they really missed the depth that Nick Johnson gave them the year before.
Ultimately it was their 3rd/4th starters that flipped the series. Derek Lowe in the regular season was a ticking time bomb of anxiety. In the post-season he nutted up big time and gave you two huge starts in that series to save you. Kevin Brown had a very good regular season but slowly lost composure all year culminating in punching the wall and breaking his hand in September. He had a very bad ALCS and Torre probably should've started Vazquez or El Duque in game 7 given the pressure they were under.
Having watched those games again, I get the feeling that if in the 2nd inning of game 7 Torre went to a real reliever like a Quantrill or even Gordon to get out of that bases loaded jam, even if they tack on a run and make it 3-0, the Yanks probably would've chipped away at that lead like the year before and then could turn it over to Vazquez or Loaiza for long relief starting with a clean inning. But bringing in Vazquez in cold to work out of the stretch with the bases loaded, in hindsight not his best decision. Not that I'm complaining.