Yes, hockey stats require context. That's why when posting R-ON/OFF data I've made comments such as the following:
"On the other hand, I recognized that looking strictly at the improvement (from when a player is on the ice to off the ice) is punitive to players who were on strong teams, like Lidstrom and Messier (which I talked about in my discussion of both)... As with any data I post, the intention is to start a conversation and maybe get people to consider things from a different perspective (which I think is happening). But I wouldn't take the data... as definitive."
"I think this metric might be somewhat unfair to him [Malkin]. Since he usually doesn't play with Crosby at ES, his R-OFF number contains a lot of Crosby's performance. You see the same thing with Mark Messier in Edmonton - his R-OFF contains a lot of Wayne Gretzky, so his ON/OFF ratio wasn't very good. I think, in a case like this, we can probably conclude that this metric isn't fair for Malkin (it wasn't fair for Messier either, until Gretzky was traded away)".
"Of course it's not a perfect metric -Guy Lapointe has 2nd worst absolute difference, and one of the 20 worst ratios, because we're comparing him to Robinson-Savard."
"A mixed bag, but more disappointing than not. Horton's R-on during the Leafs dynasty was objectively quite strong, but the R-Off was even stronger... Still, I don't think this is uncommon for primarily defensive defensemen (the results for Stevens and Langway look similar)."
"Messier's results aren't great... but we need to take a deeper look. He consistently had a strong R-ON ratio while he was on the ice, but he doesn't look great according to this metric because his teams generally had strong R-OFF ratios. Not surprising as, in many of these years, Gretzky and Kurri were dominating at even-strength."
"Bossy's numbers are pretty similar to Trottier's for the years they overlap... their R-ON and R-OFFs are virtually identical during the Drive for Five (not surprising as they were linemates). Bossy looks better than Trottier during 1978 and 1979 (pre dynasty). Their R-ON and R-OFF are, again, almost identical during Bossy's last three seasons... Lesson - R-ON ratios probably won't tell you much when comparing long-time linemates."
"I agree that Lemieux's ability to drive his team's goal differential at ES, although great, is much less than Orr and Gretzky's [based on R-ON/OFF data]... But Lemieux is almost certainly the greatest PP performer of all-time. PP goals, of course, count just as much as ones at even-strength."
"Remember that these numbers take into account ES play only - MacInnis was a major contributor on the powerplay and penalty kill."
"[re Chris Neil] - occasionally a depth player will post phenomenal results and will appear to be the best player on their team. This sometimes happens with players getting limited and sheltered minutes (so we have a combination of small sample sizes, which makes it more likely for there to be flukes, and easier-than-average match-ups, which means we don't have an apples-to-apples comparison). When we're comparing top-pairing players who are routinely playing 12-18 minutes at ES per game, generally facing challenging match-ups, we can probably compare them to each other (with the caveats I had before about how no stat is all-encompassing). But I wouldn't use this data to compare Alfredsson to Neil (Andre Roy and Wade Belak are two other players who have, surprisingly, phenomenal results by the metric)."
I'll refrain from posting other examples. I try to be careful - probably excessively so - in talking about the limitations, and key assumptions, for any stats that I'm presenting. (These are posts on a hockey message board... not a doctoral thesis).
Maybe the response to that is there's so much context that the stat should be discarded. But if the stat is showing that the greatest "two-way" players ever include Orr, Clarke, Howe, Forsberg, Crosby, Bergeron, Trottier, Bourque, Pronger, Mark Howe, Modano, Gilmour, Datsyuk, Francis, Kopitar, Zetterberg, etc- there's probably some value in it. (Plus some one-dimensional forwards like Gretzky, Jagr, Lindros and Hull - who were so good offensively, their lack of conventional defense didn't matter). And if a player looks surprisingly bad (ie Malkin, Messier and Lapointe) - let's try to understand why, rather than ignoring the data altogether.