Tangent from the Beck Malenstyn stuff - Beck is a textbook example of why with/without models are flawed. The RMNB article gives this player card from Evolving Hockey which evaluates him as a bad offensive player and an atrocious defender.
At the same time, it rates Dowd and NAK as both being average offensively and elite (90+) defensively. It thinks Beck was the passenger on that line, but in reality
Beck was the player that defined the shutdown line. You can see through
Natural Stat Trick's Line Stats tool that when that line was together, they got 7.65% OZ Faceoffs. So what happens when they break up?
- NAK solo: 53.1%
- Dowd solo: 35.7%
- NAK+Dowd without Beck: 35.3%
- Beck solo: 20.4%
For some league-wide context, if Malenstyn had 20.4% for the season that would drop his rank amongst forwards from 1/771 to 8/771, min 100 minutes played - still 1st percentile for hardest deployments.
What does a with/without model see in this context? Well when Dowd and NAK play away Beck they get significantly easier deployments with more skilled teammates, so their numbers improve away from him. Meanwhile, when Beck's away from Dowd and NAK he's being asked to play the same shutdown role, but now with teammates who aren't as well suited to it, so those teammates who are normally being sheltered by Beck's line see their numbers plummet. The model interprets this as Beck being a drag on all the players around him, so it gives him low scores.
This is the fundamental issue with these kinds of models. They'll always underrate players who consistently get tough, defensively-focused deployments, and they'll always overrate players who consistently easy, offensively-focused deployments. Hence why these models always seem to love the sheltered third pairing offensive defenseman or the speedy young winger the coach doesn't trust in his own zone and hate the steady, D-first guys that coaches lean on. I think there's some value to these models for the guys in the middle, but it inherently can't handle the edge cases.