Micklebot
Moderator
- Apr 27, 2010
- 58,416
- 36,500
Cool stuff, I'm not a fan of the hockey ref point share model (it's a bit dated and imo favours players on good teams over equal players on bad teams) but every all in one model will have its flaws and this one is readily available which is always a big plus..As mentioned in the draft thread I've been tracking draft rankings in a spreadsheet since 2013 and was trying to figure out a way to evaluate how the various experts have performed. This is what I've come up with.
1. Evaluating from 2013-2019 as I started recording rankings in 2013 and anything past 2019 seems premature (do we really know what Quinton Byfield or Alexis Lafreniere is yet?)
2. Using the first round players (so 30 or 31 players depending on the year) from each list as some rankings I don't have further than that and we mostly care about the top of the draft
3. Evaluating based on the actual results of the players in the NHL - not where they were drafted as these lists aren't meant to be mock drafts but rather are how the expert thinks the player will perform. To achieve this I decided to use the career Point Share calculation from HockeyReference.com as it allows comparison across positions. This also means that players with similar results will have similar values.
4. Weight the results based on where the player was ranked - getting a good player in your ranking at #25 is fine, but if the player you put #2 was a bust that should count against you pretty heavily. I decided the simplest way to weight the rankings was by multiplying the point shares by the inverse of the slot - so for a year where the first round had 30 picks for the #1 ranked player's point shares are multiplied by 30, the #2 player by 29, etc. I'm not convinced that the top players in the ranking shouldn't be weighted even more heavily but this is at least a start.
5. For a baseline I then calculated the same value based on the actual draft positions for the year
6. To allow comparison across years I then took each list's score as a percentage of the baseline. So if the score for the actual draft was 9279 (the 2017 score) and Bob McKenzie's list scored 9204 he would end up with 9204/9279*100 = 99.2 as his score for that year. Above 100 means the list outperformed the actual draft, below 100 is worse.
My assumption going is was that Bob McKenzie would be right around the actual GM results as his list is an average of NHL scouts. I figured Craig Button and Corey Pronman would probably vary quite a bit from year to year as they tend to go against the consensus more than most. So, how did the various evaluators do?
Code:
View attachment 877206
As expected, McKenzie is generally right around 100. Future Considerations had a few very good years. Button had a terrible 2013 and excellent 2014 and 2016 rankings. Pronman was awful in 2017 and 2018. ISS is not good. Wheeler didn't do well, but it was only a 2 draft sample for him. Overall the NHL GMs get better results than the expert rankings.
Evaluating Past Draft Rankings - Calgarypuck Forums - The Unofficial Calgary Flames Fan Community
Evaluating Past Draft Rankings Fire on Ice: The Calgary Flames Forumforum.calgarypuck.com
Button is a bit surprising to see so high, I always found his lists to have some head scratchers but credit where it's due.
I've always liked FC and McKeens, this seems to validate that.