Do 'Expected' goals statistics suck?

[Bear of Bad News]

Anyone looking for clout around here deserves what they get.

 

[Summer Rose]

Anyone looking for clout around here deserves what they get.

My expected disdain from replying to this post is in the neighborhood of 20-90%. I will submit to John Chayka for clarification.

 

[FerrisRox]

clout chasing.

You think there is "clout" from randomly being paired with a couple of analytics guys in a golf game?

Are you serious with this?

 

[bambamcam4ever]

I'm not sure that's true. I remember looking up Crosby's career goals versus xG and I believe it was something like 15 out of 17 seasons (when I last checked), he outperformed his xG and sometimes by quite a bit.

The problem with expected goals is, at least the ones the general public has access to, it doesn't seem to take into account a player's actual skill. An unscreened shot from the slot has the same expected chance of going in regardless of whether the person shooting is Auston Matthews or Ryan Reaves.

I also believe (though not sure on this second one) that it doesn't take into account who is defending the shot. For instance, if Jaccob Slavin is standing between the shooter and the net or Erik Gudbranson is defending. Slavin likely either gets a stick on the shot or angles his body that the shooter has to shoot wide, while Gudbranson probably plays it horribly and the shooter gets a clear shot on net.

The stat just seems to come up with a percentage based on shot location without taking into account all the other variables that can affect the danger of a shot from going in.

Yep. xG consistently underestimates Crosby's impact and consistently overestimates Erik Karlsson, for example. There is clearly something being missed

 

[Summer Rose]

You think there is "clout" from randomly being paired with a couple of analytics guys in a golf game?

Are you serious with this?

It would be neither the first time nor the last time that someone in a profession admits to some guys with whom they're "shooting the breeze" admit that their entire profession and life's work is at least a little bit of bullshit (and for the record, I don't doubt you).

My dad has said the same. He was a botany professor at Arizona State University for pretty much his entire career. As a kid, I have distinct memories of him admitting to me that some of his work was to appease the people who gave him his research grants. Your story isn't much of a stretch of the imagination. Every job comes with bullshit. Mine did too, so I'm not immune to this. And as I said upthread, don't underestimate what kind of things people will randomly talk about over a round of golf. If you're not sure of this, go play a round of golf with some random people yourself. You'll get a lot of inane drivel as people ramble about their work, but do it long enough and a story like this will eventually pop up.

 

[Bear of Bad News]

It would be neither the first time nor the last time that someone in a profession admits to some guys with whom they're "shooting the breeze" admit that their entire profession and life's work is at least a little bit of bullshit (and for the record, I don't doubt you).

My dad has said the same. He was a botany professor at Arizona State University for pretty much his entire career. As a kid, I have distinct memories of him admitting to me that some of his work was to appease the people who gave him his research grants. Your story isn't much of a stretch of the imagination. Every job comes with bullshit. Mine did too, so I'm not immune to this. And as I said upthread, don't underestimate what kind of things people will randomly talk about over a round of golf. If you're not sure of this, go play a round of golf with some random people yourself. You'll get a lot of inane drivel as people ramble about their work, but do it long enough and a story like this will eventually pop up.

I won't admit to stretching the truth at work, but I will tell you that there's no fun in telling a boring story.

 

[Summer Rose]

I won't admit to stretching the truth at work, but I will tell you that there's no fun in telling a boring story.

Stretching the truth is apparently a "people skill" or so I've heard.

One of my jobs was at a church. How they originally hired me and what I ended up doing for them is a matter of interpretation, but I'm going to put that on my resume as "music director" and nobody will challenge me on that title. I call that a win.

 

[Bear of Bad News]

Stretching the truth is apparently a "people skill" or so I've heard.

One of my jobs was at a church. How they originally hired me and what I ended up doing for them is a matter of interpretation, but I'm going to put that on my resume as "music director" and nobody will challenge me on that title. I call that a win.

 

[Machinehead]

I mean, can you test their expected goal numbers against anything you can measure to make sure they didn't screw up the calculation somehow? Like missing a variable that should have been accounted for, or using the wrong formula. I certainly wouldn't trust GPT to be able to get everything right.

There is no "right" answer. Baseball is the most boiled down to a formula sport in the world. They still have to play the games. They still spend millions of dollars on scouting. As I touched on yesterday, the standard of "this has to be correct" is not a standard we would apply anywhere else.

I'm not saying you do this, but many people still treat viewings and the eye test as the most sacrosanct form of assessment. Meanwhile, if your eyes bat .250, I'm impressed. Look at what professional front office execs get wrong every single year -- even things that the fans were screaming "don't do this!" ahead of time.

Every time we trade for a guy one of the first things said is "their fans said this," and if I said "how do we test that?" I would get looked at funny.
So, nothing is ever going to be perfect and we should never hold any assessment to the standard of being correct. In terms of what we can test it against, expected goals are more predictive of future goals, than actual goals. That's a fact. It has shown this in multiple sports.

Soccer:

Hockey:

For the purposes of layman's terms, think of MSE as randomness and R^2 as certainty. xG scores lower for randomness and higher for certainty.

This is not to say that they're "correct." I don't believe that exists. All assessment is needed including viewings. The original question presented is "do they suck?" and the answer is obviously not.

 

[Summer Rose]

You ever had a job where you were hired on as one thing, presumably kind of paltry, then you suddenly had a bunch of extra responsibilities thrown your way and no increase in pay? That's what it was like for me.

 

[Machinehead]

Also, I should add that, in the second chart, after a bunch of games, you'll notice that the xG eventually comes back down to where the goals are.

So yeah, if you can shown me [name player here] who outperforms his xG over a large sample, then that's fine. I care more about his actual goals and his goal differential.

 

[Baksfamous112]

Please don’y say this! You’re going to take away @dgibb10 entire personality!!

 

[Summer Rose]

Also, I should add that, in the second chart, after a bunch of games, you'll notice that the xG eventually comes back down to where the goals are.

So yeah, if you can shown me [name player here] who outperforms his xG over a large sample, then that's fine. I care more about his actual goals and his goal differential.

I think that's what a lot of people sometimes miss. These kinds of stats are "long term" stats that indicate some kind of regression towards the mean. If someone can show me an example of a player who outperforms his "expected goals for" over the course of his whole career, I'm all ears. Ovechkin seems like that kind of guy, but to me, he's one of those cases who's an exception to the norm. I'll say, I suppose, that these kind of stats aren't actually that bad at predicting long-term trends. But to what I said earlier, if I did (I honestly forgot), they simply predict things. Liam O'Brien isn't scoring many goals off one-timers from the left circle. Alex Ovechkin is. Why? Ovechkin is good at shooting from that area (and other areas too). "Expected goals" doesn't take that into account though. The statistic "expects" that O'Brien will score as often as Ovechkin from that area of the ice. If someone told me that s/he expects Liam O'Brien to score as many goals as Alex Ovechkin from that area, I'd ask him/her what they were smoking, and if s/he would share.

TLDR: context is key. If you don't interpret stats in the proper context, you're doing it wrong in the first place.

 

[Machinehead]

I think that's what a lot of people sometimes miss. These kinds of stats are "long term" stats that indicate some kind of regression towards the mean. If someone can show me an example of a player who outperforms his "expected goals for" over the course of his whole career, I'm all ears. Ovechkin seems like that kind of guy, but to me, he's one of those cases who's an exception to the norm. I'll say, I suppose, that these kind of stats aren't actually that bad at predicting long-term trends. But to what I said earlier, if I did (I honestly forgot), they simply predict things. Liam O'Brien isn't scoring many goals off one-timers from the left circle. Alex Ovechkin is. Why? Ovechkin is good at shooting from that area (and other areas too). "Expected goals" doesn't take that into account though. The statistic "expects" that O'Brien will score as often as Ovechkin from that area of the ice. If someone told me that s/he expects Liam O'Brien to score as many goals as Alex Ovechkin from that area, I'd ask him/her what they were smoking, and if s/he would share.

TLDR: context is key. If you don't interpret stats in the proper context, you're doing it wrong in the first place.

It depends on the guy/team as well.

Ovechkin has somewhere around 545 individual xG not counting when the net is empty (because that defeats the whole purpose of expecting to beat the goaltender). xG are only tracked since 2007-08 so you have to ignore the 98 goals he scored before that, and the 65 empty netters, which puts him at 723. He's outscoring his xG by almost 200, and for Ovechkin, I absolutely believe that.

But then you have the Rangers, who have been an analytics black hole since I'm in high school, and people will say "but Machinehead, they win games over large samples!" and like, I don't know, I've seen the Rangers. I still think they suck.

And that's where a bit of the eye test does come in. How much to believe they're gonna outrun this thing? Ovechkin could run forever. The Rangers get bodied in the playoffs every year. And I mean bodied. The series they lose are laughably lopsided as soon as they run into somebody.

I think players are way more likely to outperform than teams. Players is less of a matchup thing. They get to perform somewhat independently. Teams run into other teams that are going to actively take the puck away from them. When you can't possess the puck, you invite trouble, and the things you do well go away.

 

[IWantSakicAsMyGM]

There is no "right" answer. Baseball is the most boiled down to a formula sport in the world. They still have to play the games. They still spend millions of dollars on scouting. As I touched on yesterday, the standard of "this has to be correct" is not a standard we would apply anywhere else.

I'm not saying you do this, but many people still treat viewings and the eye test as the most sacrosanct form of assessment. Meanwhile, if your eyes bat .250, I'm impressed. Look at what professional front office execs get wrong every single year -- even things that the fans were screaming "don't do this!" ahead of time.

Every time we trade for a guy one of the first things said is "their fans said this," and if I said "how do we test that?" I would get looked at funny.
So, nothing is ever going to be perfect and we should never hold any assessment to the standard of being correct. In terms of what we can test it against, expected goals are more predictive of future goals, than actual goals. That's a fact. It has shown this in multiple sports.

Soccer:
View attachment 991543

Hockey:

View attachment 991546

For the purposes of layman's terms, think of MSE as randomness and R^2 as certainty. xG scores lower for randomness and higher for certainty.

This is not to say that they're "correct." I don't believe that exists. All assessment is needed including viewings. The original question presented is "do they suck?" and the answer is obviously not.

The fact that they still need to play the games and still spend millions on scouts would suggest that the numbers aren't very trustworthy, and like everything else claiming to predict the future, should be taken with pretty big grain of salt.

I also don't dispute that these stats have value, when used correctly. A team with a high xG number for a season has been taking shots from historically dangerous spots, which increases the likelihood of them scoring more goals. Taking more dangerous shots generally leads to more goals. Or, for Corsi, more shots than your opponent generally leads to more goals for than against. At their core, they are both really just common sense things being quantified, and higher numbers should produce better results more often than not. The GF% numbers aren't nearly as good as predicting success because it shows the results, and not the behavior that helps lead to those results.

Where these numbers stop being useful is when people don't understand the limitations of them, and want to use these stats are something more than they are, which unfortunately seems to a frequent occurrence around here. That's why we should be wary of the advanced statistic being paraded out as meaning one thing or another until there's a demonstrable reason to believe it actually shows what is being claimed.

 

[Gaylord Q Tinkledink]

I'd bet an incredible amount of money that this convo didn't happen. Two people working in analytics willingly telling others that their job is completely worthless. No shot.

Well, yeah. You don't telling people you work for.

The amount of people who willing tell others that they do f*** all at work is quite a lot of people.

They're mostly work from home people, though.

IT people like tell others how they do very little day in and day out.

 

[Machinehead]

The fact that they still need to play the games and still spend millions on scouts would suggest that the numbers aren't very trustworthy, and like everything else claiming to predict the future, should be taken with pretty big grain of salt.

I also don't dispute that these stats have value, when used correctly. A team with a high xG number for a season has been taking shots from historically dangerous spots, which increases the likelihood of them scoring more goals. Taking more dangerous shots generally leads to more goals. Or, for Corsi, more shots than your opponent generally leads to more goals for than against. At their core, they are both really just common sense things being quantified, and higher numbers should produce better results more often than not. The GF% numbers aren't nearly as good as predicting success because it shows the results, and not the behavior that helps lead to those results.

Where these numbers stop being useful is when people don't understand the limitations of them, and want to use these stats are something more than they are, which unfortunately seems to a frequent occurrence around here. That's why we should be wary of the advanced statistic being paraded out as meaning one thing or another until there's a demonstrable reason to believe it actually shows what is being claimed.

As a concept, I think we all agree that stats have limitations and should be taken with a grain of salt.

Where I differ from a lot of folks is that I feel analytics provoke an outsized reaction, particularly negative ones.

That's just my perception. In my time here, I've seen people be sanctimonious about analytics, and perhaps you're seeing an outsized reaction in that direction.

But again, I come back to this thread, which is "do analytics suck?" based on this random ass one game, and I would ask you, can you point me to any of the following thread topics?

-"Does this expert who had one bad take suck now? Should we disregard him entirely?"
-"My coach is moron. Does coaching suck? Are coaches useful?"
-"My friend watches the Flyers every night and his opinions are dumb. Is watching hockey pointless?"
-"I really thought the Red Wings played well, but the other team scored 4 more goals. Should we take the final score with a grain of salt?"

That all sounds dumb, right? That's exactly what "analytics produced this one result I didn't expect so do they suck entirely?" sounds like.

That's what "this shot should go in NINETY-SIX(!!!) percent of the time" sounds like. And I'm not trying to call out OP, but I mean c'mon, if you shot the puck off a pier, you wouldn't hit the water 96% of the time.

We don't have these very silly discussions about anything else being wrong.

 

[dgibb10]

See, it's interesting you mention this, because I really think the issue is just semantics sometimes.

Whether you subscribe to stats or not, we all have an idea of who's carrying the play in a game.

Some people will just beat themselves over the head with the old "the team that won played better" club all day (and then those same people will talk about "having a goalie that can steal a series" like WTF), but most of us will concede that teams getting outplayed win all the time, and vice versa.

In a series, anything can happen in one game, two games; over the course of that entire series, the team getting the lion's share of the chances probably wins that series. I don't think anybody really disagrees with that.

Once you call it "expected goals," they're mad about it.

Also, people's "eye tests" are WAY more influenced by statistics than they'd like to admit.

Those statistics just tend to be ones that are old and worn into their minds.

If they hadn't been tracked for 100+ years already, and people tried to introduce the concept of "secondary assists" today, they'd be viewed as outlandish advanced stats that don't mean much (I mean, being the 2nd guy to touch the puck before a goal? that doesn't mean shit). (see anytime the concept of Hockey assists is brought up in basketball, it's viewed as an "advanced stat" there).

 

[dgibb10]

If "expected" is a thing that people have an issue with, what should they call it?

See, it's interesting you mention this, because I really think the issue is just semantics sometimes.

Whether you subscribe to stats or not, we all have an idea of who's carrying the play in a game.

Some people will just beat themselves over the head with the old "the team that won played better" club all day (and then those same people will talk about "having a goalie that can steal a series" like WTF), but most of us will concede that teams getting outplayed win all the time, and vice versa.

In a series, anything can happen in one game, two games; over the course of that entire series, the team getting the lion's share of the chances probably wins that series. I don't think anybody really disagrees with that.

Once you call it "expected goals," they're mad about it.

"advanced stats" was set back a decade by the names corsi and fenwick convincing people they were some advanced thing and not the incredibly simple concept of counting "shot attempts" and "shot attempts-blocked shots".

 

[Machinehead]

Also, people's "eye tests" are WAY more influenced by statistics than they'd like to admit.

Those statistics just tend to be ones that are old and worn into their minds.

If they hadn't been tracked for 100+ years already, and people tried to introduce the concept of "secondary assists" today, they'd be viewed as outlandish advanced stats that don't mean much (I mean, being the 2nd guy to touch the puck before a goal? that doesn't mean shit). (see anytime the concept of Hockey assists is brought up in basketball, it's viewed as an "advanced stat" there).

Also, to add onto that, I think the "eye test" is vibes more often than people like to admit.

I think vibes have a place. I heard other people say this guy is good. I noticed this guy when I watched a game he was in. You know, if it looks like a duck...I 100% think there's some validity there.

How many times have I actually sat down and watched Macklin Celebrini play? I know he's good. I don't need to pull up his card.

All I'm saying is don't act like you scouted the guy in iso when you didn't.

 

[Bear of Bad News]

The "eye test" is basically just a list of cognitive biases - confirmation bias, narrative bias, highlight bias, sampling bias. And everyone's susceptible to it - even professional scouts and managers. Highly recommend Daniel Kahneman's "Thinking Fast and Slow" for people interested in this stuff.

One "advantage" of the "eye test" is that literally no one can attack your argument and get you to admit fault if your argument is "I saw him play and I know what I saw", which I think is also why (some) people prefer it as a methodology.

 

[IWantSakicAsMyGM]

I think that's what a lot of people sometimes miss. These kinds of stats are "long term" stats that indicate some kind of regression towards the mean. If someone can show me an example of a player who outperforms his "expected goals for" over the course of his whole career, I'm all ears. Ovechkin seems like that kind of guy, but to me, he's one of those cases who's an exception to the norm. I'll say, I suppose, that these kind of stats aren't actually that bad at predicting long-term trends. But to what I said earlier, if I did (I honestly forgot), they simply predict things. Liam O'Brien isn't scoring many goals off one-timers from the left circle. Alex Ovechkin is. Why? Ovechkin is good at shooting from that area (and other areas too). "Expected goals" doesn't take that into account though. The statistic "expects" that O'Brien will score as often as Ovechkin from that area of the ice. If someone told me that s/he expects Liam O'Brien to score as many goals as Alex Ovechkin from that area, I'd ask him/her what they were smoking, and if s/he would share.

TLDR: context is key. If you don't interpret stats in the proper context, you're doing it wrong in the first place.

When a player outscores his xG, all that really tells us is that he's scoring from locations that most other players haven't scored from as frequently. It doesn't mean that we should expect every player to score goals from that location (or really any location in the case of O'Brien), it just shows we could expect there to be that many goals scored if every shot taken was the average of all shots taken from that location. Guys with above average shots who score from difficult areas, like OV or MacKinnon or Panarin can pretty easily beat the averages for those areas and outscore the xG numbers. Guys who take more shots from the higher danger areas are going to have a harder time, which is probably why Matthew Tkachuk is below his xG numbers in 4 of the last 5 years.

 

[BigDaddyLurch]

The "eye test" is basically just a list of cognitive biases - confirmation bias, narrative bias, highlight bias, sampling bias. And everyone's susceptible to it - even professional scouts and managers. Highly recommend Daniel Kahneman's "Thinking Fast and Slow" for people interested in this stuff.

One "advantage" of the "eye test" is that literally no one can attack your argument and get you to admit fault if your argument is "I saw him play and I know what I saw", which I think is also why (some) people prefer it as a methodology.

...and yet most "advanced stats" fail to take into account the plethora of variables a 5-player driven game like hockey is...they are snapshots in Time that can be weighted to prove/disprove almost any "arguement" the "eye test" can come up with...

 

[Bear of Bad News]

...and yet most "advanced stats" fail to take into account the plethora of variables a 5-player driven game like hockey is...they are snapshots in Time that can be weighted to prove/disprove almost any "arguement" the "eye test" can come up with...

I've never under any circumstances suggested that people should only look at statistics at the expense of actually watching things. Far from it - I do advanced analytics for a living (not in hockey although I have done contracted work for hockey teams), and it's exceptionally important to pair any analysis with direct evidence.

What I usually tell people is that a good predictive model should match the eye test about 80% of the time. If it's less than 80%, it's probably fundamentally wrong. If it's more than 80% then it's not telling you anything new and why did you bother.

I can also tell you that I know quite a few people doing this for professional sports organizations, and absolutely all of them love the sports they are involved with and spend as much time as possible watching it. The myth of "stats nerds in their parents' basements who have never set foot in an arena" is flat out bullshit. I played college hockey and I'm still far from the best hockey player among those involved with advanced analytics.

 

[Lunatik]

All stats are both good and bad depending who is using them.