The Replication Project: Measuring Shooting Overperformance
/Introduction
Before Laurie Shaw was working for the esteemed City Football Group, he was gifting us with tidbits of knowledge in blog posts. One such piece of writing was “Exceeding Expected Goals”, a dive into the age-old question of finishing skill in soccer.
While expected goals (xG) tell us about the quality of the shots—accounting for the context of a shot—they’re agnostic to player identity, so we need more info to tease out individual shooting ability. Shaw points out that one way to evaluate finishing “overperformance” is to divide a player’s count of goals (G) by their xG. A ratio of 1 indicates that a player is scoring as many goals as expected; a ratio greater than 1 indicates overperformance; and a ratio less than 1 indicates underperformance.
But this measure (i.e. G/xG) wasn’t really the novelty of Shaw’s writing. Rather, it was his approach to adjusting this ratio for the fact that the volume and quality of shots vary a lot among players. Just comparing the empirical G/xG ratio of one player to another can be extremely misleading. (A direct comparison of the Crew’s Sean Zawadzki and Philly's Jose Martinez, who have completely different shot volumes and profiles, is statistically meaningless.)
Using empirical Bayes (EB), one “shrinks” the G/xG ratio back closer to 1. While the exact amount of “shrinking” depends on how many shots one has taken, broadly, the G/xG ratio of those who have taken fewer shots is going to be adjusted more. After applying the EB adjustment, G/xG becomes more comparable across players, regardless of the number of shots they’ve taken.
Further, EB provides a robust framework for estimating uncertainty, which can provide answers to questions like “Was Diego Fagundez’s positive goal conversion rate in 2022 significant?” (Note the descriptive nature of this framing. More on the predictiveness and stability of G/xG later.) Specifically, we can look to see whether the uncertainty bands around the player's G/xG ratio EB-adjusted estimate overlap with 1. If not, and the adjusted ratio estimate is greater than 1, then we might say that the player’s performance is “real” and more likely than not going to continue over the long run. (Technical note: The plots that follow show ±1 standard deviation around the mean, but a more strict and, arguably, the correct measure is a 90 or 95% credible interval.)
Measuring Overperformance
Implementing EB with understat xG data for the 2016/17 and 2017/18 English Premier League seasons, we can generate a chart similar to Shaw’s original. While we don’t have Stratgem’s xG data to do an exact replication, Eden Hazard is at the top and DC United’s Christian Benteke is at the bottom. The rest of the players, while not in the exact order, are generally similar.
Now, if we apply the same concept to the MLS for players taking shots since the beginning of the 2021 season, we see names that MLS sickos should be familiar with—the likes of Saudi-exported Lucas Zelarayán, USMNT Caribbean folklore hero Jesús Ferreira, and 2022 MLS MVP Hany Mukhtar. (Note the 100-shot minimum criteria, which, admittedly, biases the results towards attackers.)
What might be more interesting is the players who are heavily affected by the “shrinking” of the EB adjustment, yet are still found to be relatively strong shooting overperformers. We can see this if we change the shooting criteria to those who have taken between 10 and 100 shots (“low volume”) since the start of the 2021 season. Almost by definition, many of these players are midfielders, wingers, or defenders.
Júnior Moreno is an interesting case. He’s taken just over 30 shots since 2021 but has scored 6 goals on just less than 2 xG. His raw G/xG ratio would be just over 3(!), but due to the EB shrinking, that ratio is pulled back to 1.37. This is still nothing to sneeze at, as it puts him at the top among the “low volume” shooters.
Stability
So, from looking at the low volume shooters, hopefully it is evident that EB adjustments can make players shot performance more sane and comparable. (Indeed, if not for the EB adjustment and some minimum shot criteria, Sean Zawadzki would appear to be the strongest overperformer in the MLS since 2021, with an other-worldly 13.5 G/xG ratio on 7 shots at the time of writing.) Intuitively, one might think that EB shrinking might have positive implications for the year-over-year stability of G/xG of outfield players.
Unfortunately, that’s not the case. There’s basically zero correlation between a player’s EB-adjusted G/xG in one year compared to the same player’s adjusted G/xG ratio in the prior season.
While studies show that xG is perhaps the best metric to use to project next year’s performance at the team level, xG and simple mutations of it (e.g. G/xG ratio) have been rightfully criticized for lacking stability at the player level. Even an EB adjustment does not improve this.
Other approaches for quantifying shooting overperformance
The ratio of goals to xG is not the only metric used for measuring outstanding shooting skill. Taking one step back, a more common approach is to simply take the difference between goals and xG. G minus xG, like G/xG, is intended to represent the quality outcomes (goals) accounting for the quality of chances (xG). The interpretation of the difference is almost exactly like G/xG, except 0 is the “reference” instead of 1. However, like G/xG, G minus xG is also poorly predictive of itself, season-over-season. Further, it can be even more misleading than raw G/xG in its portrayal of players strongly over- or under-performing in small samples.
Another approach is to subtract one’s post-shot xG from the pre-shot xG. (Missed shots get a post-shot expected goals value of 0.) However, this has its own flaws. Devin Pleuler observed that this measure, like G/xG, has a relatively weak year-over-year stability; arguably, there is more signal in the “more repeatable ability of constantly finding themselves in goal-scoring opportunities”, i.e. shots taken and xG volume.
Recent research from Baron et al. has found that there is a non-trivial signal of finishing skill in off-target shots. This is certainly a blind spot of focusing on xG and goals themselves, which are measures of outcomes. Their metrics show much improved inter-season stability even at the player level, a desirable trait that G/xG does not have. Notably, Baron et al. include the MLS in their analysis, quieting any potential claim that their conclusions do not apply to the American league and its aggregate player skill level.