Player Rating Systems in the Invasion-Territorial Team Sports – What are the Issues?

Executive Summary

  • Player rating systems are important as a means of summarising the overall match performance of individual players.
  • Player performance in the invasion-territorial team sports is multi-dimensional so that a player rating system needs to be able to combine metrics for a number of skill-activities.
  • There are two broad approaches to the construction of player rating systems – win-attribution (or top-down/holistic) approaches and win-contribution (or bottom-up or atomistic) approaches.
  • The plus-minus approach is the most widely used win-attribution approach based on the points margin when a player is playing.
  • A player’s plus-minus score is sensitive to context especially the quality of team mates and opponents. This can be controlled using regression analysis to estimate adjusted plus-minus scores.
  • The plus-minus approach offers a relatively simple way of measuring player performance without the need for detailed player performance data. But the approach works best in high scoring sports with frequent player switches such as basketball and ice hockey.

 

A central issue in sports analytics is the construction of player rating systems particularly in the invasion-territorial team sports. Player rating systems are important as a means of summarising the overall match performance of individual players. Teams can use player rating systems to review performances of their own players as well as tracking the performance levels of potential acquisitions. Moneyball highlighted the possibilities of using performance metrics to inform player recruitment decisions. But the relatively simple game structure of baseball, in essence a series of one-to-one contests between hitters and pitchers, means that the analytical problem is reduced to finding the best metrics to capture hitting and pitching performances.

 

Once we move into invasion-territorial team sports, we are dealing with sports which involve the tactical coordination of players and player performance becomes multi-dimensional. The analytical problem is no longer restricted to identifying the best metric for a single skill-activity per player (i.e. pitching or hitting in baseball) but now involves identifying the full set of relevant skill-activities and creating appropriate metrics for each identified skill-activity.

 

There are essentially two broad approaches to constructing player rating systems when player performances are multi-dimensional. One approach is the win-contribution (or bottom-up or atomistic) approach which involves identifying all of the relevant skill-activities that contribute to the team’s win ratio, developing appropriate metrics for each of these skill-activities, and then combining the set of skill-activity metrics into a single composite measure of performance. Over the years many technical and practical problems have emerged in constructing win-contribution player rating systems. I plan to discuss these in more detail in a future blog. Suffice to say, the most general criticism of the win-contribution approach is the difficulty of identifying all of the relevant skill-activities particularly those that are not directly and/or easily observable such as teamwork and resilience.

 

The alternative approach is a more holistic or top-down approach that uses the match outcome as the ultimate summary metric for measuring team performance and then attributes the match outcome to those involved in its production. I call this the win-attribution approach to player rating systems. The analytical problem is now the choice of an attribution rule.

 

Plus-Minus Player Ratings

The best-known win-attribution approach is plus-minus which has been used for many years in both basketball and ice hockey. It is a very simple method. Just total up the points scored and the points conceded whenever a specific player is on court (or on the ice), and then subtract points conceded from points scored to give the points margin. This represents the player’s plus-minus score.

 

For those of you not familiar with the plus-minus approach, here’s a simple example. Consider the following fictitious data for the first three games of a basketball team with a roster of 10 players.

The results of the three games are:

Game 1: Won, 96 – 73

Game 2: Lost, 68 – 102

Game 3: Won, 109 – 57

The minutes played (Mins) for each player, and points scored (PS) and points conceded (PC) while each player is on court, are as follows:

 

Player Game 1 Game 2 Game 3
Mins PS PC Mins PS PC Mins PS PC
P1 32 54 58 28 35 64 12 27 18
P2 29 63 45 25 33 56 13 30 21
P3 27 48 43 20 36 47 13 29 23
P4 33 58 52 27 32 63 15 33 22
P5 35 63 54 36 37 82 25 54 33
P6 22 49 24 28 44 43 33 72 30
P7 20 45 20 22 35 37 35 76 32
P8 16 37 27 24 38 51 33 77 36
P9 15 35 23 23 36 50 35 82 38
P10 11 28 19 7 14 17 26 65 32

 

A player’s plus-minus score is just the points margin (= PS – PC). So in the case of player P1 in Game 1, he was on court for 32 minutes during which time 54 points were scored and 58 points were conceded. Hence his plus-minus score is -4 (= 54 – 58). Given that the team won the game with a points margin of 23, the plus-minus score indicates a well below average performance. The full set of plus-minus scores are as follows:

 

Player Plus-Minus Scores Average Benchmark Benchmark Deviation
Game 1 Game 2 Game 3 Total
P1 -4 -29 9 -24 8.50 -32.50
P2 18 -23 9 4 10.27 -6.27
P3 5 -11 6 0 12.85 -12.85
P4 6 -31 11 -14 12.94 -26.94
P5 9 -45 21 -15 18.35 -33.35
P6 25 1 42 68 26.46 41.54
P7 25 -2 44 67 31.92 35.08
P8 10 -13 41 38 26.42 11.58
P9 12 -14 44 42 28.81 13.19
P10 9 -3 33 39 28.48 10.52

 

As well as the plus-minus scores for each player in each game, I have also reported the total plus-minus score for each player over the three games. I have also calculated an average benchmark for each player by allocating the final points margin for each game pro rata based on minutes played. So, for example, player P1 played 32 out of 48 minutes in Game 1 which ended with a 23 winning margin. An average performance would have implied a plus-minus score of 15.33 (= 23 x 32/48). His average benchmarks in Games 2 and 3 were -19.83 (= -34 x 28/48) and 13.00 (= 52 x 12/48), respectively. Summing the average benchmarks for each game gives an overall average benchmark of 8.50 for player P1. The final column reports the deviation from benchmark of the player’s actual plus-minus score.

 

In this example players P1 – P5 were given the most game time in Games 1 and 2 but all five players have negative benchmark deviations. The allocation of game time in Game 3 better reflects the benchmark deviations with players P6 – P10 given much more game time.

 

Limitations and Extensions to Plus-Minus Player Ratings

The advantage of the plus-minus approach is its simplicity. It is not dependent on detailed player performance data but only requires information on the starting line-ups, the timing of player switches, and the timing of points scored and conceded. The very first piece of work that I did for Saracens in March 2010 was to rate their players using a plus-minus approach. I focused on positional combinations – front row, locks, back row, half backs, centres, and backs – and calculated the plus-minus scores for each combination. Brendan Venter, the Director of Rugby, was very positive on the results and commented that “your numbers correspond to our intuitions”. It was on the basis of this report that I was engaged to work as their data analyst for five years. The plus-minus approach was used for player ratings in the early stages of the 2010/11 season but was eventually discarded in favour of a win-contribution approach.

 

One of the problems with the simple plus-minus approach is that it will give high scores to players who regularly play with very good players. So, if a particular player was fortunate enough to be playing regularly alongside Michael Jordan, they would have had a high plus-minus score but this reflects the exceptional ability of their team mate more than their own performance. My dear friend, the late Trevor Slack, one of the top people in sport management and a prof at the University of Alberta in Edmonton, used to call it the Wayne Gretzky effect. Those of you who know their ice hockey history will know exactly what Trevor meant. Gretzky was one of the true greats of the NHL and brought the best out of his team mates whenever he was on the ice. The Edmonton Oilers won four Stanley Cups with Gretzky in the 1980s.

 

Similarly it can be argued that the basic plus-minus approach does not make any allowance for the quality of the opposing players. Rookie players given more game time against weaker opponents will have their plus-minus scores inflated just as those players who get proportionately more game time against stronger opponents will see their plus-minus scores reduced. One way around the problems of controlling for the quality of team mates and opponents is to use Adjusted Plus-Minus which involves using regression analysis to model the points margin during a “stint” (i.e. a time interval when no player switches are made) as the function of own and opposing players. The estimated coefficients represent the adjusted plus-minus scores. There have also been various attempts to include other performance data to create real adjusted plus-minus scores which represent a hybrid of the win-attribution and win-contribution approaches.

 

Overall the plus-minus approach offers a relatively simple way of measuring player performance without the need for detailed player performance data. But the approach works best in high scoring sports with frequent player switches such as basketball and ice hockey. The plus-minus approach is not well suited to football (soccer) which is low scoring and teams are restricted to only three substitutions.

 

15th September 2016

The Real Lessons of Moneyball

Executive Summary

  • Moneyball was a game-changer in raising general awareness of the possibilities for data analytics in elite sport.
  • Always remember that Moneyball is only “based on a true story” and does not provide an authentic representation of how data analytics developed at the Oakland A’s.
  • The conflict between scouting and analytics is exaggerated for dramatic effect.
  • The real lesson of Moneyball is the value of an evidence-based approach. This goes beyond the immediate context of player recruitment in pro baseball to embrace all coaching decisions in all sports.

blog-8-graphic-1                     blog-8-graphic-2

 

 

 

The publication of Moneyball in the Fall of 2003 proved to be a real game-changer both for sports analytics and myself personally. The book, and subsequent Hollywood film with an A-List cast, has probably done more than anything else to raise general awareness in elite sport of the potential competitive advantages to be gained from data analytics.

 

I was visiting the University of Michigan to give some presentations on what business could learn from elite sport in September 2003, just after Moneyball was first published. At that point I was making good progress in the analysis of player performance data in football (soccer) and had constructed what I later called a structural hierarchical model of invasion team sports. As I was being driven to Detroit Airport at the end of my visit, Richard Wolfe, a sport management prof, told me that I had to read Moneyball, saying “it’s you but baseball”. I picked up the book at the airport around 6pm that Friday night and had completed my first read by 6am the following morning. Here was someone actually using data analytics in elite sport to gain a competitive advantage. And there’s nothing like success to persuade others to adopt an innovation. I now had real evidence of what analytics could do; I just needed access to coaches to spread the word – easier said than done. I lost count in the coming months of the number of conversations I started with “Have you read Moneyball?” But people started to take notice and the invitations to meet directors and coaches began to follow.

 

The first coaching staff to invite me into the inner sanctum was at Bolton Wanderers managed by Sam Allardyce, now the England manager. I made a presentation on Moneyball and the implications for football in early October 2004 at their quarterly coaches’ away day organised by Mike Forde, the Performance Director who subsequently became Chelsea’s Performance Director. Big Sam remained pretty quiet during the presentation, restricting himself to points of information and summarising the discussions, but never revealing his opinion on what I was saying. It was Sam’s assistant, Phil Brown, currently manager at Southend United, who was the most vocal and concerned that I seemed to be advocating the use of algorithms for team selection (which I wasn’t). Bolton followed up by getting me to do some analysis of the FA Premier League including identifying the critical success factors in away performances. I also outlined an e-screening procedure for identifying prospective player acquisitions to be prioritised in the scouting process. Although it was something of an achievement to have a Premiership team ask for an analytics input in 2005, the frustration was that I was kept at arm’s length from the coaching staff and only received limited feedback on my reports. Being told that your report had provoked an “interesting discussion by the coaches” was satisfying but nothing more. What I really needed to know was what precisely had interested the coaches and how could I expand and improve the analysis to deal with any limitations they saw in it. It is an important lesson – data analytics only really works when there is full engagement between the coaches and the data analysts. My subsequent experience at Saracens showed how much I could improve the analysis by having direct contact with the coaches and being included in their discussions. As one senior member of the coaching staff at Saracens put it, I effectively became an “auxiliary member of the coaching staff” in the same way as the performance analysts and sports scientists.

 

Of course the biggest impact of Moneyball for me personally was to eventually connect with Billy Beane and to work with him in exploring the potential for analytics in football – the Oakland A’s own the MLS San Jose Earthquakes franchise. Seeing Billy and his staff at work at the A’s was a great education and allowed me to fully appreciate the “true story”. The book and the film are after all only “based on a true story” and make no claim, particularly the film, to be an authentic representation of the development of data analytics at the A’s.

 

Having seen close-up how the A’s actually operate, I’ve been better placed to respond to the criticisms of Moneyball. For example, a head of scouting at a leading European football club recently put to me that “perhaps Moneyball had become a bit of an albatross”. This head of scouting is a former player and is a very progressive individual, open to innovation to improve how things are done. But when we met he was initially very wary of adopting a more analytical approach to scouting since he thought that this would mean a reduced role for the scouts. He was won over when I pointed out that an evidence-based approach would actually enhance the role of scouts since their scouting reports would become key data that would be used in a much more meaningful way rather than just gathering dust as I suspect happens to most scouting reports.

 

The Oakland A’s managed by Billy Beane operate in a fundamentally different way to the Hollywood A’s managed by Brad Pitt. There is an over-emphasis in the film for dramatic effect on the conflict between the traditional scouting approach and the analytical approach. But in reality what differentiated the A’s under Billy Beane was the commitment to an evidence-based approach and a preparedness to question conventional wisdom rather than relying on gut instinct. It was the questioning of conventional wisdom that attracted Michael Lewis to the story in the first place. He started his professional life as a financial trader, trying to make profits on the financial markets by exploiting market inefficiencies caused through the over-reliance by other traders on conventional wisdom that had become outdated. Lewis applied the same lens to the MLB players’ labour market and saw Billy Beane as a kindred spirit, taking advantage of the over-reliance on traditional scouting methods. In particular, Billy Beane had recognised that the market didn’t factor into hitter salaries the ability to draw walks. Hitter salaries were mainly driven by batting and slugging averages. As economists say, walks were a “free lunch” because conventional wisdom saw them more as a pitcher error than due to the hitter’s skill in selecting when to swing and when not. Two economists, Hakes and Sauer (Journal of Economic Perspectives, 2006), have shown that on-base percentage (OBP) which includes walks had no significant effect on hitter salaries in the five seasons prior to the publication of Moneyball but in 2004 OBP was the single most significant predictor of hitter salaries. Conventional wisdom had changed because of he publication of Moneyball and, just as economic theory predicted, the ensuing market correction meant that particular free lunch quickly disappeared.

 

What is also forgotten is that, as in so many success stories, success was a long time coming. The evidence-based culture at the A’s was not created by Billy Beane (the film is very misleading in this respect) although he has played a leading role in the use of analytics by the A’s. But the possibility of gaining a competitive advantage from using sabermetrics was first recognised by Sandy Alderson, Billy’s predecessor as GM, and a long-time admirer of the work of Bill James. It was Sandy Alderson who employed the consultant, Eric Walker, to develop some “Bill James-type stuff that would be proprietary to the A’s”. Alderson passed on Walker’s report to Billy. The rest, as they say, is history.

 

It is the value of an evidence-based approach to all coaching decisions that is the real lesson of Moneyball. It is a lesson that goes beyond the immediate context of Moneyball – player recruitment in pro baseball – and is transferable to all sports. Yes the principal applications of data analytics remain in the area of player recruitment but as my experiences in football, rugby union and other sports have shown, all coaching decisions can potentially be supported to a greater or lesser extent by “knowing the numbers”, systematically analysing the available evidence both quantitative measures and qualitative assessments, and always preferring analysis over anecdote when justifying a course of action. That’s the real lesson of Moneyball.

 

7th September 2016