Pythagorean Expected Wins Revisited

Executive Summary
-The Pythagorean expected-wins model provides a very simple predictor of team win% in baseball using squared scores
-Predictive accuracy can be improved by slightly adjusting the model
-The Pythagorean expected-wins model should be a key component in the strategic planning of the team roster and evaluating the sporting “bottom line” of any player trade-
-The Pythagorean expected-wins model can be applied to other team sports but typically requires using different powers depending on the average number of scores and the average winning margin
-The Pythagorean expected-wins model can be applied in soccer with appropriate adjustments to allow for tied games but goal difference remains the best simple score-based predictor of league performance
-Ultimately what is transferable from baseball across all pro team sports is not the specifics of the Pythagorean expected-wins model but rather the discipline of projecting the expected performance gains from any significant player recruitment decision

Pythagorean Expected Wins
Of all of Bill James’s many contributions to sabermetrics, probably the best known particularly outside baseball is his notion of Pythagorean win expectation. It is a very simple idea – league performance over a season will be closely associated with total scores made and total scores conceded. James’s innovative insight from his extensive study of baseball data was that the relationship between team win%, runs scored (RS) and runs allowed (RA) followed a power relationship:
Win%=〖RS〗^2/(〖RS〗^2+〖RA〗^2 )

As an example, Pythagorean win expectation applied to the 2014 MLB regular season yields the following results:

Table 1: MLB Regular Season 2014

Blog 18.06 Graphic (Table 1)

Pythagorean win% works pretty well in predicting the actual win% as I have tried to highlight using conditional formatting – the colour coding matches up pretty closely but with some important exceptions discussed below.

Using Expected Wins as a Strategic Tool
There are a couple of practical uses of Pythagorean expected wins. The first and by far the most important is that it provides a key relationship to be considered when planning changes to a team roster. Pythagorean expected wins can be used to project the likely impact on the team win% of a series of player trades. It is a great discipline for GMs and Personnel Directors to formalise exactly what they expect a new recruit to bring to the team. What is the sporting “bottom line” of any trade?
Suppose you were acting as an advisor to the Milwaukee Brewers at the end of the 2014 season. The 2014 Brewers were the epitome of an average team certainly statistically. The MLB team average for total runs scored and allowed that season was 659; the Brewers scored 650 runs and allowed 657 runs, finishing with 82 wins and 80 losses to yield an actual win% of 0.506. The Pythagorean formula would have predicted 80 wins (i.e. Pythagorean win% = 0.495). If you had wanted to transform the 2014 Brewers into a team capable of competing for the World Series, you would have needed to target a regular-season win% of around 0.600 which represents 97 wins. To achieve this level of performance would require an improvement in hitting and pitching of approximately 10% to 715 runs scored and 591 runs allowed.
Win%=〖715〗^2/(〖715〗^2+〖591〗^2 )=0.594=>96 wins (= .594 x 162 games)
Of course an equal 10% improvement in hitting and pitching is just one scenario. If you look at the LA Angels with the best win% in the 2014 regular season at 0.605, this was achieved principally by their hitting strength with 773 runs scored (17.3% better than the MLB average). The Angels allowed 630 runs which was only 4.4% better than the MLB average.
Achieving that magnitude of performance improvement is a tall order for any organisation and would require a strategic plan over a number of seasons involving player trading, draft picks, player development and financial planning. But the key point is that James’s formula helps formalise the task more precisely and provides a means of evaluating how alternative courses of action could contribute to the strategic goal of the organisation.

Benchmarking with Expected Wins: Altitude Effects in Denver, Loss of Form in Oakland
Another application of Pythagorean expected wins is as a useful benchmarking device to identify large anomalies between actual win% and predicted win%. Two such anomalies stand out in the 2014 data. The Colorado Rockies had an actual win% of only 0.407, the 2nd lowest, but their Pythagorean win% was significantly higher at 0.460. This deviation is largely due to the impact of the Rockies playing their home games at altitude in Denver. Games involving the Rockies that season averaged 19.1% more runs than the MLB average.
The other big anomaly that season were the Oakland A’s and shows the effect of an extremely inconsistent season. The A’s had a league-best 59 wins and 36 losses at the All-Stars break (win% = 0.621) and still led the Angels in the AL West in early August (9th Aug: 72 wins, 44 losses, win% = 0.621) but thereafter their season collapsed with the A’s losing 30 of their last 46 games and only scraping into the post-season Wild Card game by winning their final regular-season game. But the A’s had been so good in the first two-thirds of the season that their season totals of runs scored and runs allowed still predicted that they should have had the best win% rather than finishing 10 games behind the Angels, their divisional rivals.

Do Squares Yield The Best Predictor?
James’s Pythagorean expected-win model has stood the test of time as a very useful and accurate predictor but its accuracy can be improved upon by adjusting the Pythagorean parameter (i.e. power used). In the case of the 2014 regular season, predictive accuracy can be improved by using 1.810 (based on minimising total squared deviations). This is in line with various other studies such as Kaplan and Rich in 2017* who found that the best fit in individual seasons 2007 – 2016 varied from a minimum of 1.63 to a maximum of 1.96. (Kaplan and Rich’s method gives a Pythagorean parameter of 1.79 for the 2014 season which is very close to my own findings.) All of which goes to confirm that James’s original insight back in the 1970s still remains a very good approximation to MLB reality 40 years later.

Does the Pythagorean Expected-Wins Model Apply to Other Team Sports?
There have been attempts to apply James’s model to other team sports. Kaplan and Rich in their study report results for 2007 – 2016 for the other Major Leagues. They found that the NFL (American Football) and NHL (ice hockey) also work well using squared scores although the expected-wins model works better in the NFL using a Pythagorean parameter around 2.8. Predictive accuracy in the NBA (basketball) requires a parameter in the range 12 – 14. Kaplan and Rich show that the differences in the Pythagorean parameter across sports depend on the average score per game and the average winning margin.
And what about (association) football? The first complication is that football allows for tied games. It is not unusual for 20% – 25% of games to finish tied. This is further complicated by football’s 3-1-0 points system. I have found that the most useful way to apply the Pythagorean expected-wins model in European football is to treat tied games as “half-wins” for the purpose of calculating team win%. The alternative is to use the points percentage (i.e. total league points as a proportion of the maximum attainable). However typically you find that even with the adjustment for tied games, squared scores do not predict very well at the extremes. Table 2 provides a good example of the problem using the FA Premier League for 2013/14. Using squared goals massively over-predicts the win% of the top three teams and under-predicts the win% of the bottom three.

Table 2: FA Premier League, England, 2013/14

Blog 18.06 Graphic (Table 2)

Predictive accuracy in European football is much improved by using a Pythagorean parameter much closer to unity. Using the method of minimising total squared deviations I find that 1.232 works best for the FA Premier League that season. A Pythagorean parameter close to unity in European football fits with the common finding that goal difference is the best simple score-based predictor of league performance. So there really is no need to complicate things; in football just use goal difference to predict league performance.

The Bottom Line
Ultimately what is transferable from MLB across all team sports is not the specifics of the Pythagorean expected-wins model per se. Rather it is the discipline of projecting the expected performance gains from any significant player recruitment decision. Given the size of the financial commitments involved in the salary, acquisition and development costs of elite players, it is only rational to try to project the expected benefits. Some will argue that sport is different and that the expected impact of a new player cannot be quantified. But as soon as you have signed the contract, you have quantified the value of that player financially irrespective of whether or not you believe sport is different. Costs are costs in sport as in business. Where sport differs is in the resistance of some to subjecting their expectations of performance gains to due diligence. You would expect the financial director to subject other major investment by your organisation to proper due diligence using project appraisal techniques. So why not adopt the same logic to sporting investments? That ultimately was the whole point of the Moneyball story which popularised the strategic possibilities for the Pythagorean expected-wins model.

*Edward H. Kaplan and Candler Rich, ‘Decomposing Pythagoras’, Journal of Quantitative Analysis of Sports, vol. 13, no. 4 (2017), pp. 141- 149.

Excelling in Analytics

Executive Summary
• Effective data analysts need to have great Excel skills; expect 80% of your analytics to be conducted in Excel
• Too often data analysts do not receive enough training on Excel especially at universities, partly because of “software snobbery” and partly because of a lack of appreciation of Excel’s extensive analytical functionality
• The six specific functions in Excel with which every analyst should be very familiar are:
o Pivot Tables
o Custom Sort and Filter
o Conditional Formatting
o Graphics
o Formulas Menu
o Add-Ins

I’ve been asked many times over the years for advice on how to be an effective data analyst. I always stress the importance of having really good relationships with the decision-makers for whom you are working. Always ensure before you start any data analysis that you are clear on how your analysis is going to be used – which decisions is it going to feed into? Analytics after all is analysis with real-world purpose.
The other practical advice I regularly give data analysts is to ensure that you have got great Excel skills. I usually support this with a couple of anecdotes. One is that of an economics graduate with an excellent degree from a well-respected business school who was turned down for a post with a central bank because of a lack of Excel skills. The other is about a meeting I attended that was hosted by the local branch of the OR Society a couple of years ago. Four data analysts were invited to talk about their work experiences. During the Q&A they were asked what they wished that they had done more of at university and all four agreed that they wished they had been taught Excel. They estimated that 80% of their work was done in Excel and only 20% in more specialised software yet all had only been taught how to use specialised statistical software at university. These analysts had gone to different universities and studied different courses but their experiences were common to most data analysts. In part the lack of Excel training at universities is “software snobbery” – it’s sexier to teach more advanced software such as R, SPSS and SAS. But I also think that it is because of a misperception that Excel has limited analytical functionality. Nothing could be wider of the mark. In my own case I use Microsoft Access to store and combine datasets; I use Excel to do the bulk of my data manipulation and analysis; for multiple regression and other advanced statistical techniques I mostly use PC-Give (an econometrics package that I have used for years and which is particularly useful for investigating structural breaks in relationships).
There are many reasons why Excel is so widely used by data analysts. Microsoft Excel and Microsoft Office are so widely available that there are very few organisations that do not operate in a Microsoft environment. Excel is very convenient for basic data preparation. It is easy to import data into Excel, clean it up, transform it as required, and export it to other packages. But Excel has extensive analytical functionality as I will discuss further and can be supplemented by a wide range of add-ins provided by Microsoft and other sources. Excel is particularly suited to analysts who prefer the transparency and flexibility of programming within a spreadsheet environment to meet their specific needs as they arise. If, like me, you prefer to check the output of your programming instantly with a real example then Excel is for you. And, crucially, given that most of your colleagues will have Excel on their PCs, laptops, tablets and mobile devices, it becomes a convenient communication tool. You might find it desirable at times to send your spreadsheets to colleagues. It is quite common for organisations to track their KPIs graphically in dashboards constructed in Excel.
There are six specific functions in Excel with which I think every data analyst should be very familiar and which I use extensively in my own work in sports analytics. The list is by no means comprehensive and I am sure others would consider other Excel functions to be just as important if not more so but these are six Excel functions without which my data analysis for teams would be much more time-consuming and probably much less effective.

1. Pivot Tables
Pivot tables are a very efficient means of combining rows of data and calculating summary statistics. They are particularly useful when you are dealing with large datasets when it could take considerable time to sort the data into sub-groups and then calculate summary statistics for each sub-group. I hate to think of the amount of time I wasted in my early days doing just that in ignorance that the Pivot Table function would do it in seconds. The example below is taken from my season review of the 2014/15 Aviva Premiership. I had a summary row of KPIs for every team in every regular season game. Only 264 rows of data (12 teams x 22 games) but it would have taken several minutes to sort the data for each team and calculate season averages for each team. The Pivot Table below did it all in seconds.

Figure 1: An Example of a Pivot Table

Blog 18.05 Graphic (Pivot Table)

2. Custom Sort and Filter
The Custom Sort and Filter icon in the Home menu is definitely one of my most clicked icons in Excel. The Custom Sort function allows you to order your rows by multiple criteria. I am forever using it to sort team and player performance data which I load as it becomes available but will often want it grouped by team/player and ordered by date. The Filter tool is also very useful. It allows you to select sub-samples of relevant data, again using multiple criteria. Only yesterday I used Filter to extract data on every possession in the Aviva Premiership this season. My full dataset for this season has around 140k rows of data from which I extracted data on 6,550 possessions. It took two clicks to get the possession data using Filter. I copied and pasted it into another spreadsheet and was ready to roll on analysing these possessions. It is also worth remembering that there is a Filter option within Pivot Tables which is particularly useful if you want to analyse different types of sub-samples. Instead of filtering the data to create spreadsheets for each sub-samples and then producing pivot tables, you can work with the initial spreadsheet of data throughout and just filter when you are constructing your pivot tables.

3. Conditional Formatting
Conditional formatting is a very useful facility available in the Home menu to visualise differences in your data using colours, data bars or a variety of icons include traffic lights and flags. I make very extensive use of colour-coding of KPIs using Conditional Formatting. I have included below an extract from a recent analysis of the EFL Championship. It immediately highlights the tactical differences between teams as regards whether or not they play a possession-based style (e.g. Brentford compared to Burton Albion and Cardiff City), as well as highlighting that Birmingham City and Burton Albion rank poorly in some of the critical shooting KPIs.

Figure 2: Colour-Coding KPIs using Conditional Formatting

Blog 18.05 Graphic (Cond Format)

4. Graphics
One of the main complaints that I and other users had with the early versions of Excel was that the graphics facility was limited in its options and not user-friendly. I found that once I had created a graphic that did the job, I would go back and just copy and edit it when I needed that type of graphic again. But as data visualisation has become more and more important Microsoft have massively improved the graphics facility in Excel to generate a wide variety of graphics that are easily edited with pull-down menus. On my last count I found 53 basic graphical templates available in Excel. Excel’s graphic facility now has smart functionality. If you highlight the data to be graphed and then click on the Recommended Chart icon in the Insert menu, Excel will identify the most useful graphic templates for your specific data.

5. Formula Menu
The functionality in the Excel Formulas menu is often not fully appreciated. There is so much more available in Excel’s Function Library beyond the extensive array of data analysis functionality in Math & Trig, Statistical (found in More Functions) and Financial. The functionality in Logical and Text can be very powerful for editing textual data.

Figure 3: Excel’s Function Library

Blog 18.05 Graphic (Formulas)

6. Add-Ins
Excel comes with some optional functionality in the form of add-ins that need to be loaded via the File menu (select Options, then Add-Ins, then Manage Excel Add-Ins, then tick the boxes for the add-ins you want to load) when first used. There are two add-ins that I use frequently – Data Analysis and Solver. The Data Analysis add-in provides a set of statistical macros that provide an efficient one-stop facility to undertake a number of related statistical functions. Options in Data Analysis include ANOVA (both single factor and two factor versions), Correlation, Regression and t Tests (both for paired and independent samples) amongst many others. The Solver add-in is an optimisation facility which is very useful not only for undertaking standard OR problems but also allows you to construct a variety of more advanced statistical methods such as those using Maximum Likelihood e.g. logistic regression. There are also a variety of “unofficial” add-ins available for Excel. I use several add-ins provided by Conrad Carlberg to accompany his brilliant guides to doing analytics in Excel (see below).

So my basic message to data analysts, both those just starting as well as old-hands, is to regularly explore the functionality of Excel. You will often be surprised as to just how much Excel can do for you. And to educators a plea that you ensure that your students get a good introduction to Excel – it is a basic life skill if you are going to work anywhere that uses data especially if you are going to have any responsibility for managing performance.

 

Some Further Reading on Excel for Analytics

There are a series of books by Conrad Carlberg providing excellent introductions to analytics using Excel:

Conrad Carlberg, Statistical Analysis: Microsoft Excel 2013, Que, Indianapolis, IN, 2014.

Conrad Carlberg, Decision Analytics: Microsoft Excel, Que, Indianapolis, IN, 2014.

Conrad Carlberg, Predictive Analytics: Microsoft Excel, Que, Indianapolis, IN, 2013.

For a comprehensive coverage of analytical methods using Excel (with excellent appendices on the basics of Excel and Access), try:

J. D. Camm, J. J. Cochran, M. J. Fry, J. W. Ohlmann, D. R. Anderson, D. J. Sweeney and T. A. Williams, Essentials of Business Analytics (2nd edn), Cengage Learning, Boston, MA, 2017.
Extensive but expensive!

Improving Performance Ratio Analysis Part 2: The Structured Hierarchy Approach

 

Executive Summary
-Practitioners are often criticised for using performance ratios in a very piecemeal and fragmented fashion
-The problem is often compounded by more holistic approaches such as the balanced scorecard which encourages reporting of a very diversified set of performance metrics often with little understanding of their interdependencies and links to strategic goals
-The structured hierarchy approach provides a systematic framework for the forensic investigation of performance trends and benchmark comparisons
-The structured hierarchy approach can use a formalised mathematical structure but this is not necessary nor may it be appropriate in some contexts
-The structured hierarchy approach requires that the analyst has a clear understanding of the overall structure of the process being analysed and of how the performance ratios are related to the various components of the process
-Statistically significant differences between ratios at one level are not necessarily evident at other levels

As well as ignoring the various statistical and other methodological issues with performance ratio analysis, another strand of criticism directed at practitioners has been the tendency to use performance ratios in a very piecemeal and fragmented fashion. Again this has been a very common criticism of financial ratio analysis. And in some ways the problem was made worse by another criticism that there is too much emphasis on financial performance in assessing business performance. This led to Kaplan and Norton proposing the balanced scorecard approach in which four dimensions of business performance are identified – financial, customer, business processes, and learning and growth – with businesses encouraged to monitor a set of KPIs for each dimension. Although the need for a more holistic approach to performance is well taken, in practice the balanced scorecard approach has just compounded the problem by leading to a greater range of performance metrics being reported but still used in a very piecemeal and fragmented fashion. Ittner and Larckner, in particular, have been very critical of the balanced scorecard approach, arguing that the performance metrics are seldom linked to the strategic goals of a business, the supposed links between the metrics and overall performance are not validated and tend to be more articles of faith than evidence-based, and, as a consequence, the balanced scorecard does not lead to the right performance targets being set.
But again the problem of a fragmented approach to performance ratio analysis has been recognised in finance and a more structured approach has been adopted by some and often referred to as the Du Pont system recognising the chemical conglomerate that first popularised the approach. Others have called it the pyramid-of-ratios approach. The basic idea is to take an overall performance ratio and then decompose it into constituent ratios. For example, the return on assets (ROA) is calculated as the ratio of profits to assets. ROA can be decomposed into two constituent ratios – asset turnover (= sales/assets) and profit margin (= profit/sales). These two ratios capture the two fundamentals of any business – the ability to “sweat the assets” to generate sales (as measured by asset turnover) and the ability to extract profit from sales (measured by the profit margin). So if you want to understand changes in a company’s ROA over time or you want to explain the difference in ROA between companies, you can use this structured approach to determine whether the changes/differences in ROA are due mainly to changes/differences in asset turnover which reflects external market conditions, or changes/differences in the profit margin which reflects internal production conditions. The simple ROA pyramid is summarised in Figure 1. It can be extended in both directions, upwards by relating ROA to other rates of return, and downwards by further decomposing asset turnover and profit margin.
Figure 1: The ROA Pyramid

zzzz

The Du Pont/pyramid-of-ratios approach is an example of what I call the structured hierarchy approach and provides a systematic framework for the forensic investigation of performance trends and benchmark comparisons. In particular the hierarchical structure facilitates a more efficient analysis of performance by first identifying which aspects of performance primarily account for the differences/changes in performance overall and then tunnelling down into those specific aspects of performance in more detail.
Although the structured hierarchy approach as applied in financial performance analysis often uses a multiplicative decomposition in which performance ratios are decomposed into a sequence (or “chain”) of ratios, the product of which equals the higher-level ratio, there is no need to impose such a formalised mathematical structure. You don’t need to adopt a “one-size-fits-all” approach to creating a structured hierarchy. Multiplicative decomposition is particularly useful when dealing with processes that can be broken down into a sequence of sub-processes in which the output of one sub-process provides the input for the next sub-process in the sequence. In some cases it might be more useful to apply a linear decomposition in which a ratio is broken down into the sum of a set of constituent ratios. Linear decomposition is useful when a higher-level performance ratio depends of two or more activities that are separable and relatively independent of each other. But in many cases the structured hierarchy approach is best seen as a much more informal structure without any specific mathematical structure imposed on the relationships between performance ratios. The key point is that the structured hierarchy approach requires a clear understanding of the overall structure of the process being analysed and of how the performance ratios are related to the various components of the process.

zzz

Most of my work is in the invasion-territorial team sports mainly (association) football and rugby union. When putting together a system of KPIs to track performance, I always adopt a structured hierarchy approach. The approach is quite generic across both sports as I have summarised in Figure 2. The win percentage depends on the score difference between scores made and scores conceded (a linear decomposition). Typically these performance metrics are reported as game averages to facilitate comparisons between teams and over time. Scores made represents attacking effectiveness and naturally leads you to tunnel down into the different aspects of attacking play. In football I tend to separate attacking play into three dimensions – passing, other attacking play (e.g. crosses and dribbles), and shooting. When it comes to scores conceded I tend to separate this into exit play and defence. Exit play is a familiar term in rugby union but little used in football. Working across these sports I am particularly interested in their tactical commonalities especially the territorial dimension. I plan to post in more detail on this in the near future. But suffice to say at the moment that my experience working in rugby union particularly with Brendan Venter has made me even more acutely aware of the importance of play in possession deep in your own half. Lose possession there and you are going to cause yourself trouble. It’s what I call a SIW (self-inflicted wound). The same tactical considerations are equally applicable in football and underpin the use of a deep pressing game to maximise the number of times opponents can be pressurised into losing possession deep in their own half. A pressing game is all about reducing the effectiveness of opposition exit play. As I said I will pursue this line of thinking in more detail in a subsequent post.
Figure 2: A Generic Structured Hierarchy Approach for Invasion-Territorial Sports

zz

One final point to bear in mind when working with performance ratios as a structured hierarchy. Statistically significant differences between ratios at one level are not necessarily evident at other levels. Again this is a problem that has bedevilled research in financial performance analysis. For example, research on the impact of location on business performance usually found significant differences in profitability between urban and rural locations but the urban-rural differences were often no longer statistically significant when profitability was decomposed. But just because there are statistical differences in ratios at one level in a structured hierarchy does not in any way imply that these statistical differences should be observed at other levels. One sporting example of this is the score difference. By definition if you analyse differences between winning and losing performances, the score difference will always be statistically significant – positive when a team wins, negative when a team loses. However when you break this down for individual teams it does not always follow that there are statistically significant differences in scores made and scores conceded. For some teams winning and losing is much more about the variation in their attacking effectiveness than the effectiveness of their exit play or defence. So in a win-loss analysis these teams will tend to have statistically significant differences in scores made but not in scores conceded. It can go the other way for teams where defensive effectiveness is the crucial performance differentiator.

Improving Performance Ratio Analysis Part 1: Some Lessons from Finance

Executive Summary
• Performance ratios are widely used because they are easy to interpret and enhance comparability by controlling for scale effects on performance
• But performance ratios are susceptible to a number of potential problems that can seriously undermine their usefulness and even lead to misleading recommendations on how to improve performance
• The problems with performance ratios are well known in finance but are largely ignored by practitioners
• Crucially performance ratio analysis assumes that scale effects are linear
• Before using performance ratios, analysts should explore the shape of the relationship between performance and scale, and check for linearity
• If the performance relationship is non-linear, group performances by scale and use appropriate scale-specific benchmarks for each group
• Remember effective performance ratio analysis is always trying to compare like with like

It is very common for KPIs to be formulated as performance ratios. The reason for this is very simple. Ratios can enhance comparability when there are significant scale effects. For example, it tells us very little if we compare the total activity levels of two players with very different amounts of game time. We would naturally expect that players with more game time will tend to do more. In this situation it makes more sense to control for game time and compare instead activity levels per minute played.

Blog 18.03 Graphic (Box)

As well as controlling for scale effects on performance levels, ratios can also control for size effects on the degree of dispersion which can create problems for more sophisticated statistical modelling such as regression (the so-called heteroscedasticity problem).
However, despite the very widespread use of performance ratios, there are a number of potential problems with using ratios, some of which can seriously affect the validity of any conclusions drawn about performance and even lead to misleading recommendations on interventions to improve performance. The problems with performance ratios are well known in finance where financial ratio analysis is the standard method for analysing the financial performance of businesses. Hence I believe that there are lessons to be learnt from financial ratio analysis that can be applied to improve the use of performance ratios in sport.
One of the key messages in the debates on the use of performance ratios in finance is the importance of recognising that ratio analysis implies strict proportionality. What this means is best explained diagrammatically. Suppose that we want to compare two performances, A and B, where B is a performance associated with a larger scale. Suppose also that we know the expected (or benchmark) relationship between scale and outcome and that both of the observed performances lie on the benchmark relationship. In this case performance ratio analysis would be useful only if the outcome-to-scale ratio is equal for A and B. Graphically the outcome-to-scale ratio represents the slope of the line from the origin to the performance point. It follows that A and B can only have the same performance ratio if they both lie on the same line from the origin. This is strict proportionality and is shown in Figure 1(a). Comparing performance ratios against a single benchmark ratio value presupposes that the scale-outcome relationship is linear with a zero intercept. If either of these assumptions does not hold, then it is no longer valid to draw conclusions about performance by comparing performance ratios. This is a really important point but ignored by the vast majority of users of performance ratios.
The problems of non-zero intercepts and non-linear relationships are illustrated in Figures 1(b) and 1(c). In both cases A and B are on the benchmark relationship but their performance ratios (represented by the slopes of the blue lines) differ. In these cases performances ratios become much more difficult to interpret. It is no longer necessarily the case that differences between performance ratios can be interpreted as deviations from the benchmark, implying better/worse performance after controlling for scale effects. Effectively the problem is that the scale effects have not been fully controlled so that differences in performance ratios are still partly reflecting scale effects on performance.

Blog 18.03 Graphic (Fig 1)
So what is to be done? It becomes even more important to undertake exploratory data analysis to understand the shape of the relationship between performance and the relevant scale measure. At the very least you should always plot a scatter graph of performance against scale. If it looks as if there a non-zero intercept (i.e. there is a non-scale-related component in performance), then re-calculate the performance ratio using the deviation of performance from the non-zero intercept. If the performance relationship looks to be non-linear, then categorise your performances into different scale classes and use a range of values for the benchmark ratio appropriate for different scales. For example, in association football, the number of passes is often used as a scale measure for performance ratios. But we would expect very different ratio values for teams playing a possession-based, tiki-taka passing style compared to teams adopting a more direct style. Unless the underlying benchmark relationships exhibit strict proportionality, different benchmarks should be used to evaluate the performances of possession-based teams and direct-play teams. Always try to compare like with like.
There are two other statistical problems with ratio analysis that should also be noted. First, if the same scale measure is used in several performance ratios, this can influence the degree of association between the ratios. It is called the spurious correlation problem and was first identified in the late 19th Century in studies of evolutionary biology. Using common denominators in ratios can create the appearance of a much stronger relationship between different aspects of performance than there actually exists. In some circumstances common denominators can obscure the degree of relationship between different aspects of performance. Another statistical problem with ratio analysis is that ratios can exaggerate the degree of variation as the denominator gets close to zero and the ratio becomes very large. It is crucial to be aware of these outliers since they can have undue influence on the results of any statistical analysis of the performance ratios.
Some researchers in finance have recommended abandoning financial ratio analysis and using regression analysis. But regression analysis brings its own methodological issues and is not always applicable. It also ignores the reasons for the widespread use of performance ratios, mainly their simplicity. What is needed is better use and better interpretation of performance ratios informed by an awareness of the potential problems. In addition, we need to use performance ratio analysis in a more systematic fashion which is the subject of my next post.

Ranking Teams by Performance Rather than Results: Another Perspective on International Rugby Union Rankings for 2017

Executive Summary

• Competitor ranking systems tend to be results-based
• Performance-based ranking systems are more useful for coaches by providing a diagnostic tool for investigating the relative strengths and weaknesses of their own team/athletes and opponents
• Performance-based rankings can be calculated using a structured hierarchy in which KPIs are combined into function-based factors and overall performance scores
• A performance-based ranking of international rugby union teams in 2017 suggests that the All Blacks are still significantly ahead of England mainly due to their more effective running game

Most competitor ranking systems are results-based and use either generic ranking algorithms such as the Elo ratings (first developed to rank chess players) or sport-specific algorithms often developed by the governing bodies. As well as their general interest to fans and the media, these rating systems can often be of real practical significance when used to seed competitors in tournaments. These results-based ranking systems can be very sophisticated mathematically and usually incorporate adjustments for the quality of the opponent as well as home advantage and the status of matches/tournaments. These ranking systems also tend to include results from both the current season and previous seasons, usually with declining weights so that current results are more heavily weighted. A good example of an official results-based ranking system in team sports is the World Rugby rankings.
From a coaching perspective, results-based ranking systems are of very limited value beyond providing an overall comparison of competitor quality. What coaches really need to know is why their own team/athlete and opponents are ranked more highly or not. Opposition analysis is about identifying strengths and weaknesses of opponents in order to devise a game plan that maximises the opportunities created by opponent weaknesses, and minimises the threats from opponent strengths (i.e. SWOT analysis). Opposition SWOT analysis requires a performance-based approach that brings together a set of KPIs covering the various aspects of performance. A performance-based rankings system can provide a very useful diagnostic tool that allows coaches to investigate systematically the relative strengths and weaknesses of their own team/athletes or opponents, and help inform decisions on which areas to focus in the more detailed observation-based analysis (i.e. video analysis and/or scouting).
As an example of a performance-based ranking system, I have produced a set of rankings for the 10 Tier 1 teams in international men’s rugby union (i.e. the teams comprising the Six Nations and the Rugby Championship) for 2017. These rankings are based on 36 KPIs calculated for every match involving a Tier 1 team between 1st January 2017 and 31st December 2017. In total the rankings use 118 Tier 1 team performances from 69 matches. The ranking system comprises a three-level structured hierarchy. It is a bottom-up approach which starts with 36 KPIs which are combined into five function-based factors which, in turn, are combined into an overall performance score.

Blog 18.02 Graphic (Fig 1)

There are several alternative ways of combining the KPIs into function-based factors and an overall performance score. Broadly speaking the choice is between using expert judgment or statistical methods (as I have discussed in previous posts on player rating systems). In the case of my performance rankings for international rugby union, I have used a statistical technique, factor analysis, to identify 5 factors based on the degree of correlation between the 36 KPIs. Effectively factor analysis is a method of data reduction that exploits the common information across variables (as measured by the pairwise correlations). If two KPIs are highly correlated this suggests that they are essentially providing two measures of the same information and so could be usefully combined into a single metric. Factor analysis extracts the different types of common information from the 36 KPIs and restructures this into a smaller set of independent factors. The five factors can be easily interpreted in tactical/functional terms (with the dominant KPIs indicated in parentheses):
Factor 1: Attack (metres gained, defenders beaten, line breaks, Opp 22 entry rate)
Factor 2: Defence (tackles made, tackle success rate, metres allowed)
Factor 3: Exit Play, Kicking and Errors (Own 22 exit rate, kicks in play, turnovers conceded)
Factor 4: Playing Style (carries, passes, phases per possession)
Factor 5: Discipline (penalties conceded)
The factors are calculated for every Tier 1 team performance in 2017, averaged for each Tier 1 team, adjusted for the quality of the opposition, rescaled 0 – 100 with a performance score of 50 representing the average performance level of Tier 1 teams in 2017, and normalised so that around 95% of match performances are in the 30 – 70 range. The results are reported in Table 1 with the results-based official World Rugby rankings included for comparison. (It should be noted that the official World Rugby rankings cover all the rugby-playing nations, allow for home advantage and include pre-2017 results but exclude the tests between New Zealand and the British and Irish Lions.)

Blog 18.02 Graphic (Table 1)

Despite the differences in approach between my performance rankings and the official World Rugby rankings, there is a reasonable amount of agreement. Based only on 2017 performances, the gap between New Zealand and England in terms of performances remains greater than suggested by the official rankings. Also Ireland rank above England in performance but not in the official rankings, suggesting that Ireland’s narrow win in Dublin in March to deny England consecutive Grand Slams was consistent with the relative performances of the two teams over the whole calendar year.
Of course, the advantage of the performance-based approach is that it can be used to investigate the principal sources of the performance differentials between teams. For example, England rank above New Zealand in three out of five of the factors (Factors 2, 3, 5) and only lag slightly behind in another factor (Factor 4). The performance gap between England and the All Blacks is largely centred on Factor 1, Attack, and principally reflects the much more effective running game of the All Blacks which averaged 517m gained per game in 2017 (the best Tier 1 game average) compared to a game average of 471m gained by England (which ranks only 5th best). It should also be noted that All Blacks had a significantly more demanding schedule in 2017 in terms of opposition quality with 8 out 14 of their matches against top-5 teams (with the Lions classified as a top-5 equivalent) whereas England had only 2 out of 10 matches against top-5 opponents.

 

Small is Beautiful: Big-Data Analytics and the Big-to-Small Translation Problem

Happy New Year. And apologies for the lack of posts on Winning With Analytics over the last year. Put it down to my Indiana-Jones-type existence, a university prof by day, and a sports data analyst by night. This duality of roles became even more hectic in 2017 as I returned to rugby union to work again with Brendan Venter now Technical Director at London Irish as well as assisting South Africa and Italy. I have also continued my work with AZ Alkmaar in Dutch football. To some I might seem to be a bit of a dilettante, trying to work simultaneously at an elite level in two different sports. Far from it. Much of the insights on game tactics and analytical methods are very transferable across the two sports. The last 12 months have probably been one of my most productive periods in developing my understanding of how to best use data analytics as part of an evidence-based approach to coaching. I hope to share much of my latest thinking with you over the coming months with regular posts.

Executive Summary
• Data analytics is suffering from a fixation with big-data analytics.
• Big-data analytics can be a very powerful signal-extraction tool to discover regularities in the data.
• But big-data exacerbates the big-to-small translation problem; big-data, context-generic statistical analysis must be translated into practical solutions to small-data (i.e. unique), context-specific decision problems.
• Sports analytics is most effective when the analyst understands the specific operational context of the coach, produces relevant data analysis and translates that analysis into practical recommendations.

The growth in data analytics has been closely associated with the emergence of big data. Originally “big data” referred to those really, really big databases that were so big as to create significant hardware capacity problems and required clusters of computers to work together. But these days the “big” in big data is, much like beauty, in the eye of the beholder. IBM categorise big-data analytics in terms of the four V’s – Volume (scale of data), Velocity (analysis of streaming data), Variety (different forms of data), and Veracity (uncertainty of data). The 4 V’s capture the core problems of big-data analytics – trying to analyse large datasets that are growing exponentially with data captured from multiple sources of varying quality and reliability. I always like to add a fifth V – Value. Big-data analytics must be relevant to the end-user, providing an evidential base to support to the decision-making process.

Sports analytics, just like other applications of data analytics, seems to have been bitten by the big-data bug. In my presentation last November at the 4th Annual Sportdata & Performance Forum held in Zurich, I called it the “big-data analytics fixation”. I don’t work with particularly big datasets, certainly not big in the sense of exceeding the capacity of a reasonably powerful PC or laptop. The basic XML file produced by Opta for a single football match has around 250k data points so that a database covering all matches in a football league for one season contains around 100m data points. This is pretty small compared to some of the datasets used in business analytics but sizeable enough to have totally transformed the type of data analysis I am now able to undertake. But I would argue very strongly that the basic principles of sports analytics remain unchanged irrespective of the size of the dataset with which the analyst is working.

Big-data analytics exacerbates what I call the big-to-small translation problem. Big-data analytics is a very powerful signal-extraction tool to discover regularities in the data. Big-data analytics, like all statistical modelling, attempts to decompose observed data into systematic variation (signal) and random variation (noise). The systematic variation captures the context-generic factors common to all the observations in a dataset while the random variation represents the context-specific factors unique to each individual observation. But while analytical modelling is context-generic, decisions are always unique and context-specific. So it is important to consider both the context-generic signal and the context-specific noise. This is the big-to-small translation problem. Understanding the noise can often be just as important, if not more so, as understanding the signal when making a decision in a specific context. Noise is random variation relative to the dataset as a whole but random does not necessarily mean inexplicable.

I disagree profoundly with the rather grandiose end-of-theory and end-of-statistics claims made for big-data analytics. Chris Anderson in an article on the Wired website back in 2008 claimed that the data deluge was making the scientific method obsolete. He argued that there was no longer any need for theory and models since in the world of big data correlation supersedes causation. Indeed some have argued that big-data analytics represents the end of statistics since statistics is all about trying to make inferences about a population from a sample but big data renders sampling irrelevant when we are now working with population data not small samples. But evidence-based practice always requires an understanding of causation. Recommendations that do not take into account the specific operational context and the underlying behavioural causal processes are unlikely to carry much weight with decision-makers.

There is a growing awareness in sports analytics of the big-to-small translation problem. In fact the acceptance by coaches of data analytics as an important source of evidence to complement video analysis and scouting is crucially dependent on analysts being able to translate the results of their data analysis into context-specific recommendations such as player recruitment targets, game tactics against specific opponents, or training session priorities. It was one of the themes to emerge from the presentations and discussions at the Sportdata & Performance Forum in November 2017 (yet again an excellent and very informative event organised by Edward Abankwa and his team at the Pinnacle Group). As one participant put it so well, “big data is irrelevant unless you can contextualise it”. And in a similar vein, a representative of a company supplying wearable technologies commented that their objective is “making big data personally relevant”. Sports analytics is most effective when the analyst understands the specific operational context of the coach, produces relevant data analysis that provides an appropriate evidential base to support the specific decision, and translates that analysis into practical recommendations to the coach on the best course of action.

Bridging the Gap: Improving the Coach-Analyst Relationship (Part 2)

Executive Summary

  1. Analytical results are usually presented most effectively to coaches by using data visualisation and story-telling.
  2. Don’t ignore external commercial data if it is available and affordable.
  3. Data analysts can make a vital contribution to the organisation of training sessions.
  4. Data analytics is only one input into decision making by coaches, albeit a potentially very important one if used effectively.

 

  1. Analytical results are usually presented most effectively to coaches by using data visualisation and story-telling.

As well as the imperative of translating analytical results into practical recommendations framed in the language of coaches, a number of speakers stressed the importance of data visualisation and story-telling as communication devices. “A picture is worth a thousand words” has become even truer in the age of data analytics where effective data visualisation has become a vital tool for the analyst. Rob Carroll (The Video Analyst) illustrated this very well with his graphics on the quality of shooting opportunities in Gaelic football, a form of expected goals model. Ann Bruen (Metrifit) suggested that we should always have in mind the story we are going to tell as we collect and analyse the data. Ben Mackriell from Opta, whose core business is providing performance data, made the same point when he said that it is possible to have a conversation about data without actually mentioning the data (or the analytical techniques). Of course when it comes to evidence-based story-telling we must remain open-minded and allow the precise details and ending of the story to emerge from the analysis. There is always a danger of not allowing the data to get in the way of a good story, of pre-judging the results of the data analysis; it is what cognitive psychologists call confirmation bias. A good evidence-based story is a story that conveys analytical results in the language of coaches, focusing on the practical implications with explanations of athlete and team performance framed in terms of skill technique and tactical decisions. As Edward Metgod (Royal Dutch Football Association) pointed out, coaches are interested in causality not correlation. Analysts must translate the evidence of statistical associations into credible stories of cause and effect with clear implications for targeted interventions to improve performance. When all is said and done analytics is actionable insight.

The discussion of the importance of story-telling reminded me of the advice of Alfred Marshall on the use of mathematics in economics. Marshall probably did more than anyone to systemise economics as a subject and much of his mathematics and diagrams still remain in the textbooks. Marshall was very aware of the uses and abuses of mathematics. Economics was intended to be a practical subject about the everyday business of life but Marshall became increasingly concerned that economists assumed good mathematics meant good economics. He advised that if the mathematics could not be translated into English and then illustrated with important real-life examples (i.e. a good story), then it should be burnt. Apart from the health and safety issues (perhaps safer to shred than burn), Marshall’s advice holds good for data analytics too. If it doesn’t produce actionable insight, it is worthless.

 

  1. Don’t ignore external commercial data if it is available and affordable.

Any discussion of data analytics must include a discussion of the nature of the data being used. The Forum was a great place for this type of discussion giving that it brought together external and internal data providers, data analysts and end-users. In the past there has been too much emphasis on different types of data as substitutes whereas now there is greater acceptance of the complementarity of data. And that complementarity will get even better as there is more and more cross-over in personnel between teams and commercial data providers. Ben Mackriell at Opta is a good case in point, now in charge of OptaPro but with years of experience working with teams in rugby union and football. External commercial data offers consistency and coverage whereas internal data is team-specific and often includes expert coach evaluation of skill technique and tactical decision-making relative to the game plan. The differences between these two types of data are variously described as objective vs subjective, frequency vs evaluation, general vs expert, small data vs big data. The differences were well illustrated in the Q&A that followed Edward Metgod’s presentation when he was asked how he would define the transition phase of play. Edward replied as a coach and scout with a subjective/evaluation/expert definition that the transition represents the period of play after a team loses possession but has not gained its defensive shape. Transition is determined by tactical factors in contrast to more objective definitions in terms of a specific time period (e.g. the first five seconds after possession is lost) or the number of passes made by the team gaining possession. What is important to recognise is that these different types of data have different but complementary functions. For example, external data, possibly in the form of a player rating system, can be used at the first stage of player recruitment to identify a target group of players for whom at the second stage internal data is then produced by the team’s scouts. This is exactly the system of e-screening of potential player acquisitions that I recommended to Bolton Wanderers in 2005. Increasingly I am finding that my greatest and most interesting challenge as an analyst is to generate expert insight from non-expert data particularly in opposition analysis. Can I get inside the minds of the opposition coaches by studying the patterns in their data?

 

  1. Data analysts can make a vital contribution to the organisation of training sessions.

There were a number of speakers at the Forum whose specialism lay in strength and conditioning, and sports science. In addition the Forum also included presentations from coach educators. Both of these groups shared a concern with the optimal use of training time. As a qualified coach and university professor, I want to gain a deeper understanding of the skill-acquisition process whether it be how players learn to perform in games or how data analysts learn to be effective in teams. Nick Winkelman (IRFU) was the lead-off speaker at the Forum and made some great points on both skill acquisition and the role of analytics. As both Nick and several other speakers stressed, when it comes to effective learning, “context is everything” and randomised but relevant learning opportunities provide the most effective way of acquiring and retaining new skills. Blocked repetitions of a specific skill will improve the accuracy with which a skill is performed in a training session but this does not necessarily transform into a game context when the player must not only accurately execute the skill but also make the right decision as to when to execute that particular skill. Nick argued, rightly in my mind, that too much of the data analysis linked to training is focused on workload when what is also needed is a greater input into creating the appropriate game-related contexts.

 

  1. Data analytics is only one input into decision making by coaches, albeit a potentially very important one if used effectively.

The Forum brought together a diversity of specialisms involved in high performance sport. All agreed, albeit with greater or lesser conviction, that data analytics is potentially a very important coaching tool but its effectiveness had been often limited by poor communication particularly the failure of analysts to translate analytical results into actionable insight framed in the language of coaches. I came away from the Forum feeling positive about the future of data analytics in high performance sport. Data analytics is now being seen as another tool to complement scouting, video analysis and reporting. But analysts must guard against complacency. There is still much to do in many sports and in many teams to create a thorough-going commitment to evidence-based coaching. And we will only do that by “bridging the gap” and producing actionable insight relevant to day-to-day coaching decisions.