# Economists Do It With Models

## Fun With Math: I Need To Teach FIFA How To Perform A Cost-Benefit Analysis…Or Do I?

#### June 29th, 2010 · 16 CommentsFun With Math · Sports

For those of you who haven’t been following the World Cup, there are three notable features this time around: vuvuzelas, Landon Donovan (*swoon*) and bad calls by referees. (If you ever turned it on and wondered why it sounded like there was a swarm of bees in the stadium, now you know. Note that the sound is due to the vuvuzelas, not the bad calls.) For example, referees failed to see a shot go over the line in England’s 4-1 loss to Germany. As a result, FIFA- or more specifically the International Football Association Board- is considering adding goal judges to its arsenal of referees. I can only imagine that this would appease Bill Simmons to some degree, since he has seemed less than pleased with the World Cup officiating. For example, he tweeted the following during the England-Germany game:

Wow!!!! Was that the worst botched call in the history of sports??? How do they not have refs behind each goal? I’m in shock.

FIFA statement: “So you know, by not putting goal referees behind each net, we save upwards of \$500 per game. We stand by our decision.”

I can only assume that he was being sarcastic with the quote in the second tweet, but who knows. In any case, let’s think about how FIFA could decide whether the extra officials would be worthwhile under those circumstances.

It’s all too tempting to say “well, getting the outcome of a game right is clearly worth \$500, so FIFA is just being stubborn,” but that’s not really what the tradeoff is. If you stop and ponder for a second, you will probably realize that the above tradeoff assumes that people can know which games will have officiating problems and only pay for the extra referees for those games. Unfortunately, scientists have failed to develop a reliable crystal ball thus far, so FIFA realistically only has the options of “hire goal judges for all games” and “don’t hire goal judges.”

The real tradeoff looks something like this:

This is sort of hard- how are we supposed to know the value of a correct outcome or the probability that the extra official will matter? For the latter measure, we can at least approximate to some degree by looking at how many bad calls have been made historically divided by the number of games. (You could limit the time frame here to a period that looks reasonably similar to the present.) Note that I didn’t say that we could look at how many bad calls have actually changed the game outcome, since we can’t really know that. (I am also assuming to some degree that the extra referees would eliminate all errors, which isn’t necessarily the case but at least provides a starting point for discussion.) For example, England was denied one goal and lost to Germany 4-1, so one could argue that the denied goal was irrelevant. However, this doesn’t take into account the notion that the denied goal might have affected play in the rest of the game. (People are, after all, human.) If we are counting all bad calls made, we are probably overstating the true probability, and if we count only those bad calls that directly affected the game’s outcome, we are probably understating the true probability, but at least we can use these two numbers to get bounds on what the right number should be.

The value of a correct outcome is a more difficult problem to think about- after all, who really knows what dollar figure to place on a legitimate sports outcome? Does legitimacy impact viewship or fandom? Probably, but again, it’s hard to quantify. Does it affect sponsorship? There are a lot of similar questions that I wouldn’t even know how to approach. Luckily, I can avoid the issue by turning the underlying question on its head a little and instead asking “how much would a correct game outcome have to be worth in order for the benefits to outweigh the costs?” In this sense, if I end up with an outlandishly high or low number, I can pretty confidently answer the original question without thinking about lost merchandise sales and ad revenue directly.

For example, let’s say hypothetically that there is one (correctable) bad call in every 5 games (even that seems like a low estimate given current circumstances) and half of these actually affect the game outcome directly. This means that the probability of a bad call in a game is 1/5 and the probability of getting the wrong outcome in a game is 1/10. These numbers would imply that a correct game outcome would have to be worth between \$2500 (since \$2500 * 1/5 = \$500) and \$5000 (since \$5000 * 1/10 = \$500) in order to wake the \$500 cost worthwhile. If nothing else, I would venture to guess that the losing team would be willing to pony up five grand to not have unfairly lost, so these estimates appear to show that the benefits of the extra officials would outweigh the costs.

Note, however, that it could be the case that the actual number of officiating errors is much smaller- I’m not quite in the mood to go and do historical research on this matter. If the number of game-changing errors is 1 every 1000 games, then the value of a correct game outcome would have to be much higher (\$500,000) in order to justify the \$500 per game cost. In general, when the chance of error with the existing referees is lower(or the marginal benefit of the extra referees lower), the value of a correct outcome has to be higher in order to justify the cost of trying to ensure that outcome.

Or…you could just make the case that the \$500 per game amounts to the cost of somewhere between 3.6 and 25 tickets out of 40,000, depending on whether the ticket-holders are from South Africa. And that, people, is called price discrimination.

Tags: Fun With Math · Sports

### 16 responses so far ↓

• 1 Mike // Jun 29, 2010 at 7:17 pm

I am both a soccer referee and an economist, and I can attest from both perspectives that, as you might expect, the story is not quite as simple as you make it. It is far from certain that another \$500 on additional referees would improve the quality of officials. Let me make one of many possible points that I have been making to others recently in the simplest possible terms. Do you want an economist building your house? Me either, even if he or she is capable of handling most f the work. I want someone who builds houses for a living to build my house. Most referees are not professional referees but instead earn their livings outside the sport. Furthermore, the very best referees, even if they are professional referees, are usually excluded from officiating in advanced stages of the World Cup because they usually come from the countries whose teams remain (this is not likely a coincident, but I don’t know about causality). If you want better calls, demand and foster better officials. There are situations that may merit additional innovation in either the officiating system in place (e.g. goal line officials may actually be worth \$500) or the use of technology (e.g. fine players that video shows to have simulated a foul, but after the fact). I will conclude by saying that people who don’t really appreciate the game of soccer (I’m not trying to suggest this is you Jodi) should exercise caution in criticizing the officiating and proposing solutions based on other sports that they do appreciate. Thanks for the help, but we soccer fans really do like our game mostly the way it is and while we want good officiating we don’t want to muck up the sport either.

• 2 econgirl // Jun 29, 2010 at 8:01 pm

This is actually exactly the reason that I limited the discussion to the possibility of adding goal judges, since that is the only non-status quo option that seems to be even remotely on the table in any official capacity. (In other words, this is in no way my personal suggestion for improvement.) Technically speaking, you could do a similar analysis for, say, adding goal line technology and such, but then you would have to count not only the dollar costs on the cost side but also the “messing with the flow of the game” and similar costs as well.

It is true that I implicitly made the simplifying assumption that the additional referees would eliminate, for practical purposes at least, all mistakes. I did not mean to imply that they would be better or even different from the existing officials, rather that they would be another set of eyes looking at very specific situations.

• 3 Jorge Lara // Jun 29, 2010 at 8:40 pm

Mike is totally right, you oversee a lot of things. Maybe the causality between better teams and better referees is given by the fact that the better teams have better tournaments (La Liga, Calcio, Premier League, Eredivisie, and Argentine and Brazil leagues). In that sense the referees of those countries have tough games every weekend. Probably you are kind of confused by others game that you are more familiar with (like American Football), but saying that a referee behind each goal will help, it is actually kind of silly. Their marginal productivity of those referees would be close to zero, but you need first to understand the game to know why. I though this was a fun blog, but also an intelligent one.

• 4 econgirl // Jun 29, 2010 at 9:10 pm

As an extension to my previous comments, it’s worth noting that I didn’t come up with the idea that a referee behind each goal will help. I am merely analyzing the tradeoff to say that even IF we think that the extra officials would mitigate errors, it’s not necessarily the case that it justifies the cost if errors are reasonably infrequent in the first place. Moreover, even if the benefits outweigh the costs, the “if a correct score worth \$500?” question is not the right one.

To quote the article linked to above:

“Maingot said FIFA would consider adding two extra officials who would act as goal judges, one at each end of the field, a system that the Europa League — a major European club competition — used last season.”

If they are considering it, then at least some people deep within the soccer, er, football community must believe that the extra referees would add value.

• 5 Why do soccer players take so many dives? « The Unqualified Economist // Jun 29, 2010 at 9:26 pm

[…] Beggs at Economists Do It With Models does some back of napkin calculations on whether it would be beneficial for FIFA to spend an extra […]

• 6 TUE // Jun 30, 2010 at 9:37 am

I disagree with Mike. While, I don’t want an economist building my home, I think it would be pretty terrific if said builder consulted with an economist. It might lead to the elimination of those vestigial window shutters that don’t actually function and addition of something actually usable.

There is a difference between doing something drastic like adding instant replay or goal line technology than adding an additional goal line referee. Shouldn’t the most important aspect of the game — the scoring — have its own judge when the referee has the assistant referees to aid in boundary and offside calls?

• 7 Cory // Jun 30, 2010 at 12:02 pm

econgirl: I think sports officials could work perfectly within a pigou taxing system, taxing them for making bad calls. There will always be human error, so I would suggest something like- make 3 bad calls in a season and get ___ taken out of your paycheck (or something along those lines). This will allow for some human mistakes, but it could limit bad calls if their pay check is riding on the line.
Another option could be providing monetary incentive for not making bad calls….every 4 or 5 games without a blown call gives you an extra X amount of money. I recently watched this video: http://www.youtube.com/watch?v=dgKKPQiRRag , I obviously haven’t looked at the resarch but the video seems to follow some logic and the cartoons are awesome. If this video does explain behavior and incentives well, then the sports officials (mechanical skills) should have a positive reaction to monetary incentives.

what are your thoughts?

• 8 econgirl // Jun 30, 2010 at 12:34 pm

I think it depends on whether the bad calls are due to negligence or lack of information. Incentives can help to overcome negligence, but, if the refs simply don’t always have a good vantage point from which to see the goals accurately, providing carrots on sticks isn’t really going to change that. In fact, the monetary rewards and punishments could even make referees anxious enough such that their error rates go up rather than down.

As an example of this last point, consider baseball umpire Jim Joyce and his blown call on the last out of Armando Galarraga’s perfect game. It wasn’t a particularly close call, but it was one that was very important and very notable, and he screwed it up, despite the fact that his overall performance is very good. The chance that the timing of the error is random is pretty small.

• 9 Brad // Jun 30, 2010 at 9:11 pm

Bad calls can be due to negligence, lack of information, or a bent ref. It’d be nice to know if the ref is bent (biased, paid off, on the take, crooked), but I don’t know any way to tell if the ref is bent, bad, or in a poor position to make the call.

The assistant referee (AR or sideline or touchline judge) is usually the one to call a goal like in the England game. The best position is right on the corner flag. Unfortunately, the AR also calls off-side, which means the AR is supposed to be positioned with the last defender. So, for a long shot, the AR may be in poor position to tell if it was a goal or not. Can’t be in 2 places at once.

I’m not a referee. I just watch them on TV.

• 10 Joshua // Jul 1, 2010 at 1:47 pm

Great post! You do neglect to consider the soccer market’s impact on FIFA’s decision. Though monopolistic and entrenched in tradition, if FIFA were, for example, to save a lot more money by only having one ref on the field, they would encourage the rise of competing soccer organizations.

One thing they may be weighing is their market share and the potential for losing it. How much would \$500/game improve their market share, or lessen the potential for losing their monopoly?

• 11 Pablo Garcia // Jul 1, 2010 at 4:47 pm

I think that taxing the referees by punishing them for bad calls is a bad idea. It would lead to bad calls being made, and also, who gets to choose a bad call? Like basketball, which seems to have terrible officiating as well, many fouls are subjective in soccer. The only objective call made in soccer that everyone can agree on is the start of the game.

A better idea would be to fine players, similar to celebrations done by NFL players. You don’t see many celebrations because of these fines. Sometimes it just isn’t worth it. This would keep players from flopping so drastically or so frequently. This could be done by a post game committee that reviews fouls awarded by a ref of some sort. I’m not here to offer an in depth solutoin though.

I think that we all assume that in every country, every soccer game is relevant like the world cup. Which isn’t the case. I don’t think we’d be making a fuss similar to this if it had happened in the MLS. We’d probably see a replay of it on ESPN, say that soccer is stupid (which i disagree with), and go about our way.

Bill simmons brings up a good point, and for matches like the world cup, 500 for additional less error is tiny. But applying it to every game, every leauge around the world, and someone might have said this, but the quality of reffereess goes down with each additional ref (assuming you employ the best of the best and choose the next best to fill corresponding positions).

Interesting topic non the less.

• 12 The Economics of the World Cup @ Rational Reactor // Jul 14, 2010 at 4:42 pm

[…] about the World Cup that I enjoyed came from Jodi over at Economist Do It With Models. In it she performs a cost-benefit analysis to argue the case that FIFA should have extra referees at each match, specifically behind each goal […]

• 13 EDA GRONSKY // Nov 2, 2011 at 11:43 am

I used to be very pleased to find this web-site.I wished to thanks to your time for this glorious learn Fun With Math: I Need To Teach FIFA How To Perform A Cost-Benefit Analysis…Or Do I?!! I definitely enjoying every little bit of it :).

• 14 Y88 // Jul 12, 2013 at 2:32 pm

First off I want to say wonderful blog! I had a quick question in which I’d like to ask if you do not mind. I was curious to find out how you center yourself and clear your head before writing. I have had a hard time clearing my thoughts in getting my thoughts out. I do enjoy writing but it just seems like the first 10 to 15 minutes tend to be lost just trying to figure out how to begin. Any ideas or tips? Cheers!

• 15 Patio Deck Box // Jul 6, 2016 at 5:27 am

Pretty nice post. I simply stumbled upon your weblog
and wanted to mention that I’ve really enjoyed surfing around
your weblog posts. After all I will be subscribing on your feed and
I’m hoping you write again very soon!

• 16 vmware training in pune // Dec 24, 2016 at 2:31 am

nice one…