Capital Ideas Blog

How well do umpires make decisions?

By Daniel Storey, Chelsea Vail
July 31, 2015

From: Blog


How often are umpires wrong? When they’re wrong, what kinds of errors do they make? And do they show favoritism to the home team?

In many competitive settings—say, a financial trading floor—professionals don’t typically get the chance to review their decisions and reverse them without cost, even when they’re later found out to be mistaken.

But in cricket, the Decision Review System (DRS) allows teams to challenge umpire rulings and have them analyzed by a third, off-field umpire with access to audiovisual technology, introducing a new level of precision to decisions. 

Combined with ESPN’s extensive cricket stats, the DRS has given researchers an exceptional opportunity to learn about how well umpires make decisions, and how technology is changing the game. Chicago Booth’s Ram Shivakumar uses the data to create detailed analyses of 126 cricket matches, focusing on 1,201 challenges to on-field umpire decisions. He finds that about 26 percent of on-field umpire calls that get challenged are reversed upon review—making the conditional error rate, in which only challenged decisions are counted, much higher than the unconditional 8 percent touted by the International Cricket Council.

Further, the data confirm a phenomenon some cricket writers have observed: on-field umpires have become more aggressive in calling batsmen “out” than “not out.” Cricketers attribute this swing to the institution of the DRS, as the trend breaks with the sport’s tradition of giving batsmen the benefit of the doubt.

Shivakumar uses logistic regressions to determine that outs called by the on-field umpire are 83 percent more likely to be reversed than not-outs. He also finds that LBW decisions are 50 percent less likely to be reversed than “caught” decisions, and suggests that the ICC take greater care in training umpires to call “caught,” which is more difficult to judge than LBW. With 20 outs, or wickets, in a five-day test match, these individual calls can make all the difference in the overall outcome.

Is there any evidence of bias by the third umpire in favor of the home team? There’s some evidence for umpires’ home-team bias in cricket, but it does not appear to extend to the third, off-field umpire, who has the benefit of more time and resources to make calls, according to Shivakumar. It’s equally likely for a home team or an away team to successfully challenge a decision, and that “provides cautious support for experimenting with a home on-field umpire in test matches,” he writes.

The DRS, although it has demonstrably improved the quality of decision-making in cricket, has not been met with universal approval within the sport. If only there were an equivalent technology for the workplace.

To request a copy of this paper, please email Ram Shivakumar.

Ram Shivakumar, “What Technology Says about Decision-Making: Evidence from Cricket’s Decision Review System (DRS),” Working paper, May 2015.

Archive

Latest Comments