close window Close Window

Why Business Needs More Random Experiments

Most businesses make decisions without good feedback mechanisms, said Steven Levitt, Alvin H. Baum Professor in Economics and the College and director of the Becker Center on Chicago Price Theory. The “easy parts” of business - manufacturing, inventory, and logistics - all contain feedback, Levitt told a full house during a brown bag lunch discussion hosted by the Becker Center at Charles M. Harper Center October 10.

“The beauty of those types of activities within a firm is you get immediate feedback,” he said. “You can figure out whether you’re doing something wrong and respond to it.” For example, the production line at a Big Three automaker is able to reduce the number of errors on a new vehicle by 86 percent within a month and limit them to only a few errors within about two more months, Levitt said.

The “hard problems” are advertising, pricing, and less defined processes such as new product introductions, he said. “These are activities that usually go on in a firm without any real, accurate quantitative ways of measuring whether or not what you’re doing is working,” Levitt said.

“Pricing may be the best example of all,” he said. “Firms don’t set prices in the way economists think they should, with simple formulas about marginal costs and markups. In the real world, representatives of finance, marketing, and human resources sit around the room and debate what the price will be. Typically, there is not very much experimentation.”

Businesses need to perform more randomized experiments in natural settings, Levitt said. Companies such as Subway, Starbucks, and Capital One pick random markets to experiment with prices, products, and even the presentation of their products, he said. “Then they see whether or not good things happen in those markets, relative to other markets,” Levitt said.

Such experiences are powerful because the business world is so complicated, he said. “The power of randomization is that it cuts through the complexity,” Levitt said. “All you have to do is flip a coin and put people into treatment and control groups. In the end, you just have to be able to add it up and say, ‘Am I selling more or am I making more profit in the group that got the treatment versus the group that got the control?’”

Experiments can be powerful management tools if all business players are involved in the process, he said. “Another beauty is that they are incredibly cheap. You can often do them for free as part of what already doing,” Levitt said. “Almost invariably experiments are a way of reducing your risk. By introducing something on a small scale randomly, you get immediate feedback on whether it works or not and you can avoid big disasters, like New Coke.”

As a consultant, Levitt said he has experienced successfully implemented change through random experimentation. The Ambassadors Group, Inc. has grown its direct-marketing travel service by 20 to 30 percent a year, even after 9/11, by experimenting with mailings and pricing, Levitt said.

“Our analysis suggested their customers were not price sensitive at all,” he said. “We found about a 0 to 3 percent decline in quantity when we raised prices by 10 percent. Basically, it was like minting money for this company. Just by erasing one number on a sheet and replacing it with another number, we were increasing their bottom line profits by something like 15 to 20 percent.”


- Phil Rockrohr