What is the difficulty with evidence-based policy making?

Hansen: Of course evidence is important, but evidence never speaks for itself. It requires a conceptual framework, and that is just as important as the evidence. The conceptual framework has an impact on the policy recommendations. So it’s very important to clarify the rules both for the conceptual framework and for the evidence itself. There’s always a danger that policy makers can use evidence to support different policy platforms. It’s not a neutral playing ground in terms of scientific discourse. It’s very easy to use the political arena to distort what knowledge is really telling us.

Murphy: One of the reasons to understand the underlying framework is to understand what is evidence on a given question. Evidence is not just an experiment or something we ran on that specific question, but our experience with related and similar events historically as well as other policy and nonpolicy outcomes. If we’re interested in a particular tax, we don't want to just look at that one tax as an example of how people would respond. Other taxes can be very informative, and knowing which other cases are similar or not requires understanding the underlying theory. Evidence-based policy is sometimes focused too narrowly on what the relevant set of evidence is. It doesn’t draw on our broader knowledge as economists. For example, we have a wealth of evidence that when prices rise, people buy less. If, in a given situation, prices went up and people didn’t buy less, we’d say, “Am I looking at these data correctly, or is there something I’m missing?” I like to give this analogy: You don’t want to say, “I’ve never thrown a bowling ball off the roof of my garage. I don’t know whether it’s going to fall to the ground or float in the air.” The theory of gravity probably applies to bowling balls. There’s a very slim possibility that it doesn’t. The same is true in economics. If there’s a goofy result, we need to look at a broader set of evidence.

Hansen: One of the aims of econometric models is to take situations with lots of data and extrapolate to where we don’t. If we stick to requiring in-depth evidence on the exact question we want to ask, we’ll be limited in terms of what we can provide. Any hopes of trying to draw on a bigger pool of knowledge requires conceptual frameworks. That’s a big part of understanding what the evidence tells us.

Murphy: One of the problems with specific evidence is that it’s often focused on short-term responses. One thing we know in economics is that long-term and short-term responses tend to be different. People and markets can make lots more adjustments given time. Those are the hardest things to run experiments on, so experimental evidence often is limited.

Policy makers typically prefer certainty to uncertainty, and specific numbers to ranges. How much of a challenge is that for economists who want to influence policy?

Hansen: On the one hand, economists who want to influence policy often make statements with great confidence about outcomes. And then different economists will make conflicting statements, also with great confidence. So the public thinks there’s no agreement on some of the fundamental questions. That undermines our long-term impact in terms of policy. On the other hand, politicians want to tell the public exactly why they’re doing things, based on full knowledge and understanding, so they gravitate toward people who’re more certain. When the evidence is uncertain and there’s a range of possible outcomes, they tend to tilt it in the direction that supports a particular policy outcome. It’s important to understand when that tilt is taking place, because it comes from prior beliefs or policy aims. And that’s where things get really muddy.

Murphy: As economists, we want to broaden the set of evidence, but temper our predictions. Most of the time, when we predict a response, if it’s plus or minus 50 percent, that’s not a terrible prediction. Economists have had a big effect, for example, in improving macroeconomic policy around the world. A lot of that involved learning that we didn’t know as much as we thought. Probably the greatest improvement in macroeconomic policy has been the recognition that our ability to understand and manipulate the economy is a lot weaker than we thought. We’ve responded accordingly, with largely positive results. So, often, less message can actually be helpful because it discourages people from doing things that are out-and-out harmful.

Is the economics research process problematic—the way that early drafts of working papers get a lot of attention and are seized on by the media and policy makers?

Hansen: I find that potentially problematic. What we want to avoid is the cold-fusion phenomenon that happened in physics, where there was a premature announcement of some incredible advance that was undermined very quickly. Gary Becker used to say that we should communicate some things about the stock of knowledge—things that have accumulated, things that have been replicated, things that are basic, for which we have lots of supporting evidence from a variety of sources. By contrast, the latest working papers are part of the flow of knowledge. Sometimes that flow is a bit flimsy. Sometimes all-important nuggets come out, but it takes time to distill them. The rush to the media is not the best way to communicate scientific evidence.

Murphy: I wouldn’t limit it to working papers. The stock of knowledge is valuable. We’ve learned a lot in economics that can help business people, individuals, and policy makers. The flow of knowledge is essentially toxic. You would not want to consume it until it gets filtered to the point where it ultimately becomes valuable. Most of what comes out is either not correctly interpreted or wrong. I include my own research in that. I’ve changed my mind on things over the years. I’ve learned a lot, but it took years of filtering my own research, let alone the research that’s going on more broadly. And it’s not just working papers, but even published papers. If you go back and read past journals, there are a number of articles that turned out to be profound and stand the test of time, but a lot have been forgotten, overturned, or had an interpretation that has been superseded. So we want to work with the stock of knowledge and leave the flow to percolate and become part of the stock before we really use it.

It’s also incumbent on those economists who get involved in policy to actually step back and say, “What is it that I can say with confidence, and how can I communicate the degree of confidence I have?” If they do that, I think they can be very helpful to people. They’ll find themselves making much less bold statements. They’ll find themselves saying, “Here’s what economics teaches us. It doesn’t give us the full answer, but it helps us understand.” Policy makers will value that. You need to resist the temptation to be the latest sensational news outlet. I think that’s the biggest problem.

Hansen: Part of this is about how we define having influence: whether your work is in the newspaper in the next couple of years versus what impact your ideas have 10 or 20 years down the road. In a scientific discipline such as economics, the longer-term impacts are really the important ones.

Murphy: The channel is important. I’ve had really great experiences with professional policy people at the Congressional Budget Office and at central banks. Where people have to live with the consequences of bad policy, there’s much more interest in getting that valuable input. Those are the places where economists can have the most positive impact.

What are the challenges in translating research to issues such as the minimum wage and policies to tackle inequality?

Murphy: This is where a framework really helps. You can talk about inequality in terms of the outcomes, winners and losers. But that’s not very helpful for thinking about why it happened, and what a logical policy response is. What’s driving up the wages of one group and pushing down the wages of another? We have lots of information to help understand that—for example, the relative demand for different skills. But the supply side is also important. There’s overwhelming evidence from the United States and elsewhere that if you reduce the supply of people in a group for which wages are depressed, that will push up wages. So supply responses to the growth and changes in demand are the natural response. It took us a long time to get into the situation we are in now, and it’s going to take a long time to get out, because human capital is one of the most durable assets in the economy. There aren’t that many assets we produce today that are still going to be around and still be an important part of the economy half a century from now. People are one of those. And if we’re not investing in people the way we should, we’re going to suffer the consequences of that for decades to come.

What about putting a price on carbon?

Hansen: The price of carbon has been a somewhat naive conversation in many respects. The way economists deal with it, myself included, is that we have ignored the implications of other taxes, so we envisioned that all countries would have somehow coordinated on this. And it’s not necessarily the best policy lever or the most realistic policy. The more basic question has to do with framing. Here’s where uncertainty becomes important and where empirical evidence is of limited value. We’re talking about moving economies potentially in regions in which we’ve had very little experience, so we can’t just do some regressions and magically get credible numbers using evidence from climate science. Climate-science models are elaborate and sophisticated, and they need inputs from economics. And there are divergences across the different model predictions, so to imagine we’re going to come up with some single number for the social cost of carbon that we can use to design policy is naive. Even the academic literature on this has numbers all over the map. You need to combine economic and geophysical frameworks to get any type of sound policies.

Recommended Reading

Murphy: There are two big issues here. One is the uncertainty over what the number is, or if there even is a number. The second is understanding what that number means and how you would actually go about using it. For example, the idea was that all policy decisions would have to account for the price of carbon. The problem is that the price of carbon only exists in a world in which carbon is priced. To talk about the price of carbon in a world in which carbon is not priced is economically not valid. If you haven’t priced carbon, the different margins on which you’re using carbon are going to have different social impacts. I can cut carbon emissions, but that carbon is going to flow to somebody else, and if he or she puts it into the atmosphere, I haven’t really saved anything. When other alternative uses are not priced, the idea that I should act as if it is priced is just not supported by economics. It’s an example where people took one framework and said, “Somehow, magically, that’s going to tell me how to act in this very different world.” It doesn’t, unfortunately.

Hansen: A lot of the academic literature computing the social cost of carbon does it as a Pigouvian tax rate, figuring out the right way to tax an externality. It comes out as a price when evaluated at a socially efficient outcome, taking into account that externality. That conceptual framework isn’t the one we’re dealing with on a day-to-day basis. So you can’t just extract the number computed in that fashion and pretend it’s going to be valuable in this other setting.

Murphy: You might not just be a little bit off. You could be miles off—not even close. It may the the right number in the “but for” world where we have a Pigouvian tax, but it isn’t going to be the number in the world that we actually have. That doesn’t mean you can’t do anything. You just have to say, “For this experiment, is something like that number likely to be close?” It’s going to vary depending on the experiment whether that’s going to be close or far from that number. But you can’t just grab a number from one situation and throw it into another. That’s where understanding the theory is important. You don’t just hand this number over to policy makers to use however they want.

Could machine learning help policy makers process the flow of information?

Hansen: I think of machine learning as a combination of some clever computer algorithms designed for us to go find patterns in large-scale data sets. In the private sector, these methods have been successful for short-term forecasting and not for more basic policy questions. What could be attractive to economists would be to use some of the computational tractability that has come out of computer science, and to put an explicit economic structure on things, to actually integrate in a formal economic framework. Without that, we’re back to the “let the data speak”-type mentality, and you can be stuck answering very short-term-prediction-type questions at best.

Murphy: Machine-learning tools as inputs into economic analysis that tries to combine an underlying framework with data could be very helpful. One of the problems we’ve always had with explicit models that people write down is: To what extent are those really helping us because they capture the things we are competent in, and to what extent do they appear to help us because they impose some very ad hoc restrictions on the data? In some ways, machine learning can help with that, but not as it’s typically applied now, which is much more framework free. A lot of ways people apply machine learning today is basically for prediction—“I don’t care why it works, but it works.” For policy questions, that’s rarely that helpful, because the way it works is probably going to change when you change policy. But similarly, econometrics was a very useful tool for helping us understand data. There were tremendous mistakes we were making before we made progress on econometrics. But it’s not an input; it’s not really a substitute for the overall process. It added to the tool kit we have. Economics is a fundamentally about concepts and principles and data combined, usefully, together. If you want to know how to do that, go read Milton Friedman. He was one of the best at combining data and empirical evidence.

More from Chicago Booth Review

More from Chicago Booth

Your Privacy
We want to demonstrate our commitment to your privacy. Please review Chicago Booth's privacy notice, which provides information explaining how and why we collect particular information when you visit our website.