Chicago Booth Review Podcast Are We Too Obsessed with Numerical Data?

Erika Kirgios: We have a study where we have a benchmark condition. If all the information is numeric or if none of the information is numeric, how do you choose? And what we find is that quantifying some but not all distorts choices away from that benchmark. So you make substantially different decisions when only some information is quantified than when all or none is quantified.

Hal Weitzman: We're obsessed with numerical data at Chicago Booth Review, but could that obsession sometimes be a bad thing? Welcome to the Chicago Booth Review Podcast, where we bring you groundbreaking academic research in a clear and straightforward way. I'm Hal Weitzman. Chicago Booth's Erika Kirgios and her co-authors came up with the term quantification fixation to describe how we tend to overweight numbers compared to qualitative evidence. What are the risks to that tendency and how can you use quantification fixation to your advantage? Erika Kirgios, welcome back to the Chicago Booth Review Podcast.

Erika Kirgios: Thank you again for having me.

Hal Weitzman: Well, we had to be back because we had such a fascinating conversation with you last time about Craigslist and selling cars and interesting responses that people give that we want to talk to you about another area of your research, which is quantification fixation, our obsession with numbers. Tell us more about what is quantification. Did you come up with that term?

Erika Kirgios: My co-authors and I came up with it together. We went through a lot. We had a long list of terms and we were like, "Is the rhyming too much?" No, it's perfect.

Hal Weitzman: Okay. Well, it definitely works. So what is quantification fixation? How do you define it?

Erika Kirgios: So the way we define it is the tendency to overweight or over-index on numeric information when you are making difficult trade-offs. So if you're making a decision and there are a lot of choice attributes and some of them are numeric and some of them are not numeric, they might be verbal, like you're choosing between two restaurants and you know the average price of a main dish, that's numeric and you read some reviews on Yelp and they describe the ambiance and that's not numeric. They're saying, "It's a little noisy, but overall the decor is nice." What do you pay more attention to when you're deciding between two restaurants? And we suggest that you over-index, you overweight on numeric information even when it means neglecting important non-numerical.

Hal Weitzman: So if the reviews were starred, then you would look at the number of stars.

Erika Kirgios: Yeah. You might care well. I think actually stars, even though you can quantify how many stars, it's a visual representation.

Hal Weitzman: It's not a number.

Erika Kirgios: So if-

Hal Weitzman: Does that mean they should change the Yelp or whatever should change the stars system to number ... Or maybe they have one, Yelp. I'm thinking of Amazon.

Erika Kirgios: Yeah. Some platforms only do the visual stars and some do the stars plus the number. Probably better to include the numbers if you want people to pay attention.

Hal Weitzman: Okay. Maybe Amazon's paying attention to that. So you say in this research that in recent decades there's been a migration from the qualitative to the quantitative. What do you think of that? What are some examples?

Erika Kirgios: So at a broad level, what we were kind of trying to reflect is this idea that we have become much more data driven as a society. So there are all of these websites and platforms that aggregate information and often take experiences that once would have been qualitative and transform them into metrics or numbers. So employee wellbeing is an inherently qualitative characteristic, but if you go to sites like Glassdoor, you'll get an employee wellbeing score for a company that's on a zero to 100 metric. Or if you think about people who day-to-day care about moving their body and getting exercise, now many of those people are tracking everything and they consider whether they had a good day on the basis of how many steps they got, how many calories they burned, as opposed to how good their body felt when they were exercising that day and what it felt like when they were done or during the workout.

And so we're assessing ourselves more and more based on these quantitative metrics, even at the replacement of a more qualitative sense of how we're doing, how we feel about the environments we're in.

Hal Weitzman: And is your thesis that the quantitative actually is less meaningful than we think it is?

Erika Kirgios: I'm not going to make a prescriptive claim about whether numbers are bad. What I will say is that we might lose sight very easily as we quantify things of what that means about what people focus on. So if there are things that are inherently difficult to quantify, then they might lose prominence in people's decision making whether or not we want them to. So if you think about even really tough, intense decisions like somebody deciding between two cancer treatments, and it's really easy to quantify some components of that, like expected life expectancy. It's harder to quantify expected quality of life, but because it's so easy to quantify expected life expectancy, if you present that to people and then verbally describe what the impact on their quality of life might be, they're going to over-index on the numeric information.

Hal Weitzman: Okay. So if you face a trade-off between quantitative and qualitative like you just graphically illustrated, people tend to fixate on the quantitative scale. So tell us about how you designed an experiment that discovered that or proved that.

Erika Kirgios: Yeah. So my co-authors and I actually ran 21 experiments discovering or proving this.

Hal Weitzman: We'll talk about some of them.

Erika Kirgios: Yeah. So the general design is you're facing a trade-off. So you have to decide between two job offers, two employees, two hotels, whatever it might be in all sorts of context, two policies. And you learn about two attributes of each choice and one choice is better on one attribute, the other choice is better on the other. So that's a basic trade-off. And what we vary across conditions is which of those attributes is expressed numerically and which is expressed non-numerically. And that could mean an icon, like you said, the stars that you see on Amazon or Yelp. It could be in the form of a continuous graph. So if we're giving you a zero to 100 metric, you see a bar that's filled in from zero to 100, so it's just as granular as the numeric information.

It could be if you're deciding between two students and you get information about their grades in math and their grades in English, you could see a numeric range. They're getting an 83 to 87% in English and a 93 to 97% in math, or you express those as a B or an A. So they could be really familiar non-numeric things like a grade that we often know-

Hal Weitzman: So it's really about the numbers.

Erika Kirgios: Really about the numbers.

Hal Weitzman: Because you're not measuring people with stars or letters or anything else. It's just the numbers.

Erika Kirgios: No. It's just the numbers. So regardless of how we convey the non-numeric information, we get the same effect. And sometimes we use verbal descriptions. And what we vary across conditions is which attribute is described numerically and which is described verbally. And we try to be really careful to make sure that we're conveying the same information no matter what. So we've even gone to the extent of translating verbal descriptions to numbers. So we tell you, for example, you're deciding between two cars, one has a better environmental friendliness score, one has a better safety score. And when the score is described verbally, we say, for example, very poor environmental safety score: one, poor environmental safety score: two. So we're giving you a literal mapping between the verbal score and the number, and then we show you the two options and we just have the verbal information or we just have the number and we still see that people are more likely to choose the option that dominates on the numeric dimension, even when they have a mapping available to them that would translate the verbal information to a number.

Hal Weitzman: So don't tell us about all 21 experiments. Tell us about a couple of your favorites.

Erika Kirgios: Well, I tend to like the ones where you have real incentives on the line. So we have a fun one where you're a manager and you need to decide between two employees to hire. And we have these employees play real games online. So there's a trivia game, a math game, and a visual logic game. And we tell you that you are going to receive a monetary bonus if you pick the person who has the best visual logic score and the two candidates, one of them has a better math score and the other one has a better trivia score. So how do you choose between them? And we express one of those scores numerically. So either both of the math scores are numeric and both of the trivia scores are expressed with a little visual bar where you can see the grade goes from zero to 10, just like the numeric information, and it's filled in to the level of their score.

Or in the other condition, the math is the visual numeric bar and the trivia is a number. And we say, "Who do you want to hire? And you're going to be paid if you pick the person who actually does better on the visual logic game." And we show that people are more likely to pick the person who's better at math when that's numeric and more likely to pick the person who's better at trivia when that's numeric and it comes at a financial cost to them.

Hal Weitzman: Fascinating. You're making me think of, I've heard people talk about the reason there's range anxiety for Teslas. Is that right? I don't have a Tesla, but I think they might have a number to the amount of power that you have.

Erika Kirgios: Interesting.

Hal Weitzman: So it's like a phone.

Erika Kirgios: Yeah.

Hal Weitzman: So my kids in particular get very anxious about charging their phone because it's dead to go down to 83% or whatever. And I've heard it said that one of the reasons is when you have your, in a traditional car like I have, there's no number there. It's just to gauge.

Erika Kirgios: When you think about the fuel gauge, yeah.

Hal Weitzman: So you don't worry about it till it's basically about to run out.

Erika Kirgios: So speaking at you and telling you, "Please worry about me."

Hal Weitzman: Unless you have one of those things that says so many miles to go and then you do get anxious about it. But it's interesting kind of real world example of what you're talking about that I've definitely seen. I think I've heard range anxiety attributed to exactly that makes it very salient. So tell us about another one of these experiments.

Erika Kirgios: So the other experiments that have real incentives on the line involve choices between charities where people are deciding whether they want to give of one of two similar, they're both wildlife charities. One of them has a better financial responsibility score and the other one has a better community score, so community impact score. And again, there's a choice, do you take the one with the better community impact score or the one with the better financial responsibility score where one of those is described numerically and the other one is described visually? And again, we see that people are more likely to give their money to the charity that dominates on the numerically described dimension. Or I mean, they're going to sound similar as I described them, but we have a policy choice where one policy is more beneficial to the community and the other one is more efficient.

And we say, "You're voting in your local election. Which of these policies would you rather pick?" When benefit to community is numeric, about 85% of people choose the more beneficial project. But when efficiency is numeric, about 50% of people choose the more beneficial project. So we're talking about huge differences in choice just based on which of these attributes is conveyed numerically and which of them is conveyed verbally or graphically.

Hal Weitzman: If you're enjoying this podcast, there's another University of Chicago Podcast Network show you should check out. It's called Big Brains. Big Brains brings you the stories behind the pivotal scientific breakthroughs and research that are reshaping our world. Change how you see the world and keep up with the latest academic thinking with Big Brains, part of the University of Chicago Podcast Network. Erika Kirgios, in the first half we talked about quantification fixation, this term that you and your co-authors came up with, and you gave us all sorts of examples of people where the numbers are salient, they kind of glom onto that and that seems to be more important than they choose, the numbers, whether it's a trade-off with the qualitative description of something. Now, we have heard for many years that stories are very, very powerful and they do something to your brain and the power of stories and there's a whole movement about storytelling. And so how does that work?

Because according to that theory, numbers are kind of, dare I say, a bit boring and not very compelling, and you've got to put them in a narrative framework and all this kind of business. How do you address that?

Erika Kirgios: Yeah. There is this sense in psychology that numbers are pallid. They lack the vividness of stories that if you want to convince someone to give to a charity, you don't want to give them statistics and tell them how many people are impacted by a problem. You want to tell them one story about one victim. And we don't negate that work. I think that the difference here is you're making a trade-off decision. You're not just considering in isolation, "Do I want to give to this charity or not?" And when you're making a trade-off, what you're trying to decide is, is the difference, let's say, between the benefit of these two policies big enough to warrant the difference in efficiency that goes in the opposite direction? And what that's asking your mind to do is compare and take those differences and compare the magnitude of the differences and decide whether giving up this much of one attribute is worth getting this much of another.

And for that, numbers are so intuitive to use. You ask me to compare two verbal descriptions, two compelling stories of an individual victim, and that feels hard. I don't know. Which one do I care about more? But you ask me to compare two sets of statistics, it's really intuitive. I can subtract one set from the other. I have a strong sense of exactly how much I'm giving up or how much I'm trading off for something else. And so our argument and what we find evidence of in the paper is that this thing that we call comparison fluency, how easy or intuitive it feels to make comparisons is the reason numbers dominate when making trade-offs. It's so simple for your mind to see two numbers and take a difference and consider what that magnitude means to you. It's much harder when you're just thinking of a verbal description or even you feel like it should be easy and intuitive with a graph or with star rating, and yet it seems to hold people less. It seems to sway their decisions less.

Hal Weitzman: But even with the star rating, I feel like you're exposing one of my weaknesses, my biases, because I'm thinking my own behavior. I would do exactly what you described. If I was buying a product or a service and I saw a written description, I would probably pay less attention to that. I think it's an opinion. Whereas if the person put three stars, I would... Suddenly somehow that's more important to me, which is absurd, but that's, I guess, how my brain is using what you're describing.

Erika Kirgios: And if you were to take those three stars and make them a number, you would do that even more, is what our research suggests.

Hal Weitzman: Right. Fascinating. Yeah. Right, absolutely. So maybe I'm less savvy than others, but maybe not. So do you think some people are more susceptible to quantification fixation?

Erika Kirgios: So we do find that some people are more susceptible than others. We specifically looked at something called numeracy. So numeracy is an individual characteristic that speaks to a person's facility or ease with using numbers. And there are two different types of numeracy. So there's objective numeracy. Are you literally good or bad at manipulating numbers in your mind or at subtractions, at division, at thinking about numbers? And subjective numeracy is, does it feel easy to you? Does it feel comfortable to work with numeric information or not? And we find no differences based on objective numeracy. So people who are objectively good or bad with numbers both show this effect. We find big differences based on subjective numeracy. So people who feel like numbers are uncomfortable to work with, who feel like numbers are less informative to them, they don't exhibit quantification fixation. And it makes sense, right?

If I think numbers are hard to work with, if I feel like they're uncomfortable to work with, if I feel like they're less informative, then I'm not going to exhibit this comparison fluency. I'm not going to feel like it's easier to make comparisons between numbers than verbal descriptions or images. I'm going to prefer to make those visual or verbal comparisons, and that's what's going to hold my attention.

Hal Weitzman: This is really fascinating because of the whole discussion about you should present data to people, but some people, it sounds like those who think of themselves as comfortable with numbers would be happy with that, but others really that they wouldn't like it. So when we go to my question about the stories and the numbers, it sounds like there is not a set ... I mean, a lot of that research suggests that there's a universal response, which I guess of course there isn't.

Erika Kirgios: Yeah. We're always talking about aggregate responses.

Hal Weitzman: Right. So there isn't a universal response, but maybe it depends partly on how we feel about numbers. So I want to think about the real world because ... I mean, would you classify this as a ... Is this a mistake? Is quantification fixation, is it a bias in the same sense of other psychological biases?

Erika Kirgios: So we are pretty careful not to necessarily call it a mistake, though I will say that it causes a distortion in preferences. And so if you want to think of that as a mistake, I think that's reasonable. The thing is that because we're looking at trade-offs, there often aren't right or wrong answers. Often it's about individual preferences. And so it's hard to say your individual preference here is a mistake. But what we can say is we have a study where we have a benchmark condition. If all the information is numeric or if none of the information is numeric, how do you choose? And what we find is that quantifying some but not all distorts choices away from that benchmark. So you make substantially different decisions when only some information is quantified than when all or none is quantified. So that does suggest that we're leading people to make choices that might not be reflective of their preferences if they had all the information presented in the same format.

Hal Weitzman: Did you ever test or come across the example where somebody gave a good numerical rating of something, but the verbal rating was terrible?

Erika Kirgios: Oh, man.

Hal Weitzman: Because sometimes you see that. I wonder if that's ...

Erika Kirgios: So that's interesting. We always were trying to separate the trade-off. One attribute is good, another attribute is bad. What you're kind of describing is I give a bad rating, but for the same dimension, a good verbal description or vice versa.

Hal Weitzman: Right. I'm thinking of me. This is all personal. I'm just trying to get what I can out of this, Erika. No, I'm just thinking of sometimes you would see a product where you'll see it's rated highly. And when you dig into what the people have said-

Erika Kirgios: It's actually quite negative.

Hal Weitzman: ... it's to some spurious reason or it's quite negative or the other way around. They think it's terrible because the box was slightly damaged. Something that you sort of ... I don't care about that. Or the color was slightly different to the color that was promised, I mean, I don't care about. So I'm just wondering if then I discount the numbers.

Erika Kirgios: I think it depends on how quickly you're processing. So probably most people don't dig that deeply into the reviews, and so the numbers are going to sway their decisions hugely.

Hal Weitzman: Right.

Erika Kirgios: But if you're spending the time to be like, "Oh, these negative reviews are based on the box, but all I care about are the contents of the box," then you'll probably dismiss the numbers. If it's a one-off, if it's one person, you might just think there's a mistake and there I think you're probably more likely to think the rating is a mistake because it's a button they clicked. And so again, you might dismiss the number.

Hal Weitzman: Right. Absolutely. You're making me think of ... So a few years ago I wrote a book and there was a five-star rating and the review said something like, "The book arrives in good condition. It is now on my shelf," which is one of my favorite reviews. I'm just thinking about what ... So when I said a mistake, I'm wondering, what is the damage here? What is the risk for organizations, for individuals who are affected by this, if managers just fixate, or I guess nowadays if they're using AI just to quickly process applications, RFP responses, whatever it is, what do you think?

Erika Kirgios: You might, for example, in the application context, you might miss out on great people. You might make systematic mistakes where let's imagine you're hiring for an entry level job and what you can see is a candidate's GPA and you can get a sense of the relevance of their expertise or experiences based on, let's say, the internships they've done or the types of classes they took. You're going to overweight GPA and that might be an error because you might miss somebody who has really relevant, really great work experience and a GPA that maybe is a little lower than somebody who has no relevant experience whatsoever. And GPAs can be different for all sorts of reasons. People can come in without any family members who went to college and struggled to adapt to college, or they might have really high GPA in their major and low GPA from required courses outside of it.

And if all you care about is that somebody's a great software engineer and their GPA is slightly dampened by their grades and their English classes that were required, and they have this great software engineering experience, another candidate has a high GPA and none of that experience, you might be making a mistake. And for an individual, if you're deciding between two job offers and you have information about salary, and then you have these conversations that you've had with employees about the culture, you might really overweight salary relative to culture.

Hal Weitzman: Right. Well, that's a classic one, right? Often people fixate on the nominal amount of compensation and don't think about the whole picture.

Erika Kirgios: Exactly.

Hal Weitzman: Even things like vacation, right? And so now you're making me think of the flips case, like what about people using... So if I were writing my resume, should I make sure there's lots of numbers in there?

Erika Kirgios: The stuff that you-

Hal Weitzman: Particularly if my GPA isn't that good.

Erika Kirgios: Yeah. If you want other information to gain prominence in people's minds, quantify it. And I even will say that in my negotiation classroom. I'm like, "If you have any control over how things are expressed in a deal sheet, the stuff that you want them to focus on, express it numerically. And the stuff you want them to focus on less, make a graph."

Hal Weitzman: So I could write my GPA out.

Erika Kirgios: Yeah. For example.

Hal Weitzman: 3.2. When you're writing words, then they would be less likely to focus on it.

Erika Kirgios: Yeah. They might.

Hal Weitzman: All right. Well, this has been a fascinating conversation. Thank you very much, Erika, for coming back on the Chicago Booth Review Podcast to talk to us about quantification fixation.

Erika Kirgios: Thank you for having me as well.

Hal Weitzman: We'll have you back in the next year, next 12 months.

Erika Kirgios: All right.

Hal Weitzman: Thanks again.

That's it for this episode of the Chicago Booth Review Podcast, part of the University of Chicago Podcast Network. For more research, analysis and insights, visit our website, chicagobooth.edu/review. When you're there, sign up for our weekly newsletter so you never miss the latest in business-focused academic research. This episode was produced by Josh Stunkel. If you enjoyed it, please subscribe and please do leave us a five-star review. Until next time, I'm Hal Weitzman. Thanks for listening.