Chicago Booth Review Podcast Why AI Isn’t Boosting the Economy

Anders Humlum: And we can ask and answer whether adopters have felt better in the labor markets, if they're becoming more productive. Do we see it manifesting in higher earnings, higher productivity, changed work hours? And here the key finding is that chatbots have really not made a difference.

Hal Weitzman: Companies are building their own chatbots and training their staff in how best to use the technology. You might think this will be feeding through into corporate profits and boosting the economy. So why isn't it? Welcome to the Chicago Booth Review Podcast, where we bring you groundbreaking academic research in a clear and straightforward way. I'm Hal Weitzman, and today I'm talking with Chicago Booth's Anders Humlum. His research documents the paltry effects that AI is having on company earnings and economic growth. Is it just too early or will AI's economic impact turn out to be overrated? Anders Humlum, welcome to the Chicago Booth Review Podcast.

Anders Humlum: Thank you.

Hal Weitzman: Okay, so we are here to talk about your research about AI and why it isn't exactly setting the economy on fire, yet anyway. Your research, this paper, was written about Denmark. I know you're from Denmark, so obviously it's nice to have a excuse to go back, but why is Denmark particularly important or useful for studying this phenomenon?

Anders Humlum: Yeah, absolutely. So this project is on the arrival of AI chatbots, really motivated by the rapid diffusion of ChatGPT and the similar tools that are really part of the workplace now. Denmark is really a fantastic setting to study the arrival of these tools in the workplace for several reasons. The first being that the Danish workforce have been on the forefront of the adoption of these tools with adoption rates similar to the US setting. So I think a lot of the lessons that we'll learn from the Danish settings are directly applicable to the US economy and most other Western economies.

The second being an amazing availability of data. So we collaborated with Statistics Denmark, which enabled two fantastic opportunities. First, every Dane has a digital mailbox the government uses to send tax documents. For example, other important documents. When I lived in Denmark, I had a digital mailbox. Statistics Denmark can send survey invitations through these digital mailboxes, which means that we can really run large-scale representative surveys and we can even target these surveys to the occupations where these chatbots are relevant. So we get a large sample in occupations such as software developers, teachers, HR professional, accountants, and so forth.

And then in terms of the labor market, so we are interested not only in understanding the take-up of these tools, but also the implications for the labor market. And here we can link these survey responses to fantastic data, administrative data on workers' earnings, their hours, their wages. So this is really the gold standard of labor market outcomes. So we can really assess whether these tools have made a difference. So I'm a labor economist. I want to know, do these things matter in terms of earnings, in terms of productivity? And I think this data set really offers a fantastic opportunity to take stock and look at the evidence.

Hal Weitzman: As you said, companies in Denmark are investing in AI. What exactly are they investing in?

Anders Humlum: Most employers are now really taking active actions in terms of, first, explicitly encouraging their employers to use these tools. In many workplaces, it is an expectation that workers are leveraging these tools in their daily.

Hal Weitzman: How are they encouraging them?

Anders Humlum: So they're not just putting up users policies. They're actually putting their money where their mouth is. So first, around a third of employers have their own chatbot. So this is a version of ChatGPT often, but that is customized such that it integrates more seamlessly into the workflow of the particular workplace, making it easier for employees to take advantage of this.

Hal Weitzman: So a third of the companies have developed that?

Anders Humlum: Yeah, have some version of their AI.

Hal Weitzman: Do you know how that compares to other countries?

Anders Humlum: So again, that's where actually on the employer side, we have very limited evidence on generative AI adoption. And this is the first set of evidence along these lines. I couldn't see why it wouldn't generalize to other settings. I think these tools are really becoming pervasive and a part of the general tool set in many knowledge work professions. Second, we see that, and I was stunned by this statistic, one-fourth of employees in these occupations have received training into how to use the tool.

So the neat thing with chatbots is that-

Hal Weitzman: Just to clarify.

Anders Humlum: Yeah?

Hal Weitzman: So they've received training on how to use their own internal tool or generally?

Anders Humlum: Generally. So it could be either/or. We could split it, but I think in general, employers have now offered training courses into how to start prompting. The nice thing with these chatbots is that it actually doesn't require a lot of training to get started. A one-day course is often more than enough. You can even use the chatbot to teach you-

Hal Weitzman: Right.

Anders Humlum: ... how to start prompting. So in that sense, that seems like a straightforward thing for employers to do. And indeed, we see that a quarter of employees have already received this type of training.

Hal Weitzman: So they're investing in it. They're building their own. They're encouraging people to take these training courses. And then we've covered at Chicago Booth Review, we've covered some of your earlier research about who is adopting AI. And what we basically found, if I can summarize that, was generally it's younger people, less experienced, high achieving workers. Women were quite significantly less likely than men to use it. Is that what you found here? Is that replicated here in this study as well?

Anders Humlum: Yeah, those inequalities were replicated. So we did two survey rounds. So the first round was just the year after the arrival of ChatGPT where we saw these large inequalities emerging. And these inequalities were replicated in the second round, which were exactly two years after the arrival of ChatGPT this winter. We see that these gaps in adoption are still with us today. Women are significantly less likely to use the tool relative to a comparable man in the same occupation. We can even compare co-workers in the same workplace and at least a 10 percentage point gap remains in their adoption rates.

Hal Weitzman: Does that apply even after they've had training?

Anders Humlum: Yeah. So that's the new thing in this new study is that we show that employers can actually do something about these gaps. When employers are providing training, these gaps, and in particular especially the gender gap, really shrinks. So you can half the gender gap by offering training courses. So we see this that the employee-

Hal Weitzman: Even though as you said, the training is not really strictly necessary because you could train yourself quite easily?

Anders Humlum: Yeah, I think it's something about setting explicit expectations that we want you to use it, and then, second, giving people that notch to start using it. And then we can see that women really step up the game and close a lot of the gap. So I think I was very encouraged to see this, that even though there are these gaps, employers, firms, policymakers can actually do something about it. And it seems that training in particular seems quite effective in closing these gaps.

Hal Weitzman: So one of your findings was that AI helped workers save time. That makes sense. How much time? And what was the variation between different jobs? Who is saving more time?

Anders Humlum: So on average in these 11 so-called exposed occupations, these are occupations where it's actually relevant to use AI chat, but we found an average time savings of around 3%. So that varied both across occupations and across workplaces, depending on whether employers were actually also investing in the tools. So some of the occupations with the largest time savings were, for example, programmers, software developers.

Hal Weitzman: They're using it to write code.

Anders Humlum: They're using it to write code. And that was one of the first use cases of these large language models is that they can really speed up coding. So this is really a part of the tool set of any software developer, I would say. Marketing professionals, HR professionals.

Hal Weitzman: Who's writing copy.

Anders Humlum: Writing copy, yeah. So this is really bread and butter for these large language models is writing ads or job posting for HR professionals.

Hal Weitzman: Right. Right, right. Job descriptions, right?

Anders Humlum: Yeah. So for these occupations, especially in the workplaces where they're encouraged to use it, the time savings were larger, more on the order of 7%.

Hal Weitzman: So that would be for the coders and for the marketers and HR people?

Anders Humlum: Yeah. So these are the occupations and workers with the largest time savings and productivity gains in the economy from these large language models. At the lower end of the spectrum, we have a bunch of occupations where these tools are still relevant, but they're not saving as much time. For example, teaching. I use ChatGPT every day. I love it. Teachers have also taken it up, but they're reported time savings are much smaller, more on the order of 1% of the total work hours.

Hal Weitzman: Do you know what they're using it for?

Anders Humlum: I can tell you what I'm using it for.

Hal Weitzman: Okay, yeah.

Anders Humlum: I use it to prepare lectures. I use it for research. I think it's a great learning tool. You can also use it to assist, in principle, grading essays. There are many use cases for chatbots for a teacher. Of course, they cannot automate the-

Hal Weitzman: So you'll use it to grade the essays, which presumably the students are writing using ChatGPT as well.

Anders Humlum: Yeah, you can just fully automate it.

Hal Weitzman: Just put two agents together and we don't need to be involved. Here's your degree.

Anders Humlum: Yeah. So that is actually interesting. There's a lot of teachers that say that they're now also spending time detecting whether their students are using these tools, potentially cheating on their exams.

Hal Weitzman: And they're using ChatGPT to detect whether the students are using ChatGPT?

Anders Humlum: Potentially.

Hal Weitzman: Okay. So in some jobs, it's pretty obvious the way you talk about them. They're basically writing jobs, whether they're writing code, copy, or job descriptions. And presumably we could think about lawyers. I don't know if that's part of your sample.

Anders Humlum: Yeah, legal professionals. We both have paralegals and lawyers in the sample. They fall somewhere in between the teachers and the marketing professionals in the sense that they reap on average like 3 to 4% time savings. There are specific barriers to all occupations. For legal professionals, their factuality, whether they're actually correct, what these large language models are spitting out is a key stumbling block for lawyers.

Hal Weitzman: It helps them with the boilerplate or the small print or whatever, that kind of thing. So those are the white collar office jobs. That's why I guess teachers doesn't quite fit into that. It's not an office job in the same way. And presumably, I'm guessing if you're a welder or a plumber or a carpenter, it's much less useful. So this is what we would expect, that office jobs, white collar jobs are the ones that are finding this most useful right now. And so companies are investing in this. There's widespread adoption. There is time being saved, but what you find is it's not feeding through into earnings. Why is that?

Anders Humlum: Yeah. Well, actually that's-

Hal Weitzman: Explain what's happening. Maybe first tell us what's happening.

Anders Humlum: Yeah, that's the core contribution of this paper is that we collect all this survey evidence, and this is really large scale. We have 25,000 responses that we now can link to the administrative tax records of these workers. And we can ask and answer whether adopters have fared better in the labor markets. If they're becoming more productive, do we see it manifesting in higher earnings, higher productivity, changed work hours?

And here, the key finding is that chatbots have really not made a difference for earnings, for hours, for wages. We have so much data that we can reject even small effects. Effects larger than 1% on these outcomes, we can reject. Chatbots have really not made a significant impact on labor market outcomes. We can even look at the trajectories of these workers and see, are they trending better? At least that could provide some signs that maybe in the longer run they'll do better. The trends are flat. To me, that's an early indication that even in the medium run, we should maybe not expect these tools to be very transformative in terms of labor market outcomes.

Hal Weitzman: If you're enjoying this podcast, there's another University of Chicago podcast network show that you should check out. It's called Capitalisn't. Capitalisn't uses the latest economic thinking to zero in on the ways that capitalism is and more often isn't working today. From the morality of a wealth tax to how to reboot healthcare, to who really benefits from ESG, Capitalisn't clearly explains how capitalism could go wrong and what we can do about it. Listen to Capitalisn't, part of the University of Chicago podcast network.

And as in the first half of the show, we talked about how your big finding was that even though people are using chatbots, they're saving time, they're not generating more earnings for their organizations using those chatbots. Why? Why is that? Why do you think that hasn't shown up yet? Or is it that it's just yet we're early on in the cycle. What do you think is going on?

Anders Humlum: Yeah, I was personally surprised, I must say. There is really good evidence from experiments showing these tools do have remarkable potential to boost productivity. And that stands in very stark contrast to the zero effects we find in our administrative data in the real economy. And I think there are two reasons why we don't see it showing up in earnings. The first is that the productivity gains we observe in the real economy are modest and significantly smaller than what people have found in experiments. And the second thing is that even though we do find some time savings, there is a very weak pass-through to earnings, and the pass-through is particularly weak in workplaces where it's not encouraged. So our evidence point to-

Hal Weitzman: Not discouraged, just not encouraged.

Anders Humlum: Not encouraged. There is very few workplaces now that explicitly bans ChatGPT and chatbots. These are mostly banks that are handling very confidential data or legal professionals where being correct is it is a first order importance. Just not encouraged seems to weaken this pass-through from productivity to earnings.

And our evidence point to a couple of reason why that link to earnings might be weak. The first one is that even though chatbots might save you some time on existing tasks, like for example, I use it to draft a document, draft an email, it speeds up that process. Whether it really boosts earnings depend on what do I use that time savings for and can I reallocate that time to a task where I'm actually productive or not? And I think that the answer to that question really depends on whether employers are encouraging it and whether employers have started to change workflows in response to these chatbots. And we see that workers in their workplaces where they're encouraged, they're better able to take on more work now that they're more productive. And I think that is the mechanism for which I expect these tools to affect earnings in the long run is that workers are able to do more or they're able to shift their work towards tasks where they have higher value added. But we just see limited evidence of that thus far.

Hal Weitzman: Okay, but your suspicion is that that's the next step?

Anders Humlum: Yeah, that's the next step I see. The arrival of chatbots have been fascinating because it's really been this bottom-up adoption process where workers have just started using it, many of them without the permission or the encouragement of their employers. But what I see now the next step is that employers are now getting on board and starting to invest in the tools, starting to reorganize workflows in response to the chatbots and the productivity potential in it. And we see that when employers are investing, both adoption rates are much higher and the benefits are larger. So the time savings are larger, they're better able to take on more work. And I think that's the mechanism for which these tools might become transformative in the medium to long run.

Hal Weitzman: Okay. So it's interesting because there's a lot of predictions out there about how AI is going to transform work. So without... It's early, but so far it's not quite living up to its billing.

Anders Humlum: I was surprised. I must say personally, I love ChatGPT. I use it all the time.

Hal Weitzman: I can tell.

Anders Humlum: But I was also in some sense negatively surprised by the fact that it has really not affected outcomes. I think it's important to cut through the hype and actually look at the economic outcomes. And when we look at the hard metrics, it has really not made a difference. And it doesn't seem like it's trending at all either, which also suggests to me that maybe this finding of small labor market effects will stick for a while.

Hal Weitzman: Okay. So you write in your research about productivity. Do you think that there's, in other words it's going to really take off at some point, do you think that that's coming? And if so, does your research suggest it might be further out than was anticipated or something like that?

Anders Humlum: I think there's an understanding, especially among economic historians, that these processes take a lot of time. And I think this is really where history is very valuable. Going back to the steam engine or electricity, it took several decades before we really harnessed the full potential of these very transformative tools. And I think large language models falls into that category of general purpose technologies where it'll take a lot of time before we really see the large productivity gains manifesting in the economy.

The reason why I was still surprised is that there are very well executed experiments showing remarkable effects in a very short period of time. So if you had just taken that evidence and extrapolated to the economy, you would expect it to show up in the economic outcomes we observe, but we don't. So that's why I think, I still think these technologies have tremendous potential, and the fact that people are really adopting it is in some sense confirming that. But in terms of the economic outcomes, productivity earnings, I think it will take longer time than people might anticipate.

Hal Weitzman: Yeah. You talk about two different things there. Productivity and earnings, they're different, aren't they? And a lot of the concern about AI is that it won't help workers boost their productivity, which is the companies will focus on profits, not earnings, and so they'll just get rid of workers. And that's how they'll use AI to improve productivity, not by giving people more tasks to do but by giving them nothing to do and getting rid of them, and that's how they'll improve their profit. So I don't know if you feel like this sheds any light on that debate because the other counterpart guess is, no, AI would be great. We'll all be doing more and we won't be doing boring things like drafting boilerplate language. What do you think?

Anders Humlum: Yeah, that's a very valid concern. And actually something where the Danish data is also fantastic because we can link these responses up to the workplace level. So we can look at workplaces that have adopted these tools faster, and there is tremendous variation across workplaces, even within very narrow occupations. Some marketing companies have really jumped into these tools and they provide an early sign of, well, do they then start laying off workers or freeze hiring? We don't find any evidence of that. High-adopting workplaces have not fared differently in terms of their total wage bill, in terms of total employment. We can also look at incumbent workers. That's something you might worry about is that, for example, customer support agents that now the incumbents, those that were before chatbots arrived, they're laid off and then it's just fully automated by AI chatbots answering these customer inquiries. We don't find any evidence of that. So again, both at the worker level as well as at the firm level, it seems that the impacts are much smaller than some might have anticipated and that it will take a longer time.

Hal Weitzman: So we don't need to fear the chatbots?

Anders Humlum: I think that's a positive spin of this set of evidence is that there's still time to act. There's-

Hal Weitzman: Even for an old man like me. And you talk about some of the new job tasks that are going to come from AI. What are some examples.

Anders Humlum: Yeah. So in the short run, there is a lot of work that goes into just integrating these tools into the workplace. So teachers are now adapting their exams in order to take into account tool usage. Legal professionals are now drafting new legal policies in response to these chatbots. Workers are also now shifting more gears towards overseeing the quality of AI generated output. So this could be, for example, a IT support specialist. Now instead of drafting the initial response to some worker or to some customer, and now it's a chatbot, the draft, that initial response, but then the specialist is there to oversee the quality of their output. So more being a middle manager than being on the lower level-

Hal Weitzman: So the future is we're all going to be managers.

Anders Humlum: I think we're all going to be managers of chatbots. I think that's what we are heading towards.

Hal Weitzman: Okay. Well, I think that's a good place to leave it. The future of management will be everyone. That'd be great for Chicago Booth because everybody will have to come here to get their MBA.

Anders Humlum, thank you very much for coming on the Chicago Booth Review Podcast.

Anders Humlum: Thank you.

Hal Weitzman: That's it for this episode of the Chicago Booth Review Podcast, part of the University of Chicago Podcast Network. For more research, analysis, and insights, visit our website at chicagobooth.edu/review. When you're there, sign up for our weekly newsletter so you never miss the latest in business-focused academic research.

This episode was produced by Josh Stunkel. If you enjoyed it, please subscribe, and please do leave us a five-star review. Until next time, I'm Hal Weitzman. Thanks for listening.