Good Studio/Adobe Stock
Artificial intelligence is rapidly changing jobs and industries, causing no small amount of consternation as it does. But on the bright side, it has the potential to greatly aid economists by streamlining experiments’ design and implementation and leveraging behavioral insights, suggests research by University of California at Santa Barbara’s Gary Charness, Chicago Booth principal researcher Brian Jabarian, and University of Chicago’s John A. List.
Recent advances in generative AI, mainly through large language models, have sparked considerable interest. For one example, after OpenAI launched LLM-based ChatGPT, its valuation exploded, competitors rushed to keep up, and Microsoft kicked in $10 billion. Across the world, people are scrambling to understand how LLMs will transform jobs, the labor market, and various companies and sectors.
Science, as many researchers have noted, is not immune. And as Charness, Jabarian, and List explain, LLMs can help revolutionize how it is practiced. Addressing economists in particular, they write that LLMs could be harnessed to scale up experiments, make findings more accessible, and foster a culture of critical thinking of evidence-based analysis. LLMs could be used to improve nearly every step of an experiment, they explain—and they propose specific approaches for doing so. “All these offered directions require experimental benchmarking before becoming established scientific policies,” qualifies Jabarian.
They group their recommendations into three categories: the design phase of an experiment, the implementation phase, and the analysis phase. Design involves crafting and coding an experiment, and here, they write, LLMs offer a groundbreaking approach to literature review, hypothesis generation, and experimental setup. LLMs could be used to analyze extensive data sets, identify gaps in knowledge, and help generate research ideas. AI could speed up the brainstorming phase while ensuring that research hypotheses are well-grounded.
Machine learning can help identify new hypotheses to test.
What’s the Next Big Question in the Social Sciences? Ask an Algorithm
The decisions of companies, governments, and educators will help to shape the ultimate outcomes of the A.I. revolution.
A.I. Is Going to Disrupt the Labor Market. It Doesn’t Have to Destroy It.Once a research question or hypothesis is in hand, LLMs could recommend a suitable experimental design, be it an economic game, market simulation, or something else. Drawing on knowledge learned from their training data, they could guide whether an experiment should be conducted in the lab or the field (or both). AI could help determine the optimal sample size for study and calculate the minimum number of participants needed to achieve statistically significant results—balancing the need for robustness with practical considerations such as cost and time limitations.
In the implementation phase of an experiment, the real-time capabilities of LLMs become particularly useful, the researchers write. By functioning as interactive chatbots, LLMs could provide immediate support to participants, clarify instructions, answer questions, and ensure compliance with the experimental protocol. They would produce a better experience for participants while also safeguarding the integrity of and monitoring an experiment. If a participant were to misunderstand instructions, become less engaged, or even cheat, LLMs could detect that and take steps to address it—all while reducing the workload for human researchers and minimizing the potential for errors.
And LLMs would significantly expand the scope and depth of data interpretation in the analysis phase, according to the research. Through state-of-the-art natural language processing techniques, they could analyze qualitative data such as participant feedback or chat logs, and extract insights that traditional statistical methods might miss. They could organize and clean data efficiently, which not only speeds up the pre-analysis process but allows researchers to focus on interpreting results and drawing conclusions. And LLMs could be used to conduct statistical tests, generate visualizations, and identify patterns or correlations.
Ultimately, generative AI opens up new avenues for exploration and discovery, the researchers write. But while outlining these and other advantages, Charness, Jabarian, and List acknowledge risks to using LLMs in experiments, “including concerns about intellectual property (IP), digital privacy issues, user deception, scientific fraud by fabricating data or strategies to hide data manipulation, hallucinations,” and more. Reliance on LLMs could result in less creative research questions, they posit, as standardization in prompts and other processes “could, in principle, create research drones” and “lead to lost opportunities for new wisdom, thought, hypotheses, and scholarship needed in the face of every new societal challenge.”
But the advantages of using LLMs, they conclude, outweigh these drawbacks—and the scientific community should adopt a structured approach that amplifies the benefits and reduces the risks. Creating such a framework would hopefully, they write, “foster a culture of policy and industry experimentation at scale.”
Gary Charness, Brian Jabarian, and John A. List, “Generation next: Experimentation with AI,” Working paper, October 2023.
Two panels of economic experts consider the disparate career impact of bearing and rearing children.
What Drives Gender Gaps at Work?
A.I. identifies a high amount of relatively uninformative content in companies’ annual reports and earnings calls.
The Equation: How ChatGPT Revealed ‘Bloat’ in Corporate Disclosures
History suggests inefficient economies benefit the most.
Where Does New Technology Have the Biggest Impact?Your Privacy
We want to demonstrate our commitment to your privacy. Please review Chicago Booth's privacy notice, which provides information explaining how and why we collect particular information when you visit our website.