How Bigger Data Diminish the Value of Research to Business

Most academic research claims a link to the world outside the ivory tower. Biology research may advance medical science; philosophers may offer insight into the human condition; and research by accounting professors can put new tools, methods or insights, and findings into hands of accountants, auditors, CFOs, and a wide variety of other businesspeople.

However, this last connection appears to be fraying. University of North Carolina’s John Hand, a visiting professor at Chicago Booth since 2017, examined a random selection of publications in the accounting field’s top research journals over the past 30 years and finds a large and steady weakening of the explanatory power of what researchers have historically focused on, and continue to.

“The real world cares about stuff that matters with a capital M, or at least a medium-sized M, yet we in accounting seem to be documenting things with tiny, tiny, tiny Ms,” he says.

Hand’s analysis included more than 200 articles published between 1995 and 2024 in the four top academic accounting journals. He identified the key independent variable in each paper along with the paper’s t-statistic. In the world of academic publishing, the t-stat denotes how well a chosen variable explains the effect that is being studied; t-stats over a certain threshold are considered significant.

However, explanatory power of the t-statistic is typically closely related in an inverse manner to the number of observations in a study’s dataset. When the number of observations used in accounting research papers grows larger (as it has in the past three decades, increasing 60-fold), it becomes easier to sift through data and uncover findings that are statistically significant based on the t-stat. However, those findings are tiny in terms of their explanatory power, and Hand argues that this is what matters most to the business world. The explanatory power of key independent variables in recent accounting papers is nearly 50 times smaller than in studies conducted in the mid-1990s, he finds.

The problem, Hand suggests, is that growing datasets send researchers hunting for minutiae—what he terms “over-krilling,” or homing in on the microscopic “krill” in the ocean of ideas. “People are losing sight of the big economic picture when they focus on these tiny results,” he says. In prior decades, Hand posits, when data were not as abundant and cheap to obtain, researchers had to be more deliberate and thoughtful about the kinds of questions they asked and the data they gathered. In doing so, they tended to focus on factors that were likely to be important to businesspeople.

Hand finds that more recent papers with less powerful findings are also more heavily footnoted and contain lengthier appendixes. While recognizing that these metrics are “imperfect proxies” for the tendency to over-krill, he argues that researchers who dive into the weeds spend more time carefully annotating their work, potentially as a means of exaggerating the perceived importance of their findings.

He proposes a list of recommendations to correct what he sees as a movement in the wrong direction. It includes straightforward measures, such as having journals require that the explanatory power of each key independent variable be reported, along with more substantial fixes. These include encouraging researchers not to simply use larger, digitally collected datasets but rather to collect data manually. This slower process would help the researcher think through the data being collected and what analyses, economic or otherwise, could prove informative and meaningful to business.

“If the work we do continues to decrease in explanatory power such that our scholarly findings in 30 years’ time explain just one thousandth of 1 percent of what’s happening to the outcomes that people care about, we risk becoming wholly irrelevant to business,” he says. “This isn’t a smart direction for us to be heading.”