The Best Experts May Be People-Assisted Machines
- By
- November 17, 2025
- CBR - Artificial Intelligence
Artificial intelligence can be taught to reason like an expert—including an experienced case manager making decisions about a prison-diversion program, suggests research by University of Illinois PhD student Bingxuan Li, Purdue’s Pengyi Shi, and Chicago Booth’s Amy Ward. Instead of just examining basic defendant information such as employment status and criminal history, the AI system they built attempted to make the same logical connections that case managers do. (For more, read “Can a machine-learning model reason like an expert?”)
The researchers dub their tool FLAME. They started with 40 reasoning templates that they created based on expert input and then used to ask a large language model to generate 3,000 synthetic training cases. They then fine-tuned the LLM on those newly generated data and applied the improved model to intuit key challenges and risk factors from data about new potential participants.
When the team applied FLAME to data from a diversion program, the algorithm was able to assess the risk that someone would reoffend and predict whether or not that person would complete the program. The accuracy rate for the first task was about 75 percent, versus 45–60 percent for traditional machine-learning models, the researchers find.
This has potential real benefits, as AI could be used to aid in decision-making. But the 75 percent accuracy rate makes clear that humans still have a role in guiding algorithms. The question is when to trust automation and when to involve people.
How to enhance human input
A tool called FLAME uses large language models to infer hidden features from observed data, enriching prediction models. In incarceration-diversion programs, it helps improve estimates of who is likely to complete the program successfully.
Booth PhD student Zhiqiang Zhang, Shi, and Ward researched admission decisions—often made by a judge and a team of court stakeholders—to Adult Redeploy Illinois, a statewide diversion program. They developed a “priority score” for each individual based on observable characteristics such as age, criminal history, employment status, and housing stability and predicted the likelihood of their belonging to different risk classes.
The priority-score calculation weighed the potential cost savings of admitting someone against the expected costs of doing so. That meant pitting the savings from avoiding incarceration against the costs to run a program, plus the amounts related to any potential reoffense. Individuals with higher priority scores represented better investments of the program’s limited capacity—ranked according to who would give the public the biggest benefit for each slot that was available, essentially.
An individual’s true risk level was unknown. The researchers used machine learning to estimate the probability of belonging to each risk class, and then calculated an expected priority score based on all possible outcomes weighted by how likely each one was to occur.
They also simulated having humans review a model’s decisions to test how many cases needed such a review, which cases benefited most from it, and whether human intervention helped or hurt. Human review was most helpful in gray-area cases when the automated system was more uncertain about whether to admit or deny someone, they find.
And the research offers a reminder that neither AI nor humans are infallible. The simulations assumed that people make perfect decisions, which they don’t. Even with that idealized assumption baked in, the research finds that people can create complications. Say a human reviewer correctly overturns a decision where an algorithm has wrongly recommended admitting a candidate. That opens a valuable slot in the diversion program. But should the slot go to the next person in line? That might seem reasonable—but it would be wrong if the algorithm’s original priority rankings were wrong. “Correcting one error with human review could propagate new errors,” the researchers write.
- Bingxuan Li, Pengyi Shi, and Amy Ward, “Latent Feature Mining for Predictive Model Enhancement with Large Language Models,” Preprint, arXiv, October 2024, arXiv: 2410.04347.
- Zhiqiang Zhang, Pengyi Shi, and Amy Ward, “Admission Decisions Under Imperfect Classification: An Application in Criminal Justice,” Working paper, May 2025.
More from Chicago Booth Review
More from Chicago Booth
Your Privacy
We want to demonstrate our commitment to your privacy. Please review Chicago Booth's privacy notice, which provides information explaining how and why we collect particular information when you visit our website.