It’s a formula that vexes researchers and patients alike: spend years conducting a clinical trial on a new drug, only to discover at the end of the trial that the drug is useless—or, worse, harms patients. Also, testing new drugs is expensive, and getting more so. More than 90 percent of the cost of a drug’s development comes from Phase III clinical trials, according to a report last year from the Manhattan Institute, a center for policy research. The average length of a clinical trial increased by 70 percent from 1995 to 2005, and the cost of a single trial has ballooned to as much as $100 million.
For years the medical community has maintained this status quo—a traditional clinical trial that
can be long, expensive, and inconclusive—but John R. Birge, Jerry W. and Carol Lee Levin Professor of
Operations Management, and PhD student Vishal Ahuja are trying to change that. The operations management experts are developing a model for clinical trials that could speed up research, save money, get better treatments to patients, and save lives. The details for these humble aspirations are in the math.
Birge and Ahuja’s research into clinical trials grew out of the frustration felt by Anirban Basu, now an associate professor at the University of Washington. Three years ago, when he was an assistant professor in the Department of Medicine at the University of Chicago, Basu was analyzing data from a large national
trial that compared secondgeneration antipsychotic drugs used to treat schizophrenia with older drugs. The trial found the new drugs weren’t more effective, but Basu wondered if that conclusion was due to design flaws that are inherent in a conventional trial.
In such a trial, researchers divide patients into roughly equal groups. Then in various rounds of study, they randomly assign patients either a treatment or a placebo. At the end of the trial,
researchers use statistical analysis to understand how well each treatment worked. Birge has some
firsthand knowledge of this: in the mid-1990s, he participated as a patient in one such trial, a study for
the blockbuster drug Lipitor, which treats high cholesterol. Every two weeks, he was randomly assigned
either Lipitor or a placebo, with equal chances of receiving either. That meant that he was often off
But could trials work more like doctors, who switch patients from one treatment to another based on how each patient reacts to a drug or device? As researchers began to understand that Lipitor was highly effective and safe, could they have randomly assigned a greater percentage of the patients to be taking the drug, potentially improving patients’—including Birge’s—overall health, without sacrificing any learning?
An adaptive design for trials
Trials can work this way when they are designed to be “adaptive.” Adaptively designed drug trials
were pioneered in the 1970s by Don Berry, a faculty member at the University of Texas MD Anderson
Cancer Center. In an adaptively designed trial, patients continue to be randomly assigned treatments
or placebos, as they would be in a traditional study. However, the proportions of patients assigned a given treatment can change in each round of assignments as researchers learn more about each treatment’s safety and efficacy.
Suppose researchers are comparing two diabetes drugs, and drug A isn’t working well for a number of patients in the first round of a trial. In a traditional trial, the same proportion of patients would be randomly assigned drug A in the next round. By contrast, in an adaptive trial, researchers might see that more patients taking drug B were improving relative to the number of patients taking drug A. In the next round of assignments—which could include either a new group of patients or the ones already enrolled in the trial—a higher percentage of patients would randomly be assigned drug B.
The problem with adaptive trials, and the main reason they’re not more widely used, is that the trials have been slow and small by necessity. Researchers learn how a treatment affects a single patient, and that knowledge is incorporated into how the next patient is treated. But the trials don’t learn from multiple patients participating in a study simultaneously. That makes it hard to run large adaptive trials, which are frequently needed to study new drugs and treatments.
To solve that constraint, Birge and Ahuja combine two mathematical frameworks. The first is a Markov Decision Process (MDP), which can be applied when event outcomes are partly decided and partly random. The second is a Bayesian learning framework, which involves using new data to update the probability that an event will occur. The result, to borrow a term from a 2003 paper by Michael O’Gordon Duff, is called a Bayesadaptive Markov decision process. In this, the probabilities of an event happening are unknown and may vary over time as more information is observed. While Duff’s work was largely theoretical and focused on computer programming, Birge and Ahuja apply the Bayes-adaptive Markov decision process to drug trials. The probabilities at the beginning of a trial are derived from what clinicians know and believe at the time. As the trial progresses and clinicians obtain more information, they update their beliefs dynamically.
To test their model, Birge and Ahuja apply it to data from a 2008 trial that was conducted at 50 medical centers nationwide and involved 451 patients. The trial tested a stent, a device designed to improve blood flow to an artery in the brains of stroke patients, but the trial was halted when researchers discovered that patients receiving the stents were more than twice as likely to have a second stroke or die than those treated with conventional medical therapies. By the time the study was terminated, five people who had received stents had died, and a total of 46 participants in the trial had experienced a stroke or died within 30 days of receiving treatment.
The researchers in the trial ultimately learned that the stent was riskier than the alternative treatment. But their new model, the Booth researchers believe, would have allowed them to gain the same knowledge in less time, at less cost, and with less harm to patients. A trial “failure” is defined as a patient who suffers a stroke or dies within 30 days of treatment, and their research says the model would have prevented 17 failures, more than a third of the total.
Moreover, the design would have provided an additional layer of protection to patients who participated in the trial. Generally speaking, regulators scrutinize testing on medical devices less than they scrutinize it for pharmaceutical drugs. The stent being tested had been approved for general use by the Food and Drug Administration (FDA) in 2005 under a fasttrack process called the Humanitarian Device Exemption, meant to help devices reach the market that would benefit a few patients (fewer than 4,000 annually). Devices approved under this exemption tend to undergo less rigorous testing than those that go through the regular approval process. So patients participating in a study involving such a device take on additional risk, as they did in this one.
From mathematical model to reality
The new model has received some attention: Birge and Ahuja’s working paper, which they are revising for publication, last year won the Pierskalla Award from the Institute for Operations Research and the Management Sciences for the best research paper in healthcare management science.
But publishing a better model would be just the first step towards actually implementing it, which requires FDA approval. Beyond that, the model has to be refined. It works well with diseases and treatments when effects reveal themselves quickly, says Elbert Huang, director of the University of Chicago’s Center for Translational and Policy Research of Chronic Diseases. But the model, and adaptively designed studies in general, work less well when it comes to diseases and treatments whose effects manifest more slowly.
At his primary care practice, Huang has many older patients with Type 2 diabetes, and he tries to find the best treatment for each patient. That can be tough because there are many drugs to choose from, and they’re often used in combinations that haven’t been studied. In March the FDA added to the complexity doctors face when it approved the first of a new class of medicines to treat diabetes. On top of that, diabetes can take different forms that aren’t fully understood, and many patients have other health problems that complicate treatment.
To address the challenge of applying the model to diabetes, Birge and Ahuja are doing some more research, studying doctors who treat diabetes to understand how they determine the best sequences of treatments to offer patients. The researchers are using data from a large, government-funded study Huang is overseeing, which is investigating what drugs doctors have prescribed for roughly 500,000 diabetes patients treated through the US Department of Veterans Affairs.
The research, as it is further developed, could find applications beyond health care. Companies such as Amazon and Netflix are using mathematical models to determine what products to recommend to individual customers and how much to charge. This Booth-led model could potentially improve those recommendations by considering and learning from individual preferences.
But the researchers are in no rush to seek new applications. After all, the health-care industry is burdened by skyrocketing expenses, and doctors’ offices are full of people in need of treatments. Creating a better drug trial is a fine place to start.
Vishal Ahuja and John R. Birge, “Fully Adaptive Designs for Clinical Trials: Simultaneous Learning from Multiple Patients,” Working paper, August 2012.
Donald A. Berry, “Modified Two-arm Bandit Strategies for Certain Clinical Trials,” Journal of the American Statistical Association, June 1978.
Michael O’Gordon Duff, “Design for an Optimal Probe,” Machine Learning-International Workshop then Conference, 2003.