In a typical regression analysis, it’s accepted practice for researchers to identify and consider these covariates. But should they do the same when running a regression discontinuity analysis? Without a formal approach, researchers again made their own call. But Calonico, Cattaneo, Farrell, and Titiunik addressed this too. Their research provides theoretical grounding for considering covariates in RDD analysis through “rigorous, applicable, transparent tools for research,” Farrell says. And their study finds that including covariates will largely improve precision just as it would in a typical regression.
The researchers also find, however, that there are certain covariates that should not be included, namely those that are directly affected by the treatment itself. For example, a researcher assessing Head Start wouldn’t want to separately consider a child’s educational performance because that, of course, would be influenced by participation in the programming.
To demonstrate how the method could be used, Calonico, Cattaneo, Farrell, and Titiunik revisited Ludwig and Miller’s work on Head Start, using the same data but designing an experiment according to their methods. They find that using covariates improved the precision of the regressions by up to 10 percent. They also demonstrate how various parts of RDD, including setting the bandwidth, should be done differently when covariates are used.
RDD in practice
The researchers have coded up their work and put it online, making it far easier to use their method. Researchers no longer have to set parameters or hire a programmer and work through bugs. They can simply download the software, feed it data, and generate scientifically sound and precise answers. This software has become the standard in academia, used in thousands of analyses. “It is hard to overstate the impact this work has had on empirical research in the past decade,” says Farrell.
Boston College’s Samuel Hartzmark and Booth’s Abigail Sussman, in a paper published in 2019, used RDD to determine that investors value sustainability. In 2016, Morningstar began scoring funds on a 100-point scale according to their sustainability, then divided the funds into five groups marked by visually salient globe icons. The top 10 percent of funds in terms of sustainability were put in the five-globe group, and the bottom 10 percent in the one-globe group.
The researchers looked at what happened when the globe rating system was introduced and used RDD to identify discontinuities at the cutoff lines between groups. Funds that are close to a cutoff line can have similar sustainability scores but be placed in different groups, and the researchers used this to demonstrate that investors react more strongly to the easy-to-grasp globe icons than to the scores underlying them.
“Prior to the rating publication, funds received similar levels of flows. After the publication, funds rated highest in terms of sustainability experienced substantial inflows of roughly 4 percent of fund size over the next 11 months. In contrast, funds rated lowest in sustainability experienced outflows of about 6 percent of fund size,” write Hartzmark and Sussman.
RDD informed research by Purdue’s Jillian B. Carr and Vanderbilt’s Analisa Packham, who find that food-purchasing assistance payments from the Supplemental Nutrition Assistance Program affected the timing and level of crimes committed in Illinois and Indiana. About the software, “we didn’t use it for the main specification, but we did use it for some robustness checks and for all bandwidth selection,” says Carr.
Granted, not every situation calls for RDD. At many businesses, experimentation has gotten so easy that there has to be a reason, such as cost or risk, to use something like RDD, says University of Chicago’s John A. List. And the software isn’t appropriate for every RDD project.
Villamizar-Villegas, who helped document the two waves of RDD growth, says, “I think that for us it was easier to pinpoint ‘formalization waves’ when the method was still in the beginning stages of development.” He’s unconvinced, but also wouldn’t rule out, that there’s a third wave underway.
For now, the use of the RDD software can be tracked through a growing list of academic citations. Simon, coauthor of the research on dental coverage, says, “I was familiar with the method previously, but read Calonico et al.’s papers extensively to better understand what the methodology is and why using their packages was the most rigorous approach to my research question.” Her research findings haven’t succeeded in moving the Medicare-coverage debate forward—but that may be because there’s not yet an equivalent, scientifically advanced tool that can be used to find answers hidden in politics.