Law and Order and Data

Such tools work by analyzing massive stores of data, including data about past criminal events and outcomes, to predict where crimes will occur, who’s most likely to fail to appear in court, and who’s most likely to be a repeat or violent offender. The promise of these tools is better outcomes using less resources while locking up fewer people. The risk is that they will make existing inequalities even more pervasive and difficult to address.

The academic literature contains support for both notions: cautionary tales and warnings about unintended consequences, as well as promising glimpses of new possibilities. These widely varying outcomes hint at the heterogeneity that underlies the broad banner of machine learning.

University of Chicago Harris School of Public Policy’s Jens Ludwig illustrates this point by contrasting algorithms with vaccines. Unlike a vaccine—which is a single, clearly defined thing—A.I. is not a “thing” but rather an umbrella term for a collection of tools that are heterogeneous in their design and in their implications for crime, fairness, and the harm the justice system can inflict. “In practice, A.I. algorithms actually vary enormously,” he says, including in their overall quality and in the amount of attention that gets paid to anticipating and addressing concerns that may be general (such as discrimination) or specific to their use. “The key challenge for the field,” he says, “is figuring out how we can get more of the good ones and fewer of the bad ones.”

Predictive policing and its risks

The criminal-justice pipeline starts with policing, where machine learning is being deployed in some cases to detect or solve crimes. Noise sensors installed throughout many cities feed ML algorithms trained on audio data to listen for and report gunshots—a significant task, as research by Purdue’s Jillian B. Carr and Texas A&M’s Jennifer L. Doleac has found that the vast majority of gunshots go unreported. Some departments also use ML-powered facial recognition systems to help identify the perpetrators of crimes.

ML is widely used for crime prediction too. The Chicago Police Department is using products sold by the policing-technology company ShotSpotter, including gunshot sensors but also a patrol management software called ShotSpotter Connect. Connect uses a combination of local crime and gunshot-detection data, as well as other data such as weather information, census data, and the locations of schools and parks, to identify discrete zones for police to patrol on the basis of the likelihood of a crime occurring there. The system creates risk assessments for numerous types of crime, from homicide to auto theft, and even suggests tactics officers should use on their patrol.

There’s also the threat of predictive-policing systems creating feedback loops as they ingest the data generated by their own predictions. Crime data come in part from police observations of crimes—so if an algorithm sends a patrol to a particular neighborhood, it becomes more likely that new data will be generated for that location. The data will then be used to update the algorithm, which becomes more likely to send a future patrol to the same area, which can lead to overpolicing.

Danielle Ensign, formerly of the University of Utah, Sorelle A. Friedler of Haverford College, University of Michigan PhD student Scott Neville, University of Arizona’s Carlos Scheidegger, and University of Utah’s Suresh Venkatasubramanian analyzed a model based on PredPol, a widely used policing platform, and find that it is vulnerable to such feedback loops.

They also find that by filtering the data used to update the algorithm over time, it’s possible to mitigate the problem. Under their proposed solution, the probability that a crime is fed back into the algorithm goes down as the probability of the area being patrolled goes up, thereby counteracting the tendency toward overpolicing. However, the findings underscore the threat posed by feedback, Venkatasubramanian says, “because we have to throttle the feedback to prevent the system from drifting away.”

Encouraging evidence on bias

To investigate whether predictive policing in Los Angeles resulted in more arrests of Black and Latinx people, University of California at Los Angeles’ P. Jeffrey Brantingham, Louisiana State University’s Matthew Valasik, and George O. Mohler of Indiana University–Purdue University Indianapolis reviewed the data from a randomized controlled experiment conducted earlier by Mohler, Brantingham, and five coauthors. (Brantingham and Mohler are cofounders of PredPol.) In that 2014 experiment, officers in select divisions of the Los Angeles Police Department were given a list of 20 target areas to patrol and told that crime was expected to be highest in those locations. Some lists were generated by human crime analysts using “all of the technological and intelligence assets at their disposal,” and some were generated by an algorithmic forecasting tool. Whether officers received a human-generated list or an algorithm-generated list varied randomly by day.

The algorithm outperformed the human analysts in terms of its impact on crime: for patrols of average duration, the algorithmically generated ones resulted in a 7.4 percent drop in crime, as compared with a 3.5 percent drop for human-directed patrols. Furthermore, in their follow-up analysis, Brantingham, Valasik, and Mohler find that “there is no significant difference in the arrest proportions of minority individuals” between the human and algorithmic patrols.

ML could even make policing less biased, other research suggests. Stanford’s Sharad Goel, Booking.com’s Justin M. Rao, and NYU’s Ravi Shroff used ML to derive more effective guidelines for New York City’s controversial stop-and-frisk policy. Data from 2008 to 2012 show that the overwhelming majority of stop-and-frisk incidents involved people of color and didn’t result in any further action. Goel, Rao, and Shroff developed an algorithm that, had it been used, would have enabled the police to recover 90 percent of the weapons they had confiscated using only 58 percent of the stops. The algorithm would also have improved the equity of the racial balance of stops.

In Chicago, ML is used in Strategic Decision Support Centers, which were introduced in 2017 following a civil-rights investigation into the police department by the Obama administration’s Department of Justice. The centers were meant to provide a way to combine better integration and use of technology with new management and deployment policies and procedures. District 7, in the Englewood neighborhood on the city’s south side, was one of the first to receive an SDSC; there, police leaders and analysts trained by the University of Chicago Crime Lab work side by side, using ShotSpotter products in real time to create more targeted deployment strategies for the district. Though gun violence in the city fell generally in 2017 after an exceptionally brutal 2016, it fell almost twice as steeply in District 7. The CPD has so far rolled out SDSCs to 21 of its 22 police districts.

The deployment recommendations made in each SDSC don’t divert resources from other districts—they are used to determine how to allocate patrols within districts, not across them. Therefore, even if the people who live in a particular district are overwhelmingly Black—many neighborhoods in Chicago, as in other cities, are highly segregated by race—there’s reason to think the algorithm isn’t moving the needle much on the racial balance of police exposure. And to the extent the SDSC helps reduce crime within that district, ML is helping to reduce disparities in public safety within Chicago.

Jay Stanley, a senior policy analyst at the ACLU, agrees that the narrower that predictive policing is used, the less concern there is. “It’s reasonable that changing the area reduces the concerns,” he says, “but I don’t think it eliminates them.”

Algorithms and the prison system

Algorithms’ reach into criminal justice goes beyond policing, extending into the US’s large and expensive prison system, and into decisions about who goes into it.

Some judges use algorithmic tools to help them decide who should await trial at home. This decision—often come to on the basis of the defendant’s flight risk, risk of recidivism, or both—has enormous consequences, as Chicago Harris’s Ludwig explained in a 2018 presentation as part of the Talks at Google lecture series. “If the judge jails you, on average you’ll spend two to four months in a place like the Cook County Jail,” he said. “You can imagine what that does to your job prospects. You can imagine what that does to your family.”

Greater predictive power could help focus pretrial detention on individuals who pose the greatest risk, and thereby minimize the number of people who have to endure this disruptive experience.

In 2017, Cornell’s Jon Kleinberg, Harvard’s Himabindu Lakkaraju, and Stanford’s Jure Leskovec, with Ludwig and Chicago Booth’s Sendhil Mullainathan, constructed an algorithm to explore whether they could improve on the results of the pretrial-release system then being used in New York City, in which judges could reference risk assessments made by an older predictive tool (rolled out in 2003) as part of their decision-making. They find that, had their algorithm been used during the date range they studied, it could have offered substantial benefits over the existing system.

Kleinberg and his coresearchers examined arrest and bail data in New York between 2008 and 2013, and their findings indicate that pretrial judges were out of sync with the predictions of the researchers’ new algorithm. The judges released nearly half of the defendants the new algorithm picked out as most risky—more than 56 percent of whom then failed to appear in court. With the new algorithm’s help, the researchers find, the judges could have maintained the same failure-to-appear rate while jailing 40 percent fewer people, or lowered the failure-to-appear rate by 25 percent without jailing a greater number of people. And they could have done all this while reducing racial disparities.

That research served as a proof of concept that a new algorithm could potentially improve on pretrial judgments in practice. Following that study, New York City engaged the University of Chicago Crime Lab and a private company called Luminosity to develop a new algorithmic pretrial assessment tool, which it began using in 2019. The algorithm uses eight factors to generate a 26-point risk score judges can consider in their bail decisions—higher scores correspond to a greater predicted likelihood that defendants will appear in court. Data from November 2019 to March 2020 (at which time New York suspended pretrial court appearances due to COVID-19) indicate that these risk scores predicted defendants’ behavior with a high degree of accuracy: appearance rates tracked risk scores closely, with nearly 98 percent of those who received the highest possible score—a group that included roughly four in 10 defendants represented in the data—subsequently appearing in court.

The algorithm recommended defendants in 85 percent of cases be “released on their own recognizance,” or without paying bail, in contrast to the city’s previous pretrial-release tool, which recommended such release in just 34 percent of cases. What’s more, the rate at which the new algorithm suggested ROR varied little across races: 83.5 percent of white defendants, 83.9 percent of Black defendants, and 85.8 percent of Hispanic defendants were recommended for ROR—in contrast with recommendations made by the old tool, which recommended ROR 30 percent more frequently for white defendants than Black defendants. The early evidence suggests judges’ decisions generally aligned with the new algorithm’s recommendations: nearly 90 percent of defendants recommended for ROR were ultimately released without bail.

Just as pretrial judges are often asked to predict defendants’ future behavior, trial judges may be asked during sentencing to forecast defendants’ risk of recidivism and future harm to the community. This opens another window for algorithmic involvement.

In 2013, a Wisconsin judge relied on COMPAS—an algorithmic tool, also sometimes used in pretrial-release decisions, that predicts the risk that a criminal defendant will commit another crime in the future—in his sentencing of Eric Loomis for attempting to flee from the police after being found driving a car that had been used in a shooting. Loomis argued that the sentencing violated his due process. The Wisconsin Supreme Court disagreed, and the US Supreme Court declined to hear the case.

The risks of allowing algorithms to weigh in on judicial decision-making—whether in pretrial decisions, sentencing, or parole decisions, where they’ve also been used—are obvious. Just as with predictive policing, biased data pose a threat to the equitability of outcomes.

A 2016 ProPublica analysis of COMPAS risk scores highlights that concern: the results indicate that the tool was more likely to misidentify Black defendants as high risk than white defendants, and more likely to mislabel white defendants as low risk. The company that owns COMPAS has disputed those findings, and other analyses have called the broader conclusion that COMPAS produces racially biased results into question.

But Loomis’s case highlighted a concern beyond bias: transparency. He argued that it violated his constitutional right to due process for a trial court to rely on the results of a proprietary instrument. He couldn’t challenge the accuracy or science of COMPAS because he couldn’t see it, much less analyze its results. And what he did know about it concerned him: Loomis alleged that it unjustly took gender and race into account.

Transparency is another issue that the ACLU and other watchdogs have raised in terms of predictive policing—transparency about the algorithmic systems themselves, and about the procurement processes by which those systems are selected. “A lack of transparency about predictive policing systems prevents a meaningful, well-informed public debate,” they write in a 2016 letter.

Whenever automated predictions are considered for policing, all stakeholders must understand what data is being used, what the system aims to predict, the design of the algorithm that creates the predictions, how predictions will be used in practice, and what relevant factors are not being measured or analyzed. The natural tendency to rush to adopt new technologies should be resisted until a true understanding is reached as to their short and long term effects.

They further argue that products and vendors need to be subject to independent, ongoing scrutiny—and currently are not getting that, and in fact too often claim trade secrets.

“A lot of what we’re seeing is in the form of commercial products that are proprietary and opaque by design, and also probably way oversold,” says the ACLU’s Stanley. “I think some of the commercial products have relatively simple nuts and bolts but use secrecy to evoke magical results that aren’t really reflecting what’s going on under the hood.”

And he warns that algorithms could aggravate discriminatory patterns by giving biased police and judges the appearance of digitally sanitized objectivity. “There’s a problem of people reifying the data and algorithms, and both obscuring and reifying decisions that are in fact highly questionable, but making them seem as though they’re objective,” he says. “That is the fundamental problem: a bunch of people running around playing with data and algorithms in all kinds of ways that have the potential for enormous destructivity. People are fast, loose, and out of control here.”

Regulation is the factor

This, then, is the crux of algorithms and ML in criminal-justice applications: they’re tools that, like any other, can be well made or poorly made, used for good or used for harm. Concerns about bias, transparency, and more boil down to the contents of individual algorithms, some of which are better than others—and to how those algorithms are implemented. In theory, those contents and that implementation could be strongly guided by a robust set of rules.

“Part of the problem is that this technology is still so new, we haven’t yet developed the right regulations for guiding its use in public-sector policy applications,” says Ludwig. “We need to get those regulations right.”

Kleinberg, Ludwig, Mullainathan, and Harvard’s Cass R. Sunstein argue in 2019 research that algorithms have the advantage of being explicit in a way human decision-making can never be—as long as regulation is in place that encourages transparency. The researchers suggest that such regulation would require the producers of algorithms to store the data used to train their algorithms; to make the algorithms available for regulators to test in order to see how changing certain factors, such as a job applicant’s or criminal defendant’s race or gender, would affect the algorithms’ predictions; and to make clear the algorithms’ “objective function,” or the specific outcome they’re asked to predict, to scrutinize whether that outcome is fair and reasonable.

Access to those things, the researchers argue, would bring much-needed clarity to some of the questions that are often unanswerable in cases of suspected discrimination: What factors were considered in making a particular decision, and why were those factors chosen? Algorithmic bias, under these conditions, becomes easier to detect than human bias.

Kleinberg, Ludwig, Mullainathan, and Sunstein note that when it comes to the elements that regulators need to scrutinize an algorithm, “at a minimum, these records and data should be stored for purposes of discovery.” In other words, they should be available to resolve legal questions, even if they’re not made public.

But for algorithms used in the public sector, transparency can go well beyond storing information for private inspection. The New York Criminal Justice Agency, which administers the pretrial-release algorithm developed in part by the University of Chicago Crime Lab, maintains a website where the general public can see data about its performance, read about how it’s used in practice, and even use the tool themselves. The site also describes the agency’s plan for assessing and updating the algorithm over time.

Given that public-sector use of algorithms is an issue of not only technical and regulatory competence but also popular acceptance, this kind of visibility into the function, performance, and maintenance of algorithms could play a key role in making them palatable to a skeptical public. Another key could be greater public oversight of how algorithmic products are selected for use. Given these tools’ potentially high impact, susceptibility to negative unintended consequences, and variation in quality, a thorough and transparent process for deciding which algorithm to use, and how to use it, may be appropriate. “We should be wary about the government procuring algorithms the same way we procure phones for the police department,” Ludwig says. “Having a private company say, ‘We can’t tell you how the algorithm works—that’s our [intellectual property]’ is not an acceptable answer for algorithms.”

The 2016 joint statement issued by the ACLU and its 16 cosigners echoes this sentiment:

Ludwig says one solution may be to rely less on the private sector and more on in-house or nonprofit development of algorithms. Mullainathan agrees that there’s still too little oversight of how A.I.-driven tools, from facial recognition systems to pretrial-decision algorithms, are selected by public decision makers, and that there should be far greater transparency about the performance of algorithms purchased by police departments and other public agencies. “The biggest gains in public governance that we’ve had in any country come from transparency and accountability, and we simply do not have that” when it comes to public-sector use of A.I., Mullainathan says.

Algorithms aren’t everything

As machine learning and other algorithms become more pervasive, their presence in and influence on criminal justice will likely continue to grow. Of course, it’s only part of the increasingly complicated picture of law and order in the US. Embracing algorithms, or abolishing them, will not take the place of broad and thoughtful reconsideration of how the police should function within a community, what sort of equipment and tactics they should use, and how they should be held accountable for their actions.

The question, then, is whether ML and other algorithms can be part of the future. If algorithms are to help improve American justice, the people adopting and using the tools must be fully aware of the potential dangers in order to avoid them.

Stanley concludes that while predictive policing and other examples of ML in criminal justice hold promise, it could take decades to work through the problems. Regulation is necessary, he says, but it is also a blunt tool, and legislators are rarely tech savvy.

He compares implementing ML to building the US transcontinental railroad system in the 19th century, which took many years and involved many train wrecks. “There’s no question there are ways that this could be socially useful and helpful, but it’s something that needs to be approached with great caution, great humility, and better transparency,” he says, adding that “a lot of the institutional patterns and incentives and cultures in law enforcement don’t lend themselves especially well to the kind of transparency that’s necessary. . . . It’s not foreordained that data and algorithms are going to bring some big social benefit compared to the nuts and bolts that need to be addressed to fix American policing.”

Kleinberg, Ludwig, Mullainathan, and Sunstein acknowledge that algorithms are fallible because the humans who build them are fallible. “The Achilles’ heel of all algorithms is the humans who build them and the choices they make about outcomes, candidate predictors for the algorithm to consider, and the training sample,” they write. “A critical element of regulating algorithms is regulating humans.”

Getting this regulation right could be the key to realizing the often striking performance benefits of algorithmic systems without aggravating existing inequalities—and perhaps even while reducing them. But it remains to be seen whether regulatory structures will develop that can meet this goal. Such structures are, after all, maintained by humans.