Democracy Dies in Silicon Valley: Platform Antitrust and the Journalism Industry

Newspapers are classic examples of platforms. They are intermediaries between, and typically require participation from, two distinct groups: on the one hand, there are patrons eager to read the latest scoop; on the other hand, there are advertisers offering their goods and services on the outer edges of the paper in hopes of soliciting sales. More than mere examples of platform economics, however, newspapers and the media industry play an irreplaceable role in the functioning of our democracy by keeping us informed. From behemoths such as the Jeff Bezos–owned Washington Post to outlets like the Hungry Horse News in the small town of Columbia Falls, Montana, the press lets us know what is happening on both the national and local levels. However, the age of the Internet and the corresponding emergence of new two-sided platforms is decimating the media industry.[1] In a world where more users get their news on social media platforms like Facebook than in print,[2] the survival of quality journalism depends in large part on whether the media industry can tap into the flow of digital advertising revenue, the majority of which goes to just two corporations founded around the start of the new millennium.

Facebook and Google, formed respectively in 2004 and 1998, are new types of platforms aiming to accomplish what newspapers have done for centuries: attract a large consumer base and solicit revenue from advertisers. However, unlike the fungible papers newsies once distributed hot off the presses, Facebook and Google connect advertisers and consumers in a more sophisticated, yet opaque manner. Facebook and Google are free to consumers insofar as users do not pay with money to surf the web or connect virtually with their friends. Instead, the companies collect information about users based on their online activity, and complex algorithms connect those users with targeted advertisements.[3] This new method of connecting Internet users and advertisers has been wildly successful, creating a tech duopoly profiting from nearly sixty percent of all digital advertising spending in the United States.[4]

          [1].      Throughout this Note, I refer to the journalism industry also as the “media” industry and the “news media” industry. Although there are undoubtedly nuanced differences between journalism and news media, for the purposes of this Note, I draw no distinction between them.

          [2].      Elisa Shearer, Social Media Outpaces Print Newspapers in the U.S. as a News Source, Pew Rsch. Ctr. (Dec. 10, 2018), [].

          [3].      Although I may not be interested in an upcoming Black Friday deal for chainsaws posted in a physical publication of the Hungry Horse News, Facebook and Google are—based on my history and activity on the platforms—aware of my affinity for things like antitrust law and coffee, and so their algorithms are likely to present advertisements to me for items such as books written by Herbert Hovenkamp and expensive burr coffee grinders.

          [4].      Felix Richter, Amazon Challenges Ad Duopoly, Statista (Feb. 21, 2019), https:// [].

* Executive Senior Editor, Southern California Law Review, Volume 95; J.D. Candidate, 2022 University of Southern California, Gould School of Law. I would like to thank Professor Erik Hovenkamp for serving as my advisor. All mistakes are my own.

On Electronic Word of Mouth and Consumer Protection: A Legal and Economic Analysis by Jens Dammann

Article | Computer & Internet Law
On Electronic Word of Mouth and Consumer Protection: A Legal and Economic Analysis
by Jens Dammann*

From Vol. 94, No. 3
94 S. Cal. L. Rev. 423 (2021)

Keywords: Internet and Technology Law, Product Reviews, Consumer Protection

The most fundamental challenge in consumer protection law lies in the information asymmetry that exists between merchants and consumers. Merchants typically know far more about their products and services than consumers do, and this imbalance threatens the fairness of consumer contracts.

However, some scholars now argue that online consumer reviews play a crucial role in bridging the information gap between merchants and consumers. According to this view, consumer reviews are an adequate substitute for some of the legal protections that consumers currently enjoy.

This Article demonstrates that such optimism is unfounded. Consumer reviews are—and will remain—a highly flawed device for protecting consumers, and their availability therefore cannot justify dismantling existing legal protections.

This conclusion rests on three main arguments. First, there are fundamental economic reasons why even well-designed consumer review systems cannot eliminate information asymmetries between merchants and consumers.

Second, unscrupulous merchants undermine the usefulness of reviews by manipulating the review process. While current efforts to stamp out fake reviews may help to eliminate some of the most blatant forms of review fraud, sophisticated merchants can easily resort to more refined forms of manipulation that are much more difficult to address.

Third, even if the firms operating consumer review systems were able to remedy all the various shortcomings that such systems have, it is highly unlikely that they would choose to do so: by and large, the firms using review systems lack the right incentives to optimize them.

*. Ben H. and Kitty King Powell Chair in Business and Commercial Law, The University of Texas School of Law. For research assistance or editing, or both, I am grateful to Jael Dammann, Elizabeth Hamilton, Stella Fillmore-Patrick, and Jean Raveney.

View Full PDF

Administering Artificial Intelligence – Article by Alicia Solow-Niederman

Article | Technology
Administering Artificial Intelligence
by Alicia Solow-Niederman*

From Vol. 93, No. 4 (September 2020)
93 S. Cal. L. Rev. 633 (2019)

Keywords: Artificial Intelligence, Data Governance

As AI increasingly features in everyday life, it is not surprising to hear calls to step up regulation of the technology. In particular, a turn to administrative law to grapple with the consequences of AI is understandable because the technology’s regulatory challenges appear facially similar to those in other technocratic domains, such as the pharmaceutical industry or environmental law. But AI is unique, even if it is not different in kind. AI’s distinctiveness comes from technical attributes—namely, speed, complexity, and unpredictability—that strain administrative law tactics, in conjunction with the institutional settings and incentives, or strategic context, that affect its development path. And this distinctiveness means both that traditional, sectoral approaches hit their limits, and that turns to a new agency like an “FDA for algorithms” or a “federal robotics commission” are of limited utility in constructing enduring governance solutions

This Article assesses algorithmic governance strategies in light of the attributes and institutional factors that make AI unique. In addition to technical attributes and the contemporary imbalance of public and private resources and expertise, AI governance must contend with a fundamental conceptual challenge: algorithmic applications permit seemingly technical decisions to de facto regulate human behavior, with a greater potential for physical and social impact than ever before. This Article warns that the current trajectory of AI development, which is dominated by large private firms, augurs an era of private governance. To maintain the public voice, it suggests an approach rooted in governance of data—a fundamental AI input—rather than only contending with the consequences of algorithmic outputs. Without rethinking regulatory strategies to ensure that public values inform AI research, development, and deployment, we risk losing the democratic accountability that is at the heart of public law.

*. 2020–2022 Climenko Fellow and Lecturer on Law, Harvard Law School; 2017–2019 PULSE Fellow, UCLA School of Law and 2019-2020 Law Clerk, U.S. District Court for the District of Columbia. Alicia Solow-Niederman drafted this work during her tenure as a PULSE Fellow, and the arguments advanced here are made in her personal capacity. This Article reflects the regulatory and statutory state of play as of early March 2020. Thank you to Jon Michaels, Ted Parson, and Richard Re for substantive engagement and tireless support; to Jennifer Chacon, Ignacio Cofone, Rebecca Crootof, Ingrid Eagly, Joanna Schwartz, Vivek Krishnamurthy, Guy Van den Broeck, Morgan Weiland, Josephine Wolff, Jonathan Zittrain, participants at We Robot 2019, and the UCLAI working group for invaluable comments and encouragement; to Urs Gasser for conversations that inspired this research project; and to the editors of the Southern California Law Review for their hard work in preparing this Article for publication. Thanks also to the Solow-Niederman family and especially to Nancy Solow for her patience and kindness, and to the Tower 26 team for helping me to maintain a sound mind in a sound body. Any errors are my own.

Data Protection in the Wake of the GDPR: California’s Solution for Protecting “the World’s Most Valuable Resource” – Note by Joanna Kessler

Note | Privacy Law
Data Protection in the Wake of the GDPR: California’s Solution for Protecting “the World’s Most Valuable Resource”

by Joanna Kessler*

From Vol. 93, No. 1 (November 2019)
93 S. Cal. L. Rev. 99 (2019)

Keywords: California Consumer Privacy Act (CCPA), General Data Protection Regulation (GDPR)

This Note will argue that although the CCPA was imperfectly drafted, much of the world seems to be moving toward a standard that embraces data privacy protection, and the CCPA is a positive step in that direction. However, the CCPA does contain several ambiguous and potentially problematic provisions, including possible First Amendment and Dormant Commerce Clause challenges, that should be addressed by the California Legislature. While a federal standard for data privacy would make compliance considerably easier, if such a law is enacted in the near future, it is unlikely to offer as significant data privacy protections as the CCPA and would instead be a watered-down version of the CCPA that preempts attempts by California and other states to establish strong, comprehensive data privacy regimes. Ultimately, the United States should adopt a federal standard that offers consumers similarly strong protections as the GDPR or the CCPA. Part I of this Note will describe the elements of GDPR and the CCPA and will offer a comparative analysis of the regulations. Part II of this Note will address potential shortcomings of the CCPA, including a constitutional analysis of the law and its problematic provisions. Part III of this Note will discuss the debate between consumer privacy advocates and technology companies regarding federal preemption of strict laws like the CCPA. It will also make predictions about, and offer solutions for, the future of the CCPA and United States data privacy legislation based on a discussion of global data privacy trends and possible federal government actions.

*. Executive Senior Editor, Southern California Law Review, Volume 93; J.D. Candidate 2020, University of Southern California Gould School of Law; B.A., Sociology 2013, Kenyon College. 


View Full PDF

The Impact of Artificial Intelligence on Rules, Standards, and Judicial Discretion – Article by Frank Fagan & Saul Levmore

Article | Legal Theory
The Impact of Artificial Intelligence on Rules, Standards, and Judicial Discretion
by Frank Fagan & Saul Levmore*

From Vol. 93, No. 1 (November 2019)
93 S. Cal. L. Rev. 1 (2019)

Keywords: Artificial Intelligence, Machine Learning, Algorithmic Judging



Artificial intelligence (“AI”), and machine learning in particular, promises lawmakers greater specificity and fewer errors. Algorithmic lawmaking and judging will leverage models built from large stores of data that permit the creation and application of finely tuned rules. AI is therefore regarded as something that will bring about a movement from standards towards rules. Drawing on contemporary data science, this Article shows that machine learning is less impressive when the past is unlike the future, as it is whenever new variables appear over time. In the absence of regularities, machine learning loses its advantage and, as a result, looser standards can become superior to rules. We apply this insight to bail and sentencing decisions, as well as familiar corporate and contract law rules. More generally, we show that a Human-AI combination can be superior to AI acting alone. Just as today’s judges overrule errors and outmoded precedent, tommorrow’s lawmakers will sensibly overrule AI in legal domains where the challenges of measurement are present. When measurement is straightforward and prediction is accurate, rules will prevail. When empirical limitations such as overfit, Simpson’s Paradox, and omitted variables make measurement difficult, AI should be trusted less and law should give way to standards. We introduce readers to the phenomenon of reversal paradoxes, and we suggest that in law, where huge data sets are rare, AI should not be expected to outperform humans. But more generally, where empirical limitations are likely, including overfit and omitted variables, rules should be trusted less, and law should give way to standards.

*. Fagan is an Associate Professor of Law at the EDHEC Business School, France; Levmore is the William B. Graham Distinguished Service Professor of Law at the University of Chicago Law School. We are grateful for the thoughtful comments we received from William Hubbard, Michael Livermore, and Christophe Croux, as well as participants of the University of Chicago School of Law faculty workshop. 


View Full PDF

The New Data of Student Debt – Article by Christopher K. Odinet

Article | Regulations
The New Data of Student Debt
by Christopher K. Odinet*

From Vol. 92, No. 6 (September 2019)
92 S. Cal. L. Rev. 1617 (2019)

Keywords: Student Loan, Education-Based Data Lending, Financial Technology (Fintech)



Where you go to college and what you choose to study has always been important, but, with the help of data science, it may now determine whether you get a student loan. Silicon Valley is increasingly setting its sights on student lending. Financial technology (“fintech”) firms such as SoFi, CommonBond, and Upstart are ever-expanding their online lending activities to help students finance or refinance educational expenses. These online companies are using a wide array of alternative, education-based data points—ranging from applicants’ chosen majors, assessment scores, the college or university they attend, job history, and cohort default rates—to determine creditworthiness. Fintech firms argue that through their low overhead and innovative approaches to lending they are able to widen access to credit for underserved Americans. Indeed, there is much to recommend regarding the use of different kinds of information about young consumers in order assess their financial ability. Student borrowers are notoriously disadvantaged by the extant scoring system that heavily favors having a past credit history. Yet there are also downsides to the use of education-based, alternative data by private lenders. This Article critiques the use of this education-based information, arguing that while it can have a positive effect in promoting social mobility, it could also have significant downsides. Chief among these are reifying existing credit barriers along lines of wealth and class and further contributing to discriminatory lending practices that harm women, black and Latino Americans, and other minority groups. The discrimination issue is particularly salient because of the novel and opaque underwriting algorithms that facilitate these online loans. This Article concludes by proposing three-pillared regulatory guidance for private student lenders to use in designing, implementing, and monitoring their education-based data lending programs.

*. Associate Professor of Law and Affiliate Associate Professor in Entrepreneurship, University of Oklahoma, Norman, OK. The Author thanks Aryn Bussey, Seth Frotman, Michael Pierce, Tianna Gibbs, Avlana Eisenberg, Richard C. Chen, Kaiponanea Matsumara, Sarah Dadush, Jeremy McClane, Emily Berman, Donald Kochan, Erin Sheley, Melissa Mortazavi, Roger Michalski, Kit Johnson, Eric Johnson, Sarah Burstein, Brian Larson, John P. Ropiequet, the participants and the editorial board of the Loyola Consumer Law Review Symposium on the “Future of the CFPB,” the participants of the Central States Law Schools Association Conference, the faculty at the University of Iowa College of Law, and Kate Sablosky Elengold for their helpful comments and critiques on earlier drafts, either in writing or in conversation. This Article is the second in a series of works under the auspices of the Fintech Finance Project, which looks to study the development of law and innovation in lending. As always, the Author thanks the University of Oklahoma College of Law’s library staff for their skillful research support. All errors and views are the Author’s alone.


View Full PDF

Binary Governance: Lessons from the GDPR’s Approach to Algorithmic Accountability – Article by Margot E. Kaminski

Article | Regulations
Binary Governance: Lessons from the GDPR’s Approach to Algorithmic Accountability
by Margot E. Kaminski*

From Vol. 92, No. 6 (September 2019)
92 S. Cal. L. Rev. 1529 (2019)

Keywords: Algorithmic Decision-Making, General Data Protection Regulation (GDPR)



Algorithms are now used to make significant decisions about individuals, from credit determinations to hiring and firing. But they are largely unregulated under U.S. law. A quickly growing literature has split on how to address algorithmic decision-making, with individual rights and accountability to nonexpert stakeholders and to the public at the crux of the debate. In this Article, I make the case for why both individual rights and public- and stakeholder-facing accountability are not just goods in and of themselves but crucial components of effective governance. Only individual rights can fully address dignitary and justificatory concerns behind calls for regulating algorithmic decision-making. And without some form of public and stakeholder accountability, collaborative public-private approaches to systemic governance of algorithms will fail.

In this Article, I identify three categories of concern behind calls for regulating algorithmic decision-making: dignitary, justificatory, and instrumental. Dignitary concerns lead to proposals that we regulate algorithms to protect human dignity and autonomy; justificatory concerns caution that we must assess the legitimacy of algorithmic reasoning; and instrumental concerns lead to calls for regulation to prevent consequent problems such as error and bias. No one regulatory approach can effectively address all three. I therefore propose a two-pronged approach to algorithmic governance: a system of individual due process rights combined with systemic regulation achieved through collaborative governance (the use of private-public partnerships). Only through this binary approach can we effectively address all three concerns raised by algorithmic decision-making, or decision-making by Artificial Intelligence (“AI”).

The interplay between the two approaches will be complex. Sometimes the two systems will be complementary, and at other times, they will be in tension. The European Union’s (“EU’s”) General Data Protection Regulation (“GDPR”) is one such binary system. I explore the extensive collaborative governance aspects of the GDPR and how they interact with its individual rights regime. Understanding the GDPR in this way both illuminates its strengths and weaknesses and provides a model for how to construct a better governance regime for accountable algorithmic, or AI, decision-making. It shows, too, that in the absence of public and stakeholder accountability, individual rights can have a significant role to play in establishing the legitimacy of a collaborative regime.

*. Associate Professor of Law, Colorado Law School; Faculty Privacy Director at Silicon Flatirons; Affiliated Fellow, Information Society Project at Yale Law School; Faculty Fellow, Center for Democracy and Technology. Many thanks to Jef Ausloos, Jack Balkin, Michael Birnhack, Frederik Zuiderveen Borgesius, Bryan H. Choi, Kiel Brennan-Marquez, Giovanni Comandé, Eldar Haber, Irene Kamara, Derek H. Kiernan-Johnson, Kate Klonick, Mark Lemley, Gianclaudio Maglieri, Christina Mulligan, W. Nicholson Price, Andrew Selbst, Alicia Solow-Niederman, and Michael Veale for reading and for detailed comments. Thanks to the Fulbright-Schuman program, Institute for Information Law (“IViR”) at the University of Amsterdam, and Scuola Sant’Anna in Pisa for the time and resources for this project. Thanks to the faculty of Tel Aviv University, the Second Annual Junior Faculty Forum on the Intersection of Law and Science, Technology, Engineering, and Math (STEM) at the Northwestern Pritzker School of Law, and my own Colorado Law School faculty for excellent workshop opportunities. Extra thanks to Matthew Cushing, whose incredible support made this project possible, and to Mira Cushing for joy beyond words.


View Full PDF

Unlock Your Phone and Let Me Read All Your Personal Content, Please: The First and Fifth Amendments and Border Searches of Electronic Devices – Note by Kathryn Neubauer

Note | Constitutional Law
Unlock Your Phone and Let Me Read All Your Personal
Content, Please: The First and Fifth Amendments and
Border Searches of Electronic Devices

by Kathryn Neubauer*

From Vol. 92, No. 5 (July 2019)
92 S. Cal. L. Rev. 1273 (2019)

Keywords: First Amendment, Fourth Amendment, Fifth Amendment, Border Search Exception, Technology


Until January 2018, under the border search exception, CBP officers were afforded the power to search any electronic device without meeting any standard of suspicion or acquiring a warrant. The border search exception is a “longstanding, historically recognized exception to the Fourth Amendment’s general principle that a warrant be obtained . . . .” It provides that suspicionless and warrantless searches at the border are not in violation of the Fourth Amendment merely because searches at the border are “reasonable simply by virtue of the fact that they occur at the border . . . .” The CBP, claiming that the border search exception applies to electronic devices, searched more devices in 2017 than ever before, with approximately a 60 percent increase over 2016 according to data released by the CBP. These “digital strip searches” violate travelers’ First, Fourth, and Fifth Amendment rights. With the advent of smartphones and the expanded use of electronic devices for storing people’s extremely personal data, these searches violate an individual’s right to privacy. Simply by travelling into the United States with a device linked to such information, a person suddenly—and, currently, unexpectedly—opens a window for the government to search through seemingly every aspect of his or her life. The policy behind these searches at the border does not align with the core principles behind our longstanding First and Fifth Amendment protections, nor does it align with the policies behind the exceptions made to constitutional rights at the border in the past.

In order to protect the privacy and rights of both citizens and noncitizens entering the United States, the procedures concerning electronic device searches need to be rectified. For instance, the border search exception should not be applied to electronic devices the same way it applies to other property or storage containers, like a backpack. One is less likely to expect privacy in the contents of a backpack than in the contents of a password- or authorization-protected devices—unlike a locked device, a backpack can be taken, can be opened easily, can fall open, and also has been traditionally subjected to searches at the border. Moreover, there are many reasons why electronic devices warrant privacy.

*. Executive Notes Editor, Southern California Law Review, Volume 92; J.D., 2019, University of Southern California Gould School of Law; B.B.A., 2014, University of Michigan. My sincere gratitude to Professor Sam Erman for his invaluable feedback on early drafts of this Note as well as to Rosie Frihart, Kevin Ganley and all the editors of the Southern California Law Review. Thank you to Brian and my family—Mark, Diane, Elisabeth, Jennifer, Alison, Rebecca, Tony, Jason, Jalal, Owen, Evelyn, Peter and Manny—for all of their love and support. Finally, a special thank you Rebecca for reading and editing countless drafts, and to Jason for bringing to my attention this important issue.


View Full PDF

An Uneasy Dance with Data: Racial Bias in Criminal Law – Postscript (Comment) by Joseph J. Avery


From Volume 93, Postscript (June 2019)


an uneasy dance with data: racial bias in criminal law

Joseph J. Avery[*]


Businesses and organizations expect their managers to use data science to improve and even optimize decisionmaking. The founder of the largest hedge fund in the world has argued that nearly everything important going on in an organization should be captured as data.[1] Similar beliefs have permeated medicine. A team of researchers has taken over 100 million data points from more than 1.3 million pediatric cases and trained a machine-learning model that performs nearly as well as experienced pediatricians at diagnosing common childhood diseases.[2]

Yet when it comes to some criminal justice institutions, such as prosecutors’ offices, there is an aversion to applying cognitive computing to high-stakes decisions. This aversion reflects extra-institutional forces, as activists and scholars are militating against the use of predictive analytics in criminal justice.[3] The aversion also reflects prosecutors’ unease with the practice, as many prefer that decisional weight be placed on attorneys’ experience and intuition, even though experience and intuition have contributed to more than a century of criminal justice disparities.

Instead of viewing historical data and data-hungry academic researchers as liabilities, prosecutors and scholars should treat them as assets in the struggle to achieve outcome fairness. Cutting-edge research on fairness in machine learning is being conducted by computer scientists, applied mathematicians, and social scientists, and this research forms a foundation for the most promising path towards racial equality in criminal justice: suggestive modeling that creates baselines to guide prosecutorial decisionmaking.

I.  Prosecutors and Racial Bias

More than 2 million people are incarcerated in the United States, and a disproportionate number of these individuals are African American.[4] Most defendants—approximately 95%—have their cases resolved through plea bargaining.[5] Prosecutors exert tremendous power over the plea bargaining process, as they can drop a case, oppose bail or recommend a certain level of bail, add or remove charges and counts, offer and negotiate plea bargains, and recommend sentences.[6]

When it comes to racial disparity in incarceration rates, much of it can be traced to prosecutorial discretion. Research has found that prosecutors are less likely to offer black defendants a plea bargain, less likely to reduce their charge offers, and more likely to offer them plea bargains that include prison time.[7] Defendants who are black, young, and male fare especially poorly.[8]

One possible reason for suboptimal prosecutorial decisionmaking is a lack of clear baselines. In estimating the final disposition of a case, prosecutors have very little on which to base their estimations. New cases are perpetually commenced, and prosecutors must process these cases quickly and efficiently, all while receiving subpar information; determining what happened and when is a matter of cobbling together reports from victims, witnesses, police officers, and investigators. In addition, prosecutors must rely on their own past experiences, a reliance that runs numerous risks, including that of small sample size bias. Given these cognitive constraints, prosecutors are liable to rely on stereotypes, such as those that attach to African Americans.[9]

II.  Predictive Analytics in Criminal Justice

The use of predictive analytics in the law can be bifurcated into two subsets. One involves policing, where what is being predicted is who will commit future crimes.[10] Embedded in this prediction is the question of where those crimes will occur. In theory, these predictions can be used by police departments to allocate resources more efficiently and to make communities safer.

Dozens of police departments around the United States are employing predictive policing.[11] Since 2011, the Los Angeles Police Department (“LAPD”) has analyzed data from rap sheets in order to determine how best to utilize police resources.[12] Chicago officials have experimented with an algorithm that predicts which individuals in the city are likely to be involved in a shooting—either as the shooter or as the victim.[13]

The second subset primarily involves recidivism. Here, we have bail decisions in which predictions about who will show up to future court dates are made.[14] Embedded in these predictions is the question of who, if released pretrial, will cause harm (or commit additional crimes).[15] This subset also includes sentencing, such that judges may receive predictions regarding a defendant’s likelihood of recidivating.[16]

The Laura and John Arnold Foundation (“Arnold Foundation”) designed its Public Safety Assessment tool (“PSA”) to assess the dangerousness of a given defendant.[17] The tool takes into account defendants’ age and history of criminal convictions, but it elides race and gender and supposed covariates of race and gender, such as employment background, where a defendant lives, and history of criminal arrests.[18] Risk assessments focusing on recidivism are consulted by sentencing courts.[19] These statistical prediction tools make use of a number of features (factors specific to a defendant) to produce a quantitative output: a score that reflects a defendant’s likelihood of engaging in some behavior, such as committing additional crimes or additional violent crimes.[20]

III.  Against Predictive Analytics in Criminal Justice

Statistical algorithms that have been used for risk assessment have been charged with perpetuating racial bias[21] and have been the subject of litigation.[22] A 2016 report by ProPublica alleging that an algorithm used in Florida was biased against black defendants received nationwide attention.[23] The subsequent debate about whether the algorithm actually was biased against black defendants pivoted on different definitions of fairness, with a specific focus on rates of false positives, true negatives, and related concepts.[24] Overall, the fear is that, at best, algorithmic decisionmaking perpetuates historical bias; at worst, it exacerbates bias. As one opponent of the LAPD’s use of predictive analytics said, “[d]ata is a weapon and will always be used to criminalize black, brown and poor people.”[25]

Professor Jessica Eaglin has argued that risk itself is a “malleable and fluid concept”; thus, predictive analytics focused on risk assessment give a spurious stamp of objectivity to a process that is agenda-driven.[26] Furthermore, Professor Eaglin argues that the agenda of these tools is one of increased punishment.

Critics also address the creation of the models. Some argue that the training data is nonrepresentative.[27] Others argue that recidivism is difficult to define[28] and that some jurisdictions are improperly defining it to include arrests, which may be indicative of little beyond police bias.[29] Still others debate which features such models properly should include.[30]

IV.  The Importance of Data for Criminal Justice Fairness

While it is important to question how data is used in criminal justice, the importance of data’s role in diminishing racial disparity in incarceration should not be underestimated. First, without robust data collection, we have no way of knowing when similarly-situated defendants are being treated dissimilarly. If we cannot clearly identify racial bias in the different stages of the criminal justice system, then we cannot fix it. And there is still a ways to go before prosecutorial data is properly organized and digitized.[31]

Second, data is essential for collaborative intelligence, which shows significant potential for improving prosecutorial decisionmaking. Prosecutors’ offices are in possession of information that can be used to form clear and unbiased baselines: hundreds of thousands of closed casefiles. Using advanced statistical and computer science methods, these casefiles can be used as a corpus from which to build a model that, based on an arresting officer’s narrative report and suggested charges, produces a prediction as to how a case would resolve if the defendant were treated race-neutrally. This is a classic machine-learning task: train an algorithm to produce a prediction function that relates case characteristics to case outcomes. This model can then be used to guide prosecutorial decisionmaking to make it more consistent (less variance across attorneys and across time) and less biased.

Algorithms will produce biased outcomes when the training data (the historical record) is biased and the algorithm is designed to maximize predictive accuracy. It should be obvious as to why this is the case: if predictive accuracy is the goal and the data is biased, then bias is a feature of the system, not a bug. In other words, bias must be taken into account if the prediction is to be accurate.

This is the reason why, in my research, I do not optimize prediction. My colleagues and I have different goals. Our models are not predictive models but “suggestive” models. One of our primary goals is to remove suspect bias from the model, bringing its suggestions into closer accord with Constitutional mandates for racially equal treatment of criminal defendants by state actors.

Can this be done? It is no easy feat, but researchers around the country are diligently working to build models that correct for suboptimal historical records.[32] Some of these approaches involve a weak version of disparate treatment in which the protected attribute (for example, race) is accessed during model training but omitted during classification.[33] Such approaches build from the recognition, long established in the scholarly community, that not only does blindness not entail fairness,[34] it often is a poor notion of fairness.[35]

Lastly, such models can themselves be used to identify racism that is endemic to the historical record or which emerges in the construction of the model. One strength of machine learning is that it is able to make connections between inputs and outputs that elude human actors. Social science long ago established that the human mind itself is a black box, and human actors have poor insight into their reasons for acting.[36] The black box of human decisionmaking, however, can be unpacked through careful use of statistics. Local-interpretable-model-agnostic explanations,[37] for instance, can be used to identify the aspects of input data on which a trained model relies as it makes its predictions, which should, in turn, offer insight into historical human reliance.[38]


When it comes to racial disparities, the U.S. criminal justice system is failing, and it has been failing for many years. In addition, charges of racial bias have been leveled against various organizations that are employing predictive analytics in their legal decisions. Scholars are right to question how data is being used. Past discrimination must not become enshrined in our machines. But movement away from data is also movement away from identification of unequal treatment, and it represents abandonment of the most promising path towards criminal justice fairness. While it is tempting for prosecutors’ offices to maintain the status quo and not augment their processes with data science, this would be a mistake. Collaborative intelligence has the potential to render prosecutorial decisionmaking more consistent, fair, and efficient.


[*] *. Joseph J. Avery is a National Defense Science & Engineering Graduate Fellow at Princeton University; Columbia Law School, J.D.; Princeton University, M.A.; New York University, B.A.

 [1].               Ray Dalio, Principles: Life and Work 527 (2017).

 [2]. Huiying Liang et al., Evaluation and Accurate Diagnoses of Pediatric Diseases Using Artificial Intelligence, 25 Nature Med. 433, 433 (2019),

 [3]. Karen Hao, AI is Sending People to Jail—and Getting It Wrong, MIT Tech. Rev. (Jan. 21, 2019),

 [4]. Danielle Kaeble & Mary Cowhig, Correctional Populations in the United States, 2016, Bureau Just. Stat. 1 (Apr. 2018),

 [5].               Lindsey Devers, Plea and Charge Bargaining: Research Summary, Bureau Just. Assistance 1 (Jan. 24, 2011), Plea bargaining is a process wherein a defendant receives less than the maximum charge possible in exchange for an admission of guilt or something functionally equivalent to guilt. See Andrew Manuel Crespo, The Hidden Law of Plea Bargaining, 118 Colum. L. Rev. 1303, 131012 (2018).

 [6]. Scott A. Gilbert & Molly Treadway Johnson, The Federal Judicial Center’s 1996 Survey of Guideline Experience, 9 Fed. Sent’g Rep. 87, 88–89 (1996); Marc L. Miller, Domination & Dissatisfaction: Prosecutors as Sentencers, 56 Stan. L. Rev. 1211, 1215, 1219–20 (2004); Kate Stith, The Arc of the Pendulum: Judges, Prosecutors, and the Exercise of Discretion, 117 Yale L.J. 1420, 142226 (2008); Besiki Kutateladze et al., Do Race and Ethnicity Matter in Prosecution? A Review of Empirical Studies, Vera Inst. Just., 3–4 (June 2012),

 [7]. See Besiki Kutateladze et al., Cumulative Disadvantage: Examining Racial and Ethnic Disparity in Prosecution and Sentencing, 52 Criminology 514, 518, 527-537 (2014).

 [8]. See Gail Kellough & Scot Wortley, Remand for Plea: Bail Decisions and Plea Bargaining as Commensurate Decisions, 42 Brit. J. Criminology 186, 194–201 (2002); Besiki Kutateladze et al., Opening Pandora’s Box: How Does Defendant Race Influence Plea Bargaining?, 33 Just. Q. 398, 410-419 (2016).

 [9]. Decades of research at the nexus of law and psychology have identified stereotypical associations linking blackness with crime, violence, threats, and aggression. See Joshua Correll et al., The Police Officer’s Dilemma: Using Ethnicity to Disambiguate Potentially Threatening Individuals, 83 J. Personality & Soc. Psychol. 1314, 1324-1328 (2002); Jennifer L. Eberhardt et al., Seeing Black: Race, Crime, and Visual Processing, 87 J. Personality & Soc. Psychol. 876, 889-891 (2004); Brian Keith Payne, Prejudice and Perception: The Role of Automatic and Controlled Processes in Misperceiving a Weapon, 81 J. Personality & Soc. Psychol. 181, 190-191 (2001).

 [10]. See Albert Meijer & Martijn Wessels, Predictive Policing: Review of Benefits and Drawbacks, Int’l J. Pub. Admin. 1, 2-4 (2019).

 [11].               Issie Lapowsky, How the LAPD uses Data to Predict Crime, Wired (May 22, 2018, 5:02 PM),

 [12]. Id.

 [13]. Jeff Asher & Rob Arthur, Inside the Algorithm That Tries to Predict Gun Violence in Chicago, N.Y. Times: The Upshot (June 13, 2017),

 [14].               See, e.g., Public Safety Assessment: Risk Factors and Formula, Pub. Safety Assessment [hereinafter Risk Factors and Formula], (last visited June 6, 2019).

 [15]. See Bernard E. Harcourt, Against Prediction: Profiling, Policing, and Punishment in an Actuarial Age 1 (2007); Jessica M. Eaglin, Constructing Recidivism Risk, 67 Emory L.J. 59, 61 (2017); Sonja B. Starr, Evidence-Based Sentencing and the Scientific Rationalization of Discrimination, 66 Stan. L. Rev. 803, 808–18 (2014).

 [16].               Melissa Hamilton, Adventures in Risk: Predicting Violent and Sexual Recidivism in Sentencing Law, 47 Ariz. St. L.J. 1, 3 (2015); Anna Maria Barry-Jester et al., The New Science of Sentencing, Marshall Project (Aug. 4, 2015, 7:15 AM),

 [17]. About the PSA, Pub. Safety Assessment, (last visited June 6, 2019).

 [18]. Risk Factors and Formula, supra note 14.

 [19]. Timothy Bakken, The Continued Failure of Modern Law to Create Fairness and Efficiency: The Presentence Investigation Report and Its Effect on Justice, 40 N.Y.L. Sch. L. Rev. 363, 363–64 (1996); Starr, supra note 15, at 803.

 [20]. John Monahan, A Jurisprudence of Risk Assessment: Forecasting Harm Among Prisoners, Predators, and Patients, 92 Va. L. Rev. 391, 405–06 (2006).

 [21]. Solon Barocas & Andrew D. Selbst, Big Data’s Disparate Impact, 104 Calif. L. Rev. 671, 674, 678 (2016); Jessica M. Eaglin, Predictive Analytics’ Punishment Mismatch, 14 I/S: J.L. & Pol’y for Info. Soc’y 87, 102–03 (2017).

 [22]. See State v. Loomis, 881 N.W.2d 749, 75760 (Wis. 2016).

 [23]. Julia Angwin et al., Machine Bias, ProPublica (May 23, 2016),

 [24]. See Anthony W. Flores et al., False Positives, False Negatives, and False Analyses: A Rejoinder to “Machine Bias: There’s Software Used Across the Country to Predict Future Criminals. And It’s Biased Against Blacks.”, 80 Fed. Prob., Sept. 2016, at 38; see also Danielle Keats Citron & Frank Pasquale, The Scored Society: Due Process for Automated Predictions, 89 Wash. L. Rev. 1, 6 (2014) (calling for predictions that are consistent with normative concepts of fairness).

 [25]. Cindy Chang, LAPD Officials Defend Predictive Policing as Activists Call for Its End, L.A. Times (July 24, 2018, 8:20 PM),

 [26]. Eaglin, supra note 21, at 105; see also Eaglin, supra note 15, at 64.

 [27]. See Eaglin, supra note 15, at 118.

 [28]. Joan Petersilia, Recidivism, in Encyclopedia of American Prisons 215, 215–16 (Marilyn D. McShane & Frank R. Williams III eds., 1996).

 [29].               See Kevin R. Reitz, Sentencing Facts: Travesties of Real-Offense Sentencing, 45 Stan. L. Rev. 523, 528–35 (1993) (arguing against reliance on unadjudicated conduct at sentencing).

 [30]. See Alexandra Chouldechova, Fair Prediction with Disparate Impact: A Study of Bias in Recidivism Prediction Instruments, 5 Big Data 153, 153-162 (2017); Don A. Andrews, Recidivism Is Predictable and Can Be Influenced: Using Risk Assessments to Reduce Recidivism, Correctional Serv. Can. (Mar. 5, 2015),; Jon Kleinberg et al., Inherent Trade-Offs in the Fair Determination of Risk Scores, Proc. of Innovations in Theoretical Computer Sci. (forthcoming 2017).

 [31].               Besiki L. Kutateladze et al., Prosecutorial Attitudes, Perspectives, and Priorities: Insights from the Inside, MacArthur Foundation 2 (2018),
news/2018/prosecutorial-attitudes-perspectives-and-priorities-insights-from-the-inside/report-1.pdf; see also Andrew Pantazi, What Makes a Good Prosecutor? A New Study of Melissa Nelson’s Office Hopes to Find Out, Fla. Times Union, (last updated Mar. 12, 2018, 11:18 AM).

 [32]. See Alexander Amini et al., Uncovering and Mitigating Algorithmic Bias through Learned Latent Structure (2019) (unpublished manuscript),
main/AIES-19_paper_220.pdf. For another approach at building a non-discriminatory classifier, see Irene Chen et al., Why Is My Classifier Discriminatory?, in 31 Advances in Neural Info. Processing Systems 1, 3-9 (2018),

 [33]. See Zachary C. Lipton et al., Does Mitigating ML’s Impact Disparity Require Treatment Disparity?, in 31 Advances in Neural Infor. Processing Systems 1, 9 (2018),

 [34]. Cynthia Dwork et al., Fairness through Awareness, in Proceedings 3rd Innovations in Theoretical Computer Sci. Conf. 214, 218 (2012),

 [35]. Moritz Hardt et al., Equality of Opportunity in Supervised Learning 1819 (Oct. 11, 2016) (unpublished manuscript),

 [36]. See Richard E. Nisbett & Timothy DeCamp Wilson, Telling More Than We Can Know: Verbal Reports on Mental Processes, 84 Psychol. Rev. 231, 251-257 (1977).

   [37].               Introduced by Professors Marco Ribeiro, Sameer Singh, and Carlos Guestrin, “local interpretable model-agnostic explanations,” refers to a computer science technique that attempts to explain the predictions of any classifier by learning an interpretable model around the primary prediction. See Marco T. Ribeiro et al., “Why Should I Trust You?”: Explaining the Predictions of Any Classifier, ACM 1 (Aug., 2016),

 [38]. See Michael Chui et al., What AI Can and Can’t Do Yet for Your Business, McKinsey Q., Jan. 2018, at 7,


Technology-Enabled Coin Flips for Judging Partisan Gerrymandering – Postscript (Comment) by Wendy K. Tam Cho

From Volume 93, Postscript (May 2019)


 Technology-Enabled Coin Flips for Judging Partisan Gerrymandering

Wendy K. Tam Cho[*]

This session, the Supreme Court heard oral arguments in a set of twin partisan gerrymandering cases, one brought by Democrats, Rucho v. Common Cause,[1] and the other by Republicans, Benisek v. Lamone.[2] This was not the first time the Court has considered this issue: partisan gerrymandering has now come before twenty-one Justices of the Supreme Court, without resolution. Over the history of these cases, it has remained uncontroversial that the Elections Clause in Article I, Section 4 of the U.S. Constitution gives states the right, and indeed wide latitude, to prescribe the “times, places and manner” of congressional elections. That includes the drawing of electoral boundaries. At the same time, the power of legislatures is not unfettered.  And, it is the role of the Supreme Court to guard against unconstitutional legislative acts.

Akin to every other legal issue that comes before the Court, reconciling the state’s discretion and the Supreme Court’s role in judicial review requires a judicially manageable standard that allows the Court to determine when a legislature has overstepped its bounds. Without a judicially discoverable and manageable standard, the Court is unable to develop clear and coherent principles to form its judgments, and challenges to partisan gerrymandering would thus be non-justiciable.

In the partisan gerrymandering context, such a standard needs to discern between garden-variety and excessive use of partisanship. The Court has stated that partisanship may be used in redistricting, but it may not be used “excessively.” In Vieth v. Jubelirer, Justice Scalia clarified, Justice Stevens says we ‘er[r] in assuming that politics is ‘an ordinary and lawful motive’ in districting, but all he brings forward to contest that is the argument that an excessive injection of politics is unlawful. So it is, and so does our opinion assume.[3] Justice Souter, in a dissent joined by Justice Ginsburg, expressed a similar idea: courts must intervene, he says, when “partisan competition has reached an extremity of unfairness.”[4]

At oral argument in Rucho, attorney Emmet Bondurant argued that “[t]his case involves the most extreme partisan gerrymander to rig congressional elections that has been presented to this Court since the one-person/one-vote case.”[5] Justice Kavanaugh replied, “when you use the word ‘extreme,’ that implies a baseline. Extreme compared to what?”[6]

Herein lies the issue that the Court has been grappling with in partisan gerrymandering claims. What is the proper baseline against which to judge whether partisanship has been used excessively? And how can this baseline be incorporated into a judicially manageable standard?

I. The Promise of Technology

Fifteen years ago in Vieth, Justice Kennedy wrote the following:

Technology is both a threat and a promise. On the one hand, if courts refuse to entertain any claims of partisan gerrymandering, the temptation to use partisan favoritism in districting in an unconstitutional manner will grow. On the other hand, these new technologies may produce new methods of analysis that make more evident the precise nature of the burdens gerrymanders impose on the representational rights of voters and parties.[7]

Indeed, more sophisticated technology has fueled the threat of gerrymandering. With the aid of computers and advanced software, map drawers now have the ability to adhere tightly and meticulously to legal districting practices while simultaneously and surreptitiously entrenching power. Moreover, computing power and software sophistication are only improving over time—a fact certainly not lost on Justice Kagan, who last year wrote in Gill v. Whitford, “[t]he 2010 redistricting cycle produced some of the worst partisan gerrymanders on record. The technology will only get better, so the 2020 cycle will only get worse.”[8]

In short, the threat of technology for gerrymandering is real and looms more ominously daily. However, it appears that the Justices are now seeing a possible glimmer of hope: the day of technology’s promise to help identify and curb gerrymandering may have arrived, or is, at least, arriving.

The Court now appears to accept the idea that in addition to aiding nefarious intent, computers may also help detect such intent in litigation through generating large numbers of maps that embody only the neutral districting criteria. When humans are drawing maps, it is difficult to enumerate all of the criteria that are considered for a particular map. However, with a computer, the criteria are well-specified and known. One must explicitly choose which criteria to include and which to exclude. At oral argument in Rucho, Justice Alito acknowledged as much:

If you make a list of the so-called neutral criteria—compactness, contiguity, protecting incumbents, if that’s really neutral, respecting certain natural features of the geography—and you have a computer program that includes all of those and weights them all . . . at the end, what you get is a large number of maps that satisfy all those criteria. And I think that’s realistic. That’s what you will get.[9]

The Court also seems to accept that one could use such a set of maps as some sort of “baseline.” Justice Kagan stated that “[t]he benchmark is the natural political geography of the state plus all the districting criteria, except for partisanship.”[10]

II. The Barriers to Connecting Technology with the Law

While the Court appears to be in agreement that a baseline of non-partisan maps can be created, it struggles with a way to incorporate this baseline into a judicially manageable standard that allows us to identify a partisan gerrymander. For the Justices, there is not yet a satisfactory connection between the baseline that they believe the technology can now create and the requirements of the Court for a judicially manageable standard.

There appear to be two main barriers. The first is what they see as a connection to proportional representation (PR). Justice Gorsuch seems particularly suspicious that the baseline of non-partisan maps provides nothing more than a test for proportional representation in disguise. When he sees the range of partisan outcomes that emerge from the baseline of non-partisan maps, he is not seeing how one can use those maps to identify a partisan gerrymander. He envisions that there must be a “cutoff” where partisanship becomes excessive. But, to identify that point, Gorsuch asks, “aren’t we just back in the business of deciding what degree of tolerance we’re willing to put up with from proportional representation?”[11] Justice Alito is similarly perplexed about how one might utilize the baseline set of non-partisan maps:

[I]f you have 24,000 maps that satisfy all of the so-called neutral criteria that you put in your computer program, don’t you need a criterion or criteria for deciding which of the 24,000 maps you’re going to choose? . . . [I]mplicit . . . is the idea, is it not, that you have to choose one that honors proportional representation? You have no other criteria for distinguishing among the 24,000 maps.[12]

While large deviations from PR may raise suspicion and seem intuitively problematic to the public eye, the judiciary is unequivocal that PR is inconsistent with geographically defined single member districts. Hence, this seeming connection to PR is obviously problematic given the long history of the Supreme Court’s emphasis that our system of government is explicitly not one of proportional representation. To be sure, any judicial standard cannot simply require PR or an outcome “close to PR.”

A second issue is that the Constitution grants wide discretion to the states in devising its electoral maps. Neither the appellants nor the appellees in North Carolina’s redistricting case disagree. The disagreement, rather, stems from how this wide discretion affects the use and interpretation of the baseline maps.

The challengers argue that “[t]he legislature has wide discretion, as long as it does not attempt to do two things, dictate electoral outcomes, [or] favor or disfavor a class of candidates.”[13] It is true that the legislature has wide discretion so long as it does not violate the Constitution. However, the challengers did not articulate a standard for how we would know that the legislature is dictating electoral outcomes other than to say that the legislature’s map has a partisan effect that is not one of the common effects in the baseline set of maps. The challengers’ argument, in essence, is that being on the tail of the distribution (i.e., producing an unusually uncommon partisan effect) is de facto evidence of the state overstepping its discretionary powers. We have already discussed Justice Gorsuch’s objection to this articulationthis characterization of unconstitutional gerrymander is conceptually indistinguishable from a PR standard.

Within the specific facts of the North Carolina case, the challengers also argue that statements made by the legislature show that partisanship was the predominant factor and a “material factor” in creating the map. In particular, David Lewis, a Harnett County Republican and the House redistricting leader at the time, stated that the map was drawn “to give a partisan advantage to ten Republicans and three Democrats because [I do] not believe [it’s] possible to draw a map with eleven Republicans and two Democrats.”[14] Chief Justice Roberts did not take issue with the particular facts present in the North Carolina case, but also did not see how they would then translate into a general principle to govern how the baseline set of maps would help identify the degree of partisanship utilized in future partisan gerrymandering cases.

The state of North Carolina, on the other hand, points out that all of the baseline maps are properly conceived of as non-partisan since they were all drawn without partisan information. Accordingly, they say, all of these maps would thus be within the legislature’s discretion to enact. The state looks at the large set of baseline North Carolina maps “with partisanship taken out entirely,” and observes that “you get 162 different maps that produce a 10/3 Republican split.”[15] From here, they argue that when the legislature is devising its particular map, it is “about as discretionary a government function as one could imagine.”[16] In other words, the legislature cannot be dictating outcomes when no partisan information is even being utilized. Therefore, the argument goes, all of these declaredly non-partisan maps and thus their partisan effects fall within the legislature’s discretion.

The dispute here is about what the tails of the distribution of partisan effects from the baseline set of maps indicate. Do they indicate “dictating outcomes” as the challengers argue or are all of the maps, tail or not, within the legislature’s “discretionary powers” as the state argues? More importantly for the Court, how does one distinguish “dictating outcomes” from “discretionary power?”

In short, the Court is not skeptical about whether a baseline of non-partisan maps can be created. It is skeptical about whether it can reconcile a baseline they believe exists with the wide latitude conferred to the states in the Elections Clause and our system of representation, which is explicitly not proportional representation.

III. A Judicially Manageable Standard

I argue that when the application of the “new technology” is properly conceived and executed, neither the issue of proportional representation nor our commitment to states’ rights in prescribing the “times, places and manner” of congressional elections remains problematic. In fact, both are part and parcel of a judicially manageable standard.

First, let us establish the relationship of PR with the baseline set of maps. Because partisan information is necessary to determine PR and no partisan information is used in the construction of the baseline maps, we can say, unequivocally, that PR plays no role in the construction of the baseline set of maps. Instead, the computer-drawn maps are constrained only by the locations where the particular people in the state reside and the neutral map-drawing criteria.

If partisans are randomly dispersed throughout the state and there are roughly an equal number in each party, PR is, unsurprisingly, a natural outcome. When partisans cluster geographically, this type of political geography undermines PR in the sense that a “natural outcome” would more likely be further from the PR outcome. The size of the discrepancy between PR and the common outcomes in the baseline non-partisan maps depends on the state and the precise pattern of political geography and degree of clustering. Sometimes political geography works strongly against PR. In other cases, the political geography may have only a small impact. This concept appears to be well understood by the Court. In Vieth, Justice Scalia wrote the following:

Consider, for example, a legislature that draws district lines with no objectives in mind except compactness and respect for the lines of political subdivisions. Under that system, political groups that tend to cluster (as is the case with Democratic voters in cities) would be systematically affected by what might be called a “natural” packing effect.[17] 

In other words, if Democrats tend to cluster in cities, rather than being randomly dispersed across the state, then this “political geography” that is created by their tendency toward urban clustering results in Democrats being “packed” into the same districts because the map drawer may be trying to keep cities and counties together—an objective that the Court accepts as neutral and not partisan per se.

In addition, if the partisans are not roughly proportional, PR is less likely to be the outcome. We have long known that if a state’s partisans are split, say, 70 percent Republican to 30 percent Democrat, then almost surely, the Republicans will win all of the state’s seats unless the Democrats are unusually clustered so that it is possible to place them in a district where they command the majority vote. Here again is an interactive effect between political geography and the degree to which PR is even possible—though this time, clustering would work in favor of the minority party.

Indeed, the reason we simulate maps is to understand how political geography and neutral map-drawing criteria affect the natural partisan outcomes when partisanship information is not present. The effect of political geography is statespecific since it depends on the particular people in the state, where they reside, and other neutral criteria that may be based on, for example, city and county boundaries. One can think of the simulation process as procedurally fair in the sense that the process has no explicit partisan information guiding it.

The idea behind employing simulations to understand a process, map drawing or otherwise, is not new. The concept of frequentist probabilities and their interpretation has been well-established since at least the end of the nineteenth century.[18] We can gain some intuition about how simulations work in the familiar context of flipping coins. Suppose we want to know what typically happens when you toss a fair coin one hundred times. Maybe in the first round of one hundred tosses, the coin lands on heads fiftysix times. In the second round, the coin lands on heads fortyeight times. We repeat this process a large number of times. These “simulations” help us understand the behavior of a fair coin. After we have properly repeated this process sufficiently many times, we have an accurate gauge of the behavior of a fair coin.

Figure 1 shows the result when a computer simulates one hundred tosses of a fair coin, and repeats the one hundred tosses three million separate times. This process illuminates that the outcome of more than sixty heads occurs less than 2 percent of the time. Indeed, for any outcome or number of heads, we can know how likely that outcome is to occur for a fair coin. To be sure, it is possible for a fair coin to land on heads one hundred times in one hundred tosses, but if it did, any sane person would question whether that coin was actually a fair coin. While this outcome is not impossible, it is an inordinately improbable outcome. Indeed, in my actual simulation, after the computer has tossed a coin one hundred times for three million repetitions, the event where all of the tosses landed on heads did not occur even once. We can see from the figure that even seventy-five heads would be an “extreme” outcome for an allegedly fair coin. In my actual simulation, seventy-five heads in one hundred tosses did not happen even once in the three million different attempts.

A similar baseline and analysis can inform judgments about maps. Of course, the mechanics of how to draw electoral maps are exceedingly more complex than tossing coins. Indeed, I have spent many years thinking and researching about how to do this properly,[19] but the logic is the same.

To simulate map-drawing, we repeatedly draw maps that adhere to neutral principles like equal population, preservation of cities and counties, and compactness, but do not consider partisan information. Just like for coin tosses, when properly executed, this process creates a baseline for understanding what types of outcomes emerge from a map-drawing process that does not involve explicit partisan information.

Of course, as we have discussed, a state is not constrained to consider only neutral map-drawing principles—many decisions go into devising a map, and a state has wide latitude to act in the interest of its people. There are any number of criteria that can be regarded as outside the set of neutral or “traditional districting principles” but still non-partisan. One example might be a claim that Representative Lynn Wachtmann, in the state of Ohio, made in the legislative record,

The community of Delphos is split with Representative Huffman and I, and let me share with you a little bit different story about what could happen with a great county like Lucas County if they care to work on both sides of the aisle. That is, they could gain more power in Washington.[20]

She is making a claim that the splitting of this county was not done for partisan reasons, but to garner more political power for the people of Ohio. Whether this is true or not, we leave aside at the moment. It could be true, and certainly, when a map is devised, the decisions that determine the boundaries should be done in the interest of the people. In this sense, that the legislature has wide latitude to work in the interest of its people is a feature, not a flaw. Indeed, there are many non-partisan decisions that may lie behind a particular map configuration. Possibly, a representative wants her church or her family’s cemetery in her district. Why a representative may want those things might be personal and completely devoid of partisan motivation. These types of decisions all fall within the wide latitude and undisputed discretionary power of the legislature to devise its electoral map.

Note that even completely non-partisan decisions have partisan effects. Every time a boundary is changed, partisans are shifted from one district to another district. This necessarily changes the partisan composition of the districts, and a partisan effect ensues. But, then, if all decisions, even non-partisan ones have a partisan effect, how do we know if the admittedly many decisions behind a map make it “excessively partisan”? It would be impossible, almost surely, and impractical, at the very least, to try to discover all the reasons and then to determine whether each one was partisan or not.

This realization that many elements influence district boundaries is not lost on the Court. In Vieth, Justice Breyer wrote that the desirable or legal criteria represent a series of compromises of principle—among the virtues of, for example, close representation of voter views, ease of identifying government and opposition parties, and stability in government. They also represent an uneasy truce, sanctioned by tradition, among different parties seeking political advantage.[21]

Partisan effect that arises from the compromise of principles is not problematic. The need for compromise among many factors is a given. It is well established that an important role of the legislature is to bargain and compromise in the pursuit of legislation. The issue is not the compromise of principles, but rather, determining when partisanship has been injected excessively.

To gain some insight into this conundrum, we can think about how this works with the coin toss simulation. A fair coin lands on heads roughly half the time because it is not biased toward heads or tails. Likewise, non-partisan decisions, by definition, are not biased toward one party or the other. Roughly half the time (with the exact probability again depending the political geography of the state), a non-partisan decision will shift partisans in a way that makes a map more Republican. Roughly the other half of the time, it will shift partisans in a way that makes a map more Democratic. To be sure, every shift provides a more favorable effect for one party over the other. However, in the aggregate, for non-partisan decisions, there should be no systematic bias in favor of one party and at the expense of the other party.

Recall that our baseline effect emerges from only neutral criteria (the “traditional districting principles” and the law). It shows what type of partisan outcomes we expect when one employs only the neutral non-partisan map-drawing criteria. If the other motivations behind a map are non-partisan, the unintended partisan effects should wash out, just as over the course of one hundred coin flips, the tallies of heads and tails will be similar. If the partisan effects from these other decisions do not wash out (or if there are many more heads than tails), then we have evidence of partisan motivation (or unfair coins).

The stronger the cumulative partisan effect is in one direction, the greater the evidence of underlying partisan motivations. If a coin lands on heads once, no suspicion is raised. If the second flip also lands on heads, I can say that I am not bothered in the least. But if that coin lands on heads one hundred times in a row, my disbelief is boundless.

 If the legislature uses only neutral criteria, then the expected effect is reflected in the baseline set of maps. Of course, the legislature will contemplate, negotiate, and compromise. No one would argue that they should “choose” one of the baseline maps that are restricted to a small set of criteria. This would be inconsistent with the Elections Clause because it would heavily constrain the legislature rather than allowing it wide latitude. Instead, many other criteria will be considered. Importantly, the political effect from non-partisan decisions should wash out if they are truly non-partisan in nature. If one non-partisan decision results in a map that leans more favorably toward the Republicans, I am not suspicious in the least. After all, every decision moves the map in one party’s favor or the other party’s favor. If a second decision moves the map more Republican, I remain unsuspicious. As the decisions pile up and they continually move the map toward the tail of the baseline distribution, my disbelief grows.

IV.  The State of Ohio

To see how my proposed test would work in an actual redistricting case, we can examine the congressional electoral map for the state of Ohio. I served as an expert witness in the state of Ohio’s gerrymandering case, A. Philip Randolph Inst. v. Householder.[22] Since the 2010 redistricting, each of the congressional races (in 2012, 2014, 2016, and 2018) resulted in twelve Republican seats and four Democratic seats. Figure 2 shows the seat split from more than three million computer-generated maps that I created on the Blue Waters supercomputer for the state of Ohio using only the neutral districting criteria with Ohio’s population and its particular political geography. In the figure, we can see that nine Republican seats is the most commonly expected outcome. Eleven Republican seats is not common at all, and twelve seats, which did occur among the more than three million maps, is an outcome that happens so infrequently that while the histogram bar at twelve seats is present, it is sufficiently minuscule that it is not even visible.

Judging from the legislative record in Ohio, the legislature considered population equality, compactness, contiguity, minority representation, and the preservation of cities and counties in the construction of the current Ohio map.[23] My simulated maps do likewise. The legislature also took a number of other unspecified criteria into account. Once all of the legislature’s criteria were taken into account, the map they produced resulted in a 12/4 Republican/Democrat seat split for every set of congressional elections run under this map.

While we do not know what each of the individual decisions behind the map were, we do know that every one of their “unspecified criteria” moved the map either toward a more favorable Republican outcome or a more favorable Democratic outcome. How did they end up on the tail of the seat share distribution? It is possible that using only the neutral districting criteria, they started at an extreme location. It is possible, but as we know, it’s extremely unlikely—just like obtaining a highly disproportionate number of heads when tossing a fair coin one hundred times. 

One could also argue that many other considerations went into the decision process. Indeed, many other decisions could have and should have entered the calculus. One could also make the claim that these decisions were not partisan. Some appear to be benign requests like splitting a military base across several districts. Other decisions may have involved an explicit attempt to protect constituents’ interests, aimed at better representation for the people of Ohio. Each of these decisions, partisan or not, changed the partisan effect of the map. But the non-partisan decisions should have no systematic bias toward the Republicans or the Democrats. Their collective partisan effect should wash out in the aggregate. On the other hand, partisan decisions surely are intended to have a specific partisan effect and move the map in the intended direction.

What we observe is a map that is all the way on the right end of the distribution of partisan effect. That means we either began on the tail, which is extremely unlikely, or we started in a more likely spot and then the subsequent decisions moved that partisan effect to that end of the distribution. If the subsequent decisions moved that map so far in one direction, it is like the coin that keeps landing on heads. If the first “decision” makes the map more Republican leaning, that is not bothersome since it has to have some partisan effect. If the second “decision” moves the map in the Republican direction again, that is also not so unusual. If the entire set of decisions move the partisan effect all the way to the end of the distribution, we have strong evidence that an increasingly small set of those decisions were actually non-partisan.

Importantly, note that there are different types of partisan unfairness. An electoral map can be unfair if partisanship is used excessively so that one party’s seat share or electoral outcomes are affected. This might be observed, as we have just seen, by how many seats favor one party over the other. However, this is not the only way in which a legislature may use partisan information to usurp power from the voters. Another option is to create districts that are not competitive. When districts are not competitive, the outcome is essentially pre-determined such that the voters are effectively disenfranchised because while they are still able to cast a ballot, their ability to influence elections has been non-trivially compromised.

In my capacity as an expert witness for the Ohio gerrymandering case, I produced not just the baseline distribution shown in Figure 2, but also the one shown in Figure 3. Here, I examined how many of Ohio’s congressional districts were competitive. I defined “competitive” as resulting in an outcome that was “within a 10% margin of victory” (i.e., the winning party received no more than 55 percent of the two-party vote and the losing party received at least 45 percent of the two-party vote). Recall that I have already generated more than three million baseline maps. To be sure, when we have a set of baseline maps, there are many facets of these maps that can be examined. We are not restricted to seat shares or even the number of competitive seats. Indeed, this set of baseline maps has depth and richness on many dimensions, allowing us to explore numerous and varied facets of an electoral map. When I examined the competitiveness of Ohio’s congressional seats, I found that, commonly, half of the districts in the simulated maps were competitive. In contrast, in the current Ohio congressional map, all of the districts are quite non-competitive. So, in addition to producing a highly unusual seat split, the maps also resulted in a highly unusual lack of competitive seats. To be highly unusual on two partisan measures, as you can easily intuit, is even more suspect than if the current Ohio map was unusual in only one way. Maybe the first time you toss a coin one hundred times, the coin lands on heads an unusually large number of times. Unusual events like this do happen. If you toss that coin one hundred times again and a second unusual outcome occurs, the strength of the evidence is undeniably stacking up against that coin being fair.


Surely a map can be unusual on only one dimension. For instance, in North Carolina, if the map resulted in a 7/6 seat split, just because this outcome is “close to PR” does not exonerate it from other possible gerrymandering claims. We see clearly here that the baseline set of maps is not about some assessment of PR. Rather, they are far richer, allowing us to scrutinize many facets of partisan unfairness. If that map is 7/6 but sufficiently uncompetitive so that the voters have very little ability to change the outcome, then that map “dictates outcomes” and can be regarded as unconstitutional in that way. What makes a map unfair is not a deviation from any sense of proportional representation. What makes it unfair is the evidence that excessive partisanship was utilized.

V.  Rigorous Identification of Partisan Gerrymandering is Possible

When subject to litigation, a state is free to protest that its legislature’s map has been improperly identified as “excessively partisan.” That state can also present exculpatory evidence. Clearly, a map drawn free of partisanship can have an extreme partisan effect that emanates from neutral considerations. A fair coin also can land on heads one hundred times, but this outcome invites incredulity. Simulations are never able to tell us definitively that a coin is not fair or that the decisions behind a map are excessively partisan with certainty. In both cases, the simulations provide evidence and give us a sense of the strength of that evidence. The greater the number of heads over tails, the greater the evidence against a fair coin. The further the partisan effect moves from the baseline maps, the greater the evidence that partisanship was used excessively.

Sometimes, one has a smoking gun. Perhaps a suspect was caught, covered in blood, standing over the victim, holding the murder weapon at the crime scene. In the case of North Carolina, one may or may not regard Representative David Lewis’s comments about purposefully drawing a 10/3 map as this type of evidence. Barring such evidence, we still have a way to develop solid, probative, and dispositive evidence through the baseline set of maps.

The ability to create a baseline set of maps, combined with a proper and theoretically sound interpretation allows us to honor the Elections Clause that provides wide latitude to the states to prescribe the times, places, and manner of its elections, support our system of geographically based single member districts, be divorced from notions of proportional representation, and maintain the Court’s oversight of the legislature by providing a judicially manageable standard that assesses whether legislative decisions are excessively partisan.

The cutoff for what qualifies as “excessive” is a legal judgment call—the bread and butter of the Supreme Court’s constitutional jurisprudence. The exact cutoff may not be clear, but the Court is the institution charged with making that judgment. What is clear is that there is a way to measure excessiveness that is consistent with the Constitution’s regard for states rights and the legislature’s mandate to legislate for the people. This measure is not related to proportional representation, and it serves as the basis for a judicially manageable standard.

Whether the Court analyzes partisan gerrymandering as a matter of First Amendment viewpoint discrimination, as a matter of vote dilution under the Equal Protection Clause, or as an abuse of the power delegated to states under the Elections Clause, recent technological developments now enable the Court to put judicially manageable limits on the excessive use of partisanship in designing election districts. Technology has surely fueled the threat and growth of gerrymandering by providing a tool for the partisan majority of a state legislature to draw self-serving electoral boundaries, but it also now fulfills its promise by providing the basis for a judicially manageable standard to help judge whether electoral maps are excessively partisan.


[*] Professor in the Departments of Political Science, Statistics, Mathematics, and Asian American Studies, the College of Law, and Senior Research Scientist at the National Center for Supercomputing Applications at the University of Illinois at Urbana-Champaign. She has served as an expert witness in redistricting litigation and has published research on technological innovations for redistricting analysis in computer science, operations research, statistics, physics, political science, and law.

 [1]. Transcript of Oral Argument, Rucho v. Common Cause, No.18-442 (U.S. Mar. 26, 2019).

 [2]. Transcript of Oral Argument, Benisek v. Lamone, No. 17-333 (U.S. Mar. 28, 2019).

 [3]. Vieth v. Jubelirer, 541 U.S. 267, 293 (2004) (alteration and emphasis in original) (internal quotation marks omitted).

 [4]. Id. at 344 (Souter, J., dissenting).

 [5]. Transcript of Oral Argument, supra note 1, at 38.

 [6]. Id.

 [7]. Vieth, 541 U.S. at 312–13 (Kennedy, J., concurring).

 [8]. Gill v. Whitford, 138 S. Ct. 1,916, 1941 (2018) (Kagan, J., concurring) (citation omitted).

 [9]. Transcript of Oral Argument, supra note 1, at 42.

 [10]. Id. at 27 (emphasis added).

 [11]. Id. at 43–44.

 [12]. Id. at 30–31.

 [13]. Id. at 43.

 [14]. Common Cause v. Rucho, 279 F. Supp. 3d 587, 604 (M.D.N.C. 2018).

 [15]. Transcript of Oral Argument, supra note 1 at 30.

 [16]. Id. at 29.

 [17]. Vieth v. Jubelirer, 541 U.S. 267, 289–90 (2004) (citation omitted).

 [18]. For the early development and discussion of these concepts, see generally A. A. Cournat, Exposition de la Théorie des Chance et des Probabilités (1843); John Venn, The Logic of Chance: An Essay on the Foundations and Province of the Theory of Probability (1888); Robert Leslie Ellis, On the Foundations of the Theory of Probabilities, in Mathematical Proceedings of the Cambridge Philosophical Society (B.J. Green et al., eds., 1844).

 [19]. Wendy K. Tam Cho & Simon Rubinstein-Salzedo, Understanding Significance Tests from a Non-Mixing Markov Chain for Partisan Gerrymandering Claims, 6 Stats. and Pub. Pol’y (forthcoming 2019),; Wendy K. Tam Cho & Yan Y. Liu, A Massively Parallel Evolutionary Markov Chain Monte Carlo Algorithm for Sampling Complicated Multimodal State Spaces, in SC18: The International Conference for High Performance Computing, Networking, Storage and Analysis (2018),; Bruce E. Cain, Wendy K. Tam Cho, Yan Y. Liu & Emily Zhang, A Reasonable Bias Approach to Gerrymandering: Using Automated Plan Generation to Evaluate Redistricting Proposals, 59 Wm. & Mary L. Rev. 1521 (2018); Wendy K. Tam Cho & Yan Y. Liu, Sampling from Complicated and Unknown Distributions: Monte Carlo and Markov Chain Monte Carlo Methods for Redistricting, 506 Physica A 170 (2018); Wendy K. Tam Cho & Yan Y. Liu, Massively Parallel Evolutionary Computation for Empowering Electoral Reform: Quantifying Gerrymandering via Multi-objective Optimization and Statistical Analysis, in SC17: The International Conference for High Performance Computing, Networking, Storage and Analysis (2017),; Wendy K. Tam Cho, Measuring Partisan Fairness: How Well Does the Efficiency Gap Guard Against Sophisticated as well as Simple-Minded Modes of Partisan Discrimination? 166 U. Pa. L. Rev. Online 17 (2017); Yan Y. Liu, Wendy K. Tam Cho & Shaowen Wang, PEAR: A Massively Parallel Evolutionary Computation Approach for Political Redistricting Optimization and Analysis, 30 Swarm and Evolutionary Computation 78 (2016); Wendy K. Tam Cho & Yan Y. Liu, Toward a Talismanic Redistricting Tool: A Computational Method for Identifying Extreme Redistricting Plans, 15 Election L.J. 351 (2016); Yan Y. Liu, Wendy K. Tam Cho & Shaowen Wang, A Scalable Computational Approach to Political Redistricting Optimization, in Proceedings of the XSEDE 2015 Conference: Scientific Advancements Enabled by Enhanced Cyberinfrastructure (2015); Douglas M. King, Sheldon H. Jacobson, Edward C. Sewell & Wendy K. Tam Cho, Geo-Graphs: An Efficient Model for Enforcing Contiguity and Hole Constraints in Planar Graph Partitioning, 60 Operations Res. 1213 (2012).

 [20]. H. & S. Rep. No 319, pts. 12, at 28 (Ohio 2011).

 [21]. Vieth, 541 U.S. at 360 (Breyer, J., dissenting).

 [22]. Ohio A. Philip Randolph Inst. v. Householder, No. 18-cv-357, 2019 U.S. Dist. LEXIS 24736, at *40–41 (S.D. Ohio Feb. 15, 2019).

 [23]. See Wendy K. Tam Cho, Expert Witness Testimony filed in Ohio A. Philip Randolph Inst. v. Householder, No. 18-cv-357, 2019 U.S. Dist. LEXIS 24736, at *40–41 (S.D. Ohio), Oct. 5, 2018.