In 2020, the Detroit police wrongfully arrested Robert Williams for shoplifting based on a match generate by a facial recognition system. This was one of the first publicized cases of AI implicating someone incorrectly in a criminal case. Naturally, his wrongful arrest sparked outrage and debate over the role of artificial intelligence in the justice system, and even more so, when the use of facial recognition technologies led to two more cases of wrongful arrest.
These incidents raise a crucial question: can AI ever truly be unbiased or accountable when applied to criminal justice?
AI has the potential to add major efficiencies in the way crime is combatted, from predicting where crimes are likely to occur to determining whether a suspect is likely to reoffend. Yet, beneath these technological promises lie significant concerns about fairness, transparency, and accountability. To better understand the promises and concerns, one has to unpack the issues surrounding whether AI can genuinely uphold justice or if it risks perpetuating the very inequities it claims to solve.
How is AI helping courts and potentially the criminal justice system?
AI can enhance courts’ efficiencies through different methods. The importance of efficiency enhancement cannot be exaggerated, especially in jurisdictions where criminal courts face huge caseloads and backlogs. At the same time, the right to a fair trial without undue delay is a widely recognized human right. Combining both premises, criminal court delays due to heavy caseloads may lead to a stay of proceedings because the delays have rendered the administration of justice no longer possible. Canada, for instance, has reported stays of criminal proceedings caused by unreasonable delays from virtually all provinces.[1]
In this context, AI would most certainly be welcome for its ability to streamline court processes. Apart from automating case management tasks such as docketing and scheduling, the exceptional data analytic ability of AI may also improve efficiency in judicial deliberation. One potential application is to predict the likelihood of recidivism. AI may assist a judge in shortening its deliberation process by providing a statistical probability of recidivism based on the comparison between the relevant characteristics of the case and database at hand.[2]
Bias and Fairness: Can AI Ever Be Truly Impartial?
AI systems are often marketed as being more objective than humans. After all, algorithms don’t have emotions or personal histories and aren’t supposed to have prejudices. But as with anything that sounds too good to be true, there’s a catch: they’re trained on data that reflects human biases. And when biased data goes in, biased outcomes come out—sometimes with devastating consequences.
Consider predictive policing, a tool designed to forecast criminal activity and guide law enforcement resources. While this sounds like a smart, data-driven approach, the reality is often different. Predictive policing algorithms frequently rely on historical crime data, which is inherently skewed. For example, if certain neighborhoods have historically been over-policed, the algorithm may flag those neighborhoods as high-crime areas, perpetuating a cycle of disproportionate surveillance and arrests. A 2020 study by MIT Technology Review highlighted how these algorithms disproportionately target minority communities, reinforcing systemic inequalities rather than addressing them. The National Institute of Standards and Technology concluded a similar result, with statistics showing that US-developed algorithms were 10 to 100 times more likely to misidentify Asian and African Americans than Caucasians.[3]
The issue extends beyond geography. Risk assessment tools, commonly used in pretrial decisions, are another area where bias emerges. These tools assign risk scores to individuals, influencing whether they are granted bail or held in custody. Studies have found that Black defendants are often scored as higher risk compared to white defendants, even when their circumstances are similar. The result? A system that amplifies societal inequities.
What’s particularly troubling is the arguably misleading level of objectiveness these systems appear to possess. When a judge or prosecutor relies on an AI-generated risk score, it may appear more authoritative than a subjective human judgment. But if the underlying data is flawed, these scores are simply digital reflections of the biases already present in the system.
Transparency and Accountability: Who’s Responsible When AI Fails?
One of the most contentious issues with AI in criminal justice is its lack of transparency. Many AI systems operate as “black boxes,” meaning their decision-making processes are opaque even to those who use them. This raises the question: how can we trust a system we don’t understand?
Take the example of ShotSpotter, an AI-powered gunshot detection system used by law enforcement agencies across the United States. The system is designed to identify gunfire sounds and alert police to their locations. However, multiple reports have documented cases where ShotSpotter produced false positives, leading officers to arrive at scenes where no gunfire occurred. In some instances, these errors resulted in wrongful arrests. Critics argue that without transparency about how the system processes sounds and filters data, it’s impossible to evaluate its reliability—let alone hold it accountable.
To begin with, it would be difficult to allocate responsibility to relevant parties when an AI system produces an unjust outcome. To hold law enforcement agencies who deploy it to account is arguably unreasonable, because they have no feasible means to examine AI’s accuracy in analysis and decision-making. On the other hand, to hold developers, as tool-providing companies to account for decisions of government could be considered inequitable, given that the government agency was tasked with being the final decision maker of arrests. This ambiguity complicates efforts to address harm and prevent future errors. In certain jurisdictions, some guidance is offered – for example, the EU’s Product Liability Directive offers some guidance, stating that AI systems must meet safety and cybersecurity standards, but even this framework leaves significant gaps when it comes to accountability for emergent or unpredictable behaviors.
This lack of transparency becomes even more problematic in courtrooms. If a defendant’s liberty hinges on evidence generated by an AI system, their legal team must be able to scrutinize that evidence. Yet, many AI companies claim proprietary rights over their algorithms, shielding them from examination. This creates a troubling scenario where justice becomes contingent on trust in technology rather than on evidence that can be challenged and verified. Without transparency in AI algorithms and the use of AI in trial preparation, the only safeguard against AI errors is the level of scrutiny employed by judges and practitioners.
A Path Forward: Balancing Innovation and Justice
Despite these challenges, dismissing AI outright is not the solution. Instead, focus should be placed on designing systems that prioritize fairness, transparency, and accountability from the ground up.
For instance, transparency could be improved by mandating AI companies to provide detailed documentation of how their systems work and the data they use for training. Independent audits and open-source models could also help build trust and ensure that systems meet rigorous standards.
Fairness, meanwhile, requires a critical examination of the data that feeds these algorithms. Diverse datasets, combined with techniques to identify and mitigate bias, are essential for creating more equitable systems. Policymakers and technologists must work together to establish clear guidelines for how data is collected, labeled, and used. For instance, Canada has implemented the Directive on Automated Decision-making in the context of administrative law. The Directive mandates the government to categorize the impact of decisions and provide different guidelines for the use of AI in decision-making in each category. In decisions that have little to no impact on individual rights, the government will apply less scrutiny to the decision and allow more room for AI to operate. On the other hand, if a decision has a high human rights implication, the decision will require stricter peer review, extra gender-based analysis, and notice of the usage of AI. In cases with high stakes, the final decision must also be made by a human.[4]
Finally, accountability mechanisms must be robust enough to address both individual and systemic failures. This includes clear liability frameworks for developers and users of AI systems, as well as avenues for individuals to challenge and appeal AI-driven decisions.
Can AI Deliver Justice?
The intersection of AI and criminal justice is both promising and perilous. On the one hand, these technologies have the potential to enhance efficiency, reduce human error, and allocate resources more effectively. On the other hand, without careful oversight, they risk entrenching biases, eroding transparency, and undermining accountability. So, can AI truly deliver justice? The answer depends on whether we’re willing to confront its flaws and commit to solutions that place fairness and accountability at the center. Because in a system as consequential as criminal justice, the stakes couldn’t be higher.
- [1] See https://www.cbc.ca
- [2] https://www.unh.edu/
- [3] See https://www.nist.gov/
- [4] https://www.tbs-sct.canada.ca/
Authors:
