Explanation and justification: AI decision-making, law, and the rights of citizens
Schwartz Reisman Director Gillian Hadfield argues that current approaches towards explainable AI are insufficient for users. What is needed instead is “justifiable AI” that can show how the decisions of an AI system are justifiable according the rules and norms of our society. This piece is the eighth in a series of posts on the features, implications, and controversies surrounding privacy law reforms in Canada and around the world in an increasingly digital and data-rich context. Photo: Sebastiaan ter Burg.
The European Union led the world with comprehensive privacy law reforms when it passed the General Data Protection Regulation (GDPR) in 2016. Most of this legislation focused on legal requirements for how data is collected, stored, and used online. But the law also introduced a key, but confusing, requirement for automated decision-making systems—systems that use machine learning (ML) to analyze big data sets and decide who is entitled to things like government benefits, medical treatments, loans, or jobs.
The GDPR said that people who are subject to an automated decision should have the right to obtain an explanation of the decision and the right to challenge it. However, the law itself did not grant this right. Rather, it granted the right not to be subject to a decision “based solely on automated processing,” unless authorized by state legislation safeguarding the data subject’s interest, or consented to by the data subject. If a fully automated decision is authorized, the law gives the data subject the right “to obtain human intervention, express his or her point of view and contest the decision,” to know of the existence of the automated decision making, and to obtain “meaningful information about the logic involved.” There is also a right to have such decision-making not based on “data revealing racial or ethnic origin, political opinions, religious or philosophical beliefs, or trade-union membership.” If genetic or biometric data is analyzed in the decision for purposes of identifying a person, “data concerning health… sex life or sexual orientation” cannot be used.
Canada’s proposed privacy reforms, Bill C-11, go further than the GDPR. The bill provides the right to an explanation, granting this right even in cases in which humans are “in the loop.” The bill defines an automated decision system as “any technology that assists or replaces the judgement of human decision-makers using techniques such as rules-based systems, machine learning, deep learning and neural nets.” The bill requires an organization to provide plain language information about its use of an automated decision system “to make predictions, recommendations or decisions about individuals that would have significant impacts on them” [Section 62(2)(d)] and to provide, on request by an affected individual, “an explanation of the prediction, recommendation or decision” [Section 63(3)].
What is not defined, in either the GDPR or Bill C-11, is the term “explanation.”
Bill C-11 requires an organization to provide plain language information about its use of an automated decision system… What is not defined, in either the GDPR or Bill C-11, is the term “explanation.”
Computer scientists working in artificial intelligence developed the concept of “explainable AI” almost 50 years ago. Researchers at Stanford University produced an interactive expert system to help physicians decide on appropriate therapies for hospital patients with bacterial infections, the MYCIN system [1]. The researchers thought doctors would be unlikely to follow the advice of a computer if they could not understand the reasons behind its recommendation. So, they built the capacity for the computer to explain its recommendations when queried. Among other things, this allowed a physician “to reject the advice if he [sic] feels that a crucial step in the decision process cannot be justified.” This type of explainable AI—allowing a user to check the reasoning of an AI system—was also a focus of the military. In the mid-2000s, researchers at the University of Southern California’s Institute for Creative Technologies built an explainable AI module into automated soldier-training systems that could query why artificial agents had taken actions in a military simulation. In 2017, the U.S. Defense Advanced Research Projects Agency (DARPA) issued a call for research on Explainable AI.
In the past five years, the study of explainable AI has grown substantially. The robustness of the field is largely a consequence of the transition of AI research from the type of expert systems that the original MYCIN paper addressed to today’s ML techniques. An expert system is fully coded by humans: every recommendation it makes can be explained by choices the programmer made. Explanation in such a system is simply a matter of recovering those choices and sharing them with the user—the doctor or soldier trainee. But modern ML systems write their own rules for the choices they make. They take in tremendous amounts of data, and then use a goal supplied by a human programmer to figure out which rules will work best to achieve that goal. Whereas the MYCIN model was programmed by medical experts, an ML system built today to recommend antimicrobial therapies is likely to be built by feeding a computer thousands, if not millions, of cases including data about symptoms, patient characteristics, treatments, and outcomes, and tasking the machine with figuring out which therapies seem to work best when. The promise of ML is that the machine can discover patterns human experts do not see—Google’s DeepMind, for example, recently developed an ML system that could diagnose breast cancer from scans more accurately than human radiologists. But that promise also generates a challenge: how can we humans understand why the machine saw something we didn’t?
The field of explainable AI in computer science has developed techniques for studying complex ML models to resolve this challenge. Some approaches check to see how the machine’s predictions shift with potential key variables—how a loan repayment prediction, for example, changes when repayment history changes. Others have developed techniques for scanning activity in a neural network to see what parts of the system activate in response to a stimulus (e.g. a picture of a tumour) in much the way neuroscientists scan the human brain to see what regions light up when a person is shown an image. And some researchers have focused on the premise of that original MYCIN paper: that explanations improve experts’ willingness to rely on machine recommendations and their ability to evaluate whether the recommendation is appropriate or not. (Schwartz Reisman Faculty Affiliate Marzyeh Ghassemi has done important work showing this premise may be incorrect.)
But a central question arises: are the explanations that computer scientists are producing the same ones that laws like the GDPR and the proposed Bill C-11 are looking for? I think that, by and large, they are not.
“are the explanations that computer scientists are producing the same ones that laws like the GDPR and the proposed Bill C-11 are looking for? I think that, by and large, they are not.”
Explainable AI is AI that makes sense to a computer scientist, expert, or company using an AI system to make a decision. Explainable AI techniques identify what is happening inside an AI system: what parts of an image it is “looking” at, how it weighs different factors. These kinds of explanations help the computer scientist understand, for example, how to make a system safer or more predictable, and they (may) help the user to decide when to follow the recommendations of a system.
But the right to an explanation contained in the GDPR, or Canada’s Bill C-11, is not a right held by the designer or the user of a system. It is a right bestowed on someone affected by a system—the individual about whom a decision is made. The explanation is supposed to serve the needs and protect the interests of a patient, for example, not a doctor.
Yet most of us are not that interested in what is happening inside the math of an ML model. We are interested in knowing whether the decision being made about us can be justified: Is the decision fair? Did the process follow the rules about how such decisions are supposed to be made? Is it based on legitimate considerations? Was it made in a reasonable and reliable way?
We want the same thing from a decision made by a computer that we want from a decision made by a human: We want reasons—justifications—that follow the rules of our community.
Suppose you have been denied a loan. The rules of the market say that a lender is allowed to do this because it thinks the risk you won’t repay the loan and the costs of administering it outweigh the revenue (interest) they will make. The lender can reach that decision in lots of ways, including lumping you into a whole category of borrowers for whom they have decided the risks and costs outweigh the benefits. So, a valid justification is: “We decided it wasn’t profitable to lend to you.”
An example of an invalid justification is: “We decided not to lend to people of your race or gender, or those who live in your neighbourhood.” Our human rights and anti-discrimination laws rule those reasons out of bounds. Other invalid justifications could include: “We made loans only to people who slipped some cash to the loan officer and you didn’t; we played a poker game, used loan applications as chips, and your loan officer lost that night; we used a random number generator and your number didn’t come up.” Decisions like those violate rules and norms of the marketplace—an implied obligation to evaluate a loan application in good faith on its merits.
We want to know that the decisions that affect us are justifiable according the rules and norms of our society.
Notice how different this is compared to looking for the kind of causal explanation that explainable AI offers regarding ML decisions. If we hook the loan officer up to an MRI while they make the decision of whether to grant us a loan, we may well get true causal claims about their decision; perhaps of the form “brain region x was instrumental in coming to the decision to reject your application.” But this is not the kind of explanation we were looking for—it’s neither a good nor a bad reason for denying the loan, it’s merely a cause.
We don’t want to know the causal specifics of how a decision was made, except to the extent that causation is relevant to a rule governing how such decisions can be made. Because lenders in our society can justify a decision based on a good faith evaluation of profitability, we don’t need (or even want) to know the mathematical details of how all the legitimate factors in our credit history were weighed. We want to know that the decision did not turn on illegitimate factors, and was not random—that it was not made to achieve goals unrelated to legitimate profit-making.
If a doctor has made a decision about how to treat our illness, we want to know that the doctor followed the rules and norms for making such decisions: that they considered all the facts available carefully, they exercised judgment with our best interests and their duty to care for us in mind, they used their expertise and sought outside expertise where their own competence fell short, and they did not discount our pain because of our race or sheer cruelty.
What I think we are looking for, as the people affected by automated decision systems, is not explainable AI but justifiable AI. While explainable AI is focused on fact, justifiable AI is focused on judgment.
“What I think we are looking for… is not explainable AI but justifiable AI. While explainable AI is focused on fact, justifiable AI is focused on judgment.”
Currently, explainable AI research is producing explanations that will be both too much and too little to meet the need for justifiable AI. They will be too much because a lot of the complexity of those explanations will not be meaningful to people affected by automated decision systems. They will be too little because, standing alone, they won’t make the connection to judgment and the rules and norms governing decisions.
Building justifiable AI requires figuring out how to integrate automated decision systems into our judgments about what is and what is not acceptable decision-making. These are what I call our normative systems, and they include the processes and procedures we have available for challenging and debating what is acceptable according the norms, rules, and laws of our society.
In the medical context, for instance, decisions are judged by professional colleagues, organizational review procedures for when things go wrong, formal complaint, investigation, and disciplinary procedures conducted by a licensing body (e.g. the College of Physicians and Surgeons of Ontario), and malpractice lawsuits.
These are all normative systems, with more or less formal procedures, for asking a decision-maker to give reasons for their decision and for exposing those justifications to scrutiny against our norms, rules, and laws. When someone who suffers a bad outcome as a result of medical care believes that a doctor did not follow the rules—exercising reasonable care to choose appropriate treatments, disregarding illegitimate considerations such as race—then our courts are available with formal procedures for determining whether the rules were followed.
As AI becomes a tool in the medical context, we have to figure out how to ensure its use is justifiable by modifying these existing procedures, and perhaps inventing new ones. In work I am doing with colleagues at the Vector Institute and the Center for Human-Compatible AI at the University of California at Berkeley, we are exploring if it is possible to develop a licensing procedure that would only allow an automated system to be deployed if human managers were able to provide adequate justifications for its decisions. This may, or may not, require that the system be “explainable” in the sense that computer scientists now use that term.
And this brings us back to the GDPR and Bill C-11. It seems clear that the impetus behind the right to explanation is linked to the goal of ensuring that decisions made with automated systems are justifiable. A right to explanation, standing alone or understood in the way that the field of explainable AI currently understands explanation, won’t accomplish that. Instead, we should be framing laws governing the use of automated decision systems in terms of the obligations of the organizations that use them to ensure that machines do not undermine fidelity to the rules by which we humans live.
[1] Shortliffe et al. (1975), “Computer-Based Consultations in Clinical Therapeutics: Explanation and Rule Acquisition Capabilities of the MYCIN System.” Computers and Biomedical Research Vol. 8, 303-320.
Editor’s note: Bill C-11 failed to pass when Canada’s federal parliament was dissolved in August 2021 to hold a federal election. In June 2022, many elements of C-11 were retabled in Bill C-27. Read our coverage of C-27.
Want to learn more?
Read about Gillian Hadfield’s research on normativity and aligning AI with human values.
Visit our video archive to explore previous SRI weekly seminars.
About the author
Gillian K. Hadfield is the director of the Schwartz Reisman Institute for Technology and Society. She is the Schwartz Reisman Chair in Technology and Society, professor of law and of strategic management at the University of Toronto, a faculty affiliate at the Vector Institute for Artificial Intelligence, and a senior policy advisor at OpenAI. Her current research is focused on innovative design for legal and regulatory systems for AI and other complex global technologies; computational models of human normative systems; and working with machine learning researchers to build ML systems that understand and respond to human norms. Her book, Rules for a Flat World, is now available on Audible.
