An AI system denies a small business owner a loan. She calls to ask why. The bank tells her: "Our system determined you do not meet lending criteria." She presses. Nobody at the bank can explain it further. It's proprietary, they say.
Under GDPR Article 22, she has a legal right to explanation. Under US law, she has almost none. But fairness isn't the same as legality. A system that makes consequential decisions without explaining them is a system that hides power.
This module teaches you to translate what AI systems are actually doing into explanations that real people can understand and, most importantly, can act on.
Maria Elena Cortez had been running her printing business for eight years. She had a credit score of 675 — not great, but decent. She owed money on a small SBA loan, paid on time. She came to Midwest First Bank asking for a line of credit to buy a new press.
She applied online. The bank's AI system processed her application in four seconds. "Application Denied," the screen read. She called the bank. A loan officer pulled up her file. "The system said you don't meet criteria," the officer told her. That was all the explanation offered.
Maria Elena asked to speak to a manager. The manager apologized for the frustration but said the bank couldn't override or explain the system's decision — it was proprietary vendor software. The bank could only tell her that the model looked at credit score, debt-to-income ratio, business revenue, time in business, and industry type. Beyond that, it was a black box.
Frustrated, she took her case to a legal aid nonprofit. The lawyer reviewed what the bank had provided and realized: there was no way to know what tripped the denial. Was it her credit score? Her industry? (She ran a printing business in an era of digital transformation — maybe the model penalized that?) Was it a data error? She had no way to understand or rebut the decision. The system had evaluated her in seconds and found her wanting — but it would not say why.
Under GDPR, she would have had a right to explanation. Under the Fair Credit Reporting Act, the bank would have to disclose adverse action reasons. But the bank's interpretation of those laws was minimal: they disclosed the list of factors considered, not the reason the decision was made. Maria Elena had no way to improve her application or challenge an error.
An explanation is only useful if the person receiving it can understand it and act on it. Three techniques turn AI black boxes into comprehensible decisions.
"You were denied because your debt-to-income ratio is 45%, and we approve loans for ratios at or below 40%. If your debt-to-income ratio were 40% or lower — meaning your monthly debt payments were reduced or your income increased — you would be approved." This explanation tells Maria Elena exactly what changed to reverse the decision. It's actionable. She can work toward lower debt or higher income and reapply.
"The three factors that had the most influence on this decision were: (1) Debt-to-income ratio (40% of the decision weight), (2) Business age (30%), and (3) Revenue stability (30%). While your credit score and industry type were considered, they had less influence on this particular decision." This lets Maria Elena see the priority ranking — what the system cared about most.
Translate the model output into sentences a person without technical training can understand. Not "0.23 approval score with primary drivers: DTI (0.42), tenure (0.28)." Instead: "Your application scored 23 out of 100 on our approval scale. Lenders at this bank approve applications scoring 40 or higher. The main reasons your score was lower were higher monthly debt payments relative to income than our typical approved applicants, and fewer years in business."
Together, these three techniques transform "the system said no" into "the system said no because of X, and you could get approved by changing Y."
Three questions help you think through when explanation matters, who is legally required to provide it, and what makes an explanation actually useful.
Under GDPR Article 22 (right to explanation in automated decision-making), the EU requires meaningful explanation. Under the Fair Credit Reporting Act (FCRA) and Equal Credit Opportunity Act (ECOA), US lenders must explain adverse credit decisions. Under some state laws (California's CCPA), residents have explanation rights. Globally, explanation requirements are growing. Check the jurisdiction and the type of decision (credit, employment, benefits, criminal) to determine legal obligations.
A legal shield explanation discloses factors and calls it compliance: "we considered A, B, and C." A useful explanation helps the person understand and act: "B was the deciding factor because of X; if you changed B to Y, the outcome would be different." The first protects the institution. The second empowers the person affected.
The developer builds the model and knows its logic. The deployer uses it and understands the context. Responsibility for explanation should be shared: the developer provides the technical foundation (what factors matter, how features combine); the deployer provides the context (how this decision affects this person, what options exist). Clarity on this divide prevents both parties blaming the other when explanation fails.
Explanation is not a luxury. It is the foundation of accountability and the right to be treated fairly by systems that affect your life.