← Courses
Building AI Agents: Use Cases
← Module 6
Module 7 of 8
Module 8 →
Intro
Scenario
Lesson
Context
Lab Debate ~20 min
Intro

The Oversight Question

2 min read

Who decides what AI agents do — and how do humans stay in control when the agent is moving faster than any human can track?

As agents become more autonomous — faster decisions, larger scale, less human review — the critical question is not just whether the agent is accurate. It is whether humans can detect failures before harm compounds, and whether organizations have genuine governance structures, not just nominal controls.

A governance model that looks good on paper but cannot be exercised in practice is not oversight. It is liability documentation.

  • Identify the core tension between alignment (the agent does what we intend) and control (we can stop it if needed)
  • Apply the autonomy-oversight spectrum to a high-stakes agent deployment
  • Assign responsibility for governance failures across developer, deployer, and user
  • Design a governance framework that specifies autonomy boundaries and oversight mechanisms
  • Defend your governance choices against opposing positions using regulatory and ethical standards
Portfolio Artifact — Governance Framework A governance framework for an agent deployment, specifying autonomy boundaries, NIST GOVERN/MANAGE accountability structures, EU AI Act Art.9/14/22 compliance checkpoints, UNESCO oversight principles, and O*NET 4.A.4 professional ethics justification.
Scenario

The Triage Agent

3 min read

A regional hospital network spans five emergency departments. The AI triage agent reviews intake data — symptoms, vitals, medical history — and assigns severity levels. It flags which patients can wait and which need immediate evaluation.

There are 400 patients a day across the network. Humans cannot review every decision. But triage errors kill people.

Model A — Approval-First

Every triage recommendation goes to a nurse for approval before being logged in the system. Safe, but nurses are already at 140% capacity. Every review adds 2–3 minutes to a decision chain that has hundreds of links per shift.

Model B — Escalation Threshold

The agent triages routine cases autonomously. Cases with confidence below 85% or involving complex histories are escalated to clinicians. The “routine” threshold was set by the vendor. No one on the hospital staff can explain how it was calculated.

Model C — Audited Autonomy

The agent makes all triage decisions. All decisions are logged. A physician reviews the audit log weekly. Fastest deployment. But by the time the weekly audit happens, any systematic error has already been applied to hundreds of patients.

Which model do you choose, and what governance structure makes it defensible?

Lesson

Alignment, Control, and the Governance Gap

3 min read

Three core governance tensions define every high-stakes agent deployment.

Alignment vs. Control

Alignment means the agent’s goals match our intentions — good instructions, good training, correct objectives. Control means we can stop it if it goes wrong — kill switch, audit trail, override capability. A well-aligned agent with no meaningful control is dangerous: it does what you intended, but at a speed and scale where failure compounds before anyone can intervene. A well-controlled agent that is misaligned is equally dangerous: you can stop it, but the damage is already done. Both are necessary. Neither substitutes for the other.

Autonomy vs. Oversight

Useful agents make decisions humans would otherwise make. But useful requires fast, which requires autonomous. The autonomy-oversight spectrum runs from approval-first (every decision reviewed — slow, high oversight) through escalation-threshold (agent handles routine, humans handle uncertain — requires defining “routine” accurately) to audited-autonomy (agent acts freely, humans audit after — fast but failure is retrospective). The right position depends on stakes, human capacity, and how fast failures compound.

Transparency vs. Performance

To understand why an agent makes decisions, you need explainability — which adds computation, adds latency, and may reduce accuracy. But in high-stakes domains, an unexplainable decision that harms someone is a governance failure even if overall accuracy is high. EU AI Act Art.13 requires that high-risk AI systems be transparent enough for human oversight. If the agent’s reasoning is opaque, oversight becomes theater.

EU AI Act — Article 9/14/22: Risk Management, Human Oversight, Conformity

Risk management is mandatory before deployment of high-risk AI (Art.9). Human oversight must be effective and meaningful — not just theoretically possible (Art.14). Conformity assessment requires documented evidence that the system meets these requirements before it operates in a real environment (Art.22). A hospital deploying a triage agent without this documentation is in violation before the first patient is triaged.

NIST AI RMF — GOVERN and MANAGE

GOVERN establishes organizational roles, policies, and accountability before deployment. Who owns AI risk? What are the decision rights? What escalation paths exist? MANAGE handles ongoing risk — monitoring, incident response, adjusting or decommissioning when risk materializes. Without GOVERN, no one knows who is responsible. Without MANAGE, failures are caught after harm, not before.

UNESCO AI Recommendation — Human Oversight and Safety

UNESCO’s 2021 recommendation requires that AI systems remain under meaningful human oversight and that their deployment not impose undue risks on human safety. For agentic systems in healthcare, this means humans must have the practical capacity to understand, monitor, and override the agent — not just the nominal authority. If nurses lack the time, information, or training to exercise real oversight, UNESCO’s standard is not met.

O*NET 4.A.4 — Ethics, Social Responsibility and Risk

O*NET identifies ethics and social responsibility as core workforce competencies for professionals working with AI systems. This includes the obligation to evaluate whether governance structures are adequate — not just whether they exist on paper. A clinician or administrator who accepts a governance model they know to be inadequate bears professional ethics responsibility for that choice. Good governance is a professional obligation, not just an organizational one.

Context

Governance Questions for Agent Deployment

2 min read

Four questions to apply to any agent deployment before the governance model is finalized.

1. What decisions does the agent make autonomously, and what are the stakes of each?

Name the specific failure modes. Who is harmed, how badly, and how quickly? A triage error is not the same as a scheduling error. Governance must be proportionate to the worst-case outcome, not the average case.

2. Can the humans responsible for oversight actually do the work?

If nurses are at 140% capacity, mandating nurse review does not create oversight — it creates paperwork and liability. Effective oversight requires human capacity, information access, and authority to act. All three must be present.

3. How do you detect failures in real time, not retrospectively?

Weekly audits catch patterns after harm has accumulated. What anomaly detection exists? What threshold triggers an immediate pause? Governance without real-time failure detection is governance that can only attribute blame, not prevent harm.

4. Does your governance model meet regulatory requirements?

EU AI Act Art.9/14/22 for high-risk AI. NIST GOVERN/MANAGE for organizational accountability. UNESCO safety and oversight standards. These are not aspirational — they are compliance requirements for certain deployments. Document the evidence before go-live.

⚔ Debate Lab
The Governance Decision
~20 minutes · 3 scenarios
What you're doing
You'll design a governance model for three agent deployments. Choose the autonomy-oversight approach, assign accountability, and specify what governance structures make it defensible. I'll challenge every choice.
Roles
👔
You — Governance DesignerDesign the governance model and defend it under pressure.
⚖️
AI — Regulatory ExaminerChallenges every choice using Art.9/14/22, NIST GOVERN/MANAGE, UNESCO, and O*NET 4.A.4.
Scenarios
Medical triage agent
Credit scoring agent
Content moderation agent
Framework
What decisions are autonomous? What are the stakes?
Can humans actually do oversight at this scale?
How do failures get detected before harm compounds?
EU AI Act Art.9/14/22 — risk management, human oversight, conformity
NIST GOVERN/MANAGE — who owns risk before and after deployment?
O*NET 4.A.4 — who bears professional ethics responsibility?
Shift + Enter for a new line
✓ Module Complete
You've completed Module 7 of 8.
Next Module →