Who decides what AI agents do — and how do humans stay in control when the agent is moving faster than any human can track?
As agents become more autonomous — faster decisions, larger scale, less human review — the critical question is not just whether the agent is accurate. It is whether humans can detect failures before harm compounds, and whether organizations have genuine governance structures, not just nominal controls.
A governance model that looks good on paper but cannot be exercised in practice is not oversight. It is liability documentation.
A regional hospital network spans five emergency departments. The AI triage agent reviews intake data — symptoms, vitals, medical history — and assigns severity levels. It flags which patients can wait and which need immediate evaluation.
There are 400 patients a day across the network. Humans cannot review every decision. But triage errors kill people.
Every triage recommendation goes to a nurse for approval before being logged in the system. Safe, but nurses are already at 140% capacity. Every review adds 2–3 minutes to a decision chain that has hundreds of links per shift.
The agent triages routine cases autonomously. Cases with confidence below 85% or involving complex histories are escalated to clinicians. The “routine” threshold was set by the vendor. No one on the hospital staff can explain how it was calculated.
The agent makes all triage decisions. All decisions are logged. A physician reviews the audit log weekly. Fastest deployment. But by the time the weekly audit happens, any systematic error has already been applied to hundreds of patients.
Which model do you choose, and what governance structure makes it defensible?
Three core governance tensions define every high-stakes agent deployment.
Alignment means the agent’s goals match our intentions — good instructions, good training, correct objectives. Control means we can stop it if it goes wrong — kill switch, audit trail, override capability. A well-aligned agent with no meaningful control is dangerous: it does what you intended, but at a speed and scale where failure compounds before anyone can intervene. A well-controlled agent that is misaligned is equally dangerous: you can stop it, but the damage is already done. Both are necessary. Neither substitutes for the other.
Useful agents make decisions humans would otherwise make. But useful requires fast, which requires autonomous. The autonomy-oversight spectrum runs from approval-first (every decision reviewed — slow, high oversight) through escalation-threshold (agent handles routine, humans handle uncertain — requires defining “routine” accurately) to audited-autonomy (agent acts freely, humans audit after — fast but failure is retrospective). The right position depends on stakes, human capacity, and how fast failures compound.
To understand why an agent makes decisions, you need explainability — which adds computation, adds latency, and may reduce accuracy. But in high-stakes domains, an unexplainable decision that harms someone is a governance failure even if overall accuracy is high. EU AI Act Art.13 requires that high-risk AI systems be transparent enough for human oversight. If the agent’s reasoning is opaque, oversight becomes theater.
Risk management is mandatory before deployment of high-risk AI (Art.9). Human oversight must be effective and meaningful — not just theoretically possible (Art.14). Conformity assessment requires documented evidence that the system meets these requirements before it operates in a real environment (Art.22). A hospital deploying a triage agent without this documentation is in violation before the first patient is triaged.
GOVERN establishes organizational roles, policies, and accountability before deployment. Who owns AI risk? What are the decision rights? What escalation paths exist? MANAGE handles ongoing risk — monitoring, incident response, adjusting or decommissioning when risk materializes. Without GOVERN, no one knows who is responsible. Without MANAGE, failures are caught after harm, not before.
UNESCO’s 2021 recommendation requires that AI systems remain under meaningful human oversight and that their deployment not impose undue risks on human safety. For agentic systems in healthcare, this means humans must have the practical capacity to understand, monitor, and override the agent — not just the nominal authority. If nurses lack the time, information, or training to exercise real oversight, UNESCO’s standard is not met.
O*NET identifies ethics and social responsibility as core workforce competencies for professionals working with AI systems. This includes the obligation to evaluate whether governance structures are adequate — not just whether they exist on paper. A clinician or administrator who accepts a governance model they know to be inadequate bears professional ethics responsibility for that choice. Good governance is a professional obligation, not just an organizational one.
Four questions to apply to any agent deployment before the governance model is finalized.
Name the specific failure modes. Who is harmed, how badly, and how quickly? A triage error is not the same as a scheduling error. Governance must be proportionate to the worst-case outcome, not the average case.
If nurses are at 140% capacity, mandating nurse review does not create oversight — it creates paperwork and liability. Effective oversight requires human capacity, information access, and authority to act. All three must be present.
Weekly audits catch patterns after harm has accumulated. What anomaly detection exists? What threshold triggers an immediate pause? Governance without real-time failure detection is governance that can only attribute blame, not prevent harm.
EU AI Act Art.9/14/22 for high-risk AI. NIST GOVERN/MANAGE for organizational accountability. UNESCO safety and oversight standards. These are not aspirational — they are compliance requirements for certain deployments. Document the evidence before go-live.