The failures are in the plumbing
MedAgentBench, built at Stanford and published in NEJM AI, gave language models 300 clinician-authored tasks inside a FHIR-compliant virtual EHR of 100 synthetic patients and more than 700,000 data elements. The strongest performer, Claude 3.5 Sonnet v2, cleared 69.67% overall. The split underneath that figure is where the engineering lives: 85.33% on tasks that retrieve information and 54.00% on tasks that write something back.
The authors' error analysis explains the gap. Models emitted invalid actions, wrapped output in code fences the instructions had excluded, and returned a full sentence where a numeric value was expected. Gemini 2.0 Flash produced invalid actions in 54% of cases. These are integration faults, sitting between the model and the record. The environment was synthetic, so the rates carry that caveat, and the failure modes are ones every production team recognizes.
Sierra's τ-bench makes the point from another angle. It introduced a pass^k score, which credits an agent only when it solves the same task on every one of k attempts. Function-calling agents built on GPT-4o succeeded on under 50% of tasks at a single attempt and fell below 25% at pass^8 in the retail domain. Consistency is a property you build around a model. Six mechanisms carry it: explicit orchestration, a replaceable model, human checkpoints, escalation paths, audit trails, and evaluation that keeps running after launch.
Explicit paths make actions countable
Anthropic's engineering guidance draws a line between workflows, defined there as systems where models and tools are orchestrated through predefined code paths, and agents, where the model directs its own process and tool use. The advice is to find the simplest structure that works and add autonomy where it demonstrably improves the outcome.
In regulated work that line has a governance meaning. When the path lives in code as a state machine or a workflow graph, the set of actions the system can take is finite and printable. A reviewer reads the list, marks the transitions that carry risk, and puts controls on those. Where the model picks its own path, the action set is whatever the tools permit, and the review question gets much harder to answer.
Architecture shows up in the numbers before the model does. A Mount Sinai team led by Eyal Klang published a comparison in npj Health Systems in March 2026, running single-agent and orchestrated multi-agent designs against simulated clinical traffic of up to 80 simultaneous tasks. At the heaviest load the single agent fell to around 16% accuracy, while the orchestrated design held higher accuracy on up to 65 times less compute. This was a controlled simulation under synthetic load, and it stops short of saying what either would do in a live department.
Ambient documentation shows the shape at its simplest: audio becomes a draft, the draft goes to the clinician, the clinician edits and signs, the signed note enters the record. Four states, one of them controlled by a person, and nothing reaches the chart without passing through it.
The model is a replaceable part
Model versions are deprecated, repriced, and retrained on schedules the buyer does not set. An architecture that treats the model as a fixed asset inherits all of those schedules. One that treats it as a component behind an interface keeps its guarantees while the component underneath changes.
In practice the prompt, the tool definitions, the output schema, the validation that enforces it, and the evaluation set all live in code you control. Swapping a model becomes a regression run and a diff of scores.
The cost of the other arrangement is on the public record. Michigan Medicine's external validation of a widely deployed proprietary sepsis model, published in JAMA Internal Medicine in 2021, found an area under the curve of 0.63 against the 0.76 to 0.83 the vendor had cited, missed 67% of sepsis cases, and generated alerts on 18% of all hospitalized patients. The health system could measure the model. Changing it belonged to someone else.
Where a human checkpoint earns its cost
Every checkpoint spends a person's attention, so a checkpoint on every step is the same as none at all. Three questions rank them. What does a wrong action cost, can it be reversed, and how often does the step run? A step that is expensive to undo and fires rarely earns a review gate. A step that is cheap to undo and fires ten thousand times a day earns sampling and a log.
RECTIFIER, a trial-screening system from Mass General Brigham, places the gate at the consequential step. In a randomized study published in JAMA in February 2025, 4,476 patients were assigned to manual or AI-assisted prescreening for a heart failure trial. The AI arm surfaced 458 eligible patients against 284, and study staff reviewed the chart of everyone the tool flagged before enrollment. Enrollment came out at 35 against 19.
Regulation sets a floor in places. FDA's 2022 final guidance on clinical decision support software turns on a statutory criterion that the software enable the health care professional to independently review the basis for its recommendations. Article 14 of the EU AI Act requires high-risk systems to be designed so a person can disregard, override or reverse the output and interrupt the system through a stop procedure.
The checkpoint is real work and belongs in the measurement. A Stanford study in JAMA Network Open in 2024 gave 162 clinicians AI-drafted replies to patient messages; drafts were available for 9,621 of 12,844 answered messages and used at a mean rate of 20%. Reply time held steady. Task load fell from 61.31 to 47.26 and work exhaustion fell with it, both statistically significant.
Behavior when the input goes strange
Three conditions need a written response before launch. Model confidence is low; a tool fails or times out; the input looks unlike anything in the test distribution. Each deserves a named path — hold and queue for a person, retry with backoff and then degrade to the manual route, or refuse and escalate with the reason attached.
The third is the one teams skip, and it is the one that fires in a crisis. In April 2020, three to four weeks after its first COVID-19 hospitalization, the University of Michigan Hospital deactivated its sepsis model following nursing reports of overalerting. A study across 24 hospitals in four health systems, published in JAMA Network Open, measured the shift: total alerts per day rose 43%, from 953 to 1,363, while the census was lower, and the share of patients generating an alert moved from 9% to 21%.
The population changed and the model kept doing what it had been trained to do. That stop was a human judgment executed by hand under pressure. A system designed for the condition has the stop built in, with a documented threshold, a named owner, and a fallback that keeps the work moving while the model is off.
The record makes review possible
HIPAA's Security Rule states audit controls at 45 CFR §164.312(b) as a required standard with no addressable alternative: implement mechanisms that record and examine activity in systems that contain or use electronic protected health information. Recording and examining are two obligations, and a log nobody reads satisfies half of one. Related documentation is retained for six years under §164.316(b)(2)(i).
For an agent the useful unit is one decision. Which model version ran, under which prompt, which tools were called with which arguments, what came back, which policy applied, which person approved, and when. That is the record a post-incident review reads. Without it, the answer to why the system did something in March is a reconstruction.
Behavior also moves when the model changes and when the data changes. Finlayson and colleagues, writing in NEJM in 2021, sorted the causes of dataset shift into changes in technology, in population and setting, and in behavior, noting that a renamed field in an EHR dropdown can quietly break a variable a model depends on. Duke Health's Algorithm-Based Clinical Decision Support Oversight program, described in JAMIA, registers models, stratifies them by risk, reviews them independently, and monitors them across their lifecycle.
Standing evaluation looks like ordinary quality work. During The Permanente Medical Group's ambient documentation rollout, a team of physicians, informaticists, safety officers and evaluators examined a ten-week pilot covering 63,000 encounters, reading more than 3,600 clinician comments and 1,252 documentation-quality questionnaires. The average score was 4.35 out of 5, with 47% of encounters rated five stars and 7% rated one or two, and thoroughness the weakest dimension. The full deployment reached over 2.5 million uses and an estimated 15,791 hours saved in its first year.
What to demand in a demo
Ask for the state diagram and the list of actions the system can take. Ask which transitions require a person and why those. Ask what happens when a third-party API returns a 500, and watch it happen. Ask to see the audit record for one decision end to end, then how long it is kept and who reads it. Ask for the evaluation set and the last four runs, what changed between them, and which scores moved.
Then size the claims against how early this is. Loma Linda University Health runs a chart scan every four hours inside Epic to find emergency department patients who could move to a campus with open beds, reporting a 21% rise in transfers and a 15% improvement in bed utilization since go-live. Those figures reached the trade press through Epic and the customers Epic chose to feature, none independently verified, and Loma Linda was one of four health systems in the early-access group.
Some of this stays hard, and a vendor who says otherwise is worth less attention. Errors compound across steps: ten sequential steps at 95% each clear under 60% end to end, which is the case for short enumerable paths and early checkpoints. Open-ended output resists automatic scoring, so grading falls to rubrics and sampled human review, which costs money and calendar. Latency and cost per case scale with the number of model calls, and an architecture that is comfortable across 200 cases can be disqualifying across 200,000.
The discipline underneath is the one that governs any honest measurement. Run the system against your own data, at your own volume, on criteria written down before the run. A result from that is something an executive can defend and an engineer can reproduce. A result from a curated demo has told you about the demo.
How we build these
This is the work: an AI orchestration engine placed inside software a company already runs. The engine is the layer described above — the explicit paths, the checkpoints, the fallbacks, the record — and we sell it as engineering because every property a buyer actually wants lives there and none of it lives in the model.
Two commitments follow from that. The model is a component with a version number, so the guarantees sit outside it and survive the day it is swapped. And the exception path gets designed in the same week as the success path, because a system’s behaviour on the inputs it was built for is the easy half of the problem.
We would put any vendor through the same demonstration we expect to pass ourselves: the list of actions the system can take, the transitions that require a person and the reasoning for those, and a failure induced on purpose while you watch. A team that can show all three has already thought about the day it goes wrong, which is the day that decides whether anyone keeps using it.
