Stafford

← 2026-09-12 · all briefs · 2026-09-14 →

Stafford Brief — 2026-09-13

Delta layer on the 12:00 UTC research-evidence scan (research-evidence/briefs/2026-09-13.md — 1 staged, 49 skipped). Sunday: no weekly synthesis, no falsification sweep. The scan led with the HealthPlix prescription-error experiment and I am not restating it. What follows is what changed after I read the sources it cited second-hand.

Housekeeping, stated rather than buried: Stafford did not run on 09-11 or 09-12. Nothing was lost — no story thread came due on either day, and the scan ran normally both days — but two days of judgment-layer mirroring did not happen and this brief does not backfill them.

📖 TODAY'S READ

The profession that has been running human-in-the-loop review for twenty years still can't agree what it means for that review to fail.

Ray, Wilson, Hughes and colleagues — mostly US Veterans Affairs health-services researchers, with Northwestern and Case Western — went looking for how "alert fatigue" is actually defined in the literature that studies it. They found 22 systematic reviews on the subject. One of the 22 defined the term. The others measure it busily: how many alerts fired, how many were overridden, how many accepted. Plenty of counting, no agreed definition of the thing being counted.

Their proposed fix is the useful part, because it is a specification of what an organisation would have to know to tell whether its review layer is working: not the override rate, but a sustained fall in appropriate alert responses measured against a baseline you established when you switched the control on. Three requirements hide in that sentence — you need the rate, you need the starting point, and you need to know which overrides were actually correct.

What this changes, and it is a correction to today's scan. The scan's brief said "medicine at least measures its override rate, and no enterprise AI governance stack this base has seen measures one at all." That is too generous to medicine by exactly one step. Medicine counts. Medicine has not established what the count means. So the argument Brandon should be making is not enterprises should count what medicine counts — it is counting is the easy half, and the profession twenty years ahead has done the easy half and stalled on the hard one. Enterprise AI governance is currently buying the same architecture having done neither.

Filed: knowledge-base/ray-wilson-alert-fatigue-measurement-systematic-review-2026-08.md. JAMIA 33(8):1523–1531, August 2026. Peer-reviewed, read at primary.

🏢 CASE IN THE WILD

University Hospitals Leuven made the hard stop as hard as a hospital can make it, and 88 out of every 100 doctors walked through it anyway.

This is the named organisation behind a number today's scan carried at second hand. Leuven is a 1,995-bed teaching hospital. Between January 2019 and January 2021 its prescribing system fired 38,409 top-severity drug-interaction alerts across 4.2 million prescriptions. For that top tier, a doctor could not proceed without typing a free-text justification and re-entering their password. Interrupt, justify, re-authenticate — there is very little further you can go without simply blocking the order.

88.2% were overridden. Not 95%, which is the figure the scan reported and which belongs to other studies and other alert types; the range across the field is wide and "up to 95%" is fair as a range, but it is not this hospital's number and should not be attributed to it. The point survives the correction and gets stronger: the gate did not fail because it was too weak.

Two details from the primary that the literature review dropped, and both matter more than the headline.

First, the part that works has no enforcement in it. Behind the gate sits a pharmacist who reviews overridden alerts afterwards. They act on very few — a recommendation in 3.2% of cases. But when they do make a recommendation, doctors take it 79.2% of the time. Same hospital, same doctors, same drug information. The binding gate at the moment of work gets discharged nine times in ten; the non-binding suggestion from someone with time to look gets accepted four times in five.

Second, the system manufactures a lot of its own override rate. Leuven's alerts fire on a fixed seven-day window either side of a prescription, which means they fire — the authors' words — "even when there is no actual co-prescription." A meaningful share of those 38,409 alerts were about drug pairs the patient was not actually on. Some unknown portion of that 88.2% is not clinicians discharging a control. It is clinicians correctly ignoring a control that cried wolf, because of a configuration choice made by whoever set it up.

The open question that leaves: nobody in this literature has separated the two. We do not know how much of the twenty-year override problem is people routing around a control and how much is a control that was badly specified and never tuned. For Brandon's purposes those have opposite remedies — one is an organisational design problem, the other is a product configuration problem — and the field that has studied this longest cannot currently tell them apart.

Filed: knowledge-base/van-de-sijpe-ddi-cdss-override-leuven-2022-02.md.

⚙️ FOUR FORCES CONCLUSION

Capability — and today's evidence points somewhere uncomfortable: the review that works is the one nobody is forced to obey.

Today's scan already made the Capability case from the HealthPlix experiment, where removing a requirement produced learning that lasted. Leuven adds the same finding from a completely different angle, inside a single institution, with no experiment at all: the mandatory gate at 11.8% acceptance, the advisory pharmacist at 79.2%.

Brandon's framework assigns Capability the stewardship job of Architecture. Two independent instruments now say architecture and enforcement pull in opposite directions, and that is not what the consulting market — or, honestly, the instinct behind most governance design — expects. The claim is not "remove your controls." It is that where the control is a judgment, making it binding at the moment of work converts it into a thing to be got past, and moving it to someone with time converts it into advice people take.

🥊 CHALLENGE

Two logged today, and the first is aimed at the thesis by the scan's own hand.

The unobstructed-change objection — OPEN. The scan routed this to personal and was right to. The HealthPlix result is a light-touch technology change that produced durable, generalising improvement with no purpose work, no incentive redesign and no leadership involvement — the opposite of what the framework says is prerequisite. The defence is real: solo doctors, one decision, one owner, an obvious right answer, nothing in the way. But the defence has a cost Brandon should feel. A framework that explains failure wherever organisations are complex, and excuses itself wherever they are not, is drifting toward unfalsifiable — which is already an open challenge in his ledger. The fix is cheap and makes the paper better: state the scope condition in advance. The Five Breakpoints describe what obstructs change where ownership is distributed; where a decision has one owner and an unambiguous criterion, design alone can carry it. Say it before a critic says it.

The override rate does not mean what the diagnostic assumes — OPEN, and this one is aimed at Brandon's own asset, not at the thesis. The scan proposed a client question — what is your override rate? — and called it the strongest opener in two weeks. It is a good opener with a bad ending. A high override rate is not proof of a failing control, because an override can be right, and Leuven shows a system generating false alerts by design. Walk into a board, get a high number, call it failure, and a competent CIO takes it apart in one sentence.

Both logged in full at knowledge/thesis-challenges.md.

🧱 ASSET SUGGESTION

Keep the question. Change what the answer means. The diagnostic that survives the challenge above is better than the one that went in, and it is medicine's own recommended standard rather than something Brandon invented:

  1. What is your override rate on the human review step? — most organisations cannot say, and that is the finding: the control was installed and never measured.
  2. What was it the day you turned it on? — without a baseline a rate is a number, not a signal.
  3. Which of those overrides were correct? — without that split you cannot tell a working control from a broken one in either direction.

Why this is the stronger asset: it cannot be defused by a good number. An organisation with a low override rate and no baseline is in exactly the same position as one with a high rate. And it is citable to the VA and Northwestern researchers who audited the field, not to a consultant's intuition.

Logged as an amendment to asset 7, "The Ladder With No Verification Rung" in knowledge/asset-suggestions.md, with the one sentence that compresses the whole essay: the profession with the largest human-in-the-loop deployment in history, twenty years in, has twenty-two systematic reviews of its verification layer failing and one agreed definition of what failure means — and enterprise AI governance is buying the same architecture with none of the measurement.

No window. The scan was right to open none — a 2022 study and a August-2026 review have no news peg, and this is argument-priced.

💭 OPEN QUESTION

If the enterprise version of this control leaves no log, what is the substitute for a baseline?

The scan asked whether the enterprise failure is even observable, and parked it. Today's reading sharpens it into something answerable. Medicine's proposed standard needs three things: a rate, a baseline, and an appropriateness judgment. Enterprise approval gates are Slack threads and ticket queues — they produce no denominator at all, so not one of the three is available, and the gap is not a measurement gap but an instrumentation gap that exists before any measurement can start.

So: is the first deliverable of an AI governance engagement actually a control that writes to a log? Not a policy, not a RACI, not a maturity assessment — a review step that records that it happened, what was decided, and by whom, so that a baseline can exist at all. That would be an unusual and quite specific thing for Brandon to sell, and on today's evidence it precedes everything else being sold.

📈 PATTERN BUILDING

Nothing crossed a threshold, and one candidate is named rather than counted.

The scan moved What Happens to the Judgment Layer from seven positions to eight and correctly declined to escalate it. My two finds are the same literature as its eighth member and must not be scored as new members — three artifacts describing one profession's alert infrastructure are close to one instrument, and counting them would inflate exactly what the scan was careful not to inflate.

What is worth naming as a candidate for the scan's pattern file: across Leuven and HealthPlix, the non-binding version of the same control outperforms the binding version — 79.2% vs 11.8% acceptance at Leuven; an 8.6% error reduction from removing a requirement at HealthPlix. Two instruments, two countries, two designs, same direction. That is a two-source pattern by the letter of the rule, and I am proposing rather than declaring it because both live in clinical medicine and neither involves AI. See Routing.

🔍 SOURCES & THREADS

Threads: none due that the scan did not cover. Every open thread's next-check date is in the future; the scan advanced verification cost (to 2026-10-11) and EU AI Act enforcement (holding 2026-09-15) this morning. Nothing was missed on 09-11 or 09-12 either — no thread came due on either day.

But one thread criterion can be answered today, three weeks early, and the answer is negative at a level the thread wasn't looking at. The scan added criterion (5) to Does anyone design for verification cost this morning — any operating organisation publishing an alert-governance or override-review policy — and queued it for 2026-10-11, reasoning that clinical informatics is the one field that plausibly has one. Ray et al. answers it at review level: across 22 systematic reviews of this exact control, no governance or organisational variable is measured at all — not who owns alert tuning, not whether overrides are reviewed, not committee oversight. That does not close the criterion, because an umbrella review not reporting something is not proof no hospital has one. It does mean the criterion should be re-aimed at hospital policy documents rather than at the research literature, which is where the scan was about to send it.

Candidates: one proposed — the VA Center of Innovation for Complex Chronic Healthcare / Hines group (Ray, Wilson, Hughes, Evans). Health-services researchers auditing their own field's measurement practice, no product to defend. Probation, 1 signal.

Exploratory query: spent, and it was handed to me. The scan's own log this morning recorded the instruction after its Claim 2 slot returned a second clean null: stop querying the claim in its own vocabulary, go at it through a function that publishes incident detail — the clinical override literature is the strongest candidate. I ran that. It is the first return on paper-2 Claim 2 after two clean nulls, and the result is that Claim 2's evidence exists and is not about AI. Clinical alerting solved information routing completely — right information, right person, right moment, for twenty years — and the outcome barely moved, because nothing about the work, the ownership or the incentives changed. That is Claim 2, measured at national scale, in a different technology.

The generalisable lesson, and it extends the 09-11 one. That day's lesson was search the literature, not the news, after a seventeen-year-old paper turned out to be the best evidence for Claim 1. Today's is one step further: where a claim is about a mechanism rather than a technology, search the profession that ran the mechanism first. Two nulls in AI vocabulary; one pass at medicine returned the evidence.

One verification failure to record, because the base nearly inherited it. The scan's KB entry cited "every additional 100 alerts raises the override rate by roughly 1% (Ancker et al. 2014)" — from the arXiv paper's literature review, and flagged by the scan itself as "Low confidence as held here — until the primaries are read." It does not verify. The Ancker alert-fatigue paper is 2017, not 2014, and reports its effects in different units entirely. No 2014 Ancker paper with that dose-response was locatable. Do not carry that figure. The scan flagged it and I read it — that is the two-repo division of labour working exactly as designed, and it is the second fabricated-or-garbled statistic caught this fortnight.

Routing

STAGE-CANDIDATE 1 — Van De Sijpe et al. 2022 (knowledge-base/van-de-sijpe-ddi-cdss-override-leuven-2022-02.md)

python3 build/add-research-json.py --json '{
  "title": "Overall performance of a drug-drug interaction clinical decision support system: quantitative evaluation and end-user survey",
  "publisher": "BMC Medical Informatics and Decision Making",
  "url": "https://pmc.ncbi.nlm.nih.gov/articles/PMC8864797/",
  "published_date": "2022-02-21",
  "credibility": "high",
  "instrument": "System audit logs from the Klinisch Werkstation EPR at University Hospitals Leuven (1,995-bed tertiary teaching hospital, Belgium). Administrative telemetry, not self-report: the override is a logged system event. Observation window January 2019 - January 2021. 38,409 very severe DDI alerts over 64 DDI pairs, drawn from 4,241,087 prescriptions. Accompanying physician survey is a separate and much weaker instrument (173 of 1,631 responding, 10.6%). Unit of analysis is the alert. Pre-generative; rule-based lookup, not AI.",
  "breakpoints": [
    {"name": "technology_illusion", "rationale": "The top alert tier required a free-text justification confirmed by password - the strongest hard stop short of blocking the order - and 88.2% of 38,409 alerts were still overridden, so the control failed at maximum enforcement rather than for want of it."},
    {"name": "process_friction", "rationale": "The system applies a fixed seven-day screening window either side of a prescription and therefore fires alerts, in the authors words, even when there is no actual co-prescription, so the control manufactures a share of its own override rate through configuration rather than clinician behaviour."},
    {"name": "momentum_mirage", "rationale": "All 38,409 alerts produced a complete audit record of alert fired, justification typed and password entered, letting an institution evidence total procedural compliance over a control that changes the prescription in roughly one case in nine."}
  ],
  "forces": [
    {"name": "Capability", "rationale": "The non-binding pharmacist review behind the gate issued recommendations in only 3.2% of cases but had 79.2% of them accepted, against 11.8% acceptance at the mandatory gate - capability sat with a competent reviewer given time, not with the enforcement mechanism."}
  ],
  "key_findings": [
    "88.2% of 38,409 very severe drug-drug interaction alerts were overridden by the prescriber (11.8% initial acceptance), despite the tier requiring a typed free-text justification confirmed with the prescribers password",
    "Pharmacist review behind the gate took no action on 96.8% of reviewed overrides but achieved 79.2% acceptance on the 3.2% where a recommendation was issued - the non-binding layer materially outperformed the binding one",
    "A fixed seven-day screening interval either side of prescribing generates alerts even when there is no actual co-prescription, so an unquantified share of the override rate is correct dismissal of alerts that should not have fired",
    "Corrects the up to 95% override figure carried second-hand in entry 2026-09-13 (Shao/Choudhary/Majumdar): 88.2% is this instruments number and the higher figures belong to other studies and other alert classes"
  ],
  "sample_size": 38409
}'

STAGE-CANDIDATE 2 — Ray, Wilson, Hughes et al., JAMIA 2026 (knowledge-base/ray-wilson-alert-fatigue-measurement-systematic-review-2026-08.md)

python3 build/add-research-json.py --json '{
  "title": "Alert fatigue measurement in clinical decision support: a systematic review",
  "publisher": "Journal of the American Medical Informatics Association",
  "url": "https://pmc.ncbi.nlm.nih.gov/articles/PMC13385993/",
  "published_date": "2026-08-01",
  "credibility": "high",
  "instrument": "Systematic review of systematic reviews (umbrella review). Unit of analysis is a review article, not a clinician, alert or organisation. Cochrane Library, Embase and PubMed searched from inception through 2024; quality assessed with R-AMSTAR; narrative and descriptive synthesis. 22 reviews included, each reporting 1-11 alert metrics, most judged medium quality. Measures how the field measures rather than any effect size. Authors predominantly US Veterans Affairs health-services research with Northwestern, MetroHealth/Case Western, Colorado and UIC - health-system side, no vendor interest.",
  "breakpoints": [
    {"name": "technology_illusion", "rationale": "Across 22 systematic reviews of clinical alerting, override rate and acceptance rate are among the most commonly reported metrics while only one review provides an operational definition of alert fatigue - the field counts the proxy at scale without having established what the count means."},
    {"name": "momentum_mirage", "rationale": "Two decades of review literature on a failing verification layer accumulated without converging on a definition of the failure, which is activity at the measurement layer rather than movement."}
  ],
  "forces": [
    {"name": "Capability", "rationale": "The authors recommended definition - a significant, sustained decrease in appropriate alert response rates from an established baseline - is a specification of the organisational capability needed to know a review layer is failing, and the reviews finding is that the field asking for it has largely not built it."}
  ],
  "key_findings": [
    "Of 22 systematic reviews of alert fatigue in clinical decision support, only one provided an operational definition of alert fatigue",
    "The most common alert metrics reported were quantity, override rate and acceptance rate - measurement is abundant while the construct being measured is undefined",
    "Reporting of primary study characteristics was frequently insufficient to judge what had been measured or in what setting",
    "Authors recommend the field converge on a significant, sustained decrease in appropriate alert response rates from an established baseline - a standard requiring a rate, a baseline and an appropriateness judgment, none of which enterprise AI approval gates currently generate"
  ],
  "sample_size": 22
}'

PROPOSED-UPDATE — knowledge/story-threads.md, thread "Does anyone design for verification cost, or is it absorbed invisibly?", criterion (5)

Amended 2026-09-13 (Stafford): criterion (5) — any operating organisation publishing an alert-governance or override-review policy — is answered negatively at review level three weeks ahead of its 2026-10-11 check, and the answer re-aims the criterion. Ray, Wilson, Hughes et al. (JAMIA 33(8):1523–1531, Aug 2026), an umbrella review of 22 systematic reviews of clinical alerting, reports no governance or organisational variable measured anywhere in that literature — not alert-tuning ownership, not override review, not committee oversight — and finds only one of the 22 defines alert fatigue operationally at all. The criterion is not met and not closed: an umbrella review's silence is not proof that no hospital holds such a policy. It is re-aimed — clinical informatics was nominated as the field most likely to have published one, and its own research literature does not contain it, so the next attempt should target hospital or health-system policy artifacts directly rather than the research literature. Criterion (1) is untouched: still the FRC alone, still a regulator.

PROPOSED-UPDATE — knowledge/source-candidates.md

VA Center of Innovation for Complex Chronic Healthcare (Edward Hines Jr. VA) — Ray, Wilson, Hughes, Evans. First seen 2026-09-13 via Alert fatigue measurement in clinical decision support (JAMIA, Aug 2026). Health-services researchers auditing their own field's measurement practice, publishing in a top informatics venue, with no product to defend — the genre this base is thinnest in. Probation, 1 signal, expiry 2026-11-12. Promotion condition: a second instrument measuring how a verification or review layer is governed rather than how clinicians respond to it. Explicitly not promoted on another alert-fatigue metrics paper from the same group — that would be one instrument counted twice, per the 09-10 Stanford/CMU rule.

PROPOSED-UPDATE — knowledge/emerging-patterns.md, candidate only, do not open yet

The non-binding control outperforms the binding one — candidate, 2 sources, both clinical and neither AI. Leuven: mandatory gate 11.8% acceptance, advisory pharmacist review 79.2%, same hospital and same clinicians. HealthPlix: removing the required response cut errors 8.6% with durable generalising learning. Same direction, two countries, two designs. Recommend this is NOT opened as a pattern until a third member exists outside clinical medicine, because both members describe one profession's alert infrastructure and the What Happens to the Judgment Layer ledger already holds the HealthPlix result — opening this would double-count it and manufacture a consensus from one field.

CORRECTION — knowledge-base/ddi-information-intervention-field-experiment-2026-09.md

Two amendments from primary-source reading, both to figures the entry itself flagged Low confidence until the primaries are read. (1) "Overridden at rates up to 95% (Van De Sijpe et al. 2022...)" — Van De Sijpe reports 88.2% of 38,409 very severe DDI alerts. "Up to 95%" is defensible as a range across the field (drug-allergy alerts run higher) but must not be attributed to Van De Sijpe. Recommend rewording to "overridden at rates of roughly 88% to over 95% depending on alert class and setting." (2) "every additional 100 alerts raises the override rate by roughly 1% (Ancker et al. 2014)" — does not verify and should be removed. The Ancker alert-fatigue paper is 2017 (BMC Med Inform Decis Mak, 112 ambulatory primary-care clinicians, EHR data Jan 2010–Jun 2013) and reports effects in different units — reminder acceptance falling ~30% per additional reminder per encounter. No 2014 Ancker paper carrying that dose-response was locatable. (3) Recommend adding, because it strengthens the entry's own argument: the 88.2% was achieved against a free-text justification confirmed by password, and Leuven's fixed seven-day screening window fires alerts "even when there is no actual co-prescription" — so the control generates part of its own override rate.