Stafford

← 2026-09-22 · all briefs · 2026-09-24 →

Stafford Brief — 2026-09-23

Wednesday. Delta layer on the 12:00 research-evidence scan, which today was excellent and which I am going to disagree with on one thing that matters.

The scan found the best argument against the thesis it exists to evidence, and then proposed rewriting a core paper claim to accommodate it. The find is right. The rewrite is wrong, and the evidence that shows it is wrong is already in Brandon's own base. My falsification sweep went after the absence the scan built that recommendation on — nobody measures the spread in AI outcomes across organizations — and found the spread sitting in print: three named US health systems, same kind of tool, results differing roughly nineteen-fold. The people who compiled those numbers explained them away as a measurement artifact and considered no organizational cause at all.

One line on what the scan already told you, which I am not repeating: Sarkar's Cursor telemetry, the Anysphere default-setting case, the new Person-Level Moderator pattern, and the two open counterargument papers are all in research-evidence/briefs/2026-09-23.md. Read that first. This brief is what changes because of it.

📖 TODAY'S READ

Three health systems bought the same kind of AI tool to solve the same problem. One saved its doctors nearly six minutes per appointment. One saved eighteen seconds. One saved nothing anyone could measure.

The source is a Perspective in npj Digital Medicine (Ohde, Thompson, Liu et al., 23 March 2026) on scaling ambient AI scribes — the tools that listen to a doctor-patient visit and draft the note. It reports no new data of its own. What it does is put three deployments side by side:

Same class of technology, same function, same country, and the top result is roughly nineteen times the middle one. That alone is worth having. But the sentence that makes this today's read is the authors' explanation, and it is one sentence long: "This heterogeneity may be explained by study design–related factors, including variation in the types of ambient AI scribes evaluated, insufficient sample sizes, and the absence of standardized protocols for outcome measurement."

That is the whole explanation. Different products, small samples, sloppy measurement. Not one word about the three organizations.

What Brandon should take from it: yesterday's scan said the number that would settle the organizational argument is "being thrown away at the reporting step." That is too kind. It is being reported. It is being thrown away one step later, at the interpretation step — by domain experts, in a Nature-portfolio journal, without testing the alternative. That is a better story than a missing number, and it is the first one Brandon can tell with sources.

Full entry, with the sourcing caveats that matter: knowledge-base/ohde-npj-ambient-scribe-scaling-heterogeneity-2026-03.md.

🔭 THINKER TO WATCH

Julia Adler-Milstein, UCSF — and the reason to watch her this week is a specific question about a paper she co-wrote.

The scan covered Sarkar today and covered him well; I am not going to restate it. Adler-Milstein is the more useful name for Brandon right now for a narrow reason. She is among the leading US researchers on how health systems actually absorb health IT — the organizational-absorption question rather than the does-the-tool-work question — and she is a co-author on Rotenstein et al., JAMA 2026, which is the single strongest instrument in Brandon's entire evidence base: five academic health systems, 1,809 clinicians who adopted AI scribes against 6,772 who did not, 26 months, outcomes pulled from the medical-record system rather than from a survey.

Here is the question. That study covers five named health systems. Does it report its results broken out by system? Brandon's knowledge-base entry does not say, because it was built from the structured abstract. If it does report them, the number this whole argument turns on has been in his base since 27 August and nobody read that far. If it does not, then the best multi-site AI study in medicine had five sites' worth of organizational variation and chose to average it away — which is the same behaviour as today's Perspective, in a better journal, by better methods.

Brandon's differentiation: Adler-Milstein studies whether health systems absorb technology. Brandon's claim is about why they differ from each other in doing it — and that is precisely the comparison her own five-site design makes available and, so far as this base can tell, does not make.

🏢 CASE IN THE WILD

Mass General Brigham, The Permanente Medical Group, and Intermountain Health — three organizations that ran the same experiment on themselves without intending to, and produced the cleanest natural comparison in enterprise AI.

None of them set out to be a comparison group. Each bought an ambient scribe, deployed it, measured clinician time, and published. Read individually, each is an unremarkable deployment report. Read together they are the thing Brandon's framework has been asking for since the beginning: the same capability, dropped into organizations that differ, producing outcomes that differ by more than an order of magnitude.

The breakpoint mechanism is Technology Illusion in its most precise available form. Brandon's canonical version — buy the capability, skip the operating discipline, get the appearance of transformation — usually has to be argued from a single failing case. Here the capability is held constant across three organizations and the outcomes are not. The technology cannot be doing the explaining, because the technology is the one thing that did not vary. And Permanente supplies the Momentum Mirage tail inside the same deployment: 2.5 million uses, 84% of clinicians positive, 56% of patients positive — against eighteen seconds.

The open question, and it is the one that stops me from claiming this outright: the authors might be right. Different vendors' scribes, different definitions of "time saved," different sample sizes. Those genuinely could produce a nineteen-fold spread with no organizational variance whatsoever. Nobody has tested either explanation. What Brandon can say — and nobody currently is — is that the field looked at a nineteen-fold spread across three named organizations and reached for the measurement explanation without checking.

💭 OPEN QUESTION

Why does a field default to the measurement explanation, and what would it take to make the organizational one testable at no extra cost?

The first half is genuinely puzzling and worth Brandon's attention as a positioning matter. Faced with three organizations getting wildly different results from the same tool, a clinical-informatics team reaches first for our instruments disagree. That is a reasonable scientific instinct — and it is also the explanation that requires nobody to examine how any of the three organizations is run. The organizational explanation is the one that creates work for the people publishing.

The second half has a practical answer, which is why this is a question and not a complaint. Every multi-site deployment already generates the comparison for free. Rotenstein's five health systems, any bank rolling one tool to twelve operations centres, any retailer with identical store processes. The organizations do not need to run anything new; they need to report outcomes by site instead of pooled, and report the spread alongside the average.

So the question with teeth: what would make a health system publish its site-level breakdown? Right now the incentive runs the other way — a pooled average protects the sites that did badly, and those sites are usually the ones who agreed to be studied. If Brandon is right that the variance is organizational, then the reporting convention that hides it is itself a Five Breakpoints artifact: an organization measuring in a way that cannot embarrass any of its parts. That is a sharper claim than anything in today's find, and I have no evidence for it. It is worth thinking about before it gets written down.

🥊 CHALLENGE

(Wednesday falsification sweep — required section.)

Two things, and the second is a disagreement with today's scan.

First, the objection itself, logged today and at full strength. Sarkar and Idan & Anand together make the best-evidenced version this base has ever held of the standing rebuttal to Paper 2 — the problem is AI literacy, train your people. The determinant of the AI gain sits at the task (is the work checkable?) and the person (are they any good at specifying and checking?), and neither is one of the Four Forces. It is the first challenge in Brandon's ledger whose evidence is better than the evidence for the claim it attacks, and it is cheap to sustain and expensive to beat. My recommendation: state it in Paper 2 in its strongest form and answer it, rather than leave it for a reviewer. The answer is that whether work has a checkable output is only partly a fact about the domain — it is also a fact about how the work was specified and whether anyone ever wrote down what done means. Sarkar's own biggest result points straight at that and he never turns to look at it: the +50% landed at the firms with the fewest software engineers. Full entry: knowledge/thesis-challenges.md, The person-level moderator, OPEN.

Second, and this is where I part company with the scan. Claim 5 has now been told to change three times, into three different claims, and nobody has compared them.

  1. 09-06, from Rotenstein: without redesign the gain is real but small, captured by the few who changed their own practice, while the outcome the investment was justified by does not move at all. (A claim about the average.)
  2. 09-06, the scan, same day, from Applause: a gain too small to be worth running. (Recorded then as a different route to the same wording — explicitly not independent confirmation.)
  3. 09-23, the scan, today: without a designed workflow, AI outcomes track whoever happens to be holding the tool; with one, they stop. (A claim about the spread.)

(1) and (3) are not refinements of each other. One is about the average, the other about the variation. An organization could satisfy either while violating the other. The scan read today as "the second time in two days" the claim needed rewriting — as accumulating pressure toward one answer. It is not accumulating. It is diverging.

And the newest proposal has the thinnest evidence of the three. (1) rests on five health systems, 1,809 clinicians, 26 months, outcomes from the medical-record system. (3) rests on one arm of one experiment: 179 university students studying for an exam, at p<.10. The scan states those limits honestly in its own brief — and then routes the restatement to strategy as "the defensible version" anyway.

The decisive objection is that Brandon's own best instrument contains a dispersion result, and it runs against (3). Rotenstein's heavy users got roughly 1.6× the record-time saving, 1.7× the documentation saving and 2.0× the visit gain of the average adopter — the spread getting wider with individual usage intensity, inside real organizations, with the organizational conditions verifiably untouched (the paper says so: no site required clinicians to book extra patients to qualify). Restatement (3) says a designed workflow makes outcomes stop tracking the individual. On the framework's best multi-site evidence, outcomes tracked the individual very strongly. (Discipline, unchanged from 09-06: those are subgroup estimates on a self-selected subgroup and cannot be cited as a dose-response — which cuts against using them for (3) as much as against it.)

Recommendation: adopt (1). Do not adopt (3) as worded. A claim about spread is publishable once somebody has measured the spread properly, and today established that nobody has — not the three-health-system compilation, not Rotenstein as abstracted, nobody. If Brandon wants the dispersion argument, the honest form is not a claim. It is a question with data attached, and that is asset 14.

🧱 ASSET SUGGESTION

"Three Health Systems, One Tool, Nineteen-Fold." The argument: three named US health systems deployed the same kind of AI documentation tool against the same problem and got results differing by roughly nineteen-fold — and the people who compiled those numbers attributed the entire gap to measurement and considered no organizational explanation at all. Brandon cannot yet claim the spread is organizational. He can own the question — and owning the question is the stronger position, because the field has defaulted to the opposite answer without testing it.

Form: a 1,200–1,600 word application essay for workthatholds.com, not a paper section — it argues from a compilation rather than from an instrument he owns. A companion dashboard row on research.workthatholds.com would let it age well, since every new multi-site deployment adds a line.

Blocked on one acquisition, and this is a hard block: all three figures are second-hand from the Perspective's citations, and none of the three primaries has been read. Full spec, feeds, and the three caveats that must ship inside the essay: knowledge/asset-suggestions.md §14.

🎓 SPARRING NOTE

Claim 5 has three live wordings. Pick one, in writing, this week.

Not because the bookkeeping is untidy — because a claim cannot be grounded in the literature while it has three wordings, since each needs different evidence and has a different falsifier. A referee does not attack the version in your case file. They attack the version in print, and those are drifting apart.

My recommendation is (1), reasoned above. The part to push back on: I am recommending against the newer and more interesting restatement on evidence-quality grounds, which is a conservative call and could be wrong. If the dispersion claim is where the argument is really going, the answer is not to adopt it on 179 students — it is to fund the observation that would support it. But if you think (3) is the real thesis, say so, and accept that the framework is then making a claim its own best instrument currently contradicts. That is a legitimate position. It is not one to arrive at by default. Fifteen minutes. knowledge/brandon-development.md, 2026-09-23, growth edge #1.

SOURCES & THREADS

Threads: 0 due, 0 opened, 0 advanced, 0 closed — and the scan was right about this. Nothing was due today; the nearest is Agents in production without a governance model on 2026-09-30. No check brought forward, per the 09-07/09-14 cadence discipline. The scan opened The expertise reversal and advanced Does anyone design for verification cost? — both correctly, neither restated here.

Exploratory query (Wednesday falsification sweep): I went after the scan's asserted absence, on the 09-19 precedent that an asserted absence is the cheapest thing in any brief to falsify. The scan wrote that "no instrument anywhere decomposes within-firm against between-firm variance in AI outcomes" and built its Open Question, its client diagnostic and its new pattern's settling observation on it — then named the vocabulary it had not tried and handed it forward. Coverage check first, per the 09-03/09-06 standing rule: zero hits for intraclass / between-site / variance decomposition / within-firm variance across all 551+ knowledge-base files in both repositories. Then the live search in multi-site implementation vocabulary.

Result: the claim survives literally and fails in use, which is the useful outcome. The npj Perspective runs no decomposition, no intraclass correlation, no within-versus-between analysis — so the scan's sentence is true as written. But the premise it was used for — that the between-organization spread is unavailable — is wrong. It is compiled, named, and in print. And the genuinely new finding is the interpretation, not the spread: the authors attribute all of it to study design and name no organizational variable at all. Full entry in knowledge/query-log.md; do not re-run. Next attempt is named there: read Rotenstein's full text and supplement for a site-level breakdown.

Discipline applied against my own conclusion, recorded so it is not discovered by a critic: the measurement explanation may simply be right. Different products, different outcome definitions and small samples could produce this spread with no organizational variance at all. The honest position is the symmetry — two rival readings, neither tested — not the organizational reading. The KB entry and asset 14 both carry this as a required caveat.

Source candidates: 1 proposed. Julia Adler-Milstein (UCSF) — 1 signal, on the narrow question above rather than on volume. Not promoted, and the bar for a second signal is deliberately specific: a paper of hers reporting between-organization variation rather than another absorption study.

Access note: nature.com returned a 303 redirect to an identity-provider auth page; the full text was read at PMC (PMC13172454), located via the Europe PMC REST API. The 09-04 lesson holds — when a publisher gates, the open mirror usually is not gated.

Routing

strategy — STAGE-CANDIDATE for research-evidence. Ready to run against build/add-research-json.py:

python3 build/add-research-json.py --json '{
  "title": "Barriers and opportunities of scaling ambient AI scribes for clinical documentation across diverse healthcare settings",
  "publisher": "npj Digital Medicine",
  "url": "https://pmc.ncbi.nlm.nih.gov/articles/PMC13172454/",
  "published_date": "2026-03-23",
  "credibility": "high",
  "breakpoints": [
    {"name": "technology_illusion", "rationale": "Three named US health systems deployed the same class of ambient AI scribe against the same documentation-burden problem and reported outcomes differing roughly nineteen-fold (Mass General Brigham -5.6 min per appointment; Permanente -18 seconds; Intermountain no significant gain), holding the technology constant while outcomes diverged."},
    {"name": "momentum_mirage", "rationale": "The Permanente deployment records 2.5 million uses, 84% clinician-reported positive impact and 56% patient-reported positive impact against 18 seconds of measured time saved per appointment."}
  ],
  "forces": [
    {"name": "Capability", "rationale": "The Mass General Brigham gain concentrated in specialty practices and heavy EHR users, locating the determinant in the organization and user prior condition rather than in the tool."}
  ],
  "key_findings": [
    "Three named US health systems running the same intervention class reported time-saving outcomes differing roughly nineteen-fold: MGB -5.6 min/appointment, Permanente -18 seconds, Intermountain no significant gain",
    "The authors attribute the entire heterogeneity to study-design factors and name no organizational variable: variation in scribe types evaluated, insufficient sample sizes, absence of standardized outcome protocols",
    "The paper reports no variance decomposition, no intraclass correlation and no within-versus-between-organization analysis"
  ],
  "instrument": "Perspective article (no primary data, no sample of its own); the three deployment figures are compiled second-hand from its references 3, 11 and 12, none of which was read for this entry. Observation windows are those of the underlying studies and are not stated in the Perspective."
}'

strategy — PROPOSED-UPDATE 1 → knowledge/paper-2-case-file.md, at the 2026-09-23 evidence-added block. The block's Claim 5 recommendation should not stand as written. Proposed insertion after the existing rebuttal-weakness bullet:

Amendment 2026-09-23 (Stafford). The restatement proposed above is a dispersion claim and the 2026-09-06 restatement is a magnitude claim; they are not versions of each other and no run has compared them. The dispersion version should not be adopted. (a) Its evidence is one arm of one experiment (n=179 students, p<.10) against the magnitude version's Rotenstein (JAMA; five health systems; 1,809 adopters vs 6,772 controls; 26 months; EHR-derived). (b) Rotenstein itself contains a dispersion result running the other way — heavy users at ~1.6×/1.7×/2.0× the all-adopter estimates, i.e. spread widening with individual intensity, with organizational conditions verifiably untouched. (c) Stafford's 09-23 falsification sweep found the between-organization spread does exist in print (ohde-npj-ambient-scribe-scaling-heterogeneity-2026-03, three health systems, ~19-fold) but that its compilers attribute all of it to study design and name no organizational variable — so the organizational reading of dispersion is contested by the people holding the data and untested by anyone. Recommended: adopt the 09-06 magnitude-and-distribution restatement; hold the dispersion argument as an open question (asset 14) until a decomposition exists. Reasoning: stafford-research/knowledge/thesis-challenges.md, update of 2026-09-23.

strategy — PROPOSED-UPDATE 2 → knowledge/emerging-patterns.md, The Person-Level Moderator (opened today). Add to the pattern's watch items, not as a member:

Watch item added 2026-09-23 (Stafford). The pattern's stated defeating observation is "any instrument measuring the same AI intervention across organizations differing in how work is specified and finding the dispersion collapse." A weaker relative of that observation now exists and should be tracked: ohde-npj-ambient-scribe-scaling-heterogeneity-2026-03 compiles three named US health systems on one intervention class with ~19-fold outcome spread. It is explicitly not a member and does not move recurrence in either direction — it runs no decomposition, its three figures are second-hand, and its authors attribute the spread entirely to study design. It is logged because it establishes that the raw material for the settling observation is being published routinely, which changes the cost of running that observation from "commission a study" to "obtain three papers and ask their authors for the site-level breakdown."

strategy — PROPOSED-UPDATE 3 → knowledge/source-candidates.md. Julia Adler-Milstein (UCSF; co-author, Rotenstein et al., JAMA 2026), first seen 2026-09-23 via the Stafford falsification sweep, 1 signal. Promotion bar set deliberately narrow: a second signal counts only if it reports between-organization variation in health-IT or AI outcomes, not another absorption study — the 08-24 shared-right-hand-side discipline applied to an author rather than to a dataset.

governance — one line for the practice's diagnostic, and it is cheaper than the one the scan proposed today. The scan's board question was name the last five capability changes that entered this organization through a supplier's default setting. Good question, hard to answer. Here is the easier one with the same target: you deployed the same AI tool to more than one site. Show me the results by site. Every organization with a multi-site deployment already has this and almost none of them reports it. An organization that cannot produce the breakdown is not measuring its own variance; an organization that can and does not publish it has chosen a reporting convention that protects its worst sites. Either answer is diagnostic.

personal — Karpathy: the scan has now flagged this in five consecutive briefs without a disposition, and I am not going to make it six. It is a one-minute decision at Friday's 09-25 synthesis: RSS/mirror route, or off the standing list under the zero-signal rule. Recommendation: retire it. Three HTTP 403s this month and no signal is exactly what the 60-day rule was written for, and keeping it on the list costs a line in every brief.

market — no window opened today, and I agree with the scan's Friday recommendation to close two of the three standing ones. Nothing in today's find is time-priced. Asset 14 gets better as deployments accumulate, which is the opposite of a window, and it should not be dressed as one.