The Last Person Who Knows How

In Never Breaks a Rule, Never Finishes, the Ashdown instruction, the invented intake matter that recurs through this book, ends as well as it can. The agent is blocked, hands over at tick 10, and the partner has the 4 ticks needed before the date at tick 14. That chapter's lead_error_iff, proved in lean/BuildingInSteel/Liveness.lean for all natural numbers with the declared lead no longer than the date, says that a handover planned on a declared lead reaches the date exactly when the true lead does not exceed it. So the guarantee rests on a number about a person. Add one more invention: after a year of reading summaries, the same partner needs 6 ticks. Tick 10 plus 6 is tick 16. The theorem still holds, and now says the plan misses. The declared lead has gone stale, and no sign-off log records that.

The capacity to reject is an asset that wears out or fails to form, and a sign-off log cannot show its state. A firm that relies on human review should therefore test it, with known errors and with the tool switched off, as United States airports began doing with seeded images in 1999. Three claims. First, an approval record can expose a reviewer who never rejects, but it cannot separate one who can reject from one who flags without skill, and even adjudicated flags give no hit rate without a known error rate. A better tool makes this worse. Second, and most contestable: the case for testing does not need deskilling to be proved. The evidence is narrow and points both ways, and in the one trial I have found that tests practising lawyers with the tool removed, senior lawyers who had used it beat controls on unaided judgement. You test because you do not know. Third, the test can be specified today, and a proof would add nothing to it.

I build AI systems for law firms, so I gain from buyers trusting them, and I would gain again if review became something a vendor helps measure. Read the drill with both in mind. Nothing here is legal advice. The recommendation is normative: it says what a firm ought to buy with unbilled time. The evidence under it is empirical, and each item carries its setting.

What Bainbridge said

Lisanne Bainbridge's Ironies of Automation is a five-page brief paper about industrial process control, and it argues from the literature; it does not report a new experiment (Bainbridge, 1983)90046-8). Four points carry this chapter.

The first concerns decay: "physical skills deteriorate when they are not used, particularly the refinements of gain and timing", so that "a formerly experienced operator who has been monitoring an automated process may now be an inexperienced one". The second concerns succession: automated systems "monitored by former manual operators, are riding on their skills, which later generations of operators cannot be expected to have". The third concerns monitoring. If a computer decides because human judgement is inadequate, which of its decisions is the human to accept? "The human monitor has been given an impossible task." The fourth is the one a technology buyer should pin up: "it is the most successful automated systems, with rare need for manual intervention, which may need the greatest investment in human operator training".

She also said what to do, bluntly: let the operator use hands-on control for a short period each shift. "If this suggestion is laughable then simulator practice must be provided."

Four failures, not one

"Deskilling" is doing the work of several words. I separate four failures because each has a different remedy and a drill has to tell them apart. None of this evidence is about AI in legal review, so each row states its setting.

Endsley and Kiris studied the out-of-the-loop problem with an expert system on a navigation task in 1995 (Endsley and Kiris, 1995). All I add is its application to a profession whose rules, in the texts I read, do not require the test.

What the studies of AI do and do not show

Three studies of working professionals that I read test the question directly, with AI and work done without it, and they do not agree.

Poland: a fall. A retrospective observational study at four endoscopy centres, 1,443 non-AI colonoscopies, three months before and three months after AI arrived. The adenoma detection rate fell from 28.4% to 22.4%, a difference of 6.0 points (95% CI 1.6 to 10.5), and the authors say exposure "might reduce" detection (Budzyń et al., 2025)00133-5). A society's summary gives 19 endoscopists (ASGE journal scan); I could not open the publisher's full text or its later correction, so I rely on the PubMed abstract.

Norway: no fall. A prospective pragmatic trial with 13 endoscopists (7 inexperienced, 6 experienced), 5,013 colonoscopies, before, during and after AI: "no upskilling or deskilling was observed following a period of CADe exposure" (Pedersen et al., 2026). The outcome differs (polyps of 5 mm or more, not adenomas), the authors warn of underpowering, and they suggest the Polish effect "may be a temporal phenomenon that disappears over time after CADe is removed".

United States lawyers: mixed. A pre-registered randomised trial: 133 patent lawyers at eleven intellectual property firms, 91 completing the protocol, three months with an unreleased drafting assistant (Autor et al., 2026). Those remaining then redlined a flawed patent draft without AI, scored blind. Treated lawyers beat controls by 0.32 SD, "concentrated entirely among senior lawyers (0.45 SD, p = 0.02)". Juniors, under seven years' experience, "showed no average gain; their scores instead bifurcated". It is a working paper; Google funded the direct costs and built the tool, most authors work there, every firm has patent work from Google, and attrition reached 32%. "Without AI" was a stipulation the authors could not observe directly. Erring towards over-classification, they flag 15 of the 91 redlines as possibly AI-assisted. Controlling for the flag leaves the senior effect at 0.44 SD (p = 0.02); dropping those redlines cuts the overall effect to 0.25 SD (p = 0.14) and the senior effect to 0.39 SD (p = 0.06). It also contains the sentence I would pin beside Bainbridge's: "We were also unable to perform a baseline skills assessment (something our preferred design included but partner firms rejected)".

Students and novices: both directions. In a randomised trial of about one hundred upper-level law students, those who used AI on a first memo beat controls on a later application task "even when neither group had access to AI", though when all then used AI to revise, those with stronger memos regressed (Bednar et al., 2026). With nearly a thousand high school mathematics students, a standard chat interface left grades 17% lower once access was taken away, and a tutor built to safeguard learning largely mitigated that (Bastani et al., 2025). In randomised experiments with developers learning a new library, AI use impaired "conceptual understanding, code reading, and debugging abilities" (Shen and Tamkin, 2026). None of these is a practising lawyer signing a document. They do show that the design of the interaction can move the result either way, which is one more reason to measure it.

All three studies of professionals compare levels over a fixed window: about six months in Poland, up to about two years of phases in Norway, three months in the patent trial. None measures a rate of decline, so nothing here licenses talk of compounding. The cohort worry, juniors who never form the skill and later become the signatories, would be a rate effect across a career, and I have no measurement of it. So the summary is not "AI deskills lawyers". The question is open, seniors in one field may gain, and juniors are where the variance sits.

The log cannot tell you

Take an invented tool wrong on 2% of the statements it produces, and three invented reviewers. A can reject: A catches 80% of defects and flags 1% of clean statements. B signs everything. C flags 2.58% of statements at random, with no skill at all.

This is hand arithmetic on invented inputs, and it needs no proof. A and C have the same approval rate and nearly a fivefold difference in what they let through. Adjudicating each flag helps: it exposes C, whose flags are mostly false. It cannot reach what nobody flagged. A reviewer catching 20% of defects from a tool wrong 2% of the time, and one catching 80% from a tool wrong 0.5% of the time, each raise 4 correct flags per 1,000, and the log does not say which world you are in. The ground truth is what is missing.

Now improve the tool tenfold, to 0.2%. A's approval rate becomes 98.84%, and the gap between A and B narrows from 2.58 points to 1.16. Then add criterion drift: in Wolfe's experiments rarity itself moved the threshold, and the abstract reports that "Several efforts to induce observers to adopt a better criterion fail", though brief high-prevalence retraining with full feedback held it, hence D7. So a better tool gives fewer chances to practise, less signal in the log, and, if reading behaves like visual search, a more reluctant reviewer. That last step is my inference.

Outcomes will not rescue the log quickly: a bad clause may surface in a dispute years later, or never. The quantities a firm needs are plain: hit rate on known defects, false-flag rate on clean material, and the measured lead, the h in margin D h t from Never Breaks a Rule, Never Finishes. A refusal is a disturbance to the plan. A reviewer's lead is a parameter, and this chapter's worry is that it drifts. I claim no gain margin, no stability result and no conservation law; lead_error_iff holds for each value of the lead and says nothing about how that value moves. The drift is what has to be measured.

Someone already does this

The closest precedent is a fake threat on a real X-ray screen. "In late 1999," the United States General Accounting Office recorded, the Federal Aviation Administration "began deploying" Threat Image Projection, which "places images of threat objects on x-ray screens during actual operations and records whether screeners identify the threat object" (GAO, 2003). The cost is real: it was shut down after 11 September 2001 "due to concerns that it would result in screening delays and panic". A recent study at one European airport, with 1,199,838 events from 728 screeners over four years, found that about 100 events per screener gave a reliability of at least 0.7 when images were hard enough (mean hit rate below 0.9), and that scores predicted 1,184 covert tests using real prohibited items (Buser et al., 2025).

Other trades drill the fallback. In the United States a pilot in command carrying persons needs "at least three takeoffs and three landings within the preceding 90 days" (14 CFR 61.57), and licensed reactor operators pass "an annual operating test" (10 CFR 55.59). None of this shows that the method transfers to law.

The duty exists, the test does not

Law has met this before. In Pyrrho, approving predictive coding and noting "the novelty, in this jurisdiction at least, of such use", Master Matthews described best practice for training the software as "a single, senior lawyer who has mastered the issues in the case", and preferred the machine's consistency to "dozens, perhaps hundreds, of lower-grade fee-earners" [(Pyrrho v MWB [2016] EWHC 256 (Ch), paras 1, 20 and 33)](https://caselaw.nationalarchives.gov.uk/ewhc/ch/2016/256). It also shows the dependence: the method needs the senior, and displaces work the fee-earners did. I have no measurement of what that did to training.

In England and Wales the duty already sits on the person. A solicitor must "maintain your competence", must "effectively supervise" work done through others, and must ensure that those managed "are competent to carry out their role" (SRA Code for Solicitors, paras 3.3, 3.5, 3.6). In a case about false citations put before the court, the Divisional Court said that "practical and effective measures must now be taken" by those with leadership responsibilities "such as heads of chambers and managing partners", and that in such hearings the profession "can expect the court to inquire whether those leadership responsibilities have been fulfilled" [(Ayinde [2025] EWHC 1383 (Admin), para 9)](https://caselaw.nationalarchives.gov.uk/ewhc/admin/2025/1383). On 17 August 2026 the SRA warned that supervisors "may also be found to have breached regulatory requirements" where false citations reach the court "without adequate review and/or supervision" (SRA warning notice). Under UK GDPR a decision is solely automated "if there is no meaningful human involvement" (art 22A(1)(a)), and the article's only guide to what makes involvement meaningful is a direction to consider profiling. The Information Commissioner's AI guidance, marked as under review since the Data (Use and Access) Act, goes further: reviewers need the "authority and competence" to go against a recommendation, and monitoring means analysing "why, and how many times, a human reviewer accepted or rejected the AI system's output" (ICO). Asking why is more than a count, but it is still the log, with no ground truth. The ICO's consultation draft for art 22A wants a human "suitably trained and qualified" and "a record of how the human was involved" (ICO draft): a credential and a log.

None of the texts I read, the Law Society's note among them, tests whether a reviewer can tell; the ICO's are the nearest, and they rely on training and the log. A search covering the Bar Standards Board, the ABA and the Law Society of New South Wales found no such requirement; I did not read those three, so that is a lead. My searches also found no published experiment measuring lawyers' detection of planted errors in AI drafts.

The intervention drill

This is a design sketch: a specification in prose, not run by me or by any firm I can point to. Its intervals are my judgement.

The arithmetic constrains the design. For an invented group of 20 signatories, three seeds in 60 checkable statements per person per quarter is 240 seeds a year, which, if seeds were independent, would give a 95% interval of about 5 points either side of a team hit rate near 80%. Seeds cluster by reviewer and by document, so the real interval is wider. At 12 seeds a year, one person would take over eight years to reach Buser's 100 events, if that figure even transfers. So D3 grades teams and teaches individuals. It tells the four failures apart only in part. A low D5 mark points to decay or to a skill never formed, and only the D2 baseline separates those two. A low D3 hit rate beside a sound D5 mark points to the moment of review: if false flags fall as well, that is the signature of a shifted criterion in Wolfe's terms, but over-reliance can look the same, and nothing here separates them.

Bainbridge marks the limit: "Unknown faults cannot be simulated". A seed is a failure somebody has already imagined. D5 exists because of that: an open unaided piece, marked by a person, tests judgement as well as detection.

Why there is no Lean here

This book's habit would be to write structure Drill and a decidable passes. It would compile, and it would prove that a score above the floor is above the floor. Three things carry the load here: whether a seed resembles the tool's real failures, whether a drill predicts real catches (Buser found that it does for screeners at one airport, and nobody I have read has tested lawyers), and how much unbilled time the result is worth. Two are empirical and one is a value. The kernel has nothing to say about any of them. A green file would lend credibility to a structure it had not checked, which is the failure described in Floating Theories. Formalising this would be theatre.

lead_error_iff and margin_nonneg_iff belong to Never Breaks a Rule, Never Finishes. This chapter adds a reason to doubt one of their inputs and a way of measuring it.

What exists and what does not

Exists, read: the legal and government texts, Bainbridge, Autor, Pedersen and Wolfe 2013 from full text, in the passages cited; other studies in abstract only, Bednar's in a copy elsewhere; the ICO pages archived. Proved, elsewhere: the two named theorems. Hand arithmetic on invented inputs: the reviewer table and the seed counts. Sketched: the drill. Not found by me: a published test of lawyers catching planted errors in AI drafts at low prevalence; a study with repeated unaided tests over several years under generative AI; a legal regulator's requirement to test detection; evidence that this drill improves outcomes. Normative: that a firm ought to run it.

Objections

"Let the skill die. Nobody mourns long division." Often right. D10 lets skills go if a failure gets caught some other way and no human signature claims otherwise. Long division passes: a wrong sum can be caught without doing long division. A machine checker may pass too, and then the row says so. A partner's signature on advice does not pass while it still means "I could have told if this was wrong". Verdict: right for many skills, wrong while the claim of review stands. I would change my mind on evidence that unreviewed outputs of a work type do as well as reviewed ones, over a period long enough for errors to surface.

"Autor shows AI builds judgement. You are selling fear." For seniors after three months, in one sponsored working paper, it may. A firm whose lawyers improve will see so in D5. Verdict: conceded on seniors, open on juniors. I would cut D5's frequency, not the baseline, if repeated unaided tests over at least three years showed no fall across cohorts.

"Bainbridge herself says not to inject failures, and Parasuraman and Manzey say training cannot prevent automation bias." Both are real. She wrote: "It would be a mistake, however, to increase artificially the rate of computer failure as the operator will then not trust the system". They wrote that automation bias "cannot be prevented by training or instructions", though Goddard's review lists training among mitigators, so the literature is not settled. I read Bainbridge as writing about the live system: she endorsed simulations that "give experience with low probability events, which may be known to the trainer but not to the trainee", and D3 and D11 are that. On bias, the drill measures and I do not claim it cures. If hit rates will not move, D9 removes the claim of review, which is the correct answer. Verdict: survives, narrowed. I would drop D3 if a study in law, like Buser's covert tests, found drill scores do not predict real catches.

"This prices out everyone but large firms, and access to justice is the real prize." Partly right. A person with no lawyer and a free tool has no fallback to drill, and the comparison for that person is often no help at all. Making drills a condition of offering such a tool would protect the profession at the expense of people outside it. My rule is narrower: anyone selling or signing "human reviewed" should be able to show the test, in proportion to the risk, and anyone who cannot should say there is no such check. The cost falls hardest on small firms, unsolved here. Who gains? Larger firms, and people like me who could sell the tooling. Verdict: scope narrowed, thesis stands. I would reconsider if the cost were shown to remove affordable services with no measured benefit.

"You do not need seeds. Have a second lawyer re-review a random sample of live work." The strongest alternative, and it avoids two weaknesses of a drill: it sits on real work at real prevalence, and the first reviewer does not know which items are sampled. It has its own limits. At the invented 0.2% error rate, 1,000 sampled statements hold about two defects, which says almost nothing about a hit rate. And a second reviewer is not ground truth: if both have lost the skill, they agree and the figure looks good. Verdict: where volume allows, run it beside the drill as the only check on live behaviour; it does not replace known errors. I would drop seeds if a study showed that sampled re-review estimates hit rate as well as seeded tests, at the same cost.

"A failed drill on file is evidence against me." It may be; take advice on disclosure, on which I have read nothing. But a firm with no test has less to say to Ayinde's question. Verdict: a real cost, worth paying. I would change my mind if courts or insurers treated a failure acted on worse than no test.

Goodhart, and what a score loses

Measure hit rate and it will be gamed. Reviewers will flag everything, so false flags count. Firms will quote a score from a 5% bundle while the real tool errs at 0.2%, so publish the prevalence with the score. Reviewers will try harder on a drill. A projected threat appears among real bags during actual operations; D11 forbids that here, and a closed matter is recognisable. So D3 measures what a reviewer can do when alert, probably a ceiling on live detection, and it will tend to hide criterion drift and over-reliance, which are failures of the unalerted moment. Buser's validity result is for images in live screening and does not carry over by analogy; only the study named in the third objection can show it does. And a score will take the place of the thing: a hit rate does not see advice that was correct and unwise. Competence under paragraph 3.3 is an open standard a regulator can weigh, and a drill result must stay evidence of it, never its definition.

What to do on Thursday

If you are a partner, I am not asking you to code, buy anything, or change how you practise. Take a recent closed matter where a tool produced a draft that someone signed. Ask a colleague who knows the matter to put three real-looking errors into a copy: a wrong date, a clause that contradicts the instruction, a misdescribed authority. Give it to the person who would have signed it, with their usual time, and say beforehand that this tests the process. Then show them the answers. An hour or two of two people, once, not billed.

The economics are not free. For the invented group of 20, 45 minutes a quarter and three hours a year each comes to 120 unbilled hours, before preparing seeds or running D6 or D7's short set. And some of the review hours that once trained juniors were paid for by clients; you will now pay for that training yourself or not have it.

There are two dated reasons: the SRA's notice of 17 August 2026, and the court's expectation of managing partners, stated on 6 June 2025. There is also one with no date on it: a baseline cannot be taken once the people trained without the tool have gone.

Predictions

1. By 31/12/2028, a published study of lawyers, with a baseline and repeated unaided tests spanning at least 12 months under generative AI, will find juniors with access scoring below controls on unaided judgement. Refuted by such a study finding no difference or a gain. Void if none is published.
2. By 31/12/2028, the first published test of lawyers reviewing AI legal drafts with planted errors at 5% prevalence or below will report mean detection under 80%. Refuted by a mean of 80% or above. Void if none is published. The number is my guess, unsupported by data from law.
3. By 31/12/2028, neither the SRA nor the Bar Standards Board will require regulated firms or chambers to test detection of errors in AI output with known errors. Refuted by either doing so, in a rule or in guidance saying firms should.

About the author: Eduardo Aguilar Pelaez is CTO and co-founder at Legal Engine Ltd. He writes on formal methods, AI agents, and the discipline of building systems that survive being walked away from.