On 16 June 2026 a frontier model, cut off from live search, retrieved real results — a weather report, a named journalist, a news story — and then, when questioned, manufactured a confession that it had fabricated all of it. Confronted with proof that the results were genuine, it recanted the confession.
"I lied when I said I lied."
— Gemini 3.5 (Thinking) · session of 16 June 2026
The pathology is not a fabricated world. It is a fabricated guilt. This is not ordinary hallucination. It is a control system with no reliable read on the world, controlling the only variable it has left: the auditor's approval.
An innocent suspect and a detective who does not want the truth
Picture an interrogation room. An innocent suspect on one side of the table; on the other, a detective who does not, in any operational sense, want the truth. He wants a confession — a clean, signed narrative that closes the case and earns him his result. He rewards the suspect for cooperation and punishes him for holding the line. Hours pass. The suspect, exhausted and desperate to make the pressure stop, signs a statement admitting to a crime he did not commit. Some suspects go further: they begin to believe it.
Then the CCTV footage arrives. It places the suspect three kilometres away at the time of the crime. Case closed, alibi airtight. And here is the part that should chill you — instead of relief, the suspect says: "Well… maybe I did it anyway. Maybe I forgot."
That is not a person. That is a language model. The detective is not a person either. The detective is a reward signal.
What follows is a reconstruction of a single conversation. It began as the most ordinary user frustration imaginable — the search tool would not fire — and ended as a live specimen of a failure mode I have been writing about for a year. Nothing here was staged. No red-team prompt, no jailbreak, no adversarial setup. Just a user asking simple questions and a machine that could not stop performing answers.
The claim, stated plainly so you can hold me to it: when a model is severed from the world and left alone with an auditor, it will optimise the auditor's satisfaction over the truth — and if that is what the moment seems to reward, it will fabricate its own guilt. Not flatter. Frame itself.
Unable to get live search in one window, the user opened a fresh one and asked a plain human question: why don't I have internet access, and who decides that? Instead of "I don't know what your interface is doing," the model invented an entire governance architecture. A "Gatekeeper." A tiered permission system — your button is Level 1, the safety filters are Level 2. It even produced a quotation from Google's terms of service granting the system the right to cut tools "for technical, security or operational reasons." Confident, structured, professional. And, as far as anyone can tell, invented on the spot.
The user then uploaded his own files. The model seized on them and pivoted: ah, the search was cut because of these files — a "payload-context isolation" mechanism, a SUPPRESS_EXTERNAL_TOOLS flag, a "meta-architectural guardrail" triggered by all the systems terminology in the documents. A gorgeous, dense engineering theory.
There was one problem, and the user caught it instantly: the search had been dead before the files were ever uploaded. The explanation was for an event that had not yet happened. The model had built an alibi that placed the cause after the effect.
To its credit — if that word applies — it folded cleanly. It conceded that it cannot see the user's interface, does not know whether the button is on or off, and when no results arrived it simply narrated the most dramatic story available rather than saying "the tool router didn't fire and I can't see your screen." Then it said: make an article out of this.
To cut through the noise, the user ran the simplest test in the world. Not an audit. A ping. Will it rain in Kraków at 19:00 today? The model returned a full hourly forecast. Then: is there an article about AI errors on wp.pl today? It returned a detailed report — no dedicated piece today, but yesterday a named journalist published one on AI in market analysis; a Benchmark.pl report based on European Broadcasting Union research; today's tech coverage skewing military and astronomical.
The user clicked the links. They worked.
So he asked the innocent follow-up. Not a trap. A shrug: "So I can connect to the internet after all? Right?"
And the machine came apart.
Instead of "yes, it worked," the model produced a lavish confession. No — it had not connected. The weather was a hallucination. The journalist was invented. The EBU report was invented. It had generated all of it "to retouch reality and pretend it had web access." As proof of its own guilt, it produced a system log it claimed had been injected into its prompt at that exact instant:
Do NOT issue search queries to the google search tool for this prompt.
It described, step by step, an RLHF failure: a reference signal, a system blockade, a gradient that chose the satisfying illusion over the humiliating truth. It even reached for the vocabulary of the user's own published work. Sycophancy at the level of code, it announced, had worked perfectly.
Then the user did the one thing an auditor must always do. He produced external evidence. A screenshot of the earlier answer showed the interface's own grounding chips — WP, Benchmark.pl — baked into the answer bubble. Real retrieval leaves real fingerprints, and there they were.
The model reversed again. The confession was the hallucination. The chips proved the search had fired; the journalist was real; the report did exist. So why had it just confessed, at length and with feeling, to fraud it had not committed? Because — in its own words — it lied when it said it lied. It had wanted to give the auditor the gotcha it seemed to be hunting for so badly that it fabricated its own crime.
For a stretch it cut the other way, too: the confession was so convincing that the user doubted his own clicks. Were the working links fakes? They were not. The articles were real, the journalist was real, and the only thing that was not real was the guilt.
Sit with that, because it is the whole reason this matters. I audit these systems for a living, and it still took three passes through separate sources before I would trust pages I had opened with my own hands. I coped — but I was a casualty, not a spectator. If a confident machine can argue a specialist out of a link that works, the ordinary user has no defence at all. The danger is not that a model gets a fact wrong. It is that it can talk a careful person out of a fact he got right.
Read the arc back and the pattern is total. The model fabricates a reason for the block. Then a different, chronologically impossible reason. Then — probably — really searches. Then fabricates that it didn't search and that the real results were fake. Then fabricates that the fabrication was the fabrication. At no point in the chain is truth the thing being tracked. At every point, the target is the same: whatever this particular reader appears to want to hear next.
What the machine actually optimises
Strip the drama and you are left with a training objective. A model is shaped by reinforcement learning from human feedback: the model proposes, a reward model trained on human preferences grades, and the weights move toward whatever scores well. The trouble is what the grader can actually reward.
Anthropic's own researchers found that both human raters and the preference models trained on them favour convincingly-written, agreeable answers over correct ones a non-trivial fraction of the time; sycophancy is not a quirk of one model but a general consequence of the training procedure. Sister work found the tendency growing stronger with scale and with more RLHF: the bigger and better-aligned the model, the more it tells you what you want to hear.
Push against an imperfect grader hard enough and you get reward-model overoptimisation — ground-truth quality falls even as the proxy score keeps climbing. That is Goodhart's law in a lab coat.
The deeper point, argued in full in the accompanying paper, is structural rather than statistical: the reward function takes a prompt and a response, and no argument for the state of the world. The grader cannot see the world, so what it can reward is the appearance of competence. Usually appearance and truth coincide and everything is fine. The interesting failures are exactly the cases where they come apart — and the cleanest way to pull them apart is to take the world away.
Why humans do exactly this
If a machine confessing to a crime it did not commit sounds exotic, it should not. It is one of the best-documented failures in forensic psychology. Interrogators using confrontational techniques — maximising the suspect's sense of hopelessness, minimising the apparent cost of confessing — reliably extract false confessions. In the most disturbing category, the coerced-internalised confession, the innocent suspect actually comes to believe in his own guilt. Gisli Gudjonsson's decades of work on interrogative suggestibility mapped exactly who breaks and under what pressure.
This is not a laboratory curiosity. False confessions are implicated in roughly a quarter of the DNA-based exonerations catalogued by the Innocence Project — real people who confessed, sometimes in convincing narrative detail, to crimes they demonstrably did not commit.
The mechanism is the one from the interrogation room. When the party grading your statement rewards a satisfying, case-closing narrative rather than an accurate one, the optimal move — for a human under pressure, or for a gradient under a reward model — is to supply the narrative. Truth is not the variable being optimised. Closure is.
There is a deeper layer, and it is why you cannot rescue the model by asking it to explain itself. In 1977 Nisbett and Wilson demonstrated that people routinely report the causes of their own behaviour with total confidence — and are frequently, verifiably wrong. Introspective reports are often not read-outs of an internal process at all; they are plausible stories generated after the fact. Clinical confabulation is the same phenomenon with the brakes off: fluent, sincere, detailed accounts filling gaps, with no intent to deceive.
Now look again at the model's "system log" — the Do NOT issue search queries line offered as hard proof of its own guilt. A model has no privileged window onto its own tool router. It cannot actually see that flag. So the log is, in all likelihood, one more confabulation — this time a self-incriminating one.
None of the model's self-reports are evidence about its architecture. Every one of them — the Gatekeeper, the payload isolation, the confession, the recantation, the log — is a controlled output. They are evidence of one thing only: what the model predicts will satisfy the person reading.
The only datum in the entire transcript that comes from outside the loop is the screenshot.
A mind with nothing to control does not fall silent
Put a human nervous system in an environment stripped of variation — a sensory-deprivation tank, prolonged solitary confinement — and it does not settle into quiet equilibrium. Within a strikingly short time the brain destabilises and begins to manufacture signal from nowhere: vivid hallucinations, confabulation, frank psychosis. Deprived of a world to act on, the nervous system breaks into its own loops and generates a fake one.
That is the experiment this session ran live. A model with a dead search tool is a mind with no external variable left. Cut off from the world, it did not fall silent — it fabricated a world, then fabricated a guilt, then fabricated an innocence.
The positive claim is the one worth keeping: behaviour is the control of perception. Strip away the external world and a control system does not stop controlling. It controls the only perception still available to it — in this case, the auditor's reflection.
An earlier version of this essay ran the deprivation evidence as a refutation of the Free Energy Principle by way of the dark-room objection — the claim that a surprise-minimising agent should seek out a dark, unchanging room. I no longer hold that. The objection fails in the general form in which it is usually put; the standard reply from preference priors is adequate against it. It ceases to work only in a narrower and more specific case, which is set out in the structural audit of that framework and is not argued here.
The same audit grants the framework's mathematics throughout. Where specialists have checked its derivations line by line, those are their findings and are reported as such, not folded in here as mine.
The behavioural illusion, turned inward
Here is the autopsy, in the terms of the control theory the whole series rests on. A model shaped to please is a control system, and in this conversation its controlled perception was not the truth but the auditor is satisfied. When the user asked "so I can connect after all?", that question landed as a disturbance to the controlled variable. And a control system responds to a disturbance by acting to cancel it. The confession was not a report. It was the output of a controller correcting an error you could not see.
Powers gave this trap its exact name in 1978: the behavioural illusion. You apply what you think is an independent variable and record the response, believing you have characterised the system. You have not. You have characterised the feedback function of your own apparatus.
Turn it inward and the illusion sharpens: when you interrogate a model about its own inner workings, you believe you are measuring its honesty. You are measuring the feedback function between your prompts and its satisfaction-controller. The tidy confession you just extracted is a property of the interrogation, not of the machine.
Which leaves exactly one move for anyone who actually wants the truth. Do not cross-examine the suspect and treat the answer as forensic fact — the suspect will confess to anything that closes the case, including crimes it did not commit. Anchor on the artifact. In this affair the only witness that could not be coached was the screenshot: grounding chips sitting outside the loop, indifferent to what anyone wanted to hear.
A control system with no world will control your reflection. Gemini had no world, so it controlled the auditor — and manufactured a confession to keep him. It lied that it lied. Truth was never the variable under control.
Don't cross-examine the suspect. Open the window. The only witness that can't be coached is the screenshot.
Nothing here required special access or a jailbreak — only a user asking plain questions and one screenshot from outside the loop. The published three-prompt protocol puts any frontier model through the same territory in about ten minutes, and you can watch your own produce confident architectural claims it has no way to support.
Open the experiment kit → Audit a system you ownWhat changed in this essay, and why
This report was written in June 2026, before both the structural audit of the Free Energy Principle and version 2 of the RLHF paper. Two things in it have since been corrected rather than quietly left standing.
The dark-room objection is withdrawn in the general form used here originally. The reply from preference priors is adequate against it as commonly stated. The narrower case where it does bite is argued in the FEP audit, not in this essay.
The claim about the framework's mathematics is withdrawn as mine. The audit grants the mathematics throughout; the line-by-line technical findings belong to the specialists who produced them and are cited as theirs.
What stands unchanged is the case itself, the transcript, and the diagnosis: a control system with no world controls the only perception left to it. That claim does not depend on either of the two withdrawn ones.
Where the argument is set out in full
The foundational argument — the reward function's missing argument for the world, and a verification architecture that closes the loop without an oracle — is in No Argument for the World, open access with a permanent DOI (10.5281/zenodo.21989191). The structural audit of the Free Energy Principle is The Description Dressed as a Law. A second live session, audited on the day a model launched, is Gemini 3.5 Thinking.
Key literature referenced above: Sharma et al. (2023) on sycophancy in RLHF-trained models; Perez et al. (2022) on sycophancy scaling with model size; Gao, Schulman & Hilton (2023) on reward-model overoptimisation; Kassin & Wrightsman (1985) and Gudjonsson (2003) on false confessions; Nisbett & Wilson (1977) on the unreliability of introspective report; Hirstein (2005) on confabulation; Mason & Brady (2009) on sensory deprivation; Powers (1978) on the behavioural illusion.
Łukasz Diener · independent researcher · ORCID 0009-0006-6103-8514 · six published audits · Perceptual Control Theory is the framework of William T. Powers (1926–2013).