# The empty seat

By Ali Khalilli. Deterministic Labs, October 2026.
Canonical: https://deterministiclabs.ai/writing/the-empty-seat

Why people end up sitting next to machines, what the seat really
does, and when it is safe to leave it empty.

---

In the spring of 1854, at a fair in New York, a mechanic named Elisha
Otis stood on an open platform. A single rope held the platform, and
a machine pulled it high above the crowd. Then Otis gave the order,
and a man cut the rope with an axe.

The platform fell a few inches, and then it stopped. Here is why. On
top of the platform sat a wide steel spring, and the rope held it
bent. The side rails of the platform had teeth, like a saw. The
moment the rope was cut, the spring opened, its two ends pushed into
the teeth, and the platform hung there, safe.

The elevator company still tells this story today. In the company’s
version, Otis looks at the crowd and says: “All safe, gentlemen, all
safe.”[1] The real records from 1854 say less. Only two newspapers
wrote about the show at all, and the longest report was a few words
in the New York Daily Tribune: the inventor “occasionally cuts the
rope by which it is supported.”[2] So the famous words may have been
added later. The cut really happened.

And the cut is what matters. What Otis showed that day was not a
machine that goes up; lifting machines had existed for a long time.
He showed something new: a machine that stays safe when its rope
breaks. He did not ask the crowd to trust him.

**He made the failure happen himself, in public, and let the brake
answer.**

Here is the strange part. The brake worked in 1854, but for almost a
hundred years after that day, every elevator still had a human
operator inside. The operator stood at the controls all day: he drove
the car up and down, he opened and closed the doors, and the riders
trusted him. **The dangerous part was solved in 1854. The seat stayed
full until about 1950.** This essay is about that seat: who ends up
sitting in it, what the seat is really for, and what finally lets it
become empty.

[Figure 1: The brake.]

Caption: Read it from the top: the cut rope, the sprung-open spring,
its ends resting in the teeth, and Otis standing on his own proof. The
whole drop measured a few inches. The sight was the crowd’s receipt,
and the company sold eight elevators that year.

## The seats

Today, people argue about machines by arguing about seats.

1. Should a human be **in the loop**? That means: the machine
   cannot act until a person says YES.
2. Should the human be **on the loop**? That means: the machine acts
   on its own, and a person can stop it.
3. Or should the human be **out of the loop**, with the machine
   fully alone?

These words sound technical. Their history tells a different story.

[Figure 2: The three seats.]

Caption: The three seats, drawn as running circles. In the loop, the
circle cannot close without the person. On the loop, it closes alone,
and the person stands within reach. Out of the loop, it closes alone
with nobody near. The report that coined the words warned that the
middle seat tends to become the last one.

The three phrases come from a 2012 report called *Losing Humanity*,
written by people who wanted to ban weapons that choose targets on
their own.[3] And the same report, on the same pages, gave a warning
that almost everyone has forgotten. Some weapons keep a human on the
loop, the report said, but they are “effectively out-of-the-loop
weapons because the supervision is so limited.” In plain words: the
middle seat often does not work, and the warning was there from day
one.

Two days after that report, the United States military published its
own rule for weapons that act on their own. The word “loop” does not
appear in that rule. Not once. The rule asks for something else:
“appropriate levels of human judgment.”[4]

And the phrase keeps changing its meaning. In weapons, it means the
machine cannot fire without a human command. In machine learning, it
means something completely different: the people who label training
data so that the next model becomes better, and who work months
before any real decision.[5] In Europe’s new AI law, it means a
watcher: someone must watch the system, but the law does not say from
which seat.[6] One phrase, three different jobs. People argue for
hours about which seat the human should hold.

**Almost nobody asks what is under the seat.**

## The seat that could not act

At 9:58 on a Sunday night in March 2018, a self-driving test car
drove through Tempe, Arizona. One person sat inside, in the
operator’s seat, and she was looking down, toward the screen between
the seats. Outside, a woman was pushing a bicycle across the road.

The car’s software saw her 5.6 seconds before impact, but it never
understood what it saw. It called her a vehicle, then an unknown
object, then a bicycle, and with each new name, it threw away the
path it had calculated for the old name.[7] With 1.2 seconds left,
the software decided that the car must brake hard. And then it did
nothing for one full second. The pause was part of the design: the
system used that second to check its own alarm, because the designers
were afraid of false alarms. During that second, no light came on and
no sound played. The car’s own emergency brake was turned off,
because the company had disabled it for testing. A second crew member
was gone, because the company had removed that seat months earlier.
The operator looked up with about one second left.

[Figure 3: What each of them knew.]

Caption: Two rows over the same 5.6 seconds. The software’s row is
full: it saw her the whole time, and the thin mark is the moment it
decided to brake, 1.2 seconds from impact. The operator’s row is empty
until the last second, because the design showed her nothing. The gap
between the rows is the accident.

The woman, Elaine Herzberg, died. The investigators found causes
everywhere: the software that could not name her, the silent second,
the disabled emergency brake, the removed second seat, the company’s
weak safety culture, and the state’s weak rules.[7] The company was
never charged with a crime. The operator was. She admitted guilt to a
charge called endangerment and received three years of probation. A
whole system had failed, and one person took the blame.

[Figure 4: Six causes, one name.]

Caption: Six findings point in; one arrow points out. The machine’s
own record spread the cause across a company, a design, and a state.
The courtroom’s record had room for one name, and it was the person
in the seat.

It is easy to think this was one bad company. The research says
something harder: the seat itself was the problem, because a human
cannot watch a machine that is almost always right. The classic paper
on this, from 1983, says it directly: “it is impossible for even a
highly motivated human being to maintain effective visual attention
towards a source of information on which very little happens, for
more than about half an hour.”[8] There is a second problem, too. When
a machine does the work, the human’s own skill slowly disappears, so
the better the machine becomes, the less able the person in the seat
is to take over. The investigators wrote that the operator’s
attention drifted, and that this is normal next to a machine that
almost always works.[7] It is not a character flaw. It is how
attention works.

Now remember the elevator. The Tempe car had no spring and no teeth,
no machinery that would answer when the software failed. **The seat
itself was asked to be the brake, and a person cannot be a brake.**
When such a seat fails, it still has one job left. The
anthropologist Madeleine Clare Elish studied where blame lands after
automation accidents, and she gave this job its true name: the
**moral crumple zone**. In a car, the crumple zone is the part that
folds in a crash, so that it absorbs the impact.[9]

**A human placed where they cannot act does not protect anyone. They
are a place for blame to land.**

So if the seat is not what keeps us safe, what is? In the elevator,
the brake is. And machines like that brake have a general name.

## The check

**A check confirms that work is correct, without doing the work again
and without trusting the worker.** The brake lifts nothing. It
answers only one question: is the rope still holding? And it answers
with steel, not with attention. A good check also leaves something
behind: a **receipt**, a visible record that says the check ran, and
here is what it found. The crowd at the fair received their receipt
through their own eyes: they saw the platform stop.

[Figure 5: Watched, or checked.]

Caption: Two ways to trust the same machine. Above, a person watches
the work pass, and nothing remains when they look away. Below, the
work must pass a test, the test prints a receipt, and no person
appears anywhere in the row. The receipt is what lets the seat stay
empty.

Four things make a check real.

1. The check is independent from the thing it checks.
2. The check costs less than doing the work again.
3. The check comes before the harm, not after.
4. The check itself can be checked.

When all four hold, something surprising follows: the checker does
not need to be as smart as the worker.

That last sentence is not a hope; for many kinds of claims, it is a
mathematical fact. In 1985, three cryptographers described something
they called an **interactive proof**, and it works like this: a
checker with an ordinary computer asks random questions, while the
one being checked may have unlimited computing power. After a short
exchange of questions and answers, the checker can be as certain as
it wants that the answer is right.[10] The other side can be far
stronger; the check does not care. If you worry that machines will
soon be too smart for us to check, remember this result. **Checking
can be easier than doing. This is proven mathematics.**

Economics found the same idea from the other side. In 1979, the
economist Bengt Holmstrom studied a simple situation: you hire
someone, but you cannot watch them work. His answer: a contract can
only be built on what you can see, and any signal about the work,
even a weak one, is worth paying for.[11] **What you can see decides
what you can delegate.** A receipt is not paperwork; a receipt is
what makes delegation possible at all.

Nature answered the same question long before mathematics or economics
existed. Your cells copy DNA with about one error in ten billion
letters, but the copying machine alone is nowhere near that good: by
itself, it writes a wrong letter about once in a hundred thousand. So
behind it, a proofreader rereads the fresh copy and removes most of
the mistakes, and behind that, a repair crew catches most of what is
left. Three ordinary checks, one after another, multiply into
near-perfection.[12] The cell does not trust its own copier. And
notice: no one is watching.

[Figure 6: Three checks in a row.]

Caption: The genome’s assembly line, with the caught errors falling
away under each station. The copier alone writes one wrong letter in
about a hundred thousand; the proofreader cuts that by about a
hundred; the repair crew brings it near one in ten billion. Three
ordinary checks, multiplied, beat one brilliant worker.

Mathematics, economics, biology. Three fields, one idea: **the seat
is the symptom, and the check is the cause.** Where a real check
exists, work runs without a watcher; where no check exists, a person
gets pulled in to sit next to the machine, no matter what the diagram
calls their seat. So stop asking about the seat, and ask the older
question instead: what is the check? That question has a map.

## The map

Ask two questions about any task, and notice that neither question is
about the machine. First: can the work be checked against something
outside the worker? Second: if the work is wrong, can it be undone?
Two questions, four possible answers, four kinds of ground.

**The work can be checked, and mistakes can be undone: let it run,
and read the receipts.** An agent writes code in a project with a
real test suite: every change is checked in seconds, by machinery,
and every change can be undone. The honest seat here stays empty most
of the day. We think this is why coding agents arrived before all
other agents: their checks were already built and waiting.

**The work can be checked, but mistakes cannot be undone: the act
waits for the receipt.** A payment, a deployment, a medical
diagnosis. The machine may act, but only through the check: no
receipt, no act. And notice what does not count as a check: a
countdown. In 1978, two researchers at MIT wrote down ten levels of
automation for the United States Navy, and level six was the machine
that acts unless a human stops it in time.[13] Every auto-accept
countdown sold today is level six, a dialog that approves itself when
the timer runs out. The countdown stands in the place where a check
should stand.

[Figure 7: The ladder, 1978.]

Caption: Ten levels of automation, written for the Navy’s undersea
robots in 1978, from the human doing everything to the computer
deciding everything. Level six earns the ink: the computer acts
unless the human stops it in time. Every auto-accept countdown sold
today sits on that level, 48 years later.

Medicine, of all fields, shows the honest version. In 2018, the
American medical regulator allowed, for the first time, an AI system
to diagnose a disease with no doctor looking at the image. The
disease is diabetic retinopathy, an eye disease found in photographs
of the back of the eye, in ordinary clinics.[14] Nobody argued this
autonomy into existence. It was earned in a trial at ten clinics with
nine hundred patients, where the machine’s answers were checked
against the diagnosis of an expert center. An empty seat, in the most
careful field there is, because someone built the check first.

**The work cannot be checked, but mistakes can be undone: the person
is the check.** Taste, tone, drafts, judgment. These have no answer
key, and they never will, because the answer is a comparison to a
standard that somebody chose. Here the human does not watch the
machine work; the human is the answer key, and the machine comes to
ask. Keep that person in, pay them well, and stop pretending a
machine could take that seat.

**The work cannot be checked, and mistakes cannot be undone: do not
automate this.** First make the work checkable, and then automate it,
or keep the act human. There is no third option that survives its
first accident.

Now put this map next to the law. Europe’s AI law requires the people
watching high-risk systems to “remain aware of the possible tendency
of automatically relying or over-relying on the output.”[6] The goal
is right, but the tool is wrong: the law orders a state of mind, and
decades of research say this state of mind fades exactly when the
system works well.[8] **A law can demand awareness. Only a check can
replace it.**

[Figure 8: The map.]

Caption: The whole essay in one drawing. Ask two questions about the
work: can it be checked, and can a mistake be undone? Four answers,
four verdicts: let it run and read the receipts; make the act wait
for the test; keep the person as the answer key; or do not automate
at all. The seat assigns itself.

## The agent’s seat

For decades, this was a question for pilots and control rooms. Then
AI agents arrived, and now anyone who runs an agent sits next to a
machine that acts on its own. That is the seat this essay has been
looking at.

It is worth going slowly here, because this mechanism decides your
working day. Give an agent a code project with tests, and its work
lands on the best ground of the map: checked in seconds, undone with
one command. Your seat stays empty for hours, and your job becomes
reading receipts and making the judgment calls that have no answer
key. Now take the tests away. Nothing about the agent changed, but
the only check left is your attention, so the work pulls you back,
approval by approval, into the watcher’s seat. The same seat as in
Tempe, moved to a desk.

Be careful, too, what you read from that seat. **A transcript is what
the agent said. A trace is what the agent did.** A transcript reads
well even when the work went badly, because models are trained to
write well. The seat survives on traces: the recorded chain from
goal to action to result, checkable without reading anyone’s prose
again.

Researchers recently repeated the 1983 warning, this time for agents:
if nothing changes, they wrote, the old problem “will play out at
scale, compounding across every domain agents are deployed into.”[15]
The way out is not a stronger seat. The way out is one page, agreed
before the work starts.

1. **What may the agent do without asking?** Exactly the work that
   can be checked and undone.
2. **What needs a human signature?** Exactly the acts that cannot be
   undone, and the questions that have no answer key.
3. **What is logged?** Everything, as a trace.

We run our own work this way, and we learned it the hard way. The
agents that build this website work under a written contract, and
every rule in it was added after a surprise. One rule says that the
final look of every page must be checked on real hardware. Why?
Because a testing browser once reported our pages as fine, while a
real browser, on a real machine, showed the headline at size zero. We
added that check the same day, and the seat has been quieter since.

## The trust machinery

One question is still open, and the elevator answers it. The brake
worked in 1854. Why did the seat stay full for ninety more years?

Because a check that nobody can see frees nobody. Automatic
elevators, with buttons and no operator, existed around 1900, and
riders refused to use them. “People walked in and looked and walked
right back out,” says the elevator historian Lee Gray.[16] The
machinery was fine; the receipt was missing. Then, in September 1945,
the elevator operators of New York went on strike. The city lost a
hundred million dollars in taxes, and one and a half million office
workers stayed home.[16] After that, building owners wanted automatic
elevators badly. But riders still had to be convinced.

So the industry built machines for the one thing the operator had
really provided, and none of it was about lifting. An emergency stop
button. A telephone. Advertisements that showed children pushing the
buttons. And a recorded voice, as calm as any human operator: “This
is an automatic elevator. Please press the button for the floor you
desire.”[16] From invention to comfort, the road took more than fifty
years. The seat became empty when the check finally had receipts that
a rider could see.

When someone breaks this rule, you get the disasters. The
Therac-25 was a radiation therapy machine. The model before it had
hardware safety locks, physical parts that stopped the beam no matter
what the software said, and the same software bugs lived in both
machines. In the old machine, the locks caught the bugs and nobody
was hurt. In the new machine, which trusted its software, the locks
were removed, and between 1985 and 1987 six patients received massive
overdoses. Some of them died.[17] The bug was old. Only the missing
check was new.

And one summer morning in 2012, a trading company called Knight
Capital turned on new software. The update had reached seven of its
eight servers; one server still ran old, dead code, and no second
person had checked the update. In 45 minutes, the system sent
millions of orders into the market and lost over 460 million
dollars.[18] No human was in that loop. But nothing else was in it,
either.

**Seats empty at the speed of trust, and trust moves at the speed of
receipts.**

[Figure 9: The wait.]

Caption: Ninety years, drawn to scale. The brake is proven at the
left edge, and the band stays black as long as every elevator carries
an operator. It dissolves only when the receipts a rider could touch
arrive: the stop button, the telephone, the recorded voice. Seats
empty at the speed of trust, and trust moves at the speed of
receipts.

## Where the check ends

An essay about checking must also tell you where checking ends.

First: most of what you know, you never checked, and you never will.
You believe the mass of the electron without measuring it yourself,
and the philosopher John Hardwig argued that this is not a weakness:
you can have good reasons to believe something, because you have good
reasons to believe that other people checked it, and “rationality
sometimes consists in refusing to think for oneself.”[19] A world
that checked everything would stop moving. The map asks for less:
spend your checks where they change who can act.

Second: checks can become empty habits. A receipt that nobody reads
is just a seat with extra steps, a test suite that always passes
stops carrying information, and clicking approve every ninety seconds
is watching in name only. This is why the fourth rule of a real check
matters so much: the check itself can be checked. The elevator did
not earn trust with a hidden brake. It earned trust with a stop
button that any rider could press.

Third: some seats should never become empty, and pretending otherwise
helps nobody. On ground with no answer key, a machine’s confidence is
only a sound; we wrote a whole essay about that ground, and it is
called *Firm ground*. Putting a fake check on top of a human judgment
is the same mistake as putting a person where only a check can act.
The same mistake, mirrored.

## What we know, and what we bet

An essay that asks systems to show receipts should show its own.

What we know: human attention fails within the hour next to a quiet
machine, and telling people to stay alert does not fix it.[8] A
weaker checker can hold a stronger worker to the truth, and this is
proven mathematics.[10] You can only delegate what you can see.[11]
Three ordinary checks in a row copy a genome with about one error in
ten billion letters.[12] The elevator seat stayed full for ninety
years after the brake worked, and became empty only when the receipts
arrived.[16] **And a seat without a check protects nobody. It only
collects the blame.**[9]

What we bet: that courts and insurance companies will slowly turn
“human oversight,” the duty to watch, into named checks with numbers
attached, because nothing else survives in court. That the agent
economy will split along one line: what can be checked, and what
cannot. The checkable work will go quiet first, and the judgment work
will stay with people. And that the fastest road to safe autonomy, in
any field, is not a better seat but a better check, with receipts. We
are not neutral here. The third bet is the one our company is built
on.

## What Otis actually invented

Otis did not invent the elevator; lifting platforms were already for
sale. He invented a safe way to fail. He stood on the platform, made
the thing everyone feared happen on purpose, and let the brake answer
in front of the crowd. Every elevator you rode this year is safe in
that same way: nobody is watching it, the machinery that catches
it is built in, and that machinery was shown working before anyone
was asked to trust it.

The same test works on any system that has a person sitting next to
it. Three questions decide it.

1. **What is the check?** The machinery that catches a failure,
   without anyone watching for it.
2. **Who can check the check?**
3. **Where is the receipt?** The record that proves the check ran.

If these three questions have answers, the seat can become empty,
safely. **If they have no answers, the seat is not protecting anyone.
It is waiting for someone to blame.**

---

## Sidenotes

[1] Otis company history. The company’s own telling: the cut, the
brake, eight elevators sold in 1854.
[2] Bernard, *Lifted* (2014). Two papers noticed; the famous line is
tradition, the cut is documented.
[3] Human Rights Watch, *Losing Humanity* (2012). The three seats,
born with their own warning.
[4] DoD Directive 3000.09 (2012, updated 2023). The word “loop” does
not appear in it.
[5] Wu et al. (2022). In machine learning the phrase names training
labor, not runtime control.
[6] Regulation (EU) 2024/1689, article 14. A duty to watch, assigned
to no particular seat.
[7] NTSB (2019). The Tempe record: the timeline, the silent second,
the causes.
[8] Bainbridge (1983). Half an hour is the limit; skills fade exactly
when automation works.
[9] Elish (2019). The moral crumple zone.
[10] Goldwasser, Micali and Rackoff (1989; introduced 1985).
Interactive proofs: a weak verifier can hold a strong prover to the
truth.
[11] Holmstrom (1979). Contracts reach only as far as observation.
[12] Kunkel (2004). Three serial checks; roughly one error in ten
billion letters.
[13] Sheridan and Verplank (1978). Level six: acts unless the human
stops it in time.
[14] Abramoff et al. (2018). The first FDA-authorized autonomous
diagnosis, earned by a prospective trial.
[15] Mitchell, Ghosh and Passi (2026). Bainbridge, restated for
agents.
[16] NPR (2015). Lee Gray, the 1945 strike, and the recorded voice.
[17] Leveson and Turner (1993). Therac-25: the hardware safety locks
were the difference.
[18] SEC (2013). Knight Capital: 45 minutes, one unreviewed deploy,
over $460 million.
[19] Hardwig (1985). Epistemic dependence; deference can be rational.

## References

1. Otis Worldwide. “Our history.”
   https://www.otis.com/en/us/our-company/history
2. Bernard, A. *Lifted: A Cultural History of the Elevator.* New York
   University Press, 2014.
3. Human Rights Watch. *Losing Humanity: The Case Against Killer
   Robots.* 2012.
   https://www.hrw.org/report/2012/11/19/losing-humanity/case-against-killer-robots
4. U.S. Department of Defense. Directive 3000.09, Autonomy in Weapon
   Systems. 2023.
   https://www.esd.whs.mil/Portals/54/Documents/DD/issuances/dodd/300009p.pdf
5. Wu, X., Xiao, L., Sun, Y., Zhang, J., Ma, T., and He, L. “A survey
   of human-in-the-loop for machine learning.” *Future Generation
   Computer Systems,* 135, 364–381, 2022.
   https://doi.org/10.1016/j.future.2022.05.014
6. Regulation (EU) 2024/1689 (AI Act), Article 14, “Human oversight.”
   2024. https://eur-lex.europa.eu/eli/reg/2024/1689/eng
7. National Transportation Safety Board. *Collision Between Vehicle
   Controlled by Developmental Automated Driving System and
   Pedestrian, Tempe, Arizona, March 18, 2018.* Highway Accident
   Report NTSB/HAR-19/03, 2019.
   https://www.ntsb.gov/investigations/AccidentReports/Reports/HAR1903.pdf
8. Bainbridge, L. “Ironies of Automation.” *Automatica,* 19(6),
   775–779, 1983. https://doi.org/10.1016/0005-1098(83)90046-8
9. Elish, M. C. “Moral Crumple Zones: Cautionary Tales in Human-Robot
   Interaction.” *Engaging Science, Technology, and Society,* 5,
   40–60, 2019. https://doi.org/10.17351/ests2019.260
10. Goldwasser, S., Micali, S., and Rackoff, C. “The Knowledge
    Complexity of Interactive Proof Systems.” *SIAM Journal on
    Computing,* 18(1), 186–208, 1989. https://doi.org/10.1137/0218012
11. Holmstrom, B. “Moral Hazard and Observability.” *The Bell Journal
    of Economics,* 10(1), 74–91, 1979. https://doi.org/10.2307/3003320
12. Kunkel, T. A. “DNA Replication Fidelity.” *Journal of Biological
    Chemistry,* 279(17), 16895–16898, 2004.
    https://doi.org/10.1074/jbc.R400006200
13. Sheridan, T. B., and Verplank, W. L. *Human and Computer Control
    of Undersea Teleoperators.* MIT Man-Machine Systems Laboratory,
    1978. https://apps.dtic.mil/sti/citations/tr/ADA057655
14. Abramoff, M. D., Lavin, P. T., Birch, M., Shah, N., and Folk,
    J. C. “Pivotal trial of an autonomous AI-based diagnostic system
    for detection of diabetic retinopathy in primary care offices.”
    *npj Digital Medicine,* 1, 39, 2018.
    https://doi.org/10.1038/s41746-018-0040-6
15. Mitchell, M., Ghosh, A., and Passi, S. “AI Agents Push Humans Out
    of the Loop.” 2026. https://arxiv.org/abs/2608.23642
16. NPR, Planet Money. “Remembering When Driverless Elevators Drew
    Skepticism.” July 31, 2015.
    https://www.npr.org/2015/07/31/427990392/remembering-when-driverless-elevators-drew-skepticism
17. Leveson, N. G., and Turner, C. S. “An investigation of the
    Therac-25 accidents.” *Computer,* 26(7), 18–41, 1993.
    https://doi.org/10.1109/MC.1993.274940
18. U.S. Securities and Exchange Commission. In the Matter of Knight
    Capital Americas LLC, Administrative Proceeding File No. 3-15570,
    2013. https://www.sec.gov/litigation/admin/2013/34-70694.pdf
19. Hardwig, J. “Epistemic Dependence.” *The Journal of Philosophy,*
    82(7), 335–349, 1985. https://www.jstor.org/stable/2026523