The empty seat

In the spring of 1854, at a fair in New York, a mechanic named Elisha Otis stood on an open platform. A single rope held the platform, and a machine pulled it high above the crowd. Then Otis gave the order, and a man cut the rope with an axe.

The platform fell a few inches, and then it stopped. Here is why. On top of the platform sat a wide steel spring, and the rope held it bent. The side rails of the platform had teeth, like a saw. The moment the rope was cut, the spring opened, its two ends pushed into the teeth, and the platform hung there, safe.

The elevator company still tells this story today. In the company’s version, Otis looks at the crowd and says: “All safe, gentlemen, all safe.”[1]1Otis company history. The company’s own telling: the cut, the brake, eight elevators sold in 1854. The real records from 1854 say less. Only two newspapers wrote about the show at all, and the longest report was a few words in the New York Daily Tribune: the inventor “occasionally cuts the rope by which it is supported.”[2]2Bernard, Lifted (2014). Two papers noticed; the famous line is tradition, the cut is documented. So the famous words may have been added later. The cut really happened.

And the cut is what matters. What Otis showed that day was not a machine that goes up; lifting machines had existed for a long time. He showed something new: a machine that stays safe when its rope breaks. He did not ask the crowd to trust him.

He made the failure happen himself, in public, and let the brake answer.

Here is the strange part. The brake worked in 1854, but for almost a hundred years after that day, every elevator still had a human operator inside. The operator stood at the controls all day: he drove the car up and down, he opened and closed the doors, and the riders trusted him. The dangerous part was solved in 1854. The seat stayed full until about 1950. This essay is about that seat: who ends up sitting in it, what the seat is really for, and what finally lets it become empty.

The seats

Today, people argue about machines by arguing about seats.

1. Should a human be in the loop? That means: the machine cannot act until a person says YES.
2. Should the human be on the loop? That means: the machine acts on its own, and a person can stop it.
3. Or should the human be out of the loop, with the machine fully alone?

These words sound technical. Their history tells a different story.

The three phrases come from a 2012 report called Losing Humanity, written by people who wanted to ban weapons that choose targets on their own.[3]3Human Rights Watch, Losing Humanity (2012). The three seats, born with their own warning. And the same report, on the same pages, gave a warning that almost everyone has forgotten. Some weapons keep a human on the loop, the report said, but they are “effectively out-of-the-loop weapons because the supervision is so limited.” In plain words: the middle seat often does not work, and the warning was there from day one.

Two days after that report, the United States military published its own rule for weapons that act on their own. The word “loop” does not appear in that rule. Not once. The rule asks for something else: “appropriate levels of human judgment.”[4]4DoD Directive 3000.09 (2012, updated 2023). The word “loop” does not appear in it.

And the phrase keeps changing its meaning. In weapons, it means the machine cannot fire without a human command. In machine learning, it means something completely different: the people who label training data so that the next model becomes better, and who work months before any real decision.[5]5Wu et al. (2022). In machine learning the phrase names training labor, not runtime control. In Europe’s new AI law, it means a watcher: someone must watch the system, but the law does not say from which seat.[6]6Regulation (EU) 2024/1689, article 14. A duty to watch, assigned to no particular seat. One phrase, three different jobs. People argue for hours about which seat the human should hold.

Almost nobody asks what is under the seat.

The seat that could not act

At 9:58 on a Sunday night in March 2018, a self-driving test car drove through Tempe, Arizona. One person sat inside, in the operator’s seat, and she was looking down, toward the screen between the seats. Outside, a woman was pushing a bicycle across the road.

The car’s software saw her 5.6 seconds before impact, but it never understood what it saw. It called her a vehicle, then an unknown object, then a bicycle, and with each new name, it threw away the path it had calculated for the old name.[7]7NTSB (2019). The Tempe record: the timeline, the silent second, the causes. With 1.2 seconds left, the software decided that the car must brake hard. And then it did nothing for one full second. The pause was part of the design: the system used that second to check its own alarm, because the designers were afraid of false alarms. During that second, no light came on and no sound played. The car’s own emergency brake was turned off, because the company had disabled it for testing. A second crew member was gone, because the company had removed that seat months earlier. The operator looked up with about one second left.

The woman, Elaine Herzberg, died. The investigators found causes everywhere: the software that could not name her, the silent second, the disabled emergency brake, the removed second seat, the company’s weak safety culture, and the state’s weak rules.[7]7NTSB (2019). The Tempe record: the timeline, the silent second, the causes. The company was never charged with a crime. The operator was. She admitted guilt to a charge called endangerment and received three years of probation. A whole system had failed, and one person took the blame.

It is easy to think this was one bad company. The research says something harder: the seat itself was the problem, because a human cannot watch a machine that is almost always right. The classic paper on this, from 1983, says it directly: “it is impossible for even a highly motivated human being to maintain effective visual attention towards a source of information on which very little happens, for more than about half an hour.”[8]8Bainbridge (1983). Half an hour is the limit; skills fade exactly when automation works. There is a second problem, too. When a machine does the work, the human’s own skill slowly disappears, so the better the machine becomes, the less able the person in the seat is to take over. The investigators wrote that the operator’s attention drifted, and that this is normal next to a machine that almost always works.[7]7NTSB (2019). The Tempe record: the timeline, the silent second, the causes. It is not a character flaw. It is how attention works.

Now remember the elevator. The Tempe car had no spring and no teeth, no machinery that would answer when the software failed. The seat itself was asked to be the brake, and a person cannot be a brake. When such a seat fails, it still has one job left. The anthropologist Madeleine Clare Elish studied where blame lands after automation accidents, and she gave this job its true name: the moral crumple zone. In a car, the crumple zone is the part that folds in a crash, so that it absorbs the impact.[9]9Elish (2019). The moral crumple zone.

A human placed where they cannot act does not protect anyone. They are a place for blame to land.

So if the seat is not what keeps us safe, what is? In the elevator, the brake is. And machines like that brake have a general name.

The check

A check confirms that work is correct, without doing the work again and without trusting the worker. The brake lifts nothing. It answers only one question: is the rope still holding? And it answers with steel, not with attention. A good check also leaves something behind: a receipt, a visible record that says the check ran, and here is what it found. The crowd at the fair received their receipt through their own eyes: they saw the platform stop.

Four things make a check real.

1. The check is independent from the thing it checks.
2. The check costs less than doing the work again.
3. The check comes before the harm, not after.
4. The check itself can be checked.

When all four hold, something surprising follows: the checker does not need to be as smart as the worker.

That last sentence is not a hope; for many kinds of claims, it is a mathematical fact. In 1985, three cryptographers described something they called an interactive proof, and it works like this: a checker with an ordinary computer asks random questions, while the one being checked may have unlimited computing power. After a short exchange of questions and answers, the checker can be as certain as it wants that the answer is right.[10]10Goldwasser, Micali and Rackoff (1989; introduced 1985). Interactive proofs: a weak verifier can hold a strong prover to the truth. The other side can be far stronger; the check does not care. If you worry that machines will soon be too smart for us to check, remember this result. Checking can be easier than doing. This is proven mathematics.

Economics found the same idea from the other side. In 1979, the economist Bengt Holmstrom studied a simple situation: you hire someone, but you cannot watch them work. His answer: a contract can only be built on what you can see, and any signal about the work, even a weak one, is worth paying for.[11]11Holmstrom (1979). Contracts reach only as far as observation. What you can see decides what you can delegate. A receipt is not paperwork; a receipt is what makes delegation possible at all.

Nature answered the same question long before mathematics or economics existed. Your cells copy DNA with about one error in ten billion letters, but the copying machine alone is nowhere near that good: by itself, it writes a wrong letter about once in a hundred thousand. So behind it, a proofreader rereads the fresh copy and removes most of the mistakes, and behind that, a repair crew catches most of what is left. Three ordinary checks, one after another, multiply into near-perfection.[12]12Kunkel (2004). Three serial checks; roughly one error in ten billion letters. The cell does not trust its own copier. And notice: no one is watching.

Mathematics, economics, biology. Three fields, one idea: the seat is the symptom, and the check is the cause. Where a real check exists, work runs without a watcher; where no check exists, a person gets pulled in to sit next to the machine, no matter what the diagram calls their seat. So stop asking about the seat, and ask the older question instead: what is the check? That question has a map.

The map

Ask two questions about any task, and notice that neither question is about the machine. First: can the work be checked against something outside the worker? Second: if the work is wrong, can it be undone? Two questions, four possible answers, four kinds of ground.

The work can be checked, and mistakes can be undone: let it run, and read the receipts. An agent writes code in a project with a real test suite: every change is checked in seconds, by machinery, and every change can be undone. The honest seat here stays empty most of the day. We think this is why coding agents arrived before all other agents: their checks were already built and waiting.

The work can be checked, but mistakes cannot be undone: the act waits for the receipt. A payment, a deployment, a medical diagnosis. The machine may act, but only through the check: no receipt, no act. And notice what does not count as a check: a countdown. In 1978, two researchers at MIT wrote down ten levels of automation for the United States Navy, and level six was the machine that acts unless a human stops it in time.[13]13Sheridan and Verplank (1978). Level six: acts unless the human stops it in time. Every auto-accept countdown sold today is level six, a dialog that approves itself when the timer runs out. The countdown stands in the place where a check should stand.

Medicine, of all fields, shows the honest version. In 2018, the American medical regulator allowed, for the first time, an AI system to diagnose a disease with no doctor looking at the image. The disease is diabetic retinopathy, an eye disease found in photographs of the back of the eye, in ordinary clinics.[14]14Abramoff et al. (2018). The first FDA-authorized autonomous diagnosis, earned by a prospective trial. Nobody argued this autonomy into existence. It was earned in a trial at ten clinics with nine hundred patients, where the machine’s answers were checked against the diagnosis of an expert center. An empty seat, in the most careful field there is, because someone built the check first.

The work cannot be checked, but mistakes can be undone: the person is the check. Taste, tone, drafts, judgment. These have no answer key, and they never will, because the answer is a comparison to a standard that somebody chose. Here the human does not watch the machine work; the human is the answer key, and the machine comes to ask. Keep that person in, pay them well, and stop pretending a machine could take that seat.

The work cannot be checked, and mistakes cannot be undone: do not automate this. First make the work checkable, and then automate it, or keep the act human. There is no third option that survives its first accident.

Now put this map next to the law. Europe’s AI law requires the people watching high-risk systems to “remain aware of the possible tendency of automatically relying or over-relying on the output.”[6]6Regulation (EU) 2024/1689, article 14. A duty to watch, assigned to no particular seat. The goal is right, but the tool is wrong: the law orders a state of mind, and decades of research say this state of mind fades exactly when the system works well.[8]8Bainbridge (1983). Half an hour is the limit; skills fade exactly when automation works. A law can demand awareness. Only a check can replace it.

The agent’s seat

For decades, this was a question for pilots and control rooms. Then AI agents arrived, and now anyone who runs an agent sits next to a machine that acts on its own. That is the seat this essay has been looking at.

It is worth going slowly here, because this mechanism decides your working day. Give an agent a code project with tests, and its work lands on the best ground of the map: checked in seconds, undone with one command. Your seat stays empty for hours, and your job becomes reading receipts and making the judgment calls that have no answer key. Now take the tests away. Nothing about the agent changed, but the only check left is your attention, so the work pulls you back, approval by approval, into the watcher’s seat. The same seat as in Tempe, moved to a desk.

Be careful, too, what you read from that seat. A transcript is what the agent said. A trace is what the agent did. A transcript reads well even when the work went badly, because models are trained to write well. The seat survives on traces: the recorded chain from goal to action to result, checkable without reading anyone’s prose again.

Researchers recently repeated the 1983 warning, this time for agents: if nothing changes, they wrote, the old problem “will play out at scale, compounding across every domain agents are deployed into.”[15]15Mitchell, Ghosh and Passi (2026). Bainbridge, restated for agents. The way out is not a stronger seat. The way out is one page, agreed before the work starts.

1. What may the agent do without asking? Exactly the work that can be checked and undone.
2. What needs a human signature? Exactly the acts that cannot be undone, and the questions that have no answer key.
3. What is logged? Everything, as a trace.

We run our own work this way, and we learned it the hard way. The agents that build this website work under a written contract, and every rule in it was added after a surprise. One rule says that the final look of every page must be checked on real hardware. Why? Because a testing browser once reported our pages as fine, while a real browser, on a real machine, showed the headline at size zero. We added that check the same day, and the seat has been quieter since.

The trust machinery

One question is still open, and the elevator answers it. The brake worked in 1854. Why did the seat stay full for ninety more years?

Because a check that nobody can see frees nobody. Automatic elevators, with buttons and no operator, existed around 1900, and riders refused to use them. “People walked in and looked and walked right back out,” says the elevator historian Lee Gray.[16]16NPR (2015). Lee Gray, the 1945 strike, and the recorded voice. The machinery was fine; the receipt was missing. Then, in September 1945, the elevator operators of New York went on strike. The city lost a hundred million dollars in taxes, and one and a half million office workers stayed home.[16]16NPR (2015). Lee Gray, the 1945 strike, and the recorded voice. After that, building owners wanted automatic elevators badly. But riders still had to be convinced.

So the industry built machines for the one thing the operator had really provided, and none of it was about lifting. An emergency stop button. A telephone. Advertisements that showed children pushing the buttons. And a recorded voice, as calm as any human operator: “This is an automatic elevator. Please press the button for the floor you desire.”[16]16NPR (2015). Lee Gray, the 1945 strike, and the recorded voice. From invention to comfort, the road took more than fifty years. The seat became empty when the check finally had receipts that a rider could see.

When someone breaks this rule, you get the disasters. The Therac-25 was a radiation therapy machine. The model before it had hardware safety locks, physical parts that stopped the beam no matter what the software said, and the same software bugs lived in both machines. In the old machine, the locks caught the bugs and nobody was hurt. In the new machine, which trusted its software, the locks were removed, and between 1985 and 1987 six patients received massive overdoses. Some of them died.[17]17Leveson and Turner (1993). Therac-25: the hardware safety locks were the difference. The bug was old. Only the missing check was new.

And one summer morning in 2012, a trading company called Knight Capital turned on new software. The update had reached seven of its eight servers; one server still ran old, dead code, and no second person had checked the update. In 45 minutes, the system sent millions of orders into the market and lost over 460 million dollars.[18]18SEC (2013). Knight Capital: 45 minutes, one unreviewed deploy, over $460 million. No human was in that loop. But nothing else was in it, either.

Seats empty at the speed of trust, and trust moves at the speed of receipts.

Where the check ends

An essay about checking must also tell you where checking ends.

First: most of what you know, you never checked, and you never will. You believe the mass of the electron without measuring it yourself, and the philosopher John Hardwig argued that this is not a weakness: you can have good reasons to believe something, because you have good reasons to believe that other people checked it, and “rationality sometimes consists in refusing to think for oneself.”[19]19Hardwig (1985). Epistemic dependence; deference can be rational. A world that checked everything would stop moving. The map asks for less: spend your checks where they change who can act.

Second: checks can become empty habits. A receipt that nobody reads is just a seat with extra steps, a test suite that always passes stops carrying information, and clicking approve every ninety seconds is watching in name only. This is why the fourth rule of a real check matters so much: the check itself can be checked. The elevator did not earn trust with a hidden brake. It earned trust with a stop button that any rider could press.

Third: some seats should never become empty, and pretending otherwise helps nobody. On ground with no answer key, a machine’s confidence is only a sound; we wrote a whole essay about that ground, and it is called Firm ground. Putting a fake check on top of a human judgment is the same mistake as putting a person where only a check can act. The same mistake, mirrored.

What we know, and what we bet

An essay that asks systems to show receipts should show its own.

What we know: human attention fails within the hour next to a quiet machine, and telling people to stay alert does not fix it.[8]8Bainbridge (1983). Half an hour is the limit; skills fade exactly when automation works. A weaker checker can hold a stronger worker to the truth, and this is proven mathematics.[10]10Goldwasser, Micali and Rackoff (1989; introduced 1985). Interactive proofs: a weak verifier can hold a strong prover to the truth. You can only delegate what you can see.[11]11Holmstrom (1979). Contracts reach only as far as observation. Three ordinary checks in a row copy a genome with about one error in ten billion letters.[12]12Kunkel (2004). Three serial checks; roughly one error in ten billion letters. The elevator seat stayed full for ninety years after the brake worked, and became empty only when the receipts arrived.[16]16NPR (2015). Lee Gray, the 1945 strike, and the recorded voice. And a seat without a check protects nobody. It only collects the blame.[9]9Elish (2019). The moral crumple zone.

What we bet: that courts and insurance companies will slowly turn “human oversight,” the duty to watch, into named checks with numbers attached, because nothing else survives in court. That the agent economy will split along one line: what can be checked, and what cannot. The checkable work will go quiet first, and the judgment work will stay with people. And that the fastest road to safe autonomy, in any field, is not a better seat but a better check, with receipts. We are not neutral here. The third bet is the one our company is built on.

What Otis actually invented

Otis did not invent the elevator; lifting platforms were already for sale. He invented a safe way to fail. He stood on the platform, made the thing everyone feared happen on purpose, and let the brake answer in front of the crowd. Every elevator you rode this year is safe in that same way: nobody is watching it, the machinery that catches it is built in, and that machinery was shown working before anyone was asked to trust it.

The same test works on any system that has a person sitting next to it. Three questions decide it.

1. What is the check? The machinery that catches a failure, without anyone watching for it.
2. Who can check the check?
3. Where is the receipt? The record that proves the check ran.

If these three questions have answers, the seat can become empty, safely. If they have no answers, the seat is not protecting anyone. It is waiting for someone to blame.