What's Left For Us

You are the dataset

Two weeks before the keynote, a short Google Form went out to the room: what's one task you wish AI would take off your plate, and what's actually stopping you? 88 people answered. Nobody in the room knew, walking in, that they were about to become the data.

"You were the data set, by the way. It turned out that the majority of you said, 'I'm stopped because I have to check the results.'"

— opening the talk, reading the room's own survey back to it

The instinct going in was that the blocker would be a skills gap — people not knowing how to prompt, not trusting themselves to delegate. The data said something narrower and more specific. Fifty-one of 88 people said the same thing in different words: AI gets it wrong, and I still have to check. Even among the 22 respondents who work in Data / AI professionally — the people who least need an excuse — zero said "I don't know how." Eighteen of them still cited exactly this.

One colleague, describing the same thing from the other side, had a name for it:

"One of my colleagues called it AI Fatigue. Now we're having to change the way in which we do our job, and that's a problem."

It's a strange kind of fatigue. You ask AI to do the work because it's faster than doing it yourself. It does the work — fast, cheap, tirelessly. And the work piles up on the other side, because now someone has to check all of it, and nobody built the checking muscle. The bottleneck didn't disappear when execution got automated. It moved one step downstream, to verification — and verification is a skill almost nobody has had to practise at this volume before.

The verification tax

So: how does one actually deal with AI fatigue? In the spirit of the whole talk, the answer didn't come from a framework. It came from asking ChatGPT to go through a month of its own conversation history and report back on what it had watched Anand actually do.

"I asked ChatGPT: 'Go through as many conversations as you can in the month of August and tell me how I deal with verification.' And this is what it said I'm doing. Here are the — I don't know how many — 18 ways, prioritized."

The top four, in order:

The checklist, worked live on a contract

"Contracts need to be reviewed against a checklist of about 21 things. Take a checklist, go through it, see if the contract has it, yes or no, and provide some evidence."

For a publisher–author contract, that means: is intellectual property mentioned? Is force majeure covered? Are there hidden costs? The model doesn't have to judge the contract — it just has to point at Section 2.2 and say, in effect, "quality standards, covered, here's the sentence." That's not verification-free — but it collapses a 40-page read into 21 cited yes/no answers, fast enough to hand to another person, or another agent, for a second pass.

The cross-checking numbers

This is where the technique gets genuinely striking. Classifying customer chat messages ("could I take a quick look at my invoice?") into the right support category, cheap models made mistakes about 14% of the time on their own — but their mistakes weren't correlated. One model's blind spot wasn't another's:

"If we had five models cross-check and said 'I will only allow if all five models agree,' the error rate drops to 0.7% — that's a 72% effort saving at 99.3% quality."

The full methodology — 11 frontier models tested on Kaggle's Customer Support Intent Dataset, including which model tried to file "a quick look at my invoice" as a change order — is worked out on LLM Mental Math ↗. The finding underneath it: model errors correlate only 10–30% with each other — low enough that cross-checking is genuinely cheap insurance.

None of these four are new inventions. They are, mostly, management techniques:

"Models cannot be relied upon — but humans can't be relied upon either, and we have millennia of experience of dealing with unreliable humans. Many of those techniques transfer to models as well. We have to manage around it. That's literally what management is about."

Who gets the final call?

The survey also asked something harder to answer honestly: imagine AI became 10× more reliable and could securely touch every system you use. What would you still keep for yourself? 75 of 87 people said they'd still keep something. Stable across students (81%), non-students (89%), and the Data/AI crowd (91%) — this wasn't a confidence problem. It held even when the hypothetical erased the confidence problem entirely.

"One of the biggest things was taking the final call. Some people said 'the people part,' 'the thinking.' Some people said 'nothing, just take it all.' That's fair, too."

So the same exercise ran again — a month of Anand's own conversations, mined for how decisions actually got delegated in practice. The answer wasn't one policy. It was a ladder:

1. Directing — I choose the method, AI executes.
2. AI recommends, I decide — it proposes the method, I pick.
3. Draft only, never send — the most common pattern by far. AI does all the work, takes no action; the final call stays mine.
4. Work independently, but interrupt me — ask one clarifying question, or flag it for a live discussion, when unsure.
5. Test-gated autonomy — "you test it, and as long as your tests pass, I'm fine." The third most common pattern.
6. Act directly — rare, but real: production systems updated, purchases made, WhatsApp messages and emails sent, with no review at all.

"There are times when I've sent WhatsApp messages, emails, without review. The alternate was I don't even reply, and my logic is my AI is better than my silence. So let's go with it."

Risk tiering, in other words, isn't a formula you set once. It's a live judgment call, made per task, and the honest answer to "when do you do which" is: it depends, and everyone has to build their own feel for it.

A frontier that keeps moving

There's one thing that makes every judgment call above a moving target, not a settled one: the cost of intelligence is collapsing faster than most people can track it in their head.

"o1-preview, released September '24 — very expensive, $15 per million tokens — was roughly as smart as a master's student. Fast forward to Gemini 1.5 Flash, April 2025, less than a year later: roughly the same level of intelligence, at 15 cents. $15 to 15 cents. Less than a year."

You used to be able to hire a master's-level student, in effect, for $15 a million tokens when o1-preview launched. A year later, Gemini 1.5 Flash's price cut put the same intelligence at 15 cents. If that's the pace at which the cost of intelligence falls, the judgment built to operate at the next level — say, a tenured professor's — depreciates on roughly the same curve. And the level after that. Nobody knows if or when it stops.

The consequence isn't just "things get cheaper." It reshapes where human time actually goes. When execution was slow and expensive, most of a project's time sat inside execution — the doing. As execution collapses toward free, the time has to move to the two edges: deciding what to build before, and checking what came out after.

"Now the execution takes very little time and we have to therefore design what goes in, we have to test what comes out — specification, verification, that sort of a thing. And in the future, that will shrink even further."

Push that forward and the unit of work itself changes shape — from "review this PR" to "evaluate this agentic organization's performance on a virtual market."

Which means the skills built for today's frontier are also on notice. The intuition the room already had — watch what you delegate, watch how much — was the right one. What changes is that what counts as judgment, what counts as a skill worth having, doesn't hold still for very long.

Delegate maximally

Given a moving frontier, a verification tax, and a room full of people each drawing their own line for the final call, there's no universal answer to hand out from a stage. There's just one working strategy, offered without the claim that it's the right one:

"I have only one strategy that I'm using right now — not saying this is the best or even an effective one, but it's working for me — which is delegating maximally. Take whatever I can, give it to AI. It'll fail, it'll mess up at some point. But that's exactly what I need to learn now. Because that's what it can't do. Give it a shot, delegate maximally."

[Applause]