Talks · S Anand SUTD COIL · Verifying AI agents · 17 Nov 2026

SUTD COIL · Tuesday 17 Nov 2026 · Singapore

The agent passed. The test failed. Who verifies the verifier?

An AI agent sat a proctored hiring test while its owner ate an apple, and nobody noticed. From that one test, a class works out how to specify to, review and verify an intelligence smarter than you, and ends up with one rule about logging.

Anand S, LLM Psychologist at Straive, with a room of about 25 people at the Singapore University of Technology and Design · recording 1 h 36 min · this page is written as study notes for the people who were there

Illustration of the hiring-test report: score 52 of 59, SQL solved in 34 seconds against an average of 8 minutes 40 seconds, proctoring violations zero. A PASSED stamp lands, then an apple appears in the webcam frame, and a second stamp reads VERIFIER FAILED. DATA ANALYST · TEST REPORT Score52 / 59 SQL coding34 s average candidate8 m 40 s Data-driven question6 s average candidate36 s Proctoring violations0 PASSED WEBCAM VERIFIER FAILED
Everything green, and everything wrong. The numbers are from the report Anand showed; the apple is from the proctoring snapshots. Illustration.
Listen · full recording · every mm:ss on this page plays from that moment← → 5 s · Shift 30 s
0:00
1:35:42

How to use this page

Read it top to bottom as a story, or jump to any section from the bar at the bottom of the screen. Every 10:26 plays that moment of the recording; ← → then skip 5 seconds, and hovering the player’s bar previews which section is where. Dotted words explain themselves on hover. Green Try this boxes are exercises; red Fact-check boxes correct or extend what was said in the room. Quotes are verbatim from the AI transcript.

Prologue · Software you wrote

01:14to 07:44

How much of your software did you write?

The host introduced the talk with its title question, “When AI writes the code, who checks the AI?”, and guessed what his students would say: “AI will check the AI, right?” 00:00 Anand didn’t answer it. He started with a different one: “what percentage of the software that you use is coded by you?” 01:29

Then he counted hands, narrowing the question each time, and the room thinned out fast.

The hands in the room

Counts as Anand called them out, 02:06–02:57

One person raised a hand for “haven’t raised your hand so far”, “whereas it should have been about 25 hands up.” 02:50 The count for daily use wasn’t called out.

Anand’s own answer is a music player. VLC has no composer column to sort or filter by, which matters if you listen by composer. He asked ChatGPT for a player that did this. ChatGPT’s answer, in his retelling: VLC is open source, “You are a programmer. I can fix programs like crazy. What are you talking about? Just take it, download it, and fix it.” 04:19 So it did: here is his VLC fork with a composer column. He proudly showed it to his daughter.

“She looked at me like I was so dumb. ‘Why didn’t you have it write the entire music player?’”

Anand, on his daughter’s review of his VLC fork · 05:02

Now he has his own player, synced between phone and laptop, with a right-click “Find me 10 similar next songs”, which works as a personal recommendation engine. The day before the talk he was “hacking Gmail”: an extension where Alt+Shift+M attaches a note to an email, like “ignore for now”, so he doesn’t have to re-read and re-decide the same email every time. 06:35 The software doesn’t even have to be open source. Software is malleable.

He admitted this had “very little to do with the talk”, but it was a check on whether the room was dogfooding agentic coding at all. Then he named the real subject: “what I am going to talk about is how one goes about verifying AI agents.” 07:19 Hold on to the first question, though. It comes back at the end.

Act I · The test that failed

07:44to 11:50

Why is he eating an apple?

Straive recruits analysts using an online testing platform, iMocha, with a question bank for every skill. Anand took the data-analyst test himself. His score: 52 out of 59. He asked the room whether Straive would have hired him. A few said yes; “half a hand” said no. 09:01

Two webcam proctoring snapshots of a grey-haired man sitting in front of his laptop, eating an apple. Nobody else is in the frame.
The proctoring snapshots. From Using agents to answer exams.

The detail was even better. A SQL coding question that takes the average candidate 8 minutes 40 seconds took him 34 seconds. A data question that takes others 36 seconds took him about 6. And the proctoring report, which photographs the candidate through the webcam to check that nobody is helping, showed zero violations. 09:47

“Kind of looks fine, right? Except, how come I’m eating an apple in the middle of a test?” 10:11

Audience“The agent is writing code for you.”
Anand“Because my agent is writing my code for me. Exactly.” 10:26

The agent was Codex, driving the browser, running a deliberately cheap model, “GPT-5.6 Luna medium. … I’m not spending tokens on something as basic as this, which is incidentally why it got some of the spatial questions wrong.” 11:04 The video he played was real time.

Watch it happen

Codex sitting the practice test, in real time

Codex GPT 5 6 Luna solving an iMocha Practice Test on YouTube. The test runs on iMocha’s test platform; the agent controls the browser over the Chrome DevTools Protocol.

“If an agent is not taking your test, then you are wasting your time. If an agent is able to solve your test, you have to ask your professor, ‘What on Earth are you teaching me when my agent can do it?’”

Anand · 10:43

11:50to 14:27

The answer key was wrong

Anand sent the run to the team that maintains the questions, with two questions. The first: why does your question bank have errors? One question asked how to filter a Pandas table. The agent used DataFrame.query(), a perfectly valid method. Anand ran it by hand afterwards; it worked. The test marked it wrong. 11:50

“The question bank is supposed to be the source of truth, right? … and there is a mistake.”

Anand · 12:44

That is the talk’s central question in its smallest form: “Now, who verifies the verifier?” 12:37 A test is a verifier. It decides whether a candidate (human or agent) is right. If the verifier itself is wrong, every score downstream inherits the error, and nothing looks red.

Fact-check · more from the blog post

The Pandas question wasn’t the only one. Anand’s write-up of the run lists a compound-interest question where none of the four options was correct, a logic question about robins and sparrows that asked for a conclusion about finches that didn’t follow from the facts, and a SQL join that the test accepted although it could assign an old salary to the wrong department. Lesson from the post: “Question banks have errors. Agents can cheaply check quality.”

The second question he put to the room. Given that Codex solved the exam, should Straive have hired him? Four hands said yes, it doesn’t matter who solves it. About six said no, “and the hands definitely went up faster.” 13:25 Then he sharpened it: human A is smart enough to use an agent; human B isn’t using one. Who do you hire? 13:46

Audience“It depends on the role, right? … if the job process is about that, they’re supposed to use the AI agent to do something, then of course you hire someone, the other one. You might want someone to actually might use AI agents to start with, but you also want that person to be able to critically analyze the results, or troubleshoot things in the absence of AI.” 14:31
Anand“Good point. So, the answer is not necessarily a ‘yes’ or a ‘no’, the answer is, ‘It depends on the role.’ Fair.” 14:22
Try this

Take the last test or assignment you sat. Could an agent pass it? If yes, what is it actually measuring, and what would you change so it measures the thing the examiner cares about? Anand’s own suggestions, from the blog: ask candidates to use agents well on a hard task, and to verify agent output seeded with errors “ranging from blatant to subtle”.

The slide that came next

Anand makes his section slides with his own slide tool, a one-page title-slide maker. The talk opened with another one: “What % of software you use is coded by you?”

14:27to 19:54

What did the test-makers get wrong?

Anand laid out the charge: the test got a question wrong, and it didn’t detect the agent. “Eating an apple during the quiz apparently is not a violation, though it indicates several, several other things.” 15:01 What went wrong in verifying the verifier? The room offered answers, and each is worth keeping.

“How do you know what your assumptions are?”

Anand · 18:00

We assume dozens of things we never state, he said: that the world will exist tomorrow, that no robot will eat us. “Just like I kind of assuming that it’s not like there’s suddenly going to be an intelligence that’s as smart as a data analyst that’s going to come and take my test tomorrow. Except that it kind of happened.” 18:00 So how do you surface an assumption you don’t know you’re making?

Audience“Have another agent check.” [laughter] 18:53
Anand“That is actually a fantastic idea. … If you literally ask, ‘What am I assuming that I’m not even aware of? What are my blind spots?’ It can do a pretty good job.”

The laugh was because it sounded like the lazy answer the host predicted. It isn’t. It is the same move as asking a friend, a colleague or a professor to double-check, and it returns later in the talk with numbers. Anand added his own human version: he keeps a list of the feedback he ignores, because “if I’m ignoring it, it’s a blind spot.” 19:34

He then sorted the room’s answers into three buckets. 19:54 They organise the rest of the talk, and are worth learning as a checklist of their own.

Specify

Before the work: what does “right” mean?

  • Incomplete spec (“no eating”)
  • Unstated assumptions (“no one uses an agent”)

Verify

After the work: is the result right?

  • A wrong answer key
  • Validation that never met an agent

Review

During the work: the system, the environment

  • Webcam proctoring that sees only a face
  • An exam room that allows agents

The three buckets are Anand’s; which answer goes in which bucket is this page’s sorting.

Act II · Someone smarter than you

19:54to 24:25

How do you check someone smarter than you?

Agents are smarter than us in some ways and nowhere near as smart in others. Anand drew on Ethan Mollick’s idea of a jagged frontier: picture human ability as a circle and AI as a spiky blob growing around it. Today the blob pokes past the circle in many places but not most. 19:54 Whether it ever swallows the circle (“AGI, ASI, whatever you want to call it”) he set aside: “I’m less worried about the future. I’m more curious about the present.”

The picture he drew

The jagged edge of AI

The circle is human-level ability; the blob is AI. Press play and watch where it crosses the circle first. Open the Jagged Edge of AI animation.

“How do you specify to someone smarter than you? How do you verify someone smarter than you? How do you review someone smarter than you?”

Anand · 21:12

“You’ve met people smarter than you,” he said. “How do you specify to them?” 21:27 Two answers came quickly.

Then he took the earlier student’s advice and asked AI. Surely this isn’t the first time people have had to brief someone smarter than them. “You visit a doctor. How on Earth do you give specifications to a doctor when you don’t even know what’s wrong with you?” 24:37

Audience“The doctor asks you questions.” 24:47
Anand“The doctor asks you questions. So, interviewing seems to be a pretty good technique, where, for instance, you would tell an agent, ‘Look, I have no idea what I want. You ask me. Figure out what I want and then solve it.’”

“Interviewing is one of the most effective techniques that I’ve seen, and I learned that from doctors.”

Anand · 25:03

And how does a judge rule on a patent case without understanding the technology? They hear an expert for the plaintiff and an expert for the defendant, and compare. The agent version is one line: “Claude, this is what ChatGPT said. ChatGPT, this is what Claude said. What do you think?” 25:31

26:11to 28:13

Your prompts are a data set

Anand asked several agents two things. How do experts in other professions specify a brief, review work and verify results? And, from the 10,000 or 12,000 prompts he had typed that year, how does he actually do it? 26:11 “None of this is unusual. None of this is non-obvious. Perhaps the only one that may not be obvious is that you can run this technique on yourself.” 27:00

Why are your prompts worth mining? When you tell an agent “No, no, no, that’s not what I want,” you are doing three things at once: 27:18

  1. Stating a preference: I prefer X over Y.
  2. Spotting an error or misalignment in the agent, which is grounds for a fix: not fine-tuning, “but just even creating a skill to resolve it.”
  3. Revealing what you care about, which helps you prioritise and build your own profile of objectives.

“The single most valuable button for me on both Claude and ChatGPT is that export data button.”

Anand · ChatGPT’s export sends him a 1.4 GB archive · 27:54

The result of that mining is an interactive playbook: 22 verification techniques in six categories (Define, Show, Test, Challenge, Protect, Learn), each with prompts he actually typed, plus a second view of how professions verify experts. The talk walks through the ones that worked best on his software work. Explore it below; it is the best study companion to this page.

Study companion

22 ways I verify AI work

Click any card for its description, why it works, and real prompts. Toggle I / pros at the top to switch between Anand’s techniques and how professions do it.

Open 22 Ways I Verify AI Work in a new tab, or the professional view: 31 Ways Professions Verify Expert Work.

28:13to 32:22

Define done, or let the agent define it

The first technique is Define done: say what counts as finished before the work starts. The surprise is how Anand does it. Often he doesn’t define done himself. He tells the agent to. 28:13

Prompts Anand uses · read out at 28:13

“My aim is not to produce a time log, an activity report, blah, blah, blah. … Make sure that your definition of done does not include this.”

“If the task never defined what does not count, I want you to infer two or three tempting false victories, and confirm that the answer is not one.”

“For non-trivial tasks, define what done means and verify before concluding.”

The last line sits in his global agent instructions, used by both Claude Code and Codex.

A false victory is the most common failure: not a wrong answer, but a confident answer to an easier question next door. Why leave the definition to the agent? Because “I have no idea what done means.” He is specifying to an intelligence smarter than him, in areas where he lacks experience. 29:35

“By the time you graduate, most of the stuff you’ll be dealing with is stuff you don’t understand.”

Anand · 30:03

The intern who “was brilliant”

Ankur, Straive’s CEO, needed a demo for an important client in three days and asked Anand for someone experienced. Anand sent Varun, an intern, with these instructions: 30:43

“Varun, look, there’s no way you’ll understand a word of what Ankur is saying. Do not try. He will talk at a very different level. What I want you to do is: one, record the call; second, transcribe it; three, paste it into Claude; four, send him the output. That’s it.”

Anand, to Varun · 30:43

Varun did. Ankur asked for changes; Varun put the new transcript back into Claude and sent the result. That evening, Ankur called Anand: “Where is this guy Varun from? He is brilliant!” 31:21

“Sometimes the smartness is really about getting out of the way.”

Anand · 31:29

But how do you know the output isn’t completely off if you don’t know what it should look like? Varun didn’t. “Varun was using Ankur as the definition of done.” 32:05 The expert who would judge the result was in the loop, and the feedback was fast. Remember this move: when you can’t verify, route the output to someone who can, quickly. It returns twice more.

32:22to 35:58

Spot the difference in the bracket

Anand gave agents an engineering drawing of a forked mounting bracket and asked each to build the 3D CAD model for printing. He showed three renders side by side, Claude’s, Codex’s and the reference, and asked the room to spot a difference. Nobody could. 32:22 Neither could he, until a colleague rotated the model all the way round: “Hey, wait, wait. What’s this little thing that Codex has?” “Oh, okay. Yeah, fine. Ah, that’s a small error, let it be.” 33:38

The real lesson was in the verification. He had run two verifier set-ups:

Option B searched, found the exact part in a FreeCAD benchmark with its ground-truth geometry, and measured. Claude matched almost perfectly. Codex scored only about 31% on geometry, on measures like solid-volume overlap and face-centre alignment that Anand admitted he didn’t fully understand. 34:18 Option A, the visual check he had prescribed, was looking at exactly what had fooled the room.

Rotate it yourself

Claude, Codex, and the original CAD

Drag inside each panel to rotate, scroll to zoom, or press ▶ Play to watch each model being built. Then scroll inside the frame to “The results” and “Why the score can disagree with your eyes”: shared material volume is 99% for Codex, yet its geometry score is 31%.

Forked mounting bracket on the CAD benchmark page. Official overall scores: Claude Opus 5.5 >99.9%, Codex (GPT-6.1 Sol) 47.3%.

“One of the things that I’ve found is that I am the person that’s messing up the verification of the verifier.”

Anand · 35:32

Telling the verifier how to check had capped it at his idea of checking. “There are a series of things that I’m not aware of that I should be looking for, that all I had to do was let it loose and say, ‘You go beyond the limitations of my thinking and figure it out.’” 35:07 Is its answer right? “I don’t, but at least it’s better.” Constraining a verifier is itself an assumption, and like any assumption, it’s worth testing: run the check with and without your constraints and compare.

Look closer

The benchmark page shows something the talk didn’t have time for. The two models share 99% of their solid material with the reference. Codex’s geometry score is low because the grader also scores exact volume and the positions of face centres, and both came out near zero for Codex; a different construction order can split the same surface into different faces. A verifier can be more precise than your eye and still measure something slightly different from what you care about. Test the tester, too.

Act III · Test the tester

35:58to 39:08

Two graders, one mistake caught

The next technique is the one the class had already suggested: Test the tester. Before you trust a test, a judge or a grader, check that it works. Anand was grading a data-analytics case study at IIM Bangalore: eight student groups, each with slides, images and data in his downloads folder. He asked Claude to go through everything and score the presentations. 35:58 It gave the healthcare group 19 out of 20, the co-living group 19, and so on. 36:54

Then he gave the same question to ChatGPT, and pasted ChatGPT’s answer back into Claude: 37:09

Prompt · as Anand read it out

“I want you to fact-check what it said, what you said. Take what’s better, drop what’s worse. If that gives you any new thoughts, go along that direction.”

The same technique, refined, is Ask another AI in the playbook: “Fact-check and critically evaluate yours and theirs - only the parts that’ll change the outcome and where there’s disagreement.”

“I was wrong about Group 8’s contradictory accuracy figures.”

Claude’s first line after comparing notes, as Anand read it · 37:58

Claude had docked Group 8 for quoting 92% in one place and 98% in another. ChatGPT, “which tends to be a little more diligent about some of these things, if not as visually perfect,” had redone the analysis: drop the invalid survey respondents and you get 98%; keep them and you get 92%. Both numbers were correct. 38:18 One grader made a mistake; the other didn’t; together, the mistake vanished. Notice how the error surfaced: as actual numbers from the slides and the survey, side by side, not as a summary. The playbook calls this Show actual text: “You can recognise a wrong example instantly; you can’t recognise a wrong abstraction.” “That’s what I tell my team as well. … compare notes, tell me what you’ve done better, except that agents seem to be doing this a whole lot better.”

Note what the second AI did not do. It didn’t vote. It brought fresh evidence, a recalculation, that settled the disagreement. The playbook card puts it well: a second model is “a disagreement generator, not an oracle”.

39:08to 43:50

When five models agree, let it through

Does this scale? Anand’s team tested it on intent classification of customer-support messages from a Kaggle data set. “Help with adding some items” is a change order; Nova Lite called it placing an order, while GPT-4.1 mini and Gemini 2.5 Flash got it right. 39:21 Overall, single models were about 85% accurate, roughly a 14% error rate.

The key observation: models make different mistakes. Their errors are far from perfectly correlated. Gemma 3, for instance, rarely classified things the way GPT-4.1 nano did. So the rule is simple: if the models agree, accept the answer; if any disagree, send it to a human. 40:47 Slide to add models.

200 support tickets, 1 model

Data: Double Checking Hallucinations
automated, correctmodels disagree: human reviewsautomated, wrong (all agreed, all wrong)
14.1%error rate
0%sent to humans
100%fully automated

The trade-off across all five settings · tap a point

Redrawn from the table on Double Checking Hallucinations. The page’s own chart image plots 6.0% effort for one model; its table says 0%, which is what a single unchecked model implies, so this uses 0%.

With five models, error falls to 0.7% (Anand rounded it to 99.3% accuracy) while humans handle the 28% of tickets where the models disagree. A student asked how effort was measured; that 28% is it. 42:10 “So, I’m reducing the cost 72% at 99.3% accuracy. … For any person on the business operation side, that’s fantastic.” 42:32

Audience“But in this case, your assumption is that an error [inaudible] is not silent, right? So how do you do with silent error? … If they all agree, they are all wrong.” 42:54
Anand“Yeah, no, exactly.”

That is the right objection, and it is exactly the red squares in the grid above. Agreement can’t catch errors the models share. Voting works here because the task has a fixed list of 15–20 categories, so it is unlikely that five different models pick the same wrong one by chance. The source page adds a warning: drop models that are consistently bad (they dropped Gemma 3 27B). For open-ended answers, where everyone can be wrong in the same plausible way, agreement is weaker evidence.

Try this

Next time you use an AI for anything that matters, run it past a second model with Anand’s prompt: fact-check both answers, keep what’s better, drop what’s worse. Note where they disagree, and settle each disagreement with evidence (run it, recalculate it, look it up), not with a vote.

43:50to 50:31

“95% sure” is a claim. Test it.

Another way to check a model is to ask it how sure it is: high, medium, low, or a number like 95%. 43:50 Anand polled the room. When a model says it’s 95% sure, how often is it right? Make your own guess first.

A model says it is 95% sure. It is right…

Poll at 44:34 · data: confidence calibration

In the room, nobody guessed “more than 95%” or “about 95%”. About a dozen hands went to “less”; below that the guesses scattered. “Okay, we’re not sure.” 44:48 The good news: this is measurable. It’s called calibration. When GPT-4.1 nano said 95%, it was right about 77% of the time. 46:01 Both models were mostly overconfident. (The one dot where nano looks underconfident is the anomaly Anand flagged: “I had only 10 requests, all of which happened to be right.”)

A student suggested the obvious fix: use the curve to translate the model’s number into a real one. 47:18 That works, Anand said, with two catches. You must rebuild the curve for every model and every problem. And a calibrated number still doesn’t tell you what to do. “I am 78.3% sure this is correct.” What do you do with that? 48:23

So go back to the person with the problem. A bank already has a team checking classifications by hand. What they want isn’t a probability. It’s a list sorted from least sure to most sure, so the team starts at the bottom and works up as far as their time allows. 49:36 For that, the confidence doesn’t need to be accurate.

“I just need it to be monotonic.”

Anand · meaning: when the model says 95%, is it more often right than when it says 93%? · 50:06

And the curves are, broadly, monotonic: higher stated confidence, more often right. A wrong number can still give a right order, and an order is what a verification checklist needs.

50:31to 55:38

Joey and the marshmallows

There is a better source of confidence than asking. Models generate text one token at a time, and at each step they hold a probability for every possible next token. Many APIs return these as logprobs. When Anand asked how many had ever extracted one, his “Oh.” said enough, and he decided to “step back a bit”. 50:45 So he showed one. The question: “In which episode of Friends did Joey eat too many marshmallows?” 51:34 Click any token in the answer to see what else the model nearly said.

GPT-3.5’s answer, token by token

Data: Visualizing LLM Hallucinations · gpt-3.5-turbo-0613

Darker red = the model was less sure of that token. Click a token.

Check queue, the factual tokens sorted from least to most sure:

    “Jo” got 70%, against 28% for starting with “In”. After “Jo”, “ey” was certain. But “20” was only 45%, with 21, 11 and 13 close behind, and season “3” was 77%, with 7 at 7.7%. 53:09 So if you had to verify this answer, you would check the episode first and the season second. “That gives me a prioritized queue. And that prioritization has cost implications. Because I can decide when to stop at what level of quality.” 54:26

    “But they are amongst the best ways of predicting how good a model is, especially if you just rank-order them.”

    Anand, on logprobs · 54:46
    The twist

    Every check passes. The answer is still wrong.

    Run the check queue in order. Episode 20? Correct: “The One with the Dollhouse” is Season 3, Episode 20. Season 3? Correct. The title? A real episode. Every token the model was unsure about turns out to be right.

    But Joey never ate too many marshmallows. The Dollhouse episode is about Monica’s inherited dollhouse and Phoebe’s homemade one, and the Visualizing LLM Hallucinations page uses this very question because its premise is false. The error sits in the one phrase the model wrote at 100%, “too many marshmallows”, which it copied from the question. A reviewer who works up the queue from the least-sure token would confirm three facts and ship a fiction.

    It isn’t a one-off. In the calibration study, 435 answers had effectively 100% token probability, and 18 of them were wrong.

    Low confidence tells you where to look first. High confidence doesn’t tell you where you can stop.

    That is the apple again: the agent passed, the test failed. Logprobs measure how sure the model is of its next word, not whether the question made sense. Catching a false premise takes a different check, one on the question itself. It is the same lesson as the iMocha question bank in Act I, where the bugs were in the questions, not the answers.

    This, Anand said, is “yet another way of testing the tester itself. Is the validator giving us good results?” 55:28 The calibration study found the same thing on the banking data: raw logprobs are worse as probabilities, but better at ranking which answers are risky.

    Line chart: error among auto-passed answers against the share auto-passed. Ranking by token logprob lets 65% of answers skip review before error passes 5%; ranking by prompted confidence, only 43%.
    Rank by logprob, and more answers can skip review. Trust the highest-scored answers first and keep going until error among them reaches a 5% budget: ranking by token logprob auto-passes 65% of answers, ranking by the model’s stated confidence only 43%. Chart from the calibration study (GPT-5.6 Luna, banking requests). The study’s caveat: pick the cutoff on past cases, then test that exact cutoff on new ones.

    Act IV · When there is no spec

    55:38to 59:04

    How do you verify without a spec?

    “How do you verify your software usually?” A student gave the textbook answer: check against specifications, write automated tests. “How do you know your tests are right?” Benchmark against previous results and industry practice. 56:02 Anand pointed out the consequence: software that already has test cases is easy to clone, because “somebody’s taken the effort of defining the spec for you. And then the verification follows.” 56:27

    So, he asked: does specification matter more than verification? About five hands said yes. When he asked who thought verification mattered more (“if it goes wrong, then who cares how you specified it?”), a student flipped the question instead:

    “How do you verify without specification?”

    Student · 57:07

    “Good question,” said Anand, “and trust me, that happens quite often.” 57:10 Nobody writes a spec for the browser they download, yet they file bug reports. Nobody gives the builder specs for an apartment, yet they complain to the landlord. A student answered: “the user will have a subset of specifications.” 57:50 Some are latent, held in “the shared fabric of society”. Some are blind spots. Many are “I’ll know it when I see it.” 57:59 How do you capture what nobody specified?

    58:43to 63:21

    Users don’t have to be human

    “User testing?” offered a student. 58:43 Share it with 50 or 100 people and collect their inputs. Anand saw a problem with how that works in practice: the testers are usually assigned to test. “That’s basically like giving you homework. And we know exactly how much you enjoy homework.” 59:18

    His daughter built a Telegram bot and asked him to be the user tester. He agreed, set Codex to test it, and watched an episode of Matlock (the 2024 series, on Netflix).

    “I may be a bad parent, but Codex was doing a good job, and it created a fairly good report.”

    Anand · 59:46

    It scales up. Shell told Anand’s team that each survey costs “hundreds of thousands of dollars”, with Nielsen charging 100K to 500K, and asked whether agents given a personality could answer surveys instead, as a Google report had suggested. 60:22 (Probably the Stanford and Google DeepMind study that simulated 1,052 real people from interviews.) The team built 30 to 300 persona agents per segment and ran the survey on them. The demo was slow to load in the room, so here it is to try yourself.

    Synthetic Persona Survey demo: results charts for a fuel retailer survey, such as what matters most when choosing a fuel station, with filters for borough, age, vehicle type and price sensitivity.
    Synthetic Persona Survey: pick the “Top Fuel and Convenience Retailer” demo, generate personas, run the survey, then filter the results. It is one of the LLM Demos Anand searched for live.

    Did the agents produce anything new? The value turned out to be in the verbatim comments. One persona wrote: “As someone with high price sensitivity, I prioritize … finding the cheapest fuel station to save money.” Another: “My commute involves a round trip of 23 km, so I usually refuel after work.” 61:56 The client read them and thought, yes, a person might say that. It surfaced assumptions they hadn’t questioned. 62:21

    “User testing helps. Users don’t have to be human.”

    Anand · 62:34

    Coding agents already do a version of this. Claude Code on a high thinking setting, he said, opens the browser, takes screenshots and tests the page by default. His prompts just add what it misses: “Test the accessibility tree as well.” 63:09

    63:21to 69:49

    The logs are the asset

    “When it goes into production, we know software fails. What do we do then?” 63:26 Collect feedback from production, said a student, and it depends whether the failure is technical or from users. Technical: server logs, error logs. 64:04 Anand uses Sentry for errors. But: “one of the top skills that I have for my agents is the observability skill. What does it take to log every single thing that you produce?” 64:27

    “The digital exhaust is probably the single most valuable asset that we can collect.”

    Anand · 64:49

    Agents can analyse everything, so the more they are given, the more context they have. Collecting that input, he argued, may be the highest-return human effort of the next few years. His own example: he records the transcript of every meeting, now over 2,000, and uses them like this: 65:17

    Questions he asks his own logs

    “Go through all my conversations with my boss. Tell me what I should say.”

    “Go through all of the talks that I run in classrooms. What are the mistakes that I make?”

    “Go through all of the logs on my server-side code. What are the kinds of mistakes you make?”

    “Go through all of the … session logs for coding agent sessions. What are the kinds of skills you need to avoid repeating yourself?”

    65:17 – 65:55. This page itself was written from the talk’s transcript.

    The bug report that ships a fix

    For user feedback, a colleague added a “Report an error” button that lets users type, speak or paste screenshots, and files an issue on GitHub. 66:34 Then they went two steps further. An agent picks up the bug, fixes it, deploys a personal version for that user, and emails them a link: we fixed this for you, does it work? About half the time users said “Yeah, this is exactly what I want”; the other half they refined it. 67:24 The product manager then looks across these personal variants and promotes common ones to alpha and beta releases. And users who want their variant can keep it permanently, with new changes patched on top. 67:35

    Which brings back the opening question. If each person can have their own version, “how much of the software that you write is yours?” “Personalized software is not something that’s too far away.” 68:29

    A · Data

    “Observability matters a lot simply for the data. Just collect all the logs that you can.” 69:14

    B · Better spec

    “It can be useful to improve the technical specification. Just look at the logs, ask the agent, improve the stuff that’s going wrong.” 69:22

    C · Personal fixes

    “Also ask the agent to fix based on that, deploy a personalized version, and get feedback from the user.” 69:31

    70:03to 74:18

    Nurses, checklists and auditors

    “What did I miss?” Anand returned to how other professions verify. 70:03 First, The Checklist Manifesto by Atul Gawande. His retelling: hospitals with low infection rates differed from those with high ones in a surprising way: the nurses had more authority than the doctors. A nurse could say, “No, doctor, you cannot operate unless…” and stop the procedure if a check on the list failed. Pilots work the same way. 71:24

    Fact-check

    Two details from the room need correcting. Gawande wasn’t US Surgeon General; he is a surgeon and writer who later led global health at USAID. And the nurse story in the book is Peter Pronovost’s central-line checklist at Johns Hopkins: five simple steps, with nurses authorised to stop doctors who skipped one. Line infections in that ICU fell nearly to zero, and across Michigan ICUs they dropped by about two-thirds. The lesson Anand drew stands.

    “A deterministic checklist is a very effective way of making sure that a system does not fail.”

    Anand · 71:55

    For software, he said, checklists are your linters and your test cases. Nothing new. What is new is who builds them: an agent can keep reading your logs and ask what needs checking, which items work, and how to improve the list. 72:14

    Second, audits. Breaking work into steps and showing receipts is how you check someone smarter. But who checks the receipts? Professions solved this by bringing in an auditor occasionally to spot-check a random sample and the edge cases. Not every report gets audited, so the system scales. 73:12 The AI version: let agents audit, and have a human audit the AI’s audits. His team asks agents to always prepare an auditor report, the minimum information a human needs to check the receipts. 73:50

    74:18to 82:39

    “We aren’t even allowed to use AI”

    Straive bundles all of this into its StraiveX AI-native delivery methodology (an internal document, not public). Anand told it as the story of Tanuj, a colleague placed inside a client, who pushed back: “we aren’t even allowed to use AI in this organization.” No Codex, no Claude. Only Snowflake Cortex, the AI built into the data warehouse. 75:03

    Anand’s suggestion: ask Cortex to search online for the client’s business context, look at all the internal data, and list the most important problems to solve. 75:20 In a few days, Cortex produced a report of use cases for this waste-management company: revenue forecasting, pricing anomaly detection, churn prediction, a revenue analyst agent, with approaches for each. Tanuj took it to Cal, the head of analytics.

    “There’s an 80% overlap between their recommendation and what you said.”

    Cal, comparing it with a four-month consulting exercise, as Anand retold it · 76:42
    AI Use Cases on Commercial Data: an executive summary table of four categories (predictive analytics, intelligent automation, agentic copilots, advanced ML models) with impact, effort and 15 numbered use cases.
    AI Use Cases on Commercial Data: the kind of report Cortex produced, 15 use cases from 174 columns across six source systems.

    Tanuj planned to write a proposal for each use case. Anand stopped him: by the time a proposal reaches the client, “it’s going to take forever. … Why don’t you just put all of this into Cortex itself and tell it to solve the problem? Just go ahead, forge the solution yourself.” 77:47

    The pricing anomaly detector found 105,000 transactions at $0.16 for a service that costs $175 to $179, and 84,000 charged at 6 to 14 times the median rate. 78:03 The churn analysis became an email to the client titled “We’re losing $117 million in revenue, and here’s the fix.” 79:39 Of the big accounts that had left, two were contract renegotiations, four followed an account manager leaving, and five were customers nobody knew were leaving.

    Finding the problems was easy; fixing them wasn’t hard. “Where the human starts coming in is: did we get this right?” 79:17 And the validators were the client’s own experts, exactly as Ankur had been for Varun: “Leverage the capability of the experts who are good in the field.” If they can see the result at a glance and validate it instantly, “your problem is solved.” 81:14

    The method, as Anand walked through it

    From prospecting to evolving

    Sense

    Prospect: ask “what do you want built?”Let an agent read the business context and the data, and propose the problems worth solving.

    Forge

    Write a proposal per use caseDon’t propose. Have the agent build the solution and show results.

    Validate

    Put results in front of the client’s experts, in a form they can judge at a glance.

    Scale

    Build the same across each of their plants, letting agents adapt to local context.

    Run & evolve

    Watch the operational queries for drift, read the logs, revise, and feed what you learn back into Sense.

    Stages one to three are Anand’s words. “Scale” and “Run & evolve” are this page’s labels for the two stages he covered “very briefly”. 81:28

    81:48to 85:00

    The one irreversible thing

    The last stage needs observability again. The pricing detector flagged anything outside the 10th–90th percentile band. Two and a half months later, the client introduced a pricing policy that roughly doubled prices for some services, and the agent started flagging too many errors. It has to notice that from the logs and revise its own rule. 82:11 A verifier that was right at launch can be wrong two months later.

    So where are humans still needed as verifiers? Anand saw two cases. The end user who says, “I don’t know what I want, but I’ll know it when I see it.” And the expert who says, “I know exactly what I want, and I will know it when I see it.” In almost every other case, he finds systematic ways to let agents verify, often by turning what he has learned into a skill and saying “now you go apply these”. 83:17 For those two cases, what matters is getting their reactions as data, without intermediaries.

    “Make sure that every single reaction or even thought process of the users that are using the software gets logged. Agents will figure out what to do with it later. The one irreversible thing is the logging of the user reaction at that point of contact.”

    Anand, on his one must-have rule for software · 84:16

    Everything else, he said, you can derive later, or agents will derive it for you. A reaction you didn’t record is gone.

    Act V · Questions from the room

    85:26to 89:38

    Interview the end user. Describe feelings.

    The first question went to the heart of “let the agent interview you”: doesn’t that assume the human has expertise? And what should fresh graduates do, when the temptation is to hand everything to “another agent, to another agent” instead of building expertise of their own? 85:26

    Anand’s answer was to disintermediate. If the software is for him, the agent should interview him. If it is for someone else, the agent should interview them. 86:25 A finance manager who runs invoice reconciliation can name half a dozen rules (“I don’t want this entire thing costing me more than so much”, compliance with government regulations) that are “the ultimate truth”, and that Anand could never have produced himself. 87:20 If there’s no expert, ask how to find one. If there truly isn’t one, your opinion counts as much as anyone’s.

    “I don’t describe specifications anymore as how to do it. The agent is smarter than me.”

    Anand · 88:05

    Instead he describes how the result should feel: “build me something that makes me feel light. At the end of it, I want the experience to be such that I’m not hindered. It doesn’t feel like a lot of hard work.” He lets the agent translate that, asks it to interview him about what he means, “and sometimes I just tell it, ‘I don’t know.’” 88:15 His hunch about why this works: whatever expertise can be put into words, agents can learn. “Maybe it is the stuff that we can’t even express well enough that remains to be expressed.” 88:59

    89:38to 95:19

    Who’s most unique?

    A student had read one of Anand’s blog posts: if AI can program everything, let it program, and we come up with ideas. But if all of us apply for the same job with the same AI, and produce similar outputs, how does anyone stand out? 89:38

    Anand’s answer came from his own course, where one of the grading criteria is “who’s most unique? If everyone’s going to come up with the same stuff, that’s boring.” 90:30 About 300 students each made a wordless picture story, then rated each other’s. He plotted uniqueness against how much people liked them. The best-loved stories: a dog adopted by a little girl. A dog adopted by a little boy that saves him. A dog adopted by a little boy. A dog adopted by a big boy. 91:08

    “I am not going to hire any of you. What kind of a joke is this?”

    Anand, on the fourth dog-adoption story · 91:42

    The other common pattern was the thirsty crow dropping pebbles into a pitcher. The unique ones were different: the Bhagavad Gita told without words; a satire where a businessman bribes a policeman and shoos away a beggar but happily hands money to a politician who doesn’t need it; a girl saying “Oh, I love it” about a gift she plainly doesn’t want. 92:49 “These are whom I’d hire.” (The full chart, and how students voted, is below, from the TDS design meeting story, “The crow beats the astronaut”.)

    Explore the chart

    329 picture stories: original or loved?

    Guess which story classmates preferred, then explore the scatter: uniqueness across, preference up. The familiar crows and dogs sit top-left; the stories Anand would hire for sit to the right. Tap a highlighted dot to read the story.

    From The Score Can’t See Who Built the Solver, the story of the TDS course design meeting a day earlier. Scroll inside the frame for the rest of that section.

    “AI can code, you can code. I’m not looking for the coding capability. What did you decide to draw? What did you decide to code? That’s what I want.”

    Anand · 93:10

    The student still looked thoughtful, so Anand gave the implication he lives by. “What skills should I learn in the AI era? My rule of thumb is this: learn what AI cannot do.” 93:52 The prerequisite is to delegate maximally: put everything through AI, because only then do you see where it goofs up, and those are the skills worth building. 94:03

    “The intuition there is delegate maximally, do what AI fails, and absolutely do not bother learning what AI can do or will very likely do in the future.”

    Anand · 94:45

    There is a short-term catch, and the talk began with it: interviews still test things AI does well, “as you saw in the very first interview that I had.” But that isn’t a durable skill. 94:58 His current list of skills that AI is slow to learn (relationships, accountability, scepticism, noticing, taste, problem selection, and more) is on his AI Advice page, under “What skills should I learn?”

    Notice the loop

    The talk opened with a test that measured the wrong thing and closed with a grader looking for the thing AI can’t supply. A good test, like a good verifier, should measure what you actually care about. If an agent can pass it, it is measuring the agent.

    Eight-panel comic. Panels: software is malleable, with a VLC-like player with a composer column and a Gmail inbox; when AI builds, who checks the AI; Anand eats an apple while an AI agent passes a hiring test 52 of 59 with proctoring active; the question bank, proctoring and assumptions all miss the real cheating; verify the verifier with a checker AI and a critic AI comparing arguments over a bracket; let the agent interview the real user or expert, like a doctor or a judge; at scale use checklists, confidence ranking, user tests and multiple agents with humans handling disagreements; log everything, observability turns reactions into fixes, personalisation and better AI, and learn what AI still can't do.
    AI-generated comic of the talk. Click to open it full size.

    Coda

    The comic, verified

    In a talk about checking AI output, it seemed only fair to check the comic. Mostly it holds up.

    1. Right: the apple, 52 / 59, proctoring that saw nothing, a buggy answer key, and the blind-spot assumption “human is answering” (panels 3–4).
    2. Drifts: the Gmail panel shows custom filters and auto-labels. What Anand built was notes on emails, added with Alt+Shift+M (panel 1).
    3. Wrong: panel 5 has a critic AI declaring the part “not a bracket”. Nothing like that happened. The part was a forked bracket, the three renders looked identical, and the verifier that was left free found a benchmark and scored Codex’s geometry at 31%. The lesson was about Anand’s too-narrow verification instructions, which the panel’s “my checklist (too narrow?)” does catch.
    4. Right: doctor-style interviews and judge-style competing experts (panel 6); voting agents with humans resolving disagreements (panel 7); “log everything” and “learn what AI still can’t do” (panel 8).

    Takeaways

    Six things to remember about verifying AI

    01

    Verifiers fail silently

    A test with a wrong answer key and blind proctoring still reports 52/59, zero violations. When everything is green, ask what the check could not see. The apple
    02

    Ask for your blind spots

    “What am I assuming that I’m not even aware of?” is a prompt. So is a list of the feedback you keep ignoring. What went wrong
    03

    Don’t cap the verifier

    Let a smarter agent define done and choose how to check, then compare with your own constraints. You may be the bottleneck. The bracket
    04

    Disagreement is the signal

    Cross-check with another model and settle disagreements with fresh evidence. Five agreeing models cut errors to 0.7% for 28% human review, but can’t catch errors they share. Five models
    05

    Rank confidence, don’t trust it

    “95% sure” meant 77% right. Confidence and logprobs are useful as an order: check the least-sure items first. But confident can still be wrong: Joey’s marshmallows were 100% sure and false, so check the premise too. The twist
    06

    Log every reaction

    Where there is no spec, users and experts are the verifiers. Their reaction at the point of contact is the one irreversible thing; record it. The one rule

    For students

    Four exercises to try this week

    Anand’s rule is to delegate maximally and learn from where AI fails. These put the talk’s techniques on your own work.

    Test the tester

    Let an agent sit your test

    Pick a past quiz or assignment. Have a coding agent attempt it. Where it succeeds, what was the test really measuring? Where it fails, is the agent wrong, or the answer key?
    Ask another AI

    Make two models argue

    Answer a real question with two models. Paste each answer into the other: Fact-check what it said and what you said. Take what’s better, drop what’s worse. Settle each disagreement by running or recalculating.
    Mine your prompts

    Export your chats

    Use the export button in ChatGPT or Claude. Ask an agent: what do I keep correcting? Turn the top answer into one line of standing instructions, like define what done means and verify before concluding.
    Check your confidence

    Build a check queue

    Take 30 questions you know the answers to. Ask a model for an answer and a confidence. Is it right more often when it’s more sure? If yes, you have a ranking you can use; if not, you’ve learned that too.

    Related talks

    Sources and method

    SUTD COIL session · 17 Nov 2026 · Story written with Claude from the recording’s transcript and the material linked above. More talks.

    Transcript

    Open full size ↗