Make It Prove Itself

Teach it to test itself

Wednesday was the required checkpoint — the one day nobody could skip — and Anand opened with the same problem everyone had already lived through: you tell the agent to fix a bug, it fixes the bug, and a week later something that used to work has quietly stopped working. Building is fast now. Trusting what got built is not.

"It's easy to build an application, AI can help us create the product, but what's trickier is to make sure that it's useful and make sure that it's usable."

— Anand

The specific fear he named is the one anyone who has iterated with an agent for more than a day recognizes on sight:

"You'll find that something you wanted in the first place is no longer working. Meaning, it adds new mistakes as we go along, which we don't want."

— Anand

So the question for the session was blunt: can ChatGPT or Claude test an application the way a person would — clicking here, pressing that — reliably, repeatably, and without burning tokens every time? Rather than explain the answer, Anand picked two apps at random from the pile students had already shared, opened ChatGPT Desktop, and asked it to find out for itself. He explained the choice of desktop app in passing: "if it can use my computer, maybe it can also use my browser, which may require some of these sites to be logged in, and I have a little more control." The instruction he typed was almost a spec in itself:

"What I'll do is go through this site like a real user, find out what it does, and test it. Then, create an automated set of test cases that I can rerun anytime without needing any AI application. […] Also, create a small document explaining what exactly you're testing so that if I need to change something in the future, I will know. Make sure it's easy for me to add new tests in the future by just telling you."

— Anand, prompting ChatGPT to test a stranger's app

The two sites in play were City, By You — Mayu's single-page quiz that turns eight small answers into an imagined city — and, a few minutes later, Pawlog, a pet health diary. Anand picked City, By You to demonstrate live, and a parallel Codex session did exactly what he'd asked, unattended, while the class kept talking.

What the agent came back with wasn't just "it works." It read the source, mapped every selector, and flagged real behaviour worth locking down before it quietly changes: eight forced-choice questions feeding a deterministic city archetype, state that lives only in page memory, and — because it actually read the code rather than only clicking through it — three genuine product defects it chose to preserve as known, expected failures rather than paper over:

Anand's own reaction, watching the coverage list scroll past, was that it was thorough but unreadable — a list of internal test names, not something a non-engineer could sanity-check at a glance. So he asked for exactly one more thing:

"Can you rewrite the coverage as simple questions that explain what you tested, like: 'Is the text on the landing page okay?' That sort of thing."

— Anand

The rewritten guide reads like a checklist anyone could review, each question traceable straight back to an assertion in the suite:

It names the exact principle behind the request:

"The agent is doing the work; we have to verify. Usually, that means that we end up doing a lot of verification work, and that's painful. I like to make it easy for me to verify. So I tell the agent, 'Give it to me in a way that I can understand. Make my job easier.'"

— Anand

And once the suite exists, verifying a future change stops being a chore at all — it becomes one more thing to delegate: "I might say, 'I've changed the site; run the tests again and see if it works, and tell me what's changed.'" Twenty-five minutes had produced something more durable than a demo: a rerunnable contract for what "still works" means, that costs nothing to check again.

The app that couldn't leave the sandbox

Nine minutes in, Johan raised a problem that had nothing to do with testing yet, and everything to do with why testing an app in place, as an artifact, is only half the battle: the AI he'd put inside his app was the exact thing stopping him from shipping it.

"I've been trying to implement the AI itself inside the artifact, but that hinders the artifact from being shareable. Is there a workaround around that, or how else should I implement the AI inside the product that I'm making?"

— Johan

This was Johan's Stock Ticker app, deployed on Cloudflare — a different shape of problem than the Lentera Bursa tracker from Tuesday, and one Anand couldn't just answer from memory. So, true to Day 2's method, he didn't try:

"Part of the reason I'm doing this rather than sharing an answer with you is: A, I'm not sure if I know the best answer. B, even if I did, knowing the answer and telling you is not the important point. What I want to say is that you could try different ways of asking AI yourself and learn. It's that approach to how you might learn that would help."

— Anand

He typed the question into ChatGPT almost verbatim as Johan had posed it, and watched the answer arrive live.

The mental model underneath the answer is the one worth keeping, more than the specific Cloudflare plumbing:

"A public AI app needs only two things: a webpage and a tiny private function that makes the LLM call."

— ChatGPT, answering Anand's question live

Anand was candid that it wasn't the answer he'd have given himself — he'd have reached for a hand-rolled API key and a Cloudflare Worker — and that the gap was itself the point:

"This is not the solution I had thought of. This is a good solution, and it's probably an inexpensive solution as well… Some people know exactly what it's talking about; some people have no clue what it's talking about, and it doesn't matter. Because what you could do is take this entire thing, put it back into your original chat, and say, 'Look, ChatGPT said this. Do it.' And if it works, great; if not, you wasted some time. Big deal."

— Anand

It's a small moment with a large implication: the class had just watched a teacher ask a question he didn't know the answer to, get a better answer than his own instinct, and hand it off without needing to fully understand the Cloudflare Workers AI binding underneath it. That's a preview of the argument the very next student was about to force into the open.

"Is this a black box? Is this dangerous?"

While the City, By You test suite kept running in the background, KK — who'd opened Tuesday's session with the news-summariser that impressed his parents and then didn't — asked the question underneath every demo so far, and asked it without softening it.

"Sorry for always asking questions, but I feel that one thing — doing my process of trial and erroring — I feel that is this really the right thing to do? Because if you don't even know what you're doing and then you're just throwing whatever the suggestion is into Codex or whatever, then it's building whatever based on an AI suggestion, but we don't really know what the heck is going on. So it's a black box. Is this really right? Are we like — is this actually more of a dangerous thought?"

— KK

Anand's first word was "Possibly." Not a deflection — an actual concession that the worry is sometimes justified. He reached for a story instead of a rule:

"I drive a vehicle, a car. I have no idea how the engine works. If it breaks down, I can't even change the tires. Is that a bad thing? It is, because there have been a couple of occasions when I was stuck and I needed to just at least change the tire or know that the engine was heated. […] The total time that I spent on this was half a day. The amount of time that I might have taken to learn about engines and changing a spare tire, etc., would probably have been a couple of days."

— Anand

KK pushed back, and the pushback was sharper than the analogy could absorb cleanly:

"I understand the perspective that you're learning a kind of niche skill, you know, car repair, for occasional three times' use. But since we're undergrads and students, isn't this technically more important because we're potentially delivering it to our bosses and stuff? […] I will most likely encounter more bugs and issues and stuff that would be more crucial for me to know rather than car repair. So how?"

— KK

Anand's answer didn't concede the car analogy — it reframed the entire question of what's worth learning around demand and supply, using a story from his own workplace rather than another car story. He tells interns at Straive to record a call with the CEO, transcribe it, and hand the transcript straight to Claude or ChatGPT:

"Do not try and understand what the CEO is saying. You won't understand it. Don't try and interpret the output; it may not make sense to you. Feed it back to him and let him figure it out."

— Anand

Then he ran the thought experiment one step further — what does the intern actually learn if the agent's first draft is already right?

"Suppose Ankur, who's the CEO of Straive, sees the output and says, 'Yeah, this is what I want. Great.' Then there was nothing for the intern to learn. […] But if he had said, 'You know what, the interface isn't quite like what I want. […] the fonts have to be a little larger, it's got to look a little more like a slide deck,' — now Pavan has picked up something. He's learned that there are a few things that he should tell the agent even when not asked for."

— Anand

That's the whole answer to KK's "is this dangerous" compressed into a rule of thumb: the correction is the curriculum. Everything the agent already gets right on the first try isn't worth anyone's learning time — it's the narrow gap between what the agent hands back and what a real person actually wanted where the useful skill lives.

"It's worth learning what there is a demand for and what there is a supply gap for. If the agent already does it, there's no point learning it. If the human doesn't ask for it, there's no point learning it. By delegating as much as possible, we get to find out what the gap is and focus our learning on that."

— Anand

He turned it back on the room's own frustration — the fact that everyone's app was still hitting a wall wasn't evidence they were doing it wrong, it was the whole method working as designed:

"You're already hitting that barrier so easily because your apps don't work perfectly. […] And if you find that you're perfectly happy with the outcome, you're not ambitious enough. […] A lot of the things that we've been taught so far are things that agents can do. […] What it can't do is what you should be learning."

— Anand

And, characteristically, he refused to let the answer harden into dogma before the segment even closed: "Is that always true? No, but here's the thing — I'm just one teacher amongst several dozen. The others will tell you what you should be learning; I'm just telling you what you shouldn't be learning." KK's black-box worry didn't get resolved so much as relocated — from can I trust this to where, specifically, is my judgment still the scarce ingredient.

Hire three personas overnight

Testing verifies that an app does what it's supposed to do. It says nothing about whether anyone wants it to do that. So Anand turned to the second kind of verification — one that looks a lot more like a market survey than a QA pass, and does not require finding real people at all.

"When you ask different people, you are effectively getting user reactions, like how does a person react to it, how do they feel about it. That's more like a market survey. […] It's possible to have agents run that kind of a market survey themselves, and I'll show you how."

— Anand

The target this time was Study Walker [スタディウォーカー] — Yuri's Japanese-language app that turns a vague learning goal into one tiny, doable step for today, via a three-question wizard. Anand dictated the brief to ChatGPT, then handed the whole thing to Claude, and the instruction is worth reading in full because it's really three delegations stacked on top of each other:

"I want you to run the equivalent of a market survey on this site. Go through the site briefly and understand who are the top three personas that this could be a product for. And then create sub-agents using a lightweight model like Haiku, and pass them the persona for each of the audiences. Tell them to act like those personas and browse and test the application and share feedback exactly like they would in a market survey. […] To be frank, I don't know much about market research, but you are the expert, so you tell me how a good product market research should be conducted and actually conduct it that way. After these three sub-agents have finished, I'd like you to consolidate the feedback and give it to me in the form of a simple report where you start by sharing the most important feature that is easiest to action first, and then the second most, and so on."

— Anand, dictating the brief live

He named exactly what made this different from the previous scene's delegation: "I'm delegating not just the market research, but even the structuring of the market research." Claude took the instruction and, without being told the specific personas, read the app's own four example goals and five "stuck state" options and built three people out of them.

Three Haiku sub-agents then actually drove a browser against the live site above, in character, in Japanese, end to end:

The result Anand read out to the class landed almost exactly where he'd hoped: the core idea — "take one small step today instead of learning X" — worked for all three invented people, and the agent had converged on one clear, cheap, highest-leverage fix.

"It identified three personas: a Choice-Paralyzed Beginner, a Serial Habit Quitter, and an Exam-Driven Pragmatist. […] The core idea — 'take one small step today instead of learning X' — it really worked well for all three people. But the main gap is that the sessions don't have memory between the visits."

— Anand, reading the report to the class

What the transcript of the research session shows, and what Anand didn't have time to walk the class through live, is the part that makes this more than a cute trick: two of the three personas independently reported what looked like a serious bug — the app silently swapping their goal for someone else's mid-session. A weaker process would have shipped that straight to the top of the report as a "critical trust-breaking bug." Instead, the consolidating agent flagged both claims as suspicious because two different personas hit the same odd failure, re-opened the live app itself, inspected the raw localStorage, and found the real cause: three test sessions racing on one shared, un-namespaced browser-storage key — an artifact of the test setup, not something a real single user would ever encounter.

That single downgrade rewrote the whole priority list: instead of an emergency data-loss fix, the top item became something almost embarrassingly cheap — a suggestion template that tells a user to "recall an expression you already learned" while showing a brand-new example sentence, a one-line mismatch that Aiko's persona flagged the moment she hit it. Anand's framing for the whole exercise doubled as a cost argument, and he was explicit about why the choice of Haiku mattered:

"Market research with a model like Haiku — and I specifically chose Haiku because it's an inexpensive model — can be done very efficiently. […] It's at 29% of usage; I started the session with about 12%, so it consumed roughly 17% of my five-hour limit, which is nothing."

— Anand

And, echoing the line that closed Tuesday's session almost exactly, he was careful not to oversell what this replaces: "This won't necessarily replace human response… but it certainly provides useful additional feedback at very low cost. […] If there are certain kinds of people that I can't reach out to — for instance, a football coach, and we didn't have one — I'd take that site and say, 'Imagine you're a football coach.'" A market survey you can't afford to run with real people becomes a market survey you can run three times before lunch. Read the full consolidated report ↗ · or the raw agent session log ↗.

Breaking the "simplify" feedback loop

While students filled in a shared peer-feedback spreadsheet, Dora typed a question into the chat that named a frustration almost everyone in the room had felt but hadn't put into words yet:

"For context, I'm building a root mapping tool to customize their journey. While simplifying my experiment and deploying the website, the agent adds extra steps, making it more complicated to deploy. Then it turns into a cycle of me telling it to simplify the process with varying success. How do I break out of this feedback loop?"

— Dora

Anand's response followed the pattern from Scene Two almost exactly — ask a capable model first, then add his own view on top. This time the model was Claude, specifically "Fable", at medium effort.

Before reading out what the model said, Anand gave his own answer — one he'd clearly arrived at the hard way, across many students' stuck deployments before this one:

Then he read Claude's answer, and it converged on the same diagnosis from a different angle — the vagueness of the word itself:

"'Simplify' is vague, and the agent doesn't know what 'simple' means, so it guesses. […] Stop asking it to simplify and tell it exactly what the end state looks like. […] One command, one HTML file, one JavaScript file, no build step."

— Claude, in the shared chat

Two more concrete moves came out of that same reply: naming a fresh session as one real option ("starting a fresh session is something that I mentioned a short while ago"), and asking for deletion rather than simplification — "remove this functionality, remove that functionality" — because deleting is a concrete, checkable action, where "simplify" is a judgment call, and the agent's judgment is exactly what produced the mess in the first place. Anand's closing line to Dora doubled as permission for the whole room: "Don't hesitate to start from scratch; nothing wrong with that."

Record yourself getting confused

The third kind of verification needed neither an automated suite nor a sub-agent — it needed a real, confused human, on camera, using someone else's app for the first time. Anand set up a shared spreadsheet — "FT AI Products peer-to-peer feedback" — where every student would volunteer their name against two classmates' apps, then record themselves giving feedback on video. Before asking anyone else to do it, he did one himself, live, choosing Study Walker — the same app three AI personas had just tested overnight — as his subject.

First, the practical question of how to even record a screen-plus-camera video without installing anything. He asked Google AI Mode directly, on stage:

Then he actually recorded it: opened Study Walker translated into English, typed a real goal — "learn Rust programming language" — and narrated his reactions exactly as they happened, unscripted, mistakes included.

"First step to start doing what you want today. […] What do you want to be able to become? […] I tried it but it was difficult. […] I didn't quite understand it, so I'm not sure what to do, but I'll try."

— Anand, narrating live, in character as a confused first-time user

The recording captured a real, small, useful moment of discovery: the app pointed him at "Learn Rust in 1 minute," he decided that was more ambitious than one minute really allowed, but tried the Rust Playground it linked to anyway — typed println!, watched "Hello World" print, then broke his own toy example on purpose (1 + 2 as a printable string) and let the compiler's own error message teach him the fix.

"What I noticed was the error messages are more useful than I thought. […] So in a minute or so, I was actually able to take one step: Hello World. […] That really did help me. It managed to make me move forward on something that I wanted to do."

— Anand

Two rules closed the segment, and both cut against the instinct to perform a clean take rather than a true one:

"Please feel free to speak in any language you're comfortable in, because translators these days can do an excellent job. […] The more you share how you feel in a way that's easy for you to express, the easier it is for the audience to understand. […] And please don't try and record multiple times to get the feedback right. Mistakes when giving feedback — that is probably the most useful feedback. […] Just record it one shot; that is the best feedback."

— Anand

Three sources of verification, stacked on top of each other by the end of the hour: an automated test suite that never gets bored, an AI-run market survey that costs a fraction of a real one, and a peer's honest, unedited, on-camera confusion. None of them replace the others; each catches something the other two structurally can't.