Should a student even readskip? the exam?

The question nobody needs to read

Adwitiya, a student, had a good idea. Frontier models, he had noticed, now try so hard to be correct that they test and re-test until your weekly limits run dry. A useful skill might be telling an agent where to stop. Could an assignment teach that?

He even had the mechanism. Students in TDS call models through AI Pipe, a proxy that the course can observe, so a question could check whether the model ran tests it was told not to run. 22:11 Anand, the instructor, liked the goal. Then he played the question forward, the way a student actually would.

The instructions say “tell the model to stop”, so the agent writes a prompt telling the model to stop. Full marks. Nobody learned anything, because nobody needed to.

What makes the line startling is the next one. Anand did not call this cheating. And that’s not a bad thing, he said, because if an agent can read it and solve it, that particular thing is not what we want to teach. 23:44 In a course that already tells students to use agents, friends and anything else to get work done, a question an agent can finish unaided is a question about the agent.

He then set the room the problem the team faces every week. Students should learn to conserve tokens, because that is a real problem today. But ChatGPT can write a token-saving prompt on request. So, question one: does this still need teaching? Question two: if it does, how do you teach it so that pasting the question into ChatGPT does not solve it? 24:00

That is the thread through this hour. On 5 October the course explained its philosophy: let agents execute; humans specify, orchestrate and verify; the exams are the curriculum. On 12 October the team designed agent-era assessments in public. On 19 October those designs met running code and the cost of checking. On 2 November the teachers asked whether there is anything worth teaching once the agent can read the exam — and found four problems on the way:

1. The course changes while students are inside it, sometimes during the meeting.
2. The teachers doubt their own job: if a lesson can be taught, an agent can teach it.
3. Every “agent-proof” question has a leak: a meta-agent, a shared solver, or a WhatsApp group.
4. A score can reward the wrong thing, and the students noticed before the teachers did.

It starts twenty minutes earlier, with a teacher changing the thing his students are trying to predict.

A teacher changes the number his class is predicting

Attendance was low; it was Diwali week. JK and Carlton were away, and Jaideep, a TA, wanted the next assignment’s unpublished questions discussed offline rather than on a public recording. 00:40 Then he said what everyone setting these exams had been feeling.

Anand turned it into the session’s question — How do we teach when the agent can solve the problem? 02:32 — and then the conversation wandered somewhere revealing: to software that will not hold still. He described Patrick Collison’s “liquid software” post: a feedback box in the app, wired to an agent, so the software rewrites itself when users ask. 03:24 Source (It is Collison’s anecdote about his personal apps, not a Stripe product.) Jaideep raised DHH’s Omarchy, an Arch-based desktop that, as he put it, tells you to ask an agent if you find a bug. 03:40

Adwitiya, who said in chat that he had “daily drove it for a while”, supplied the caution. An agent with full system access is fine if you are very precise; otherwise, if it messes something up, it gets tricky to debug or to revert to the state before. 05:15 On the call Hold on to that word, revert. It comes back at the end of the hour as a test design.

Then Anand did something to his own course. He asked Agrim, a TA, whether the next assignment, GA4, had an AI Pipe question. It didn’t. He said he would add one, because it will be interesting to see the impact of that on project one, question one, which was about predicting the AI Pipe token usage. 07:34 And then, on screen:

He asked the TAs to tell everyone, because most people would want to revise their Project 1 forecast. 08:40 That forecast is a class prediction market: students trade virtual credits on how much metered spend AI Pipe’s public usage feed will report for 2–8 November. Course docs The meeting happened on the first day of the very week being predicted.

There are two ways to read this moment, and both are fair. One: it is a lesson. Real forecasts are made inside systems whose operators can change the rules, and the skill is noticing, re-estimating and acting. Later in the hour Anand says exactly this, and tells students to build agents that watch the course for changes. 51:44 Two: it is a fairness problem. A rule changed on a Monday-evening recording reaches students only as fast as the announcement does. Students who check the forum hourly, or who have an agent watching for them, get the news first.

Our reading: both are true at once. A course that rewards monitoring a changing environment owes students a clear announcement of every change — otherwise it is grading who heard first.

“There is literally no point teaching what is teachable”

The course is changing so fast, Anand said, that the team now uses agents for much of the work of changing it — and those agents need briefing. So he had asked ChatGPT to draft a guidebook for the exam repository from the team’s transcripts, emails and course material: an agents.md in all but name, for any agent asked to write TDS questions. 09:00 He shared the draft and read out its first principle.

The next line turns the provocation into a design rule: what do we teach? What is it that humans need that agents won’t learn. 11:28 Agents are a tool on intelligence; give them tools, other agents and people, and drive them to get the job done — and verify it, because accountability stays with the human. The course’s older rule, “exams are the curriculum”, stays; the content pages are optional.

A caution in the margin: this is a deliberately provocative proposition, not a finding. Some things are teachable but unlearnable from text — motivation, values, the feel of a real deadline — and some things agents “won’t learn soon” may arrive next quarter. The useful question isn’t what will agents never do? but what must a person still decide, check and answer for?

Then he applied the rule to his own staff. He had dropped into a TA workshop the day before, where Mayank, a TA, was sharing how to create valid JSON. There must have been a reason for it. But I was thinking, hold on, why should we teach that? 12:56 Better, he said, to have an agent make an explainer, so that students learn to make their own explainers, and the explainer is published for any human or agent to reuse. Then came the challenge.

A teacher asking his TAs to automate themselves away sounds like an efficiency drive. The next six minutes showed it was something stranger, because the TA who answered had already done the opposite of preparing a lesson.

The TA who came unprepared on purpose

Jaideep had run one session, on WebMCP, a proposed standard for letting agents use web pages. It ended, he said, with him using Codex to build things in front of the students. 14:04 Then he explained how he had prepared.

The point was to start where his students were, with the concept equally new to everyone, and let them watch how he took it on with agents: neither “just tossing everything” at the agent nor “micromanaging things to the point where it’s better to do it by yourself” — how to find that balance. 15:32 Anand agreed at once: under preparation is an advantage in such sessions, because you’re transferring the broader skill, which is how do you deal with such a question. 15:56

This is a very old idea in new clothes. In 1991 Allan Collins, John Seely Brown and Ann Holum called it cognitive apprenticeship: in school, an expert’s thinking is invisible; apprentices learn by watching the expert work on problems the expert hasn’t solved yet, then being coached, then left alone. Research A polished tutorial hides the very part Jaideep was trying to show: the searching, the false starts, the decision to delegate or to look yourself.

Adwitiya, meanwhile, typed a practical counter-proposal. Hunting through a two-hour recording for one twenty-minute segment wastes time; why not have each TA publish four or five prompts per topic, so students can learn the content by chatting with a model? 16:16 Anand agreed with the principle and split the audience in two.

Extracting the content, he added, is useful — but it is the students’ job, not the TAs’: write an agent that pulls every prompt out of the session videos, post it on the forum, and let someone else build a site on top. He pasted a link into the chat as an example of what such a site can look like: his talks page, which turns recordings into navigable stories. 19:16 Our note You are reading one.

Jaideep pushed further than either of them.

A list of what humans are still for

After the invitation, Anand went back to the draft guidebook and its list of what the course thinks still has value when delegation is the default. 26:04 Read it as the team’s hypotheses — a working list for the agent era, not a permanent inventory of things machines can’t do.

1. Judgment“In some way, shape or form.” Choosing what matters, and which tool or agent to give it to.
2. Risk management“We don’t even know what all ways in which an agent can go wrong, and we still have to … drive towards getting the job done.”
3. Failure diagnosis“How will we know why it’s failed and where it’s failed?” Spotting the AI’s errors.
4. AccountabilityThe one that ends the argument, below.
5. Unstated requirements“Implementing not just what is written but what is intended. This is the remaining human skill.”
6. Sourcing contextFrom people or systems: “context is the tough part.”
7. Guess before you askTo learn by yourself: “you guess, see if the agent disagrees, agrees, and use that to build your judgment.”

His example of an unstated requirement is worth stealing. Suppose a question says only: a CIO wants a dashboard with this data. An agent can produce a dashboard. A person with judgment splits the work — analysis to the model that analyses better, design to the one that presents better — and then adds what nobody wrote down: the CIO will not read that much analysis, and will almost certainly open it on a phone, while the agent optimised for desktop. 27:16

Why teach this to undergraduates, when it used to come from years of experience? Anand told a story about a classmate who now hires, and who said, roughly, that he needs people who have judgment, so he can only hire senior people. 28:16 His anecdote The conclusion he drew: students will have to build that experience faster, by using agents to put themselves through problems that would normally need years.

Then he asked the room what was wrong or missing, said he would publish the guidebook and invite comments, and on screen asked an agent to add a guidebook.md and link it prominently from the course home page. 30:36 Its purpose, he said, is that agents creating questions refer to it: It will soon become an agents.md for preparing exams. 33:12

Status check, 3 Nov: the public course README now links a “TDS Guidebook for AI”, but that link returned “not found” when we tried it. Treat the guidebook as a draft in progress, not a published document.

Every trap has a leak

Adwitiya came back with ideas for forcing students to read the question — prefacing them, honestly, with of course, it’s not a very ethical thing to do, but as long as teaching is involved. 33:44 The assignment page has a “Copy to clipboard” button for pasting questions into an AI. What if the copied text were altered on its way to the clipboard? Anand: Mayank had already done something like that, in the last ROE, he thought. 34:24

His second idea was subtler. A human reads “make sure your answer is correct” and checks twice. A frontier model reads it on a fiddly floating-point problem and may burn through a large share of its budget before declaring itself confident — and a student pasting the question would never suspect that line. 34:36

Anand’s answer is the most useful paragraph of the hour for anyone designing assessments. The idea is promising, he said, but not agent-proof: a capable agent will try a whole series of prompts and see which works. It could make a good ROE question — TDS’s timed, remote online exam — if each student got a genuinely different question, different in more than a number or a phrase. Otherwise, here is what happens:

He is happy for people to delegate, and happy that one person used an agent well. But a question that one solver answers for everyone teaches one person. His bar was explicit, and modest:

A question to put to every “gotcha”: does surviving it build a capability the student will use again — reading the instructions, budgeting effort — or only a suspicion of this particular examiner? A trap that the next model shrugs off, or that penalises students using screen readers or translation, fails the test even if it catches copy-pasters.

Eight drawings only people could read

Then Anand showed what a question that resists solvers looks like when it has run. The bonus question in the third graded assignment, “Humans Only”, gave each student words to draw in 60 seconds — no letters or numbers — so that classmates would recognise them and a panel of three vision models would not. Classmates and models picked from the same ten similar options. 37:36

A ladder, drawn at an odd angle, fooled most of the models. Someone drew a lady plus a bird — a clever combination where you’re able to differentiate what you communicate to a human versus to an agent. 38:44 Humans saw a pressure cooker where models saw a kettle, perhaps, he guessed, for linguistic reasons; most models thought the medal was a shuttlecock or a whistle. Then he looked up the strict count, on a write-up he had posted two days earlier.

Look closely and the eight are less clean than the headline. The lamp wins by writing the word “lamp” and then scribbling over it. Anand’s write-up calls it “a good CAPTCHA idea”; the question, though, said not to write letters, and the published results flag drawings that did. Public data Whether that is ingenuity or a rule break depends on what the score was meant to reward — a theme the students would return to. And the reverse cases matter: an owl that all nine model guesses got and no human did is a reminder that “human-legible” and “AI-illegible” are different properties, and a drawing has to hit both at once.

Anand’s own reading was cool-headed: in terms of the percentage difference, it’s relatively weak, and obviously, the AI capabilities are going to keep going up. 40:44 But the skill being practised — finding where models and people diverge — is the durable one. He asked the TAs to put more such mechanisms into the next assignments, GA5 and GA6: more mechanisms by which we can get AI to fail while humans cannot. 41:16

Run it. Stage it. Check the checker.

The last idea on Anand’s list came from his own history. He had catalogued about 10,000, 12,000 prompts he had given coding and non-coding agents, to extract the techniques he actually uses to specify requirements and verify output. 41:56 He posted the interactive list in the chat; that evening he published it as 52 Ways I Specify and Verify Agents (30 for specifying, 22 for verifying, ranked by his own judgment of frequency, impact and effectiveness — not a controlled test). Public, later that day He walked through the top three ways to verify.

1 · Run it for real

Give the agent an environment where its output can be executed and validated deterministically — a code runner, a chess engine, an API, a physical laboratory. 42:36 The frontier, he argued, is verification environments beyond code. Most students solved GA3’s 3D-face question with Meshy plus a chatbot, which is fine; few found an environment where an agent could build a 3D model, look at it and improve it. On face morphing he was blunt: most submissions are simply doing an animated transition between these, that’s rubbish; any in-between frame should look like a real photo of a real person. 44:20 Even finding that such environments exist is, he said, a useful skill to teach.

2 · Stage before the irreversible step

Could students be made to decide, across a series of actions, which are reversible — and spend tokens on checking where it can’t be undone? Anand floated it as an end-term question 45:12: out of these 10 things, which ones of these would you commit and deploy, or tell the agent to commit and deploy? 46:28 Try a small version.

3 · Verify the verifier

In a number of cases, I’ve had situations where the test cases were wrong, where the judge prompt was wrong. 46:48 This is not a private complaint. In SpecGuardBench, a preprint from October 2026, four coding agents were asked to build validation harnesses from natural-language specifications across 291 tasks; none produced a faithful harness for more than 63% of them, often writing checks that mirrored the reference code rather than the spec. Research A plausible test suite is not evidence that the requirement was met.

He suggested the TAs design questions from both lists, for specifying and for verifying. 47:28 That was the end of his agenda. A student had been waiting in the chat.

“Making us waste tokens? :)”

Kartikkumar, a student, had spotted the pattern. If the course believes in finite verification budgets and noticing changes, were the last assignment’s odd questions built on purpose to make students spend? Anand said yes, partly — and that the intent behind some of the GA3 questions was trickier than that. 48:36 He opened GA3 and went through it from the bottom.

Question 8 asked which faces are easiest to morph. The point was not your ranking but whether you could predict your classmates’ ranking, because predicting another human’s behavior is something you will find really useful in the future. 49:24 There were two ways to solve it, he said: build intuition for the psychology of people — or this:

Both may be valuable. But they are different skills, and one score can’t tell them apart. If the class coordinates, the number stops measuring how well anyone predicts people — it measures how well they organised.

Question 7, the morph itself, he called a classic solver kind of a problem: one prompt to a capable agent finds a good morphing algorithm; Luna will solve it with 10,000 tokens at most. 50:48 Humans Only had been converted to a bonus because the dates were too tight and a lot of people didn’t see it — which became the lesson he most wanted to land.

Go further, he said: have your agent watch the TDS sites, solve the questions and submit on your behalf — preferably in a reversible way. That is why the questions have a limited number of checks. 10 out of 10 checks is very liberal, but if your agent happily goes checking, checking, checking, at some point, it becomes irreversible. So tell it to check daily, but leave some for my judgment at the end because you might mess up. 52:24 And to Kartikkumar’s joke, a straight answer: did you waste tokens? Yes, absolutely. As much as possible, burn it like crazy, and learn from getting burnt. 52:56

The fair version of “burn tokens to learn” needs the tokens to be affordable for everyone, and the waste to teach something — when to stop, what to check — rather than merely cost time. That is the line between a lesson in budgeting and a tax on attention.

A marketplace that paid for trading

Kartikkumar then raised a practical worry. In Project 1’s Human Exchange, students record a service they can offer other people and trade them for credits. Even though a lot of us had recorded our services, it never showed up on the marketplace. 53:42 If the offers appear only after the deadline, how can anyone buy them?

They can’t, Anand said; the platform can’t move dates without being rebuilt, so there will be a round two, with slightly different rules he was still working out. 54:06 Then, without being asked, he said what had gone wrong with round one.

Did it matter how you drew it?

Kartikkumar had one more. In Project 1’s Picture Stories, students make a picture that tells a three-line story. He had drawn his by hand. When he evaluated others’, most had used AI. Weren’t they supposed to draw? 55:40 Anand didn’t think he had said so, and Kartikkumar read the instructions aloud: draw it, photograph a drawing, make a collage, or use an AI tool. 55:54

Anand was perfectly fine with that: the point is, can you craft a good story that meets these criteria? 56:08 He would think about whether hand-drawing mattered, but I don’t think drawing is a skill that I want to teach in TDS. 56:48 Kartikkumar’s frustration is real — the hand-drawn route was harder for the same marks — but the rubric was doing what it said. The question is always which skill a score claims to measure.

Jaideep suggested a sharper version: count the strokes, and let agents draw with as few as possible, to see how well students direct them. 57:12 Anand saw the problem at once: it is more agent intelligence. Given the question and the optimization criteria, the agent will converge to a solution, and not entirely sure that will require human steering. 57:40 Jaideep proposed testing it internally first.

Change x halfway through

With time for one more, Adwitiya offered a suggestion rather than a question. In a timed exam, he said — it’s rather a crude way of putting it — part of the point is to get us to waste time, so that we are stuck on one question whereas we could be scoring on another one, which teaches prioritising. 58:20 Some questions come with a file to download and paste to an agent: I doubt whether even 1% of us would have, you know, actually opened the file. 58:56

Anand agreed with the theme and named its leak again: one student finds the catch and posts it on the WhatsApp group, telling everyone not to try too hard on that question. 59:28 Hide the information somewhere hard for an agent to see, he suggested — not white text, which models now catch, but something that isn’t text at all. What if it’s in video? 60:24

Adwitiya had something else in mind. Start the file with x = 10, y = 5 and a long calculation; at the end, edit one line — x was 5.5. An agent would have to recompute everything; a student who actually read the question would just fix the prompt. 60:36

A premise, untested. Agents may handle a late edit perfectly well, and a buried change is only fair if the question is honest that inputs can change. The version worth building rewards noticing that a dependency changed — the same skill as watching the course for announcements — rather than catching people out.

And that was the end. The recording runs another four minutes after the last words; nothing more is said. The meeting closed the way it opened: with a question about how to make a student read.