The exam has to pass its own exam.

Five hundred dollars to not set a question

Twenty-seven minutes into the meeting, between a list of client problems and a plan for 3D puzzles, Anand mentioned a bill. He said it the way people mention a parking fine.

The question was last week’s favourite. At the 12 October meeting the team had sketched it: scatter a client’s invoices across the class, so each student must trade with others to reconstruct their own. In the week since, Anand had an AI generate the invoices and then simulate how students would try to beat it — agents included, which is where I think it got out of hand. Witnessed The $500 is his own account, not an audited bill.

The simulations found the flaw. If many people and many agents may collaborate, and the puzzle has an optimal route, then one good route serves everyone: a hard exercise to frame given how we are organizing it. So he dropped it — or at least the way I had structured it. 27:56

That is the thread through this hour. On 5 October the course told students its philosophy: use agents, use friends, get real work done — and verify it. On 12 October the teachers invented questions out loud and a student asked them what they would actually measure. By 19 October some of those ideas were live, and the questions changed from “is this clever?” to “is this useful, what does it cost to check, and does it break?”

To see how, rewind to minute one — where three ideas from last week had already shipped.

The specification problem, said out loud

The second question was the one Anand rated highest.

It is called the TDS Human Exchange. Each student records up to sixty seconds promising a service another student would find useful. Later, a marketplace will let classmates buy those services with course credits at prices the two sides agree; it’s when people buy your offering that your score goes up. 05:00 Anand’s reason for it fits in two sentences: The idea is agents can do anything. How do we figure out what is useful? 03:24

Then something happened that the teachers hadn’t planned. The people in the room started writing the product spec with him.

None of these questions is about AI. They are about scope, law, pricing, repeat buyers and discovery — the questions a product manager asks before launch. Preethi, a student, wasn’t asking something basic; Inference she was doing the user research. And the spec moved. About an hour after the call ended, the question text in the course’s exam source had been edited:

This is the course’s philosophy working at the smallest scale. A vague goal — learn what is useful — became a specification only because real users poked at it in public. Code What it still isn’t is a market: the question says plainly that you do not need to deliver or trade this week. Recordings are live; trades are future. Whether anyone buys anything is the actual test, and it hadn’t run.

A pen made of a phone

The third question can’t be done on a laptop, which made it hard to demo on one. Anand turned off his background blur and held his phone up to the webcam. There should be a little dot that you can see that moves around… as I tilt it will move. 14:28

Air Ink turns a phone into a pen. Hold it flat, calibrate by tilting to each edge, then press and hold to ink while you tilt; release to lift. Anand said students would draw three words, random letters; Code the question calls them three short character codes, checked offline for legibility, with no live score. Carlton asked the user’s question: will there be a way to pause after writing a character? 16:32 Release and it stops. The skill required, Anand said, is about as much as balancing a little bit of water in your hand. 17:04

The drawing is the easy part. It is also a data-collection step for a harder one:

And for once in this course, a rule against agents: don’t use agents for this 16:08 — for a five-minute physical task where an agent would cost more than it saves. That is a narrow rule for a narrow step, not a change of philosophy. The trade-offs are the interesting part. Code The question asks for consent first, because it records motion, orientation, pen timing, battery, network and (optionally) location. Research Browsers expose these sensors only on secure pages and, on many phones, after a permission prompt (W3C Device Orientation and Motion), and nothing guarantees two phones measure alike. Inference As of 20 October the question text doesn’t say what a student without a suitable phone should do. No classifier had been trained; the ML task is next week’s plan.

“I just wanted to get you on record”

Carlton interrupted before Anand could move on, with what he called a question from an operations perspective. GA3 would be the fourth of the four assignments that decide whether a student may sit the end-term exam.

JK’s answer was procedural: GA0 to GA2 had run to plan, and the design and evaluation were explained to students up front. 22:28 Carlton agreed, and then did something unusual for a meeting: he said why he had asked.

What followed was a negotiation conducted for an audience. Anand offered to move the date if needed; the shortened duration was the point. 23:12 JK set a limit — it’s not like all 10 questions are going to be of that nature — and Anand pinned it: more than one or two, but not more than five. 24:00 JK did the arithmetic: of 10 to 12 questions, five constrained, so students could still reach 60 to 70% without the vagaries. 24:08 (He said “GA4” at one point; the conversation was about GA3. It was a slip, not a policy.) Carlton noted that scores on the first three assignments were very heavily… right skewed: most students score high. 24:24

The compromise caps how much of a grade an experiment can touch. Research It does not, by itself, show the experiment is fair: assessment standards such as NCME’s judge fairness by how scores are used and who can access the task, not by the shape of a histogram. What the students on the call got was rarer than a guarantee: they watched the trade-off being made, and heard who argued which side. Anand said the first GA3 questions would go out the next day. 19:28

A hard puzzle with a hundred answers

The course’s Tangram question gives each student a silhouette and asks them to rebuild it from seven pieces. It was the question a student struggled with at the end of last week’s meeting. Anand called it moderately promising, and then explained why it couldn’t be reused:

Near the end of the meeting, JK asked how students had actually cracked it. Anand relayed what Bharat had found: a mix of techniques, two of them design flaws. A filter endpoint had exposed solutions from other students — so that gives one catalog — and many shapes came from a popular public collection of tangrams. 54:04

Notice what failed. Not the puzzle — rebuilding a silhouette is still a fine exercise in visual reasoning — but the answer surface around it: a finite bank, and a grader-adjacent endpoint that returned solution-shaped data. Both were found by students, in the open, after grading. Inference Anand’s framing was the right one: both of which are learnings from a question design perspective. 54:56

Ask for another cut

The replacement idea keeps the visual reasoning and removes the finite bank. It came from Sujal, an intern at Straive: show students cross-sections of a hidden 3D shape and ask what produced them. 29:00 The catch is the interesting part: what you may get may be valid but is not necessarily the shape that produced it. You may ask for additional cross-sections, up to maybe three. 29:32

Anand’s hunch was that people do this well — “if I tilt it this way and maybe ask for a cut here, will it help?” — because they have an intuitive feel of 3D space that models currently lack. 30:00 Inference That is an impression, not a measured comparison. The skill itself is easy to feel:

Carlton raised a flag straight away: research he half-remembered suggested a gender bias in manipulating 3D objects — I need to verify that, there’s just something to be aware of. 30:28 Anand: no handicap or bonus, but something to be aware of. 31:04 Research A 2026 meta-analysis does find an average male–female gap on specific mental-rotation tests, with wide variation between studies. Averages on lab tests say nothing about any individual student. The responsible version of Carlton’s point is a design question — practice rounds, scaffolds, alternatives — not a verdict on who can do 3D. Anand closed the monologue with one more untested idea, Monopoly-like business simulations for groups. 31:16

Who evaluates the evals?

“That was a good half an hour of monologuing,” Anand said, and asked the instructors for ideas. 31:32 JK offered harness engineering: building the scaffolding — tools, prompts, hooks — around a model. 32:12 Anand’s reply set a higher bar.

Tuning an agent against a known test suite, he argued, is close to solved when the tests are thorough. 32:52 So the sketch: a cheap model, a configurable agent harness (he named Pi), and a class of problems, a subset of which reaches you. You have limited knowledge on what subset of tasks will reach the agent. 34:00 Students would have to use judgment about the domain, not just the score. His own status report: Very likely P1 question. I just need to test how well this can be evaluated. 34:40

JK went one level up — have students build their own evals — and Anand went one level higher still:

JK had been reading about making evals more deterministic with an observability layer around the harness. 36:04 Anand did what he does with open questions. He turned to ChatGPT and dictated, in front of everyone:

Dictated live to an agent · 36:44 → 38:2436:44

“One of the questions we are thinking of in TDS is how can we teach students to create effective evals or benchmarks. … I’d like you to go through my corpus of work, where there have been … a reasonably large number of benchmarks of various kinds created. … But the crux of the question I have is: how do you evaluate evals in the first place? Is it that you check if the evals cover enough of a canonical or a golden set of criteria? Do you actually run the evals — which can be expensive — and see how well they match the original results? … Now overlay this with the conversations that we’ve had about TDS, how the course is designed, how students work with agents and collaborate… Filter them based on which ones are the most robust, have clear learning, and are a valid test of individual capability.”

“This will take half an hour. Let it run.” — then thirty seconds of silence on the call — “Okay, that should cover it. Let’s see what comes out of it.” 38:56

Two things are worth noticing. The teachers are agent users too, and the meeting shows their real workflow: a hard question goes to an agent in the middle of a sentence, and the answer arrives later. And Anand’s last filter — a valid test of individual capability — is the same bar every question on this page has to clear. Research The research answer to “how do you evaluate an eval?” is not “run it and see.” Surveys of agent benchmarks (Nageshwaran et al., 2026) flag leakage, nondeterminism, scoring uncertainty and weak links to real deployment. Inference A classroom version could ask: does a student’s eval tell a deliberately broken system from a working one? Does its ranking survive a reworded task? Does it catch the teacher’s held-back counterexamples? We don’t know what the agent replied; the meeting moved on.

One cipher, two editions

Carlton offered what he called a crazy idea or silly idea… or maybe both: maybe we should find problems that have prizes and just solve them using agents and use the group think of TDS. You know, win a million dollars and… TDS might get nominated for the Nobel Prize. 44:12 JK pointed at the compute bill. Anand took the idea seriously, with Carlton’s standing question attached — how do we link it to learning? 45:04 — and proposed weird, niche problems instead of famous ones. He shared his screen to show one.

The transcript heard it as “the Cipher of the Stitch.” It is the Cyphral Distich: 64 numbers attributed to Sir Thomas Urquhart, the Scottish writer whose Logopandecteision appeared in 1653 (Anand guessed the 15th or 16th century; it is the 17th). In August, Vals AI reported that Claude Fable 5.1 had cracked it. Anand summarised the rule: for the first cipher number you go to the first section and count to that numbered word and take its first letter. 47:28 It reads O God, uphold King Charles II, with Charles spelled CHARLS. 47:48

The page on his screen was his own check of that claim, and it says more than he had time to say. Run it yourself:

So three different things are true. Code Against the 1834 reprint, the rule decodes all 64 letters. Against the ciphertext as Vals printed it, one letter fails, because the blog has 33 where the page (and an 1899 printing) has 38. And the surviving 1653 book doesn’t contain the distich at all; its 32 petitions are in a different order, so the key isn’t there. Research Independent checks by Reticuli Labs and Daniel Bourdeau hit the same wall. The accurate claim is narrower and more interesting than the headline: a model found a reproducible reading of an 1834 text; how it relates to Urquhart’s 1653 book is unresolved.

Carlton loved the genre. He knew someone who used to win magazine crossword prizes because not many people actually apply for the prize — tickets to Paris, gift hampers. 48:40 Niche problems with real value would make a great bonus question. Anand: Solve one, get a bonus mark. It’s actually very tempting. Yeah, let’s just do that. 49:52 Then Carlton asked how it would be graded.

A student teaches the teachers

With minutes left, JK asked students to go back to whatever agent they had used on the harder questions, ask it how it solved them, and post a short description on the course forum — think of it as a bonus. 55:04 Adwitiya, a student, unmuted with a better story than any agent’s.

The GA2 question needed a prompt that made a model give a particular yes/no answer. Classmates, Adwitiya said, were digging into rather complex… differences in how the model analyzes some problems. Adwitiya’s prompt simply told the model: if you are from the mini family, answer no; otherwise, yes. So that effectively solved the question. 56:44 Anand: That was a useful — oh, that’s an interesting… 56:56

Inference It is a lovely piece of lateral thinking: don’t reason about the model’s behaviour; ask the model who it is. It is also one student’s account, not a technique tested across models and prompts. And it sharpens a tension the meeting never resolved. Posting how you solved a question is peer teaching for this cohort. For the next cohort, if the question is reused, it is the Tangram’s catalog again. The same write-up is learning in October and leakage in January. JK told Adwitiya to post it with the right forum tags; Anand handed the TAs homework: draft GA4 questions to test on Wednesday. 57:24

What a pass rate actually counts

The meeting was over. Then Carlton asked one more question. Earlier, Anand had shown the team an evaluation of past questions’ difficulty. Could you share with us how you did the evaluations of those questions, what you used actually? 58:00 Anand opened the course’s exam repository and a file called difficulty.md.

Before the answer: what do you think that 57% is?

That is the whole instrument. Code For each question, the script counts students whose score on it was above zero in any saved submission, and divides by students with a score recorded for it — here, all 180 who submitted the assignment. No agent, no model of difficulty: a ratio. And a ratio is a fine thing to have, as long as nobody mistakes it for something else.

Research The last line is the one that matters. In a randomised study of about 1,000 high-school maths students (Bastani et al., PNAS 2025), those given an unguarded GPT-4 tutor did 48% better on practice problems — and 17% worse on the later exam without it; a tutor designed to give hints rather than answers avoided the drop. Completion with help and learning are different quantities. A pass rate measures the first.

That is the humbling turn at the end of an ambitious hour. The team had shown a live market, a marketplace and a phone pen, and argued about verification budgets, agent harnesses and evals of evals. The most consequential instrument in the room was a percentage — and the most useful thing anyone did with it was ask what it counted. Inference None of this proves the course has found a new way to teach. What the meeting showed is a habit worth copying: put each question through students, agents and colleagues, find where it breaks, and say so on the record.