# Transcript

You were the data set, by the way. You probably received a survey asking a bunch of questions like, "What do you fear about the use of AI without supervision?" and things like that. And it turned out that **the majority of you said, "I'm stopped because I have to check the results."** Many of you said it doesn't know my world, you have to be in the room, presence, and so on. But the major problem is that you have to verify the result.

In a sense, that's weird because you're asking it to do the work and it does it so fast and the work piles up, which means that now you are faced with a ton of verification and we're not used to verification. One of my colleagues called it **"AI Fatigue."** Now we're having to change the way in which we do our job, and that's a problem.

So how does one deal with AI fatigue? How does one deal with the problem of verification? I don't know, and like with almost everything with life these days, I asked ChatGPT. I said, "How do I deal with it?" The question literally was: "Go through as many conversations as you can in the month of August and tell me how I deal with verification." And this is what it said I'm doing.

Here are the—I don't know how many—18 ways, prioritized, that I'm dealing with this. And let's go through the top ones. First, **what I'm doing is running—that is executing—not reading the output.** Reading the output is painful. But if it is code, then I say: **"Create and run test cases that comprehensively and efficiently cover realistic user scenarios."** Now that's great. It will create the test cases, it will run the test cases, I glance at the test cases, it seems fine. That's not a bad idea. And this is really powerful for anything that is verifiable. Code is verifiable. Mathematics is verifiable. Data science is verifiable. And what I mean by verifiable is that the kinds of errors that an LLM makes are the kinds of errors a human makes. It'll hallucinate, it'll transpose numbers, things like that. **The kind of errors code makes are very different: syntax errors, runtime errors. And they are able to catch each other pretty well.**

So that works fine, this is a pretty good technique. But then you say, "Wait, hold on, Anand, this works for data, this works for numbers, this works for code, but would it work for, you know, contracts if I have to review them?" Maybe. We can take some of this and start applying it. One of the things that we did was say: **"Look, contracts need to be reviewed against a checklist of about, let's say, 21 things."** This is where a publisher says, "I'm signing a contract with an author," and we need to make sure that there are no hidden or additional costs, intellectual property should be mentioned, force majeure should be mentioned. Fine. Take a checklist, go through it, see if the contract has it, yes or no, and provide some evidence.

So what this is saying, for instance, is contract number two is covered in terms of quality and standards because in Section 2.2, it literally says, "If the publisher determines blah blah blah is complete, satisfactory, etc., then fine, otherwise they may reject it." Okay. At least it brings the verification down one step closer. You can also pass it to another human or even another agent and get the job done.

Which brings us to the second technique that it said that I use very often, which is **triangulating sources.** For instance, check Gmail, chat, and online. Get it from multiple places. My emails and transcripts are evidence of my position, not a proof. Verify anything that I've said against other sources. Put another way, **if I can get two or three pieces of evidence and map it, that is pretty useful. And it's not just evidence; it's also evaluators. If I can get two LLMs to evaluate, then that's pretty powerful.**

What we found, for instance, was when we asked different models to take a chat message like, "Could I take a quick look at my invoice?" or "Help adding with some items," etc., and see if they're able to classify it into the right category. We find that some of them succeed. So for instance, GPT-4o-mini managed to get it right. But Amazon's Nova Lite one said, "I will put it in Change Order," whereas it should have been in "Place Order." So some models make lots of mistakes, for some items there are lots of mistakes. **But what we found is that the mistakes are not correlated. Some models tend to make very different mistakes from others, which means that if I start cross-checking what one model says with another, there is a reasonable chance of catching the errors.**

In fact, models are not very auto-correlated. So if you ask them to catch their own errors, they do a reasonably good job. We found, for instance, that on average these very cheap models were making errors about 14% of the time. But **if we asked two models to cross-check and said, "I will only allow if both the models agree," then that reduced the error rate to about 3.7%.** That's pretty good. And the amount of manual checking that I had to do was about 12.5%. Okay, still better than 100%. **And if we had five models cross-check and said, "I will only allow if all five models agree," then the error rate drops to 0.7%, which is pretty impressive.** And the effort—the human effort—where even if one of these models disagree with any of the others, I still have to manually check, is still only 28%, which means **72% effort saving at 99.3% quality.** That's a pretty good technique.

So this happens to be the second most common that I use. The third is making sure that we **define the test first.** Spec failure, as you saw a short while ago, is one of the top reasons for failure. And therefore I make sure that—you know, well okay, somewhere I said silence is better than a mediocre reply. But that apart, create and run test cases that comprehensively and efficiently cover realistic user scenarios. Effectively, say **"define the tests."** I try and define the tests where I can, or I delegate that, but making sure that there is a reasonable basis for what we're framing is also helpful.

And it goes on. Where I'm going with this is not that models cannot be relied upon, but rather that **humans can't be relied upon either, and we have millennia of experience of dealing with unreliable humans. And many of those techniques transfer to models as well.** I wouldn't therefore say we can't use something that is unreliable. We have to manage around it. That's literally what management is about. And there are many ways of doing that.

So you're right in that what's stopping you is that you'd have to check it anyway. But don't let that stop you permanently. Just treat that as the next bottleneck in a continuously moving cycle. Another thing that you said was, "What would you hand over tomorrow?" Many people said, "I'll hand over tasks where AI writes something for me." And keep in mind that this is when AI becomes super capable. The question was, "Look, assume that it is perfect, it can really do everything, it doesn't really make mistakes, even then, what are some of the tasks that you would hand over?" And writing it up for you, chasing it up for you, crunching it for you—all of these seem to be things that you were comfortable handing over.

What would you keep? And many people said—in fact, the vast majority said—"I would still keep something for myself." Even after AI can do anything and everything, I won't delegate everything. What were some of those things? **One of the biggest things was taking the final call. Some people said "the people part," "the thinking."** Some people said "nothing, just take it all." That's fair.

Let's talk a bit about the final call. Now what we're seeing is that **the way in which we're taking decisions is also changing a fair bit.** Let me give you first a sense of how I take decisions based on AI, and I did exactly the same thing. Look, I don't really know how I'm taking final calls based on AI. Am I delegating it? Am I holding back something? So "go through my last two months' conversations and tell me how I'm doing this."

Turns out that here is my current process for delegating versus not delegating or whatever. It ranges from **directing**—that is, I ask AI to choose the method—or **AI recommends the method and then I decide.** Or **AI finishes the whole thing but I review the draft** and then say, "Okay, go ahead" or "Don't go ahead." Sometimes I just say, "**You test it and as long as your tests pass, I'm fine.**" Or sometimes I just **delegate.** There are times when I've sent WhatsApp messages, emails, etc., without review. The alternate was I don't even reply, and my logic is my AI is better than my silence. So let's go with it.

The technique that I seem to be using the most is **"Draft only, never send."** So you do all the work, take all the decisions. I will retain the final call, which seems aligned with what we just saw. But another one is to say, "Look, **you work independently but interrupt me. You can, for instance, ask me one clarifying question or recommend that we discuss live on this.** When you're not sure, you stop and pause and ask me about what we might want to do."

Another technique, and the third most popular technique, is "You go ahead, you implement, you test, you fix. Everything is fine. And what I'm going to do is just make sure that the tests that were executed were kind of okay." I'll skip these blah blah blah, I'll see if I can jump to some cases where I have, yeah, "Go with your call," "**Act directly.**"

There are times when I just let it update the APIs. Very rare, but **give it access to production systems and say, "Yeah, just go ahead," or "You can make the purchase in the account right away."** And yeah, I've let it make purchases on my behalf. I've given it a lot of details: my keys, my cards, all sorts of things. Not always, but there are times when even that works. So the set of people that have said, for instance, "Nothing, take it all," you have a point. This is valid.

And the question then becomes, when do you do which? Obviously, you need a certain amount of risk tiering, you need to understand at what point what works. Which is not to say that I have a universal formula for this; each of you will have to develop your own approach. But there's one thing that I want you to keep in mind while you are making this call. And that is that **this is a moving frontier. Anything that you learn today, any judgment that you build, any skill that you build, is in all likelihood depreciating.**

See what's happening is that the models are getting smarter and smarter. This is a chart where on the x-axis, you have the cost of models—right side is expensive, left side is inexpensive. And on the y-axis, you have the smartness of the models. And as time progresses, the models have become cheaper and smarter. This is not news to you. This is what's happening. What may be news is that, for instance, GPT-4 which was released—when was it released? Long time ago, around July of... or March 23? Oh yeah, okay well... in... No, okay, I probably got this graph wrong. Let's take o1-preview, which was released sometime in September '24. Very expensive model, $15 per million tokens. And it was roughly as smart as a master's student.

Let's fast forward a little bit and let's actually take Gemini 1.5 Flash preview. April 2025—less than a year later—Gemini Flash has roughly the same level of intelligence at 15 cents. Think about it: $15, 15 cents, less than a year. **What that means is that you used to be able to hire a master's level student in August '24 for $15; now the same master's student costs 15 cents.** If that is the pace at which the cost of intelligence is falling—and it is falling faster than many of us are able to comprehend—then the judgment that we build to be able to do the next level, maybe that of a tenured professor, will probably fall at the same pace. And the next level, and the next level, we don't know when it's going to stop or if it's going to stop.

So as a result, what used to be the case, which was the bulk of the time was spent in execution, a little bit therefore on the stuff that comes before and after, even less on the stuff that used to happen before and after—that's obviously changing. Now the execution takes very little time and we have to therefore design what goes in, we have to test what comes out: specification, verification, that sort of a thing. And **in the future, that will shrink even further. We'll probably have to say, "I want to create an agentic organization that has this broad vision and purpose," and then come back and say, "I want to evaluate its performance on the stock market or some such virtual market."** Going beyond.

Which means that the skills that we had built here are also going to fade, right? So there is a need to constantly benchmark ourselves and rapidly move. So to that earlier intuition where you were saying, "I need to watch for what I delegate, how much I delegate," I think you have the right intuition. The thing that may change is **what you pick up now as judgment, what you pick up now as skills, are not likely to stay the same for very long.** Keep that in mind.

I have only one strategy that I'm using right now—not saying this is the best or even an effective one, but it's working for me—which is **delegating maximally.** My idea is this: take whatever I can, give it to AI. It'll fail, it'll mess up at some point. But that's exactly what I need to learn now. Because that's what it _can't_ do. Give it a shot, delegate maximally. All the best.

[Applause]