AI-generated and lightly corrected against the audio. Speaker labels may be wrong. Click a time to play from there. Participants' names read out during the roll call are left out. The Markdown original is also available.
0:00HostToday’s session on leadership and scaling AI. Professor Anand is an LLM psychologist and an alumnus of IIM Bangalore. His work focuses on understanding and working with LLM models, bringing valuable perspectives to the evolving field of AI. As we begin our final module of AIPL and today’s session, we will look beyond the technology itself and explore the technology perspective with the essence of leadership and drive its implementation and scale AI initiatives. I’m sure this will be an insightful and enriching session for all of you. With that, I warmly welcome Professor Anand and hand over the session to you. Over to you, sir.
0:46AnandThank you. The session is largely interactive, so let’s start. Sorry, the session will be largely interactive, and you’ll be doing stuff. It’s more workshop mode than lecture mode. Ultimately, you’ve seen AI, you’ve used AI. Let’s start with a small survey. I am curious, and I’ll share a link in the chat window and let me also share my screen, that way you’ll know what’s on the survey.
1:33AnandWhen you open the link, you should see just one button. Click on it, and there’s this survey that you’d need to answer. If you prefer answering it on your phone, the QR code is on the top right. I’d love for you to just spend a few minutes answering these. Just a quick check, there are about 20-odd participants, so I’ll wait till we have about 20 responses, at least on question one.
2:06AnandQuestions are simple. How far have you gone with AI in the last 30 days? You may even say, "I have never used AI this month," that’s fine. Or you’ve asked it for some inputs. You’ve used its output but you were checking every single thing. Or you were checking only some of it. Or you let it act without deciding anything. All of these are perfectly valid answers, just which is the most that you have gone to in the last 30 days.
2:38AnandSecond question: Is there something that you did not ask AI to do last week because you thought it couldn’t do it? For whatever reason. Now, I’m not saying you should pick something that was not actually possible; it might have been possible, just name what you did not try and why. And if there was nothing, then nothing is fine too.
3:03AnandThird, do share: Would you trust an AI? And there are some details behind it. Just go through the question and decide if, for that scenario, you would trust the AI. I’m just going through this a little quicker for a short while. And what decisions of AI would you check? These are the four questions in the form. I’ll wait till at least the first one has—well, right now we have four responses. We have a total of 20-odd participants. I’ll wait for this to get to over 20. And the QR code, for those of you who prefer the QR code, is also available online. If anyone has any problems accessing it, please just type in the chat window.
5:50MukulHi, good morning Anand. Not able to access the post-scanning of the QR.
5:55AnandI’ll paste it on the chat again in any case, in case you joined late and weren't able to see it. No worries, I’ve just put it in the chat.
6:19MukulYes, now I can access. Thanks.
9:05AnandOkay, we’ve crossed 20. That’s a good sign. For those of you who joined in late or are not able to access it, we’ll figure it out. I am curious to see what the results are. Let’s take a look.
9:33AnandOkay, so this is what it’s looking like for the 23 who have been answering it so far. None who haven’t used AI at all, so this is obviously a group that’s active on AI. About a sixth of you are asking AI for ideas and answers. About half of you—more than half of you—are using AI’s work but meticulously checking every single thing, and that is clearly where the majority of this group is. Some of you, almost a quarter of you, are using AI’s work but not checking every single thing; you are spot checking, so to speak. And at least one of you has let AI do something without checking it at all.
10:24AnandNow, this is the spectrum on which we as a group are. And it’s not like, obviously, there is one right thing, one wrong thing, etc. It depends on which task and what level of authority you want to delegate. But what I find is that as AI’s capability is constantly growing, the more tasks that we find at every level of the spectrum, the more we are finding useful use cases to use AI for.
10:57AnandTherefore, if you start pushing yourself—saying, "Look, this week I only used AI for getting ideas, but I did not actually use it even with checking"—then you might start considering: is there a way I could push myself or my team to go a step further and see where AI fails? At least let it fail because you’re checking anything anyway. And where it fails, you know this is the limit. Now, why does this matter? That’s part of what we are going to be talking about in the first 10-15 minutes.
11:34AnandWhile that’s happening, the other questions we will be going through in a short while, but let me come back to why it matters by talking a little about how LLMs have evolved over time. Now, I’m happily assuming here that you can see my screen, but let’s do a small check. Is the small text that I’ve highlighted here legible to you? If anyone finds that you are not able to read that text, I’ll increase the font size, but in that case, please just type a message in the chat window saying "small" or something. I’m seeing the thumbs up, so I’m assuming that the text is legible. I will keep it at this font.
12:18AnandSecondly, as we go along, please just type stuff in the chat window or just unmute yourself. There isn’t a fixed thing that we have to cover. I’ve got a bunch of things that I’ll anyway end up sharing one way or the other, but if you pose a question, that way at least one, maybe a few people, would have benefited from aligning to your question, and I’ll explain it in a way that makes more sense to you. Alok has a question on a streaming lag; if anyone else is facing a streaming lag, please do share on the chat window.
12:53AnandBut what I’m going to do is talk a little bit about how LLMs have evolved over time. Now, let’s actually move the story to maybe around March 2023. There were a bunch of models available at that particular time. You can characterize models based on cost and quality; that’s one of the ways of doing it. The X-axis is the cost. Cost per million input tokens. By now, you’re probably familiar with what tokens are. The way I think about them is: tokens are the words in LLM’s language, and all the Harry Potters put together is roughly 1 million tokens, or the entire King James Bible is roughly 1 million tokens.
13:43AnandTo process—if we said, "Summarize Harry Potter in one sentence"—then the cost for a model like Llama 13B would have been 23 cents. GPT-4 would have cost $30. 23 cents, $30. That’s over a hundred times more. And the level of intelligence was also different. Some models were only as smart as middle schoolers. Some were about as smart as high school freshmen or high school graduates. GPT-4 was one of those early models that was almost as smart as a college junior.
14:19AnandThings changed over time. We had newer models coming up, and as things progressed, for instance, let’s look at O1 preview. When by September 2024 when it was released, it was about as smart as a master's student. Things progressed, and with the release of GPT-4.5 preview, we had a model about as smart as a PhD candidate. With Gemini 2.5 Pro coming in, we had a model that was about as smart as a tenured professor. And that progression has continued and gone beyond.
14:54AnandBut that is one axis. Take a look at the other axis: the cost of intelligence. GPT-4 had the intelligence of a college junior and cost $30 to read Harry Potter. Time moved, and you had the same level of intelligence coming in for $10 from GPT-4 again in November—one third the price. Things moved, and then we had GPT-4o mini: 15 cents for the same or approximately same level of intelligence.
15:28AnandNow, this is July ‘24, and that was March ‘23. So about a year and a quarter later, we have a model that is 200 times cheaper for the same intelligence. Just more than a year, 200 times reduction in cost for the same level of intelligence. Now, this is the pace at which things have been progressing constantly. You have, let’s say, O1 released as a master's student level intelligence, $15. That’s in December ‘24. And DeepSeek-V3, which of course was a big revolution at that time, two months later reduced it for 17 cents. That’s 1/50th of the cost. GPT-4.5 came in at $75, and same level of intelligence, let’s see, yeah, Gemini 2.5 Pro preview for $2.50. And that’s just in three months.
16:28AnandThis progression of moving towards the top left is so incredible that today we have models like GLM 5.3 Flash which are almost as good as a tenured professor at 15 cents per million tokens. Now, think about it. You go to a professor and say, "I want you to read this research report which is as big as the entire King James Bible. How much will you charge me for it?" And they say, "15 cents. And I’ll have it in the next few seconds."
17:00AnandThat is roughly where we are in terms of the scale of intelligence. But also, notice how fast they’ve moved from middle schooler to freshman to graduate, etc. They’re learning faster than we are. So if we are operating in that realm, then the question that we also have to ask ourselves is: How far would you go with a tenured professor? Would you not use them at all? Would you ask them for some ideas and answers and then decide what you want to do? Would you ask them to do something and then check all of their work? Or maybe check some of their work? Or just give them a consulting assignment and let them act on it independently?
17:46AnandOf course, the answer will vary based on the task, no question about it. But now, I think we have to start thinking about: have we, because we have something like a thousand professors sitting in our pockets, used them as effectively as we can? And if not, are there opportunities for us to push it one level forward? And that becomes a big theme in my interpretation of when do you let AI act and arguably how as well.
18:17AnandLet’s move on. The second question that we looked at was: What is one task that you did not ask AI to do last week because you thought it couldn’t do it? Now, let’s see. Some people said help arrange meetings, hack into an account, code review validation verification AI evals, nothing, nothing. Creating context of ideas or background for the task that I’m trying to get done from AI. Specifically for ABAP coding, it gets confused on the syntax, so I haven’t had it do it. Convert from Cypress tests to Playwright. Process and solutions only. Deep research because it mentions some files that don’t exist. Final mixing of text with images, very long prompt, etc. Booking flight tickets. Collaborative ideas and research is what you’ve been using it for, but you need help on technological work since in a Workday technology—long way to go where I can train it and get use out of it. Legacy systems it struggles with. HLDs it struggles with.
19:45AnandBut I’m also conscious that there are a whole bunch of people who’ve said "nothing," indicating either that you haven’t had a task that you thought it couldn’t do or you think it can do everything. Both are perfectly valid. This is an interesting set. Now, open question to all of you. Is there anything that any of you see on this list that you say, "Wait, hold on, I’ve tried this and it works"?
20:14AnandAnd maybe I should copy the whole thing. Is there a way I could just download—let me just see if I can make this easier for you and put it in the chat. Hold on, just processing it. I’ll remove all the "nones" and "nothings," which leaves us with about 14 responses which I will sort alphabetically and paste in the chat window, and also share on my screen so you might be able to see each of these. I’ll show you only the first parts of it so you can get a sense. Oh, I’m not able to put it in the chat window. Fine, I will put it half and half. That’s the first half, and that is the second half. Question to the audience: are there any of these that you have tried and it’s worked for you? You’re welcome to unmute yourself and speak, respond in the chat window, raise your hand if you want, whatever.
22:02AnandYes, Mukul. Please unmute yourself.
22:11MukulYeah, I think this HLDs and LLDs have worked. I’ve seen my team leverage AI to give the context, a lot of context or I would say, first a context was created before leveraging the tool or the AI to develop HLDs and LLDs. So these could be then reviewed by the team before getting into discussions or any presentation. In addition to that, I see code reviews being done because I work for a company which has got some of the AI tools. I work for Publicis Sapient, they have a tool called Slingshot which actually does code to spec or spec to design, and so they have these tools, so my teams, I’ve seen my teams using it a lot. And obviously, yes, creating context of ideas and background for the task we’re trying to do. So I think some of the questions which—rather the question that you have asked—are more how we are using, engaging with AI at different levels or different things. Quite a lot of things I’ve been seeing people using it, but these three are those which are firsthand experience for sure.
23:44MukulAnd that "help arrange the meeting" is—it’s I who told this, I mentioned this. I have not yet gone ahead and given access to AI to help arrange the meetings.
23:53AnandWhich is interesting. And let’s talk about that. Vaibhav, maybe you could unmute. I know we have a hand raised from someone—Mrinmay, I’ll just come to you in a few minutes, Mrinmay—but Vaibhav, how has arranging meetings worked for you?
24:05VaibhavIt had worked well. So I used it basically from an internal perspective, from my personal perspective—sorry, sorry, say that again? I didn't catch—could you repeat? I have used arranging meetings from my personal perspective, specifically from the capstone perspective on how—because that was one of the projects we were working on, and I wanted to leverage AI to see if I can arrange the meetings and all and it worked very well. So if it is connected with Google Calendar or maybe Outlook and all, it goes ahead, creates the meeting, properly sets up the context, and it is mainly the type of command, the prompt that we give it to them; accordingly it works well and updates it.
24:53AnandGot you. Interesting. Mrinmay, your hand is raised.
24:59MrinmayYeah, professor. So I would like to share that this point where we have "review a movie" and creating HLD diagrams and context ideas—so in my experience, if you have a transcript, for example, I’m a line manager, so people report to me. So whenever there is an—let’s say I want to put a write-up for an excellence award. Generally I’ll use AI to kind of review what content I have written for my team. So it gives out that "this is eight-rated or nine-rated, this will stand out or this will not stand out" if I give the proper context. So those things I think review and all it works, actually, if you have a proper transcript and background.
25:43AnandGot you. Interesting. And Akshika, I’m curious. You mentioned code reviews with GitHub Copilot but it also depends on the tech stack usage. Could you elaborate on that?
25:55AkshikaI mean, sometimes in legacy systems it fails, or sometimes per se like we are having ARM templates to IaC when we are doing those kind of migrations. I saw that the code reviews had certain comments which were not legitimate in terms of—it was flagging something which was actually correct. So I think sometimes it hallucinates, but with Python the confidence score for the code reviews given by Copilot, I think it is better than the legacy tech stacks. That’s my personal opinion.
26:33AnandGot you, that’s specific tasks it does better. Got you. We did have a hand raised but is now lowered from Saurav, I think. Did you want to say something Saurav or not anymore?
26:48SauravYeah, yes sir, it was about the reviewing the movie. I have not really tried reviewing a movie, but yeah, uploading video works. I mean if you upload video to ChatGPT, any kind of applications, then it does summarize what exactly is in the video and if I haven't really asked to review it, but it gives you the context and idea of what the video is about. So I have done that, uploading the video.
27:16AnandGot you. And Ketaki mentions that number four is something you’ve had success with, number 14 as well, number 10 for collaborative ideas and research, and number five. I would love to hear more, Ketaki, about your experience on maybe let’s take, yeah, 14, to do deep research on project work files.
27:46KetakiYeah, so again to the point which Akshika mentioned from the legacy system perspective, so definitely when we talk about legacy systems, the kind of code it was written and the patchwork that generally is there in the legacy systems. So instead of going through reviews, what we did is while revamping the entire system to the new format, we went through the entire code and instead of asking to fix anything, we actually worked and brainstormed with the code itself to understand how we can actually go ahead with this entire thing, what are the parameters that we need to consider from non-functional perspective or functional point of view, or even from the overall architecture point of view. So those kind of brainstorming sessions and research that we did and then came up with a final architecture and actually it went well with the solution architect as well. Of course it was reviewed, but it did help us because it was a very big application, very scattered kind of code, and a lot of dependencies on the upstream and downstream. So I think that was a very good exercise that we could do from the research perspective.
29:03AnandInteresting. Lakshmi, I was anyway going to call upon you. I know you have your hand raised. Whatever you wanted to cover, could you please also cover your experience with number seven on the text and images? But please, over to you.
29:17LakshmiOkay, so seven I’ll take second. The first one what I wanted to say is like how Ketaki just said, so we have specific instances where the customers do give us business requirements where there are a lot of personas who will use the app. Now, if we were to write it simply as text on a Word or even create a PPT where we have boxes or flow diagrams to understand, "See, this is the persona, this persona has this task, that is the business decision"—the usual flow diagram is, of course, that has been the way we use it, but if I give it to a Claude or a ChatGPT Plus, the way it creates a diagram is very different.
29:57Lakshmidiagram is very visual. As in, you know, it puts all the personas on one side, then it puts all the tasks on the other side, then it has the flow generated in the middle. So, I mean, the way it conceives a very big document and then puts it in the form of one single image is brilliant. I mean, that it really helps everybody to go through the diagram together on a call and come up with better, you know, decisions than how it would have taken, you know, maybe a four or five calls where all vendors get together and then people don't understand, everybody has their own, you know, takeaways from the calls, they have to think about it. Instead, now we have the architecture ready where, you know, "this is where I belong to, this is what I'm going to use in the app for, and this is how the business—" so the to-and-the-from becomes very clear because of these diagrams that it is able to churn and give us. It's very useful in terms of, you know, identifying that.
31:04LakshmiAnd when I say seven, this text and images—okay, so I've been using it a little for my artwork. As in, if I have a diagram and if I have something like a text and if I want something like an image to begin with, or some—something like an inspiration, but it's not not really there, but earlier I'll have to do at least three to four iterations of that same in order to come, you know, get it right. But now if I have a baseline ready, so "this is my, let's say, text and this is how I envision it," it's able to give me an exact thing that I can begin well with. So it's able to churn in—I mean, for example, I do a lot of diagrams with respect to mythology. That's a very complex subject on its own, right? So if I don't want an inspired mural art but rather want to make something on my own for depiction, etc., it really helps. It's very, very useful in that way. So...
32:06AnandAnd what I’m taking away—thank you for that, Lakshmi, by the way—what I’m taking away from this is we have, as a group—let me just stop sharing, or my screen's already stopped sharing, is it?
32:17ParticipantNo, sir, you're still sharing on the tasks.
32:24AnandAh, okay, got it. Okay, let me stop sharing. Now, what I’m taking away from this is as a group, we have about 14 responses on things that we have not asked AI to do because we believe that it’s not yet possible for it to do it. And what we’re also finding is that in this group, some of us have tried some of those things and it’s not working so bad. Maybe our standards are higher, therefore others saying that it’s a success doesn’t really apply to us, possibly. Maybe we’re using different tools. Maybe we’re providing it different context. Maybe we’re iterating differently from it. Maybe the connections that we’ve given it are different from what others have done. Put another way: there may be opportunities for us to extend the capabilities of what we believe AI can do.
33:18AnandWhat I’m saying is, because on the one hand, AI is getting smarter and smarter by itself—that is one axis. The second axis is other people, even today, with the same rough AI that everyone has access to as of date, are able to do certain things that I thought were not possible. And the latter is broadly called the Jagged Edge of AI. I’m going to go into this in a few minutes, but before that, I notice that the participant count has jumped up a little bit. So I will once again share my screen—or well, yeah, let me share my screen and just let people know that we are running a little survey form, whatever. The link to the form is going to appear on the chat window for those who—okay, first I have to find my chat window. Chat, here we go. And yeah, this link that you just see on the chat is where you need to click and start filling out the questions. It may not take long. You do need to fill out the questions because the entire session is going to revolve around this form and a few other questions that I’m going to add.
34:46AnandNow, for the tasks that we didn’t ask AI to do—obviously in such a case, we’re not going to let it act autonomously, we didn't even allow it to give us any input. What we’re seeing is: let me collate the techniques that I’m seeing so far. In some cases, maybe there are specialized tools that leverage AI that if we use, it may be able to get the job done. Or there is additional context that if we provide, it is able to get the job done. Now, even creating the context could be a problem, but in that case, what I usually do is ask it: "What context do you need? Interview me. Ask me questions. I will get you the data," which is roughly in line with yet another technique we looked at, which is iteration. You keep asking, probing, pushing, seeing if it works, if it doesn’t work, etc. Or maybe connecting it to other software, like whatever it takes to book a meeting—say a calendar or a Zoom. And maybe sometimes there are things that it doesn’t really do well, so it’s really about picking the right subset of tasks or the right framework by which I specify what needs to go to AI.
36:16AnandBasically, what I’m saying is there is a difference between all of us in terms of where we’ve had successes and where we’ve not on the same kinds of tasks. And this, broadly, is what people refer to as the Jagged Edge of AI.
36:25AnandThe Jagged Edge of AI is simply the premise that AI is not—it doesn't have the same level of advancement in all areas consistently. So it started off as being very useless, but then it became a little more useful, even better than humans in some areas, better than humans in many areas, and maybe there’ll come a time when it’s better than humans in all areas. But at any point, on most tasks, we are at a specific stage. So we may be seeing it roughly at this stage right now where in some areas it is even doing better than humans, in some areas it is good enough that I can just tell it to do what it wants and occasionally correct it, in some areas it is not doing so well, some areas it is not doing well at all. And these experiences are also personalized, meaning each person sees a different curve because you have different tools.
37:24AnandI once asked somebody, "How are you able to set up a meeting appointment on your calendar?" They said, "Oh, our organization has provided Microsoft Copilot which connects to Outlook." Another person said, "My organization also has Microsoft Copilot and I’m not able to create calendar events." Then we figured out that they had not been enabled permissions for adding events to their calendar; they could only read from their calendar. So permissions also play a part in this experience.
37:52AnandBut maybe there is more to this. So I’m curious, if I take this list and make a copy of this—let’s get rid of the idea that actually better would be to take a more updated list—but since this is the list that we discussed, I will take this list itself and ask a question to my chat agent. I’m going to use ChatGPT and paste this list and ask it: "Look, I’m in the middle of the AIPL session that we discussed a short while ago and you can go back and look at the context of this session. When I asked the question, 'What are some things that you did not ask AI to do last week?', this is what the audience replied. Now, I’m curious, which of these have I done or been able to do? And why? Go through all of my past chats, transcripts, everything that you need to analyze, and find out which ones I’ve been able to do most, which ones I’ve been able to do least, and give me a sorted list. I’m in the middle of the meeting, so I don’t want you to give me long, rambling responses. First give me the sorted list. Under each item, mention maybe one or two examples in a meeting response style of how I have used this. Keep in mind that this is a public meeting, so please don’t share anything confidential and all of that. And use relevant skills."
39:39AnandNow, you’ll notice a couple of things. One: I’m talking to it like I would talk to a person. Second: I’m asking it to do a reasonably complex and abstract task. This will take ten minutes. Let it finish, and we’ll come back because this will be my version of tasks that you see as currently not possible, for me whether they are possible or not possible. Or where possible or not possible. My answer may not transfer to you because the permissions that I’ve given my system may be different from the permissions you’ve given your system.
40:18AnandBut I think we should also be aware that when we have to let AI act, we also have to know whether it can act. And that answer changes with time and person. These are two very different axes.
40:33AnandI’m going to present an Autonomy Ladder, that is, stages of decision, which is roughly what you saw in the questionnaire as well. But as we go along, to avoid this being a monologue or even a dialogue, put in your comments, questions, etc. in the chat, because in about 10-15 minutes, we are going to go through this. Maybe some ideas are popping up and you want to discuss it, but you also don't want to lose the stream of what we're discussing. Put it on the chat. Maybe it'll be useful for others, maybe others have the same question, maybe others have the same idea, maybe others disagree in the same way—just keep the chat message flowing.
41:12AnandI’m going to share my screen and we’ll talk about the Autonomy Ladder. For which, yeah. Now, the way I see it, there are four things that we can do when we are working with AI. And what I mean by when we are working with AI is roughly along the lines of what you had answered here. I call this, let's say Level 1, Level 2, Level 3, Level 4. Nobody’s at Level 0—"I haven't used AI at all"—which is:
1. Whether you ask for advice.
2. Or you ask it to create a first version of it that you will then recreate.
3. Or you let it do the job but periodically make sure that it's doing okay.
4. Or you let it do the job and you don't even check that it's doing okay.
42:05AnandAll of these are the levels that we have of autonomy. Now, one axis that I’ve not yet mentioned but briefly touched upon in the Jagged Edge of AI is where it is good at. I said, one, it’s continuously improving; two, it may vary from person to person; but also there are certain areas where it’s doing better than others. For example, this—I’ll share the link in the chat as well—this is a study called GDPval. GDPval is OpenAI’s research on—well, and this is more than a year old, by the way—but a year ago, it was OpenAI’s evaluation of in which sectors are which models doing better than humans or worse than humans.
43:01AnandThe box size represents the total salary of that profession roughly in the US. So general and operations managers are a big box; registered nurses are a big box. The color indicates whether AI is doing better than experts (which is green) or humans are doing better than AI (which is red). So obviously, you know where OpenAI's preferences are based on the colors on the screen. But the way they did it was fairly rigorous. What they did was, for instance, let's take accountants and auditors. They had experts create five tasks, like: "You are an auditor and you have to review the accuracy of anti-financial crime metrics. Here is a spreadsheet that contains something. Based on this, you have to calculate the required sample size for audit testing for a 90% confidence level, do a variance analysis, and so on, and eventually come up with a new spreadsheet titled 'Sample' that has the selected sample and your workings of the calculation."
44:01AnandA year ago, they gave various models this task. They also gave various human experts this task. And they had another bunch of human experts evaluate these to see who’s doing better. Among the accountants and auditors, for instance, Claude among the models was the best, but it had a win rate of only 24%, meaning last year 76% of the time, it’s the humans that were beating the models.
44:36AnandWhereas if we take financial advisors, on those tasks, Claude was beating the models 64% of the time. Think about it. Today if you want to hire—no, not today—a year ago, if you wanted to hire an auditor, you were better off hiring an auditor. If you wanted to hire a personal financial advisor, you were better off paying $20 to Claude two-thirds of the time.
45:03AnandPharmacists were doing better: 26% of the time Claude won, but that wasn’t much. But sales managers' work: almost 80% of the time GPT-4—GPT-5—was beating it. That’s powerful. So different kinds of tasks obviously there are different results.
45:26AnandAnd how do we figure that out? Short answer: I think is benchmarking, which means test it, see where it works. And we'll go a little bit into benchmarking in a short while. But the idea that I want to share—and I will wrap up with one last example—is that depending on the task, performance varies. Depending on what axis, what context, what tools, what way of interacting with AI, etc., success varies. And just over time, AI just keeps getting better and better. And all three of these have to inform what you are going to delegate to AI, when you let it act and when you don’t let it act.
46:07AnandI’m going to open up for questions in a few minutes, before we go to the second phase of this session. I’ll share one personal example. Now, this said for accounting and auditing, keep an auditor, but for personal financial advice, make sure you do your own research. So that’s what I’ve been doing. I’ve retained my auditor. A few months ago, my tax needed to be filed and the auditor sent a filing. I don’t understand tax at all. Zero. I gave it to ChatGPT, sort of like how I gave this question, I dictated this question right a short while ago. I asked it to do something very similar: "My auditor has given this tax statement. Take a look at my bank account, take a look at the income tax website, log into my website, do whatever you need. Tell me if this looks right. What should I—and give me an email that I should reply to my auditor telling them whether they should go ahead or not."
47:13AnandAnd it said, "Anand, you have a discrepancy of several lakhs. This says that you should get a refund of some 2 lakhs. You actually should be getting a refund of something like 17 lakhs." I was like, "Wait, hold on, what? I will happily take 14-15 lakhs extra, but—" I said, "Are you sure?" I have no basis for evaluating. It took another half an hour and said, "Yes, I’m sure. Send this email to your auditor."
47:39AnandThe auditor calls up and says, "But Anand, that is true only if you have an NRE account; you have only an NRO account." I said, "Kamatchi, I don’t understand any of this. Why don’t you send me an email?" She sent me an email, I gave it to ChatGPT. It came back saying there is one Anushka vs. State of Maharashtra case in which the judgment said that if you are sitting in Singapore and paying Singapore tax, there is a Double Taxation arrangement between India and Singapore. As a result, even if you sit and earn money in your mutual funds in India, the tax for that is actually taxed based on the Singapore taxation laws and in Singapore there is no capital gains tax, so it should actually be 0%. And this case specifically says that it is where the person lives, not what kind of account that they have, that makes a difference.
48:31AnandSo I said, "Okay, fine. Give me an email." Dropped her an email. She said, "I’ll check with my senior," came back and said, "Yes Anand, this is true." And I got a refund. In some of these cases, what I’m realizing is that when I push AI to act on my behalf one step further, I literally make money, or at least literally don’t lose money. Might be worth a try.
48:57AnandOn that note, let me open up. Any questions, comments for what we've covered in the first hour or so? Well, 45 minutes. And if you’re thinking of directions—and I’ll come to you, Mukul, in half a second—just think about how might you apply this and where might what we’ve discussed be useful to you. Yeah, Mukul.
49:37MukulYeah, I think this has been a great—great discussion. I would say the topic itself is super, super important as we continue to progress in this space. So quickly, a background: I’m—I come with a background of a hands-on practitioner in the industry, in the financial and commodities trading world. From there, I moved into the consulting bit of both business and IT. And currently, I’m in a growth and sales role. And one of the things that actually enticed me to pick up this course is kind of discussion we are having and similar I had.
50:13MukulSo I—this I just wanted to put a background for you as well as the class so that the comments I’m going to—kind of hear it. I think it's a trust thing and the explainability which is holding us as humans to a certain extent, based on the server results or the answers to the questions. And degree of trust and degree of confidence probably varies from person to person and their own experience and what they see hands-on. For example, there are some of the people who are working directly with certain models or build agents, so they will be more confident in handing over things to AI as compared to others. For example, for me, I will test out things which are low-risk and I will hear from others who have taken certain risks and seen, okay, fine, they have fine-tuned it well and they are able to use it AI in a better way. So I will learn from let's say their experiments or their mistakes and do the things right. So—so from that perspective, I think and specifically the cost and the learnability and intelligence graph that you showed, I think that's—that's a—that's a very interesting thing to look at and it also gives a sneak peek into the future what's coming forward, which is both great and a bit dangerous. That's my take.
51:45AnandNoted, Mukul. Any question from anyone? Yeah, Ram. You're still muted.
52:06RamShould I go ahead?
52:13RamYeah, okay. So I think even the graph—like I think the one that you showed for the US Senate and that kind of Congress kind of thing, right? That analysis and in fact, even what you demonstrated here on the live ChatGPT, it—it revolves a lot around, I think just now it was mentioned, trust, but I think that trust is on your data, right? Then there is a trust on the actions. So I think that's somewhere that like, I think the capabilities are no more of an issue, the way we see. It’s probably a matter of time that we see the technology or the speed or the compute catch up. It’s already good.
52:56RamBut where we are not able to assess or I think where the usages are now getting beyond a certain trust boundary line is where it's being—it's being misused. Now, the misuse could be in two ways. One is the user intentionally asks the AI to misuse, because the AI takes it as a request, right? But it doesn't know where the user may use it. So for example, deepfake kind of videos that are generated or someone trying to hack into accounts or these kind of things, right? So AI doesn’t really understand those, I would say, ethical boundaries of the outcomes.
53:44RamThe other thing is you trust a lot about that okay, you—you were—you just gave a—a prompt, maybe on a—a voice or maybe as a text, to AI with a lot of trust from your side, but in the background, the—the models and the companies who own that, right, they essentially get everything that you are providing it, right? So a lot of time involuntarily. So in a way, if I have to give a—in a way, if I have to give an analogy, we are so ingrained to using mobile—signals, mobiles and mobile data, that I think we—we've lost a track that we are being tracked—where we go, where we move, how much time we spend. And if there is a need, someone who is, you know, having good powers can get every data of where I’ve been, in which location, for how long and what all, right, and with whom. So it's as much as that, that a lot of this data is being now given with a lot of trust from our side, but we have no clue how it's getting used. Big tech companies are using this not—not only as a—as a gold mine, but they're using as—at—as a leverage to control us. So I think these big challenges are something that I feel are more important, especially when I look beyond our generation. The younger generation are not really aware of how much this capability needs to be controlled. So I think there are big serious issues around how much power we really allow and how much we allow it to get into our, kind of, life—lifestyles.
55:31AnandAnd trust does form an important part of this. So that’s what we’re going to go into next, but after a small exercise. I’m going to share my screen to finally wrap up this section. So, same form, but now there is a new Question 5 that I’d like you to answer. For those of you who can might need a link to the chat—sorry, to the form—I’m pasting it here. This is the last question on the list right now, which is: for Question 2, there were a bunch of tasks that we said we would not even ask AI to do, right? I’ll just put that on the screen again and we had a discussion around these. Why didn't people—not just talking about yourself, but why do you think people have not asked AI to do these things? Why do we think it is not possible? Actually, that is probably the right wording of the question. Give me a second. I think people didn't ask—what made them think that this is not possible with AI? That effectively is the question.
57:00AnandDo give this a shot. And I’ll wait for a few minutes just sharing some comments about the next topic that we’re going to discuss, which is trust. And I’ll proceed once we have maybe about—we have how many people right now? 47. At least about 30 responses.
57:24AnandThe reason this question is important is it forms your prior hypothesis on why are we not maybe pushing AI to the limits of what seems to be possible for some others, perhaps. And once you know this, you can act on it. If we’re not aware of our own hypothesis on why, it’s going to be hard to act. So give it a think. Why is it that as a group, we didn't ask AI to do some of these things?
57:58Anand(Anand leaves the "impossible" tasks list on screen).
58:28AnandJust seeing, we have five responses so far. I’ll wait till we have about 25. In the meantime, there was a question from Lakshmi on whether audio is working for all. If people are able to hear me, can you maybe do a thumbs-up or something? Or any reaction?
58:58ParticipantYeah, I guess people are able to hear you. Cool, thank you.
59:11Anand(Wait for more responses). Eight responses so far.
59:57Anand...answering the last question on... put in the link here as well. forms.sanand.net. Okay. This is what we're answering. I have 10 responses so far. I'm waiting until 25.
1:01:23ParticipantSir, I just joined back. I'm sorry. I think I missed one or two minutes.
1:01:35AnandWe were completely silent for one or two minutes. We're just waiting for people to fill out...
1:01:41ParticipantOh, okay. Was there any... anything that you posted a link? Can someone please help me with the link again?
1:01:47AnandNo new link either. Same link. Last question. Everybody is answering this question which is all of us we said we didn't try something or we didn't tell AI to do something because we thought it couldn't do it. Why is it that people might have thought AI can't do something? What's your guess is the question.
1:02:12AnandWe have a dozen responses so far. I'm waiting until we get to 25.
Mostly silence until 70:38, broken by a running count of answers (added from the audio):
63:43 “We're kind of stuck at 14 responses.” ·
64:24 “16 so far.” ·
65:07 18. ·
66:37 “Stuck at 18.” ·
67:38 19. ·
68:19 “20 responses so far.” ·
68:41 21, and thanks to individuals by name.
1:10:38AnandOkay, we're at 21 responses still, so I'm going to do something a little more dramatic. What I'm going to do is ask ChatGPT who are the people who have responded to some of the other questions. And we have 32 responses for the previous one, but only 21 here. Let's do a live analysis and call them out and literally do a roll call and see if they are there. And if not, fine, then at least I'll have a different... so okay, now we know that there is I don't know what attendance requirement for this course or some kind. And fine, if they're not there, that's cool, I'll at least know that we should lower the cutoff. So let's see if ChatGPT is able to do an analysis. Okay, 13 people who have answered other questions but not question 5 yet, and you may be in the process of answering, that's fine. [names read aloud], if you're still there can you just maybe type "yes" on the chat window or "answering" or you refuse to answer is also perfectly fine. [more names read aloud], if for any reason you're not able to answer, let me know on the chat window, that's perfectly fine. And with that, we still have 22 responses. Like I said, I'm patient enough, I'll wait.
1:12:44AnandOkay, we have 23. Let me just check our most recent response. Okay, yeah, the one that I have is from [two participants], that makes it 24. [A participant], thank you for responding. We're at 24 at the moment. Basically, if anyone's just come back to your screen, if your name is on the right and you haven't responded to question 5, please respond. Okay, thanks [participant]. That takes us to 25. Rest of you, please consider responding. This is the first workshop where I've actually had to do this, take attendance so to speak, like ever. I'm going to need one minute to think about that.
1:14:28AnandI'll tell you my problem and once I maybe narrate the problem, it will get resolved. Let me share my screen. This is partly also a sense of how I'm using AI in day-to-day life, which is asking a question like this. "You know that AIPL session that we were discussing? It was planned for two and a half hours. The first 45 minutes I was monologuing, we had a few discussions, a few people pitched in and we discussed. And in the initial stages there was a survey that I launched where we had about 30 people out of a group of eventually 49 that joined had responded to the survey. Now, a bunch of people joined late, that's reasonable, it's 9 a.m. in India. Then when I introduced question 5, it was a struggle. It took about 15 minutes to get 25 people to answer the question and I for the first time had to ever almost do a roll call, call people by name, and ask them to fill it in. That tells me that the workshop format is not really working. Maybe it is the time, that is the duration, or the timing that it is morning on a Sunday or whatever. It also tells me that I should probably revise the way in which I run this workshop. There are a couple of possibilities: one is it's getting recorded, so people can just turn it on and listen to it later on, which is a perfectly reasonable way of learning. But in that case I should tune the workshop to that sort of a format, which is half the people will be looking at it offline and the rest will probably be—not even half, maybe about 20% will be engaging. But in that case I also need to have some metrics of calibration so that I can tune it accordingly. So help me with this. Frankly I don't really know even how best to ask this question, but you have a reframe question skill, so reframe my question accordingly and use all other relevant skills and guide me on the top actions. Don't give me a long response, give it to me in meeting response style compactly. I mean you can write all the stuff that you want as thinking that I can later on go back to for further reference but just end with a meeting response style response." Let that churn and I will in the meantime continue with the session.
1:17:10AnandSo, recap. We were looking at the ways in which we delegate to AI. One of the areas that we consistently get stuck with is trust. And you have also shared several responses on where you think people get stuck in having AI answer... the session... let me share your responses to this, which is here. So, some people say it's because there's hallucination, "I tried but it wasn't helpful," it... okay, there are blockers in decision variables so the AI might not have the right understanding, and so on. So I'm going to have a parallel analysis of this question happen. Hold on. Let's see. What are the top factors that people are pointing out to? Let that come back later.
1:18:19AnandNow, let me share how I've seen people deal with at least the trust question, which is one of the biggest reliability issues or biggest things that people face as a challenge when saying, "Here's why I would not let AI act autonomously." First is that it is possibly hallucinating. That was in fact one of the first responses we saw a short while ago, the hallucination issues with AI. Yes. How do we deal with it? One client came to us saying, "We have a lot of calls, we have a lot of chat messages. We would like something that can use AI and classify it but we want it to save effort and it should be accurate." So we tested it. Some of the messages were like for instance, "Help with adding some items" or "What do I need to do to register?" Now these are supposed to be classified into a specific queue. So for instance, when we asked Amazon's Nova Lite v1, old model, to classify it, it put this into "Create account." "What do we need to do to register?" it said belongs in the "Create account" category, but it actually belongs in the "Registration problems" category. That was a valid category. And Llama—sorry, Llama 3 also got it wrong. Qwen2 on the other hand got it right. Gemini 1.5 Flash got it right. So a bunch of models got this question right. This question had roughly a 50% pass rate. Some questions had a 90-100% pass rate. Lots of questions had no problems. Some questions it struggled on. There was one question had a 0% pass rate, so we thought actually the question itself was probably categorized wrongly. But many of the others, there is a percentage. So we said, do all models make the same kinds of mistakes? So if I take for instance Claude 3.5 Haiku and correlate its answers with Gemma 2. How often are they correlated? And the answer was barely correlated. Maybe 20% correlation. Some of these had 40-60% correlation. Of course the model was perfectly correlated with each other. The way to read this is the yellower the cell, the more those two models are agreeing with each other. Obviously models on the diagonal are agreeing with each other, but many other models have close to zero correlation.
1:20:49AnandSo what that is like is I ask one question to different people, different people are making different kinds of mistakes, they're not making the same kinds of mistakes. That's a useful signal. So then we said, supposing instead of asking the same classification question to one model, we ask two models. What happens? So 14% is the error that we got when we asked one model randomly. That's like randomly I'll pick somebody off the street and ask them a question, then one in seven times they're making a mistake. Okay. Now let me take two people. I will randomly call two models off the street and ask them the same question and only if both of them agree I take the answer. In that case my error rate is now down to 3.5%, just 3.7% error rate down from 14% if I ask twice. Now this causes two problems. First I still have a 3.7% error rate and I don't know when even when both models agree are they actually correct? Second it creates manual work because if the two models disagree then I will have to manually review it. And that happened about 12.6% of the time.
1:22:04AnandSo then we said okay what if we triple check it? 2.2% error rate. Quadruple, quintuple. When we get to quintuple checking, the error rate dropped to 0.7% meaning I could go to the client and tell them I'll get you 99.3% accuracy. But that is only if all the models agree. 28% of the time the models won't agree. So then you'll have to do manual work. So your 100% of work I can cut down to 28%, I will give you 72% effort savings at 99.3% quality, only 0.7% error rate. Is that okay? They said this is great. There's one more catch. We said instead of one model's cost, you now have to bear five models' cost. They said okay fine how much is that compared to one person's cost? It works out to some 2% of a person's cost. You know, these are rounding off errors. Let's get this done. And this is one pattern we've found consistently working in making AI more reliable. Which is same as with humans. Humans are not reliable. How do we handle it in a bank? Maker-checker. One person does, one person verifies. Or wisdom of the crowds. We don't know who is the right leader. Ask a hundred people, ask a million people. Based on their vote we select. We have several processes to deal with unreliable humans. Same processes work well with AI. Arguably managers are much better equipped to make AI more reliable than engineers because engineers are working with deterministic systems, managers are more experienced with indeterministic systems. And this comes with a skeptical mindset as well. Before I go ahead I should just pause and maybe check... if there are any questions or comments that you have please feel free to just post them on the chat window as we go along, we can discuss.
1:24:05AnandSo whenever somebody says something, I ask myself—at least about AI—I ask myself, "Is that really true? Let's test it out." So one of the things people have often told me is you should tell AI to use simple English. One of the tweets that I remember is from Andrew Carr and this is what the tweet says. He says, "I always use this prompt: 'Only report to me in ASD-STE100 simplified English.' That solves a problem where AI writes some confusing things that I don't even understand." This sounds like good advice. I thought maybe I should do it, but as a matter of habit I prefer not relying on human advice also. The question of is AI reliable is something that I often ask of humans as well. So I thought let's test that out. And this is the result. I said I will take five tasks. One of the tasks is for instance model benchmarking. The question was models are constantly improving, so are there some tasks that I can use to benchmark which models are doing better, even as the models keep getting better and better, the benchmark should still work. That was roughly the task. Similarly I had half a dozen other tasks and I asked these without that suffix as well as with the suffix. The suffix being: "Only report to me in ASD-STE100 simplified English." Question was: what is the quality of the results? So if I look at the correctness of the result, what are the underlying drivers, what are the mechanisms, what are the caveats, is it calibrating properly? In each of these cases I was looking at whether the simplified response is as good as the original response. The short answer is: in every single one of those cases, the simplified response was worse. There were only a few—in fact there was only one case where the simplified response beat the non-simplified response on correctness. Once on calibration. In every other case it was no better out of, well, as you can see, dozens of cases. That's a benchmark. Now how did I run the benchmark? Manual copy-paste. I took six prompts, 12 combinations, for each of those I just pasted, pasted, pasted, pasted, pasted. And then copied all of the responses, gave them all to a set of ChatGPT windows and asked them to evaluate, saying which of these is better. And I also cursorily reviewed the responses. So my policy now is let the agent do its work, let it give the answer in a confusing way. Afterwards add that. So tell it: "Now answer in ASD-STE100 or answer in a simple way that I can understand" so that you get the benefit of the smart thinking and you get the benefit of the simplified English or simplified explanation. The key theme here is benchmarking helps. In other words, whenever you have a question "Will AI do it well or not?" test it.
1:27:35AnandSo to summarize, we're answering the question "Is AI reliable?" or "How do we make it more reliable?" One technique is double checking or triple, quadruple, quintuple checking. It's pretty cheap. Another technique is benchmarking. Test is it actually better, is it actually worse, and you can use agents to do the benchmark. Let's talk about a third technique, which is effectively converting it to verifiable code. Now what do I mean by that? Well, code has the advantage that it's very easy to write something, run it automatically, test if it is working, and if it is failing then tell the agent "Iterate, correct, keep revising." And that iteration is something that it can do by itself. Ketaki had mentioned a while ago that she was iterating to make sure that the model is able to perform... I forget which task... task number 14, I think, deep research and analysis on project work and that got it to a certain degree of reliability. Now iterating as a human is time and effort. A compiler has the advantage that it can provide a correctness signal automatically. So an agent can write something, test, if it is correct, great, if not revise. Is it possible to do something like that outside of programming? And it turns out that there are surprisingly many areas where we can do that. One of them is for instance insurance contract writing. So what you have on my screen, yeah it's still shared, is a programming language of sorts called Insurely, or I-N-S-U-R-L-E. This was developed by Cambridge and is an insurance language of sorts. In other words, insurance contracts as well as claims can be rewritten in the form of a programming language-like logical structure. For example, here is one contract. It's a vehicle insurance contract which mentions that there is a driver eligibility, the vehicle ownership, sobriety tests, vehicle classification, etc. All of these are part of the contracts' policies.
1:29:57Anand...policies. It is possible to write this as a set of Insurely rules, which look like this. It says, for instance, that "driver fully eligible is true if they are age eligible, license compliant, and driving history acceptable." What does age eligibility mean? It means that the age field is more than 21. Driving experience in months is greater than 12. It's taking the same things and converting them into a set of sub-rules, which are verifiable almost like a program—not almost like a program, actually like a program—and they have written—this is the Cambridge team—have written a program where rules like this can be verified.
1:30:45AnandOkay. So if that is the case, what can we do with it? Now it turns out that claims checking is something that can be close to automated. For example, let's take the claim form for Marcia who has filed a claim in June 2024. Now, the claim form looks like this: the accident happened on 20th June 2024 and she filed two days later. There was a $12,000 claim amount, and it has all other fields that are supposed to be there and a bunch of other pieces of information. And this is the original claim form, which let me try and zoom out—how can I zoom in this... yes. This is the original claim form which ended up being rejected, but let's see how it ended up being rejected. Let us actually check that claim.
1:31:45AnandNow what the agent is doing is going through the claim one by one against each of the clauses. Is the driver eligible? Were they age eligible? Yes, Marcia was over 21 years old. The license is valid, it's not expired, and it is in the right jurisdiction in the US. And the DUI count also—they don't have any DUI history. Great. Is the vehicle authorization valid? Yes. But what seems to have been missed is incident circumstances. They should have been sober at the time of verification, but the blood alcohol count—which is supposed to be less than 0.08 grams per DL—is actually 0.14. That is a DUI violation. Therefore the claim gets rejected. All the others are fine.
1:32:31AnandNow because this is happening programmatically, what we're able to do is take a business process and convert it into something deterministic. Then you can run it at scale. At which point whoever needs to review it—or let's say an auditor comes or a lawyer comes and says, "How can you reject Marcia's claim?" Option A, the claims adjuster looks at this and says, "Here is the reason: she was not sober at the time of the accident." Or, better yet, you have a chatbot sitting there and the lawyer is interacting with the chatbot, saying, "Why didn't Marcia get my claim?" And it says, "Here is the blood alcohol report." Realistically the lawyer's agent will be talking to your chatbot. That's another story. But the verifiability becomes a really powerful theme, especially if you can take unstructured data and start mapping it into atomic verifiable claims.
1:33:29AnandThe question that we need to address is: how is this possible? How can we do this at scale? Let's take one example here and we will be breaking for an exercise in about five, seven minutes—I'll let you know what the exercise is. But the key thing is almost any kind of unstructured data can be broken down just by agents if you give them a guidance. So what we can take is, for instance, a policy guide like the European financial promotion guidance. Now what this has is a series of PDFs that say if you're doing any kind of financial marketing in the US, then you have to follow this guide. And if you're doing promotions, then here are a set of rules that you have to follow. For instance, if you're doing any kind of crypto asset or cryptocurrency kind of communication, then you should familiarize yourself with the crypto asset promotions which is summarized in Rule Number PS23/6, which is another rule that is in another document and we have to sit and scan all of this.
1:34:51AnandWhen we tell an agent, "Go through all of these and give me an ingest of all of those rules," this ingested a little quicker than I thought... let's just make a small tweak to this and try ingesting again. Reset first and then re-ingest. That might work. Yeah, it's working. So what it's doing is going through each one of these documents, identifying an individual rule which may be taken verbatim or it may have been summarized, and creating a series of rules sort of like how Insurely was created, right? Where you had the form which or the policy document which it broke into a set of rules, but this was in a specialized language. What we're doing here is just telling it, "Give me rules that are easy to verify." Nothing more than that. And the prompt is not very complex; it simply says "Extract atomic testable rules from one policy document one by one and keep it simple," blah blah blah, and we're giving it some guidance on how to provide examples, etc., and it is extracting all of the rules.
1:36:14AnandSo somebody releases a policy document or any business process guideline, any statement of purpose, you can break it down into testable elements and then take individual documents and validate them. So let us say there is a promotional email, there is a landing page, there is a notification. You can have the agent go through each one of these, test them, see if it passes, see if it fails, and where there are known failures, you investigate. But if there are no known failures, then you pass it to a human to do a second review. But if the agent itself is spotting an error, then clearly you would reject. So far the agent has not spotted an explicit error in any one of these until here. The SMS promotion has some clear issues where the rule says "promotion highlights benefit prominently: 7% of expert portfolios" but the warning is minimal. And capital protection is misleading; "capital at risk" note is less prominent. So it has not followed the rule where we must present a balanced view of risks and benefits. And you're saying, well, "I can spot the error, obviously the regulator will spot the same error, so don't."
1:37:27AnandWhich means two things. A: we can programmatically verify or convert almost any kind of statement of procedure into something that an AI can validate, and get a set of negative scenarios where the AI itself has found a flaw, in which case you can safely reject and say 'don't do this, revise it.' The rest go through human review.
1:37:54AnandTo recap what we have is:
1. Multiple agents checking can increase reliability.
2. Benchmarking yourself can increase reliability.
3. Creating a set of rules that you can have agents test atomically increases reliability. These are not the only ways of making agents more reliable—there are others—but these are three powerful ones. And once you have agents more reliable, then you will be able to let them act more autonomously. And that's what we are going to explore in a breakout room.
1:38:35AnandSo in about three, four minutes we are going to go into breakout rooms. You have one specific task which—and Saradiya I'll request you to suggest what time would be appropriate, I was thinking maybe about... half an hour would be an apt duration for this one but tell me if you think otherwise. What I'd like you to do is take one real-life task and answer the question: how can you or what would it take for you to make it reliable enough that you can have it run autonomously? And at the end of it, I will request you to fill out one survey question, which I will draft along with you right now, which is "Task Choice." Question is: "What real-life task have you picked that you'd like AI to act autonomously on? What is the business benefit of this?" Let's take the question now—that's one question. Second question that I'd like to ask is, "How to make it reliable?" The question being: "What techniques would be—or just keep it simple—how would you make sure that the AI can do this task reliably by itself? List all techniques in detail." And these would appear in the same form as questions 6 and 7. So in the breakout rooms please take the next half hour, discuss as a group, pick any task together as a group, and I'd request everyone in the group to fill out the same thing or, if you have discussed multiple items, then split it up between yourselves in the groups and submit multiple entries.
1:41:27AnandThe wrap-up for this session will be what can we learn from the various ways you are seeing tasks become more reliable. I'll pause here for a couple of minutes. Does anyone have any questions on the task that you are working on in the breakout room?
1:41:51MukulCan we take a five-minute break? It's been almost 1 hour 45 minutes, close to 2 hours. And then we can come to this exercise.
1:42:00AnandAbsolutely please. No, and sorry, everyone please take breaks whenever and wherever you want. The session is being recorded.
1:42:07AnandOkay, if not, then what time is it in India? Whatever time it is. In half an hour's time, let's regroup back here. Saradiya, would it be possible to open the breakout rooms and assign people please?
1:42:40AnandThank you. I'm going to go off video but I will be available on chat if anyone needs anything.
1:42:45SaradiyaSo learners, we are 44 of you here as of now. So we will be making breakout rooms of four each, so 11 groups will be there.
Breakout rooms. About half an hour of group work is not in the recording.
1:43:10Mukul... see a lower number of responses. I hope that’s okay.
1:43:12AnandYeah, yeah, I’m assuming that there’ll be 10. Maybe more, but 10 is what I’m expecting. That’s fine. Hello?
1:43:19AnandYeah, same thing, same thing I [inaudible].
1:43:39AnandLet me just do a quick check. Yeah, I think the recording is on and we have everyone back. Great. In the last seven minutes, what I’m going to do is take a quick look firstly at what we are seeing so far in terms of responses. I’m not going to summarize them just yet in the session; we will be doing that later once I have all the responses. But okay, 10 responses, so that’s good.
1:44:06AnandI’m going to pick some at random. The question was, how do we make sure AI can do something reliably? Define the process with complete details is one theme, giving precise instructions and verifying is another, providing the right context and instructions is another—quite a bit of responses. Building a reliability layer—that is a useful thing. I will be sharing these to us as a group. Do take a look at these offline.
1:44:39AnandAnd if anyone else has additional responses that we have not yet filled in, please put them in. For the next one hour, this form will be open for another, at most, probably half an hour, not even more than that. It runs on my laptop and I will be shutting it down post that.
1:44:56AnandBut what are we concluding from today? Well, actually, firstly, we are trying not to conclude today. This is the first of a three-session arc. And today we raised the question: When do you let AI act? This is confounded by three complexities: one, AI’s capability is increasing; two, AI’s capability is not uniform in different areas; third, our access to AI is also not uniform. So even the answer to the question, "what should we let AI do by itself?", the answer is: it depends.
1:45:42AnandFor this, we discussed a few principles. One is how do we make it more reliable. And there were three things that I shared, which is: one, you could double-check, use multiple agents to verify; you could use programmatic verification techniques; you could use benchmarking as a technique. And you have listed these and many others as part of your response.
1:46:09AnandYou’ve looked at specific tasks—about 11 of them so far—ranging from interview evaluation, KPI monitoring, meeting preparation, risk management, etc., and seeing how you can validate these. Make sure that AI becomes more reliable in each of these cases.
1:46:29AnandIn the next arc of this, the second session, what we’re going to be covering is how do we make sure that that verification process doesn’t take up all of your time and effort. Generation has now become easy. How do we make sure that verification is easy? How do we make sure that we know we’re measuring the right things? That will be part of what we’re covering.
1:46:54AnandSo as preparation for that, my ask is: Take the tasks that you have mentioned here as a group, individually, however, or pick any other task. That’s fine. Do the same exercise. Pick a real-life task, see if you can get AI to act autonomously. Write down your experiences. What is working, what is not working? Where does it do well, where does it fail? That cataloging itself is one of the most useful exercises.
1:47:23AnandThen try and answer the question: how can you make it more reliable? If possible, apply that, or at least write down how you might plan to apply that. Could be in personal use, could be at work, wherever. We’re going to be catching up for a continuation of this next week, roughly the same time, and this will be the preparation that you would need before we go in.
1:47:50AnandThat is: a task that you have chosen to have AI act autonomously and tried it—the trying is important. Ideally, do share chat messages where you can. And how can you make it more reliable? What we’ll do next is discuss which of these techniques are more effective, less effective, under which circumstances, how we can make sure that our time and effort doesn’t go into verifying it all the time, and knowing what we can measure better.
1:48:21AnandWith that, I’ll wrap up, and we can open up for any questions for the last three minutes before we close. Any questions from anyone?
1:48:42ParticipantSir, I have a question on the expectation for the next class basically. So the questions six and seven that we filled now—like, you know, what we are using in our real life as an AI and what are the, you know, or steps that we have taken to ensure that it is reliably doing it—we will be discussing that in the next class when we open it? Is that’s what...? Okay, sure.
1:49:08AnandYes, kind of, but not just as a group, but individually also. Individually try something and see how you can make it act more reliably.
1:49:21ParticipantOkay, sir. Okay, sure.
1:49:29AnandAny questions from anyone? Not necessarily just on what we’ve covered but absolutely anything AI-related.
1:49:41ParticipantSir, one question I have. When do we generally say stop? In the sense, this is related to this question that you asked, like, you have asked something to do with AI and you actually don’t want it to do. So many a times we try to do some R&D, try to do some research. It keeps on going on with AI. So I don’t know when to stop and when to say that, "Okay, I might feel like probably this guy is not going to be able to do the work which I intend to do," and but I keep on pushing it to its limit, keep on pushing, but I do not get the exact answer. I, in fact, keep roaming into places where I don’t want. In that, where a human should actually say stop, "Okay, no, AI is not really for it and I should find a manual effort to do it"?
1:50:28AnandMy personal answer to that is: wherever you are comfortable. The thing is, models change. I don’t like interrupting people, so I don’t click on the "stop" icon on ChatGPT also. That’s silly, right? It has no feelings, it won’t mind if I interrupt it, but still, I don’t interrupt.
1:50:50AnandSometimes I give up very quickly after trying with AI, same like with people. When I interact with people, I am following a certain model of, "I’ll tell somebody something. If they do it, great, I’ll continue with them. If not, I will find somebody else." I’m applying the same strategy with AI. That is, I think, what most people do—that is, go with their experience of working with people or software for that matter.
1:51:15AnandI think the answer for agents is different. So the question that you’re asking and saying is a very important one. Not because I’m saying I have an answer, but I believe that the answer is different from what we are currently doing. Rather than propose something saying, "Look, here is what you should do," or even, "Here is what I’m doing," let me share what I am going to do, and I would encourage you to try it as well.
1:51:44AnandEvery time you feel like stopping AI, just write it down. "I felt like stopping AI and I did stop it at this point," or, "I felt like stopping AI, I did not stop it at this particular point." Just log, maybe 10 times. That log will give you a good reflection when you go back and say, "When am I feeling like I should stop it? When am I actually stopping it? When am I not stopping it? Looking at these 10 instances, should I have stopped it? Should I have not stopped it?" And that will give you the inputs to decide for your circumstance.
1:52:23AnandI’m merely replaying that benchmarking exercise. General advice and all is fine, but this is a process I would follow for yourself. I have not done this myself. Thanks to your input, I’m going to try it this week. Next week, hopefully, I will be able to report at least what’s worked for me.
1:52:35ParticipantYeah, sure, sir. Thanks.
1:52:39AnandI know we are at time, so let’s catch up next week. Have a good weekend, everyone. See you soon.
1:52:49AnandThank you, everyone. Bye.
