# Transcript

**Anand**: [00:00] Thank you. Good morning, firstly, apologies I was delayed today. We won't, however, be needing to go beyond the planned time. We should be comfortable within the established time.

**Anand**: [00:17] Now, I know many of you have shared the link to the application that you had built, which is a good starting point. And you've also shared the feedback by asking people to test it out. That's again a good, important step. Where we are at, therefore, is: **it's easy to build an application, AI can help us create the product, but what's trickier is to make sure that it's useful and make sure that it's usable.**

**Anand**: [00:59] Now, you've tested the application, you've found a few bugs, a few issues. You might tell ChatGPT or Claude Code to make a few changes, and it'll make a few changes. And after some time, you'll find that something you wanted in the first place is no longer working. Meaning, it adds new mistakes as we go along, which we don't want.

**Anand**: [01:31] And we also would like it to maybe regularly test like a person would—clicking here, pressing that, making sure that the system works the way it's supposed to over time. That's part of what we do in testing an application. **Is it possible for ChatGPT or Claude to do that testing by itself?** Can it verify? And can it verify in a way that won't do different things each time or won't cost us tokens? That's part of what we're going to see in this session.

**Anand**: [02:18] So, let's do this. I'm going to share my screen. Okay, it's probably visible now. And, oh, I made a small mistake; I joined from a different browser. Let me join from this browser.

**Anand**: [03:29] Okay, just checking, am I still audible? Good, yes. Let me start sharing my screen. Okay, great. Let's take one of the pages that you had shared. I'm going to pick one at random. I'm going to pick... okay, here's a link to a ChatGPT site from Renata. And let's take another one. Here's one from Mayu. I'm picking these at random; we can and should absolutely do this for all of the sites.

**Anand**: [05:14] Okay, supposing we want to create an automated test for this, how might we go about it? I'm just going to ask ChatGPT to try that out. Specifically, I'm using ChatGPT Desktop for this. Why Desktop? Because if it can use my computer, maybe it can also use my browser, which may require some of these sites to be logged in, and I have a little more control. So, I prefer using the Desktop version of some of these tools.

**Anand**: [06:04] Secondly, you can either use ChatGPT or Codex on the ChatGPT side. If you were using Claude, then the equivalents of this would be either you could use Chat and Co-work on the one hand, or you could use Code on the other. There isn't too much of a difference, but the code version of both of these tends to write and run code a little better. So, I'm going to use those.

**Anand**: [06:36] The next thing that I'm going to do is choose—actually no, I'm going to leave the model choice. Right now, it's chosen Luna and medium effort, which means it's going to cost very little. Well, no, let me just maybe upgrade to Soul; medium effort is fine.

**Anand**: [06:58] What I'll do is go through this site like a real user, find out what it does, and test it. Then, create an automated set of test cases that I can rerun anytime without needing any AI application. That should test the entire thing. Feel free to use my browser and whatever else you need. Also, create a small document explaining what exactly you're testing so that if I need to change something in the future, I will know. **Make sure it's easy for me to add new tests in the future by just telling you.**

**Anand**: [07:49] That roughly captures what I need. Now what I'm going to do is pick one of these sites and paste that. Yeah, give it a shot, let's see if that works. And you could do the same on Claude; I just have a little more budget on ChatGPT, so I'm using that.

**Anand**: [08:15] What I'm expecting is that it will open my browser and start controlling it by itself. Now, the reason it can do that is because I've installed an extension for Claude, and I think I've installed a ChatGPT extension, I'm not sure, but somewhere I've got an extension that allows it to browse the site by itself. And that's something that you could try. If your application says, "Oh, I can't access the browser" or "I'm not able to log in" or some such thing, then make sure that you give it access to be able to run these applications.

**Anand**: [08:58] Before we go further, I'll just summarize and then hand over to you for any questions. Feel free to type in your questions in the chat or unmute yourself and ask. Feel free to type your questions in English or Japanese or any language, that's perfectly fine. But I'll summarize by saying: **if we can automate how applications are tested by using AI, that means that we know they do what we expect them to do, and we can change them safely.** And the easier it is to change something safely, the easier it is to build rapidly. That's going to be an important part of what we're doing in the second half of this course. Any questions?

**Johan**: [10:06] I've been trying to implement the AI itself inside the artifact, but that hinders the artifact from being shareable. Is there a workaround around that, or how else should I implement the AI inside the product that I'm making?

**Anand**: [10:28] Let's find out. And your site was... let's see, I had the email somewhere. Unfortunately, it's no longer shareable because of that. Oh yeah, but I think you had shared what it was about. And I have a screenshot of the... okay, yes, it was the Stock Ticker application. Great, fine. Let me keep it generic. And this was deployed on—no, it was deployed on Vercel, if I remember right. Is it? Or Cloudflare, sorry.

**Johan**: [11:05] Oh, Cloudflare, yes.

**Anand**: [11:07] Okay. So let's ask this question. I have an application that needs to use LLMs as part of its feature set, for example, answering a question from data that the user types in. When I create this as a ChatGPT site, it works fine, but then I'm not able to deploy it for others to be able to use. If I publish it say on Cloudflare or GitHub or some other place, then I'm not sure how I could go about using the AI features. Can you explain in really simple terms, the most concise way possible, what's the easiest and right solution for me? So that I can expose a simple site to at least a few people or maybe even publicly, and it should be able to incorporate AI features without too much cost on my side.

**Anand**: [12:35] Give that a shot and check it out. Part of the reason I'm doing this rather than sharing an answer with you is: A, I'm not sure if I know the best answer. B, even if I did, knowing the answer and telling you is not the important point. What I want to say is that you could try different ways of asking AI yourself and learn. **It's that approach to how you might learn that would help.**

**Anand**: [13:00] Okay, so it's saying the simplest solution is deploy on Cloudflare Pages or Workers and use Cloudflare Workers AI. So, because yours is already on Cloudflare, it's saying it can create some link which will call the LLM. And the important bit is that it shouldn't call the API directly; instead, there should be something on the server, and Cloudflare makes it simple. And there's some code here, and it's saying, "Okay, this is something that's the easiest."

**Anand**: [13:48] Some people know exactly what it's talking about; some people have no clue what it's talking about, and it doesn't matter. Because what you could do is take this entire thing, put it back into your original chat, and say, "Look, ChatGPT said this. Do it." And if it works, great; if not, you wasted some time. Big deal.

**Anand**: [14:12] But personally, this is not the solution I had thought of. This is a good solution, and it's probably an inexpensive solution as well, because Cloudflare has Workers AI functionality. The solution that I had thought of was you would create your own API key and write a Cloudflare worker—yeah, I guess this solution is what I had thought of, and this is better for Cloudflare. What I'll do is share this on the chat for what it's worth. And that was not the share link. Strange that it didn't seem to be giving me copy—okay, that worked. And this is the link you want.

**Anand**: [15:27] Now, you notice that on the right side, let me make it bigger, it's still going ahead and looking at the site—Single Page City Personality Quiz—and it's covering both the happy path, which is people do the right thing, they know what to do, and errors as well. So when they make a mistake or it tests out some weird things. Okay, so it's created an eight-step journey, and it will also, okay, keep the routine—fine, let it keep running, it could take a little bit of time. Any other questions?

**KK**: [16:36] Sorry for always asking questions, but I feel that one thing—doing my process of trial and erroring—I feel that is this really the right thing to do? Because if you don't even know what you're doing and then you're just throwing whatever the suggestion is into Codex or whatever, then it's building whatever based on an AI suggestion, but we don't really know what the heck is going on. **So it's a black box. Is this really right? Are we like—is this actually more of a dangerous thought?**

**Anand**: [17:21] Possibly. That's a fair question. Sometimes we need to know what's going on inside; sometimes we don't need to know what's going on inside. I drive a vehicle, a car. I have no idea how the engine works. If it breaks down, I can't even change the tires. Is that a bad thing? It is, because there have been a couple of occasions when I was stuck and I needed to just at least change the tire or know that the engine was heated.

**Anand**: [18:13] But when I got into those situations, it was not a terrible thing because I was able to call up somebody and they came over after an hour or so and sorted the problem out. What I'm realizing is that if I had spent a lot of time learning engine theory and how that works, etc., I would have so far in the last 30 years spent more time on the learning than I actually used because of the problems I had. I've had about three incidents where I needed some kind of mechanical support with a vehicle. The total time that I spent on this was half a day. The amount of time that I might have taken to learn about engines and changing a spare tire, etc., would probably have been a couple of days.

**KK**: [19:03] I mean, I understand the perspective that you're learning a kind of niche skill, you know, car repair, for occasional three times' use. But since we're undergrads and students, isn't this technically more important because we're potentially delivering it to our bosses and stuff? Like, let's say I study data analytics, so let's say I create data analytics apps for soccer—because of the soccer example—and I offer it. And of course, in this process, I will most likely encounter more bugs and issues and stuff that would be more crucial for me to know rather than car repair. So how?

**Anand**: [19:54] One of the advices that I give to interns that join Straive is I tell them to join a call with our CEO, and he gives them some instructions. I tell them: record the call, transcribe it, and give it to ChatGPT or Claude and send the output back. **Do not try and understand what the CEO is saying. You won't understand it.** Don't try and interpret the output; it may not make sense to you. Feed it back to him and let him figure it out.

**Anand**: [20:39] Now, in that case, you may say, "Wait, what value is the intern adding?" Well, the process could have bugs, he's sorting it out. It takes four hours. That helps the CEO because then he can just give the instruction and take the output back. So, does that mean that the intern's not learning anything? Yes. Should they learn something? Maybe.

**Anand**: [21:05] But here's the thing: suppose Ankur, who's the CEO of Straive, sees the output and says, "Yeah, this is what I want. Great." Then there was nothing for the intern to learn. There was something that eventually he's going to ask the agent to do, and the agent's going to give him back the response. There was nothing here that was worth learning. But if he had said, "You know what, Pavan, the interface isn't quite like what I want. What I really want is something that I can stand up on a stage and demonstrate, so the fonts have to be a little larger, it's got to look a little more like a slide deck," and so on. Now Pavan has picked up something. He's learned that there are a few things that he should tell the agent even when not asked for.

**Anand**: [22:02] The next time Ankur asks for something, Pavan would hopefully automatically add that context because he would remember. If not, he'd be told again and again, and eventually told off. So, put another way, I would say: **it's worth learning what there is a demand for and what there is a supply gap for.** If the agent already does it, there's no point learning it. If the human doesn't ask for it, there's no point learning it. By delegating as much as possible, we get to find out what the gap is and focus our learning on that. And if for a short while there is no learning, we should just be more ambitious. Try something harder and see at what point somebody says, "Look, this is not right."

**Anand**: [22:56] Now, here's the thing: you're already hitting that barrier so easily because your apps don't work perfectly. Functionally they may be fine, but they don't do what the user wants. People look at it and say, "Yeah, okay, so what?" And if you find that you're perfectly happy with the outcome, you're not ambitious enough. And that's part of the skill that you're developing, which is: **all of the things that we've been taught—not all, but a lot of the things that we've been taught so far—are things that agents can do.** And we think, "Shouldn't we be learning those?" and I'm saying maybe not. If an agent can do it, maybe not. What it can't do is what you should be learning.

**Anand**: [23:48] Is that always true? No, but here's the thing—I'm just one teacher amongst several dozen. The others will tell you what you should be learning; I'm just telling you what you shouldn't be learning. All right, thanks. Keep the questions flowing, please.

**Anand**: [24:13] But let's see in the meantime what we have. So, the site asks for an age and a favorite atmosphere and presents eight choices and combines them into a city archetype illustration, traits, etc., which is very interesting. The state is only in the page's memory, no backend, no saved result, etc., fine. And then it gives me a bunch of tests, which is fine. What I'm really interested in is what it tests. So, here's what it's testing: whether the text on the landing page is fine, are they...

**Anand**: [25:00] ...are there any missing profile fields, are all the questions there, can we move back and forth, are the titles present, are the results properly described, etc. This isn't very easy for me to understand. So, let me make it easy for me to understand. I'll say: **"Can you rewrite the coverage as simple questions that explain what you tested, like: 'Is the text on the landing page okay?' That sort of thing."** That makes it easier for me.

**Anand**: [25:48] And this is another important principle that I've seen: **the agent is doing the work; we have to verify.** Usually, that means that we end up doing a lot of verification work, and that's painful. **I like to make it easy for me to verify.** So I tell the agent, "Give it to me in a way that I can understand. Make my job easier."

**Anand**: [26:21] So, in this case, okay, here are the questions that it's testing: Are the age and atmosphere fields clearly labeled? Can visitors choose from all five atmospheres? Does it reject ages below 18 and above 120? Yeah, this is much more easy for me to understand. Now that we have this, we can rerun this at any point very easily. So, it's giving me a set of commands on how to rerun it. But again, I don't even want to rerun it by myself. I would just rerun it using ChatGPT or Claude, saying, "Run the test." After I change it, for instance, I might say, "I've changed the site; run the tests again and see if it works, and tell me what's changed."

**Anand**: [27:22] What we've covered in these 25 minutes is how to go about creating an automated test suite. That's a useful skill. But this isn't the only kind of verification that you're doing. When you ask different people, you are effectively getting user reactions, like how does a person react to it, how do they feel about it. That's more like a market survey. Today, I will be collating all the sites and asking you to do a mini market survey amongst yourselves, but **it's possible to have agents run that kind of a market survey themselves**, and I'll show you how.

**Anand**: [28:12] What I'm going to do is first check my Codex usage. Okay, I have 38% of my usage limit running, which might be enough if I use Haiku more than others. So let's do this: I'm going to take another site. Let's choose another one at random. This is from Yuri; let me open the site. This is a Study Walker [スタディウォーカー]. Great.

**Anand**: [29:02] Now, this site is probably meant for certain kinds of audiences. And what I'm going to do is ask ChatGPT again—or let me ask Claude for a change, let me check how much Claude—yeah, I think I have enough Claude usage. Let me ask Claude the following: I'll ask it to pretend to be different people and visit the site and use it and give feedback. So, let me start. Like I mentioned in the last session, I prefer dictating on ChatGPT, and then we'll transfer it to Claude.

**Anand**: [29:41] "**I want you to run the equivalent of a market survey on this site. Go through the site briefly and understand who are the top three personas that this could be a product for. And then create sub-agents using a lightweight model like Haiku and pass them the persona for each of the audiences. Tell them to act like those personas and browse and test the application and share feedback exactly like they would in a market survey. Have each of them save their results as a Markdown file that explains with examples what they did, how they felt, what could be different, and so on. To be frank, I don't know much about market research, but you are the expert, so you tell me how a good product market research should be conducted and actually conduct it that way. After these three sub-agents have finished, I'd like you to consolidate the feedback and give it to me in the form of a simple report where you start by sharing the most important feature that is easiest to action first, and then the second most, and so on.**"

**Anand**: [31:21] You're probably getting the idea: **I'm delegating not just the market research, but even the structuring of the market research.** So, I'm going to paste this along with the site. Make sure things are fine. Okay, let's go ahead and run this. Okay, the folder no longer exists... okay, let me create a new folder: "Market Survey." I'll select that folder and run it. Let it run.

**Anand**: [32:42] So, to recap, the second theme of verification is: **have agents augment user research.** Have them pretend that they're users, have them look at the site like they're users, and this won't necessarily replace human response—maybe it will, I don't know, but I haven't tested it enough to know and be sure that it can replace human research, but it certainly provides useful additional feedback at very low cost. So if I don't have enough people to check with, or there are certain kinds of people that I can't reach out to—for instance, in our class, if you wanted to reach out to a sports coach, a football coach, and we didn't have one, then I'd take that site, pass it, and say, "Look, imagine you're a football coach. Use this site. Find out whether this is working for you, whether this is useful for you, and give me a report based on that." That can help.

**Anand**: [33:53] Again, any questions that you have, you're welcome to just unmute or type in the chat window. We are going to move while this—okay, it is still running, let it finish. Okay, I have a feeling the computer got kicked off... power is fine, everything seems fine... oh, oh no, it is running here. Sorry, I should just stop and delete. Here we go, okay, fine. And it's running some scripts, it's interacting with the page, doing all kinds of things. Yeah, we'll let it be.

**Anand**: [35:33] Okay, so we're now getting into that stage where you will have to finalize roughly what product you want to build by the end of this week. It certainly will be a product built by AI; it could be a product that uses AI as well as part of its feature set. That's entirely your call. You will finally be publishing one or more—you can publish multiple sites as what you're delivering this week—and we'll also be sharing the tests that you have run. That is, what are the automated tests that you ran on it, what's the market research that you did on it with agents, and **what's the feedback from human users? We'll do a second round of that.**

**Anand**: [36:32] For now, what I'd like you to do is—I'm going to create a Google Sheet and share that with all of you, and we'll ask this group to conduct a market research on each other's applications, and record that in the form of a video. I'll show you how you might go about doing that. Okay, so: Your Name comes here, App Link would come here, Feedback 1 Name... let's stay... yeah, okay. The person who's giving the first feedback, and the person who's giving the first feedback will share a link, and the person giving the second feedback, and the person giving the second feedback will share another link.

**Anand**: [37:51] Let me explain how this works. I'll take "Study Walker" as the site. Yuri, I'll fill this on your behalf, please. So, let's say "Yuri" and that's the link. And let's say the first feedback comes from me, so I'll put my name. And what I'll do is record a video providing feedback to this site. Now, how do I go about recording a video? Actually, I'm not sure, but does YouTube have some feature where I can just go to the site and click a button and it will record my screen and camera and mic? Is there any other site that does it? What's the easiest?

**Anand**: [38:52] YouTube does not have such a feature. `recordscreen.io` says it is the simplest. Okay. I'm going to go there and let's test it. Yeah, I want screen and camera for what it's worth. I'm now recording microphone, camera, and my entire screen too. Okay, this is good. So let me just go here, test. I'll speak a few things, click, and then come back here and stop the recording. Okay, it recorded my video. Perfect, this is very seamless. Cool. So I'm using `recordscreen.io`, you're welcome to use whatever tool you want.

**Anand**: [39:56] But what I would like each of us to do is provide a feedback link by using the application to each other's applications. So, here is my feedback. Recording the entire screen. Okay, I have visited "Study Walker," which is in Japanese, so I'm translating that page to English. Let's see what this is about. "**First step to start doing what you want today.**" [今日やりたいことを始めるための第一歩] Okay, not sure what that means. "**What do you want to be able to become?**" [なりたい自分になるには？] Okay, so it's some kind of a self-help study thing. So I said, okay, I want to—let's see—learn Rust programming language. I want to get started.

**Anand**: [40:59] Okay, "**Find your first step today.**" [今日やるべき最初の一歩を見つける] What's stopping me right now? I tried it but it was difficult. And how much time do I have available right now? Open it and check one thing. Yeah, that's the middle of this call, that's probably all I'll have time for. What can I use? I can use my PC. And now open the page and check the first place you touch. Okay, this wasn't too clear to me, maybe the translation isn't doing a good job, but what it's saying is, "Open the page or tool and check the first place you touch." If I can do that much, that's enough. Okay, I didn't quite understand it, so I'm not sure what to do, but I'll try.

**Anand**: [41:58] "**Start now.**" [今すぐ始める] Okay. "**Learn Rust in 1 minute.**" [1分で学ぶRust] Okay, there is "Learn Rust in X minutes." And let me just go through it. "Use `let` before assigning a variable, `fn` defines a function, and everything has a type by the looks of it. Looks like we have some pointers." Hmm, more complex than I would like. So this would take me more than three minutes. But maybe there was some other site that I could have learned... half hour... yeah, nothing that I can learn in one minute, but here's something. Oh, this is useful: the AI summary says I can use `println!`. Can I just run a Rust interpreter online? Okay, there is the Rust Playground. Okay, yeah, if I just run this, it's printing "Hello World." That's not too bad. So if I change this to `1 + 2`, it might not work because those are numbers, and maybe I convert to a string... by typing something like this... no, okay. Oh, so it's giving me some help. I like this help. This tells me what I might want to use. And if I copy-paste from here, that's kind of working and—okay, it printed 3, great.

**Anand**: [43:54] So in a minute or so, I was actually able to take one step: Hello World. And yes, "Use your first attempt." I made a first attempt. **What I noticed was the error messages are more useful than I thought.** I like that there's a "Can't start well." I was tempted to give up, and I could have clicked this. But this is useful. So I was able to perform the first operation. It's interesting that I did not even read all the other options because as soon as I find the first one, I click "Yes." So does the ordering matter? Maybe. And I actually got to the point where I started. Yes, that I did. It takes—no, and from here I'll proceed on my own. Thank you. And that really did help me. It managed to make me move forward on something that I wanted to do. So yeah, that's my feedback.

**Anand**: [44:57] Now this is great. If I open this in a new tab, it still exists... yeah, maybe if I open a new browser it may not, but it's still here and I can maybe continue from where I left off. So yeah, **overall this is a starting point for something useful.** Would I pay for it? Maybe not. Is it a useful application? Yes, I think it is. Are there other things that it is not useful for? Maybe it is, maybe not. Let me stop the recording.

**Anand**: [45:34] So that took me five minutes, and we now have feedback for one application that I was trying to figure it out, and maybe I would have failed or not have understood some other application. Let me download this video and ask for some place where I can just upload videos by dragging and dropping and sharing it with someone. Ideally, no login required, no restriction, just simple and easy. And there's something called `streamable.com`. Okay. Is Claude asking me for permission for something? Still running.

**Anand**: [46:35] Okay, let me upload the video. Where's the video? This is the video. Okay, I have visited "Study Walker"... that's the video. And it does not place the video maybe because it's in a different format. So let's skip this. `WeTransfer`? Okay. No email, I just want a link. `streamable.com`... let me upload the file. Hmm. Oh, I think I started it wrong. Yeah, here we go. Okay, but then maybe Streamable was able to do it as well. Okay, yeah, it is able to. So, sorry, the site was working just fine. Streamable. And I'm going to copy this link—it will get uploaded in a few minutes—and that's the feedback link.

**Anand**: [48:28] I have finished reviewing one application. I would request somebody else review this application, and everyone fill in the names of your applications here. So, let's call this "FT AI Products peer-to-peer feedback." I am going to share this... but wait, okay, I'll just make it editable by everyone. Change the permission. Anyone with the link can edit. And I'll copy the link and paste it here. I do see a question, we'll come to that.

**Anand**: [49:20] So now I'm going to ask everyone to please put in your names and, Yuri, if you want to correct the spelling or put in the full name or whatever, please feel free. And not just put in your name, also put in your feedback name, that is, choose two sites that you will give feedback for. I would like columns A, C, and E to be completely filled out by you in this session now. Please go ahead. Let's find an empty row and start typing. Avoid typing on the same row. Avoid filling out or overwriting someone else's response. You can see who's typing in some row. It's okay if you leave some rows blank; we can sort it later. But first, fill out your name, please.

**Anand**: [50:32] And Dora, I'll come to your question once a few people have started filling this out, and I'm also going to take a quick half-minute break to get some water.

[Pause/Interlude]

**Anand**: [51:58] Okay, we have one, two, three, four, five, six, seven, eight names so far. I’m guessing we should have fifteen—from my count last time. Let's just do a quick check on that. Yeah, I'm guessing we should have fourteen or fifteen, I think.

**Anand**: [52:44] In the meantime, I’m just going to take a quick look at the question shared by Dora, which is: "**For context, I'm building a root mapping tool to customize their journey. While simplifying my experiment and deploying the website, the agent adds extra steps, making it more complicated to deploy. Then it turns into a cycle of me telling it to simplify the process with varying success. How do I break out of this feedback loop?**"

**Anand**: [53:26] I'll tell you a few things that I would do. Actually, first, before I even do that, let's ask a reasonably smart model this question. I'm going to pick Claude to answer this question, and specifically Fable [Claude 3.5 Sonnet?]. Medium is fine.

**Anand**: [54:06] But I'll give you my answer, which is what I do when the application just goes off into a loop. **Step one: ask a smarter model to make a correction.** And how would we do that? Let's say I'm working with Sonnet; midway through, I would switch to Opus or Fable—at least one level higher and with a higher level of thinking.

**Anand**: [54:41] The second thing that works sometimes, sometimes it doesn't: **start a completely new chat.** Say, "Look, take a look at this application. It's becoming too complicated. I'd like you to rewrite the whole thing and make it extremely simple."

**Anand**: [54:59] Sometimes that works. If that doesn't work, then what I do is I don't even let the new chat access the code. I just run it and say, "**Now look at the browser; don't look at the code.** See what the application does." I could even paste screenshots of the application and say, "This is roughly what I want. Build it again from scratch, but don't add extra steps. Make it easy for me to test and deploy."

**Anand**: [55:32] **The important learning here is that throwing away something for which it did the work—you didn't spend that much time and effort—is easy.** You could even say, "Don't give me one version. Give me three different versions, each of which is very easy to test and each of which is very easy to deploy."

**Anand**: [55:51] You could even take the entire conversation. You could say, "Look, this is everything that I said, this is everything that it said, and this is how we iterated back and forth, and here is the code. **Now, give me a new prompt which will do what I wanted, but it will be much simpler to test and deploy.**" Those are a few different ways I would have done it.

**Anand**: [56:16] Let's see what Fable says. Okay, saying "**'simplify' is vague, and the agent doesn't know what 'simple' means, so it guesses.**" That's a good point. **Stop asking it to simplify and tell it exactly what the end state looks like.** Okay: "one command, one HTML file, one JavaScript file, no build step." Fair enough. If you know enough to ask for something this specific, this is actually an excellent option: telling it how exactly to simplify or how exactly to deploy.

**Anand**: [57:00] Starting a fresh session is something that I mentioned a short while ago. Asking it to delete again... yeah, just "rewrite... oh, remove stuff." Remove this functionality, remove that functionality. Yes, that again is a good way of doing it. Give it a shot, let's see if it works. But don't hesitate to start from scratch; nothing wrong with that.

**Anand**: [57:28] Let's take a look at this list: three, four, five, seven, eight, nine, ten, eleven, twelve of you. I’m guessing there are at least a couple more that probably should be adding their names on here, but please go ahead when you can.

**Anand**: [57:51] Now, please start filling in "Feedback 1 Name" and "Feedback 2 Name." Just put in your name against any of these. So, **your name should appear in at least a couple of places.** I'm going to move mine out; if somebody does want to do a Feedback 3, that's optional. But please put in your name as volunteers for one Feedback 1 and one Feedback 2. So each of your names should be in Column C and in Column E wherever you want, with no overlaps.

**Anand**: [59:31] And when sharing feedback, please feel free to speak in any language you're comfortable in, because translators these days can do an excellent job. So honestly, it doesn't matter. Meaning, it should be more... **the more you share how you feel in a way that's easy for you to express, the easier it is for the audience to understand.** So make it easy for yourself when giving feedback.

**Anand**: [60:16] And **please don't try and record multiple times to get the feedback right. Mistakes when giving feedback—that is probably the most useful feedback.** When you get confused, or you slip up, or you say, "Oh, okay, I found it," whatever. Just record it one shot; that is the best feedback.

**Anand**: [60:48] Okay, three, four, five, six, seven, eight, nine, ten... almost there. Great. Just a couple more names and... I should mention that this is optional. Just make sure you have... let's see how many do we have. Okay, at least twelve names here.

**Anand**: [61:33] Okay, while you're filling that... please make sure this is filled. And yeah, so **your one of your assignments today is certainly filling out the feedback links for whatever you volunteered for.** And of course, yeah, the app link as well. Johan, please just put yours in as soon as you can so that Noka [Noa?] can review it. For the others, you have the links; please do take a look and review that. Got you, KK.

**Anand**: [62:12] Now, let's take a look at the report that we got from Claude. First, let's see what it did. So Claude spawned three agents. And from a usage perspective, let's just see our usage. It's at 29% of usage; I started the session with about 12%, so it consumed roughly 17% of my five-hour limit, which is nothing—it's very small, actually.

**Anand**: [62:56] In other words, **market research with a model like Haiku—and I specifically chose Haiku because it's an inexpensive model—can be done very efficiently.** If you did not choose Haiku, it may end up costing you a little more, which means that until your five-hour limit is finished, you may not be able to make much by way of changes. It's okay, it's just a five-hour limit, so you won't have a problem.

**Anand**: [63:31] So it did a whole bunch of things and it identified three personas: **a Choice-Paralyzed Beginner, a Serial Habit Quitter, and an Exam-Driven Pragmatist.** Yes, these are three good types of students to pick. And it tested it against the live site, and it's saying that the core idea—which is "take one small step today instead of learning X"—it really worked well for all three people. But **the main gap is that the sessions don't have memory between the visits.** So that's probably the biggest feature to fix.

**Anand**: [64:19] And it's suggesting a series of features. We’ll cover those, and it's also provided a detailed report that explains how the research was conducted. But the first thing it's saying is there is a mismatch, there's a problem in the text, which should be easy to fix.

**Anand**: [64:38] Okay, this is a very useful one: "**Always keep the user's goal visible.**" When I was going through Rust, just mentioning that I'm learning Rust would have been helpful if I'd stepped away and come back to that screen. And so on. Confirm: "how did it go?" Yes, I think these look like reasonable suggestions.

**Anand**: [64:58] Now, you may not need to fix all of these. Each of you will be getting three sources of feedback: one from an application—from a market survey like this—second, you'd be getting it from two people: a Feedback 1 video and a Feedback 2 video. **Take all of this, decide what you want to improve. You don't have to take all the feedback.** You can decide what you want to improve and come up with a newer version of the site.

**Anand**: [65:40] What you will be sharing is also the agent research log. So in my case, the link that I shared could be... let's see how I would go about sharing this. I believe I can share the entire chat. Okay, if I tell it to "Continue in the cloud"... oh, I have to commit and push; that's too complicated.

**Anand**: [66:26] Okay, at the very least, I would take this file and publish it somewhere. Where could I publish it? Anywhere; just paste it into Claude and tell it to publish it somewhere, and I would put in the link to that particular one. So **this is a column that each of you need to fill out as well.**

**Anand**: [66:51] So that gives you... and I'll move this column to column G so that once we complete our... each of you will be filling these three columns. First two are mostly done; this column is to be filled, and others will give you feedback. Use that and update your application.

**Anand**: [67:21] Tomorrow, I'll give you instructions on how you should share your session logs. You'd be running this on ChatGPT, or Codex, or Claude, or Claude Code, or whatever. What you typed and what the agent did is saved somewhere. **I'll be asking you to share the logs of those, and tomorrow I'll guide you on how to do that.** That's part of the exercise.

**Anand**: [67:48] By the day after, you would have an application that is working better than what you started with because you've incorporated others' feedback and you have tests that verify that the application will work even after you change it. I'm not asking you to submit the automated tests just yet; that will be something that you need to submit by day after, but you can test it out for tomorrow if you want.

**Anand**: [68:23] So deliverables:
1. **The agent research.**
2. **The two feedback links.**
3. **Improvements on the application, ideally.**
4. **Automated tests (optional).**

**Anand**: [68:37] Any questions from anyone?

**Shuku**: [68:47] Okay, so should I join the feedback?

**Anand**: [68:50] Up to... it's entirely up to you. Yeah.

**Shuku**: [68:54] Yeah, okay. Thank you.

**Johan**: [68:57] And should the feedback be until the Friday session or tomorrow's session?

**Anand**: [69:08] **Feedback definitely before tomorrow's session, if possible today, because that will give the other person time to improve the application** and work on it. Exactly.

**Johan**: [69:18] Okay. I'll get it up and running once my credits are back, sorry. Thank you.

**Anand**: [69:24] Sure. Tomorrow is an optional session. If you have any questions, you can also email me at `askai@s-anand.net` at any point. I will reply whenever I get around to it, or my agent will reply whenever it gets around to it. Or you could always just join the session tomorrow.

**Anand**: [69:50] What we're going to be doing is reviewing some of this work, and tomorrow I will also be giving you instructions by email on what you need to present by day after. You won't be presenting verbally in our face-to-face session day after; you will be sharing a video that you'll be creating. Again, it could be in any language; it'll just be a quick three-minute—or not even, maybe two-minute—video explaining the application and what you did, and that will form part of your portfolio.

**Anand**: [70:34] Okay, with that, we conclude the third day's session. Thanks everyone for joining in. You're welcome to drop off, and I'll see you either tomorrow or face-to-face on Friday. Bye.

**Shuku**: [70:47] Okay, thank you.