# Transcript - Prompt to Plot

Hopefully, all of you have Wi-Fi access. Please do go to the link on this QR code, which has the talk details, because you will need to do a fair bit of setup, which you can probably entertain yourself with while I'm talking. It's probably going to be more interesting to do the setup.

Question: Just two questions, sorry. One, would you need a mic? It would be easier. This is for your recording.
Answer: Yes, I'm using the recording. Are you able to hear me clearly at this... Not very clearly. Okay, then yes, I will be needing a mic. I'll speak up for a short while, and then the mic will arrive.

Let's start with why I don't use Excel anymore. Because I switched from Windows to Linux. That has nothing to do with a choice. I mean, I've been wanting to experiment with Linux, but for 30 years, I have not needed to, mainly because Excel does not have Linux.

Two things happened. One, Straive got acquired. Sorry, Gramener got acquired; Straive hasn't been acquired yet.

Sure.

Is this coming through the mic? No? Yes? No? Check, check, check. Any better? Okay, great, perfect.

So, Gramener got acquired, and the IT security team said, "We need to install all kinds of software on your laptop that will prevent you from doing any useful work." So I said, "I'm shifting to Linux. You don't have that software on Linux, so don't touch my machine." And I no longer have Excel. But **the timing was perfect because that's when I realized I could just upload whatever I want into ChatGPT and have it do whatever analysis I would have done on Excel. And it's even easier.**

The other advantage is I could do this while walking. I'd just upload from Google Drive into ChatGPT and say, "You know, create this pivot table, drop that column, blah, blah, blah." And it was so fortuitous.

There's a lot more that we can do. Today's workshop is about how we can, together, in some sense, come up with a few data stories. When I say together, I don't necessarily mean that groups of us need to be working together to come up with one or a few data stories. You can individually come up with data stories. But **let's publish a few today, and ideally only using LLMs.**

For this, we need a little bit of collaboration, and I've got two channels out here: a Google Group and a WhatsApp, for both of which you can either get the link from the earlier the talk slides, which you have access to on the site you would have gotten to earlier. And if you missed noting the QR code for the talk slides, everything on the talk slides is available in the QR code on the top right, which will be there on every slide. And the WhatsApp invite is the second one. The Google Group that you need to, that you could join, is the first one. I'll share the longer conversations on that. You will probably be sharing more stuff than I will as we go along. For questions that you may have, we will use the WhatsApp for quick communication. Anything that you want to persist for the future, we'll use the Google Group. I haven't figured out how to create a public space. We will create a repository.

I'll move on to the next slide, but if you need this, please feel free to take a picture or scan the QR code on the top right at any point. I'll wait for about 10 seconds.

Let's go on to the next, which is what are the tools we'll be using. GitHub. Some of you may have GitHub accounts. If not, please create a GitHub account now, later, whenever we get there, because that's where we will be having LLMs explore your data set, code, whatever, create the code, and publish it. And the instructions on how to create these, we'll come to that in a bit.

We'll be using, I will be using, three chatbots and four coding agents, two of which are listed here. Chatbots you're familiar with. I would recommend getting the plus, $20 version of ChatGPT if you do not have one. Who already has a $20 or a $200 version? About 60%. And just for completeness, who does not have a $20 or a $200 version? Fine. For those who don't, I would encourage it for two reasons. One, it's, for the cost of a meal, it's not very expensive. B, if you have it, you are more likely to push yourself to use it, and increased usage is the best way of learning things, because **there is a big difference between the models in the free version versus the $20 version.** There isn't as much of a difference between the $20 and the $200 version, in the sense that you get a little more usage, a few people will need it, you get access to the latest models. And there are cases where the latest model, O3 Pro in particular, is much better than O3, but only in certain areas. You may not be able to tell the difference there.

If you are heavily into coding, then I would recommend going into the Claude Pro version. **ChatGPT is generally the smartest of these chatbots for applications. Claude is particularly good at code of most kinds, but extraordinarily good at front-end code.** So that's how I'm using it. And **Gemini's primary selling point is that it's free**, which is not a bad selling point. It's not a bad model either. And then there are others that are less popular, not necessarily less bad, but I'm just going with the popular ones. And here's also a rough rule of thumb on how you would choose between these. By default, go to ChatGPT unless you need something free, or Gemini is also particularly good at audio, certain kinds of visuals, and video. So for those applications... I should specifically say Gemini is good at photos, and if you want to transcribe something, if you want to have it process a video, that it's very good for that.

There are a few coding agents that have been released this month, last month, whatever. Jules by Google is free, and Codex by ChatGPT is available for the $20 and above version, which is why for this session, I'm recommending you go for the $20 ChatGPT. This apart, we will potentially also be using a few tools that I can run on the command line, Claude Code and Gemini CLI. For some of you who may have used and certainly many of you who would have heard of tools like Cursor, WING, GitHub Copilot, these are tools that let you write programs. And sometimes even from scratch, they can also do a certain amount of research, they can do a certain amount of analysis. But we are not going to be using those today, for no major reason. You could, but I'm just going to show how to use some of these on the terminal.

Given the mix of the audience, we may spend less time on these. Most of what these can do, the ones at the bottom, which are remote, can do. They're also a whole lot easier to use. But I will, in case I get stuck, take the help of some of these, which have one major advantage that the other two do not. These can access the files in my machine and from my machine. Those, unless I upload them, cannot. And if today I haven't decided what story I'm going to write, but if it turns out that it's a story that's based on a one terabyte data set that sits on my machine, or even let's say a 10 gigabyte data set that sits on my machine, I'm not going to waste time uploading it to those tools to do the analysis. That is one reason. And you'll also notice that the best front-end coder, Claude, is not there among the online coding agents yet. Who knows, it's probably going to come next month or next week or next month. But until then, I may choose to take the help of Claude Code to, if I try two times, three times it fails, give up. Move on to something that works better. That's roughly how we will be going ahead.

**Today we are going to be using LLMs for everything.** I wrote this deck this morning using LLMs. We will be, together, finding, and individually finding a data set to create a story about using LLMs, figuring out what to do with it by asking the LLM, analyzing the data with the LLM, visualizing it, publishing it, all of that will be done through LLMs. What we'll do is start with maybe me taking something simple and going through the entire workflow, which you're welcome to follow along or watch and repeat. I am recording the audio as well as the screencast, so short of my face, you will get this published at the same links that you scanned, and everybody will have this as well. So, I leave it to you on how you want to do this.

Now comes the hard part, which is finding a data set. Any suggestions? No, why am I saying? You're my LLM. ChatGPT.

Now, choice of models. I usually go for O3, and if I exhaust my limits, I will then go for the other models. A little worried at this early stage, I probably have about 100 uses of O3. I might not use 100 in this session, it's very unlikely. Therefore, I will reach out. I was going to say, but I want to play it safe. Who wants to play it safe?

Now, it's so much easier. I'm running a workshop as part of this. This workshop is called "Prompt to Plot." I'm going to be showing a very mixed audience how they can take a data set and convert it into a data visualization entirely using LLMs. So the LLMs will be doing the analysis, the LLMs will be creating the code, the whole works. Now, I want you to suggest a bunch of data sets, like it, searching online. But you know from my past conversations my interests. So pick something nice and quirky. This will be the first iteration, so I want to pick something that is small, let's say about 100,000 rows or 10,000 rows, not too big. And with a reasonable variety of columns, text, longest text, categorical text, numbers, that sort of thing. But most importantly, at the end, it doesn't even matter what kind of analysis the LLM does or we pick, the story should be interesting for a wide variety of audiences, otherwise, what's the fun? Now, give me something really cool. Actually, give me a whole bunch of stuff that's really cool.

**The advantage of speaking to an LLM is you can do stuff like this. There are three advantages that I see. One, I can be a whole lot more verbose, which I would not be if I were typing. Secondly, I can talk to an audience while doing this, and it is less distracting. If I were typing, it would be a whole lot more. Thirdly, I do this while walking or cycling or sometimes eating, sometimes in movies when I'm not too close to other people. It's amazing that voice is now, in 1996 or 97, I used Dragon's voice recognition for dictation. I gave up. I started using it with the Android phones. Voice recognition became much better with, I guess Siri is pretty good, but I don't use iPhones these days. With Android's Google voice recognition became good enough that I started using it, but I have to correct it occasionally. ChatGPT's Whisper, OpenAI's Whisper, which is part of the back end of ChatGPT, is excellent.**

So, voice as a medium of input, at the very least, I have completely switched over to. Voice as an output mechanism is also extremely useful, but we probably won't be exploring that much in this session. Actually, we could. Why have visual stories? We can publish audio stories. We can publish a podcast. I might show you how to do that.

So, it's saying there's a wine reviews data set and not much talk about... Okay, against startups, projects, maybe art, Chicago. Why does it even attempt? Why do you even think these are okay? Books, yes. Spotify, maybe. Okay, books. Fine. I'm going to go with books simply because off the list that it gave, this one matches my interest, but I'm going to ask it, "Go deeper into my interests."

On this list, only the Goodreads data set was really close to my interest. The rest, I wasn't particularly impressed by your choice of the rest. Some people say that talking down to an LLM helps. Some other people say praising an LLM helps. I've tested it, it doesn't seem to make much of a difference. It's smart enough to just do the job. Okay, but in the meantime, it's thinking about Tamil movies, Bollywood, which would make a good set for me. Calvin and Hobbes, good, but not a data set. Hmm.

Maybe, maybe Calvin and Hobbes. Okay, so I actually have a somewhat unique data set. I spent seven years typing out every single Calvin and Hobbes strip. I published it on my website, and then uComics came and did a DMCA takedown on that. So if I use that here, it's going to get published and blocked. But what if I didn't publish it? Okay, this screen recording will still be there.

But yeah, okay, Goodreads data set. We'll go with Goodreads. And where the thought process of this was thinking. But let's go with the Goodreads data set, and it's available on Kaggle. Good.

So we have accomplished step one for myself, which is getting a data set that I want to play with. You are welcome to play along with this data set, pick another data set now, pick another data set later, whatever. So I'll send you the prompt for this onto the WhatsApp group in case you want to use it. But basically...

Question: Which one? Which WhatsApp?
Answer: I've created a WhatsApp group as part of this. The first slide had the link. I'm hoping everybody is on the WhatsApp group. If you are not, can you tell us? Or if you press the QR code of the group, WhatsApp also there.

You would obviously want to modify the prompt to find your choice of data set. There are two advantages that I have here. One, I have enabled ChatGPT's memory feature. The way one does that is... but on the top right on ChatGPT, there are settings where one of them includes personalization. And in personalization, you can allow it to reference saved memories, reference chat history. The second feature in particular is what it would have used here. It would go through all of the chat conversations that I have had with it. I have had a bunch of Calvin and Hobbes chat conversations, Minecraft conversations, movie conversations. So it roughly knows what I'm interested in. You may not have that advantage. The second is the length and breadth of conversations that I've had. I've taken advice about every aspect of life. So it knows me very well, to the point where once I asked it, "Give me all my personality flaws," and it did an excellent job of it, and I'm working on them. Not I'm not.

Let's now take the Goodreads data set. The link for the Goodreads data set, I will put on the WhatsApp group. I know I promised that I might do this on email as well. Looks like WhatsApp is proving to be a little more convenient for me. Hopefully it won't be too inconvenient for you. So this data set requires us to download it from Kaggle. For those who may not be aware, Kaggle is a website that runs competitions. These competitions are where they publish data and ask people to do stuff with it. Somebody wins a prize, or sometimes it's just for the fun of it. You would need to create an account for it. It should be easy because you can just log in with your Google account or some such thing. And after you log in, from this page, you will be able to download the link from the top right. In this case, it's a 46 megabyte data set, borderline of uploadability. I don't... Let's see. We might want to upload it and figure out what to do with it. But it's downloading, and given that the network might not be too fast, it's going to take three minutes.

However, rather than waste three minutes, I'm going to pass the metadata, if I could, to the LLM. Remember I said, I don't want to do stuff. **I want the LLM to do as much stuff as possible. In this case, ideating on what to do with it is also the domain of ChatGPT.**

Quick catch-up for those who had joined in late. This is the link that has the slides. Please take a picture and go to the link. You will be able to follow through. I'll give you about 10 seconds. Does anyone need any more time to take a picture? Gone.

Next, we are working on this WhatsApp invite, which... the links for the WhatsApp invite are on slide two. And once you get to the this slide deck, you'll have all of that. We are going to be using these LLM tools. The list is on slide three. You can go through that. For now, we'll be using ChatGPT.

So, step one, I've found a data set. You're welcome to follow along with the same data set or pick another one and publish. The prompt is on the WhatsApp group. The next thing that we're going to explore is what kind of stories could it take... could it give to us. Now, rather than send the entire data set...

So, I didn't want to send the entire data set because it was large, and it might be slow. Large is probably not a problem. 46 MB it can handle, ChatGPT can handle it. But slow might be an issue. So let's do both. The answer with LLMs is I can create a thousand copies, or at least a dozen copies of LLMs on a dozen tabs. That's easy enough. So on one tab, I'm going to upload this data set. And... it has actually uploaded it reasonably quickly. I have to call the story. And on another tab, I'm going to talk to this. I have a feeling that my talking will be slower than its uploading, but let's take the column subscription and copy all of this. Copy the whole thing, paste it into this ChatGPT.

These are the columns in the Goodreads 100k data set. I would like to create interesting data stories out of it that can also be visualized nicely. Can you give me a dozen suggestions on, A, what stories to write, B, who they would be interesting to and why, and C, how I might go about doing this by having an LLM write code to do it? That's the prompt.

Now, you notice that I didn't bother formatting the output. I used to do all of that. I used to carefully structure things. But nowadays, I don't even bother. The only thing that I'm still a little obsessive-compulsive about is my instructions go on top, and the data goes at the bottom. This is 10% for the LLM's benefit, because OpenAI in their system card said, "In general, if you have to give instructions, prefer giving them at the top because we find that the models perform slightly better." And I optimize for things like these. But mostly for my benefit, because when I'm searching for this chat later on, I rarely scroll to the bottom to see what I was looking for. I go to the top. So this chat, I want to tell me what I'm looking for, and that it will do. Let's go ahead and I'm going to copy this and put it into the WhatsApp group, which... And in the meantime, this is also uploaded, the data set. So, instead of these are the columns, I'll say, "This is the Goodreads data set. So I'd like to do the same thing." And yeah, let it run.

Now, the first window created a bunch of story ideas. Big books, big love? What does it even say? Load pages and ratings, drop outliers...

Question: People are obsessed with long books.
Answer: Oh, okay.
Question: Longer page numbers. So is that actually true? Do the longer books score higher?
Answer: Yeah. Fine, yes. And what you pointed out is one of the major emerging drawbacks of LLMs. Bad enough that I stopped reading when I was watching Netflix. Now with LLMs, it's gone even worse. Here it is. And I was going to ask for exactly the same thing again. Attention spans will reduce. Is that a bad thing? Yes, it's probably a bad thing. Is that more efficient when we average it over all the things that we do, like washing machines, electricity, calculators, and so on? Probably yes. But this is not... Yeah, do longer books score higher? And now that I know what to look for... Okay. Paperbacks... Hardcovers are prestige, but paperbacks win hearts. You know, that's a good hypothesis to test. I wouldn't mind writing that story. But overall, I think I like this one more because I can imagine in my mind a nice scatter plot that takes 100,000 books in some shape or form, and size... yeah, length versus rating. Actually, scatter plots in general would be a good idea. I'll probably tell it to create all scatter plots and I'll publish those. So we might actually do that. And at this point...

Question: If you would have any impact on the response because the same question is exactly...
Answer: The question is, if many people in this room ask ChatGPT the same question at approximately the same time, does it have an impact? To my knowledge, no, for a couple of reasons. One, this information certainly can't quickly go back into the model. Models take months to train. And memory, even in the free version, is not shared across people. The data is going to be used to train, if you're using the free version, unless you opt out, and you can opt out. And if you're using the paid version, then unless you explicitly opt in, whatever, it does not use the data for training. I have opted in in some cases. But therefore, I suspect it may not see much of an impact. However, that is not... I can't be sure. Now, why is that? Because increasingly, things are getting connected. And if, for instance, there was, it does a search online, and that search uses Bing, and for whatever reason, a topic around Goodreads is trending on the internet, or we have for whatever reason have caused it to trend, it may pick up a signal from there. Unlikely in this setting, but this is a mechanism by which a group of people doing something can influence ChatGPT, which is through its search mechanism, not yet through the model training.

Question: Even the same prompt would lead to different answers for individuals in the room because of memory.
Answer: The comment was, even the first prompt that I gave would probably give different results because of memory. Yes. Even without memory, it probably would have given different results because LLMs are random-ish, meaning you ask the same question, you can get potentially different answers as well.
Question: Without memory, the prompt sent to the model would be the same. Maybe use prompt caching?
Answer: The comment was, since these models cache what we send, would that not give the same result every time? Even when the prompt... So, what is prompt caching? These models provide APIs, programmatic interfaces through which we can talk to them. In those interfaces, they have a feature called prompt caching, which I will explain in a few minutes. But the interface that I'm using, which is ChatGPT, does not enable prompt caching in an explicit or a documented way. So, maybe it's using it behind the scenes, maybe it's not using it behind the scenes.

Even if it used it, the way prompt caching works is different from how we understand normal caching. **Caching is where you give something a request, it stores the answer, and then sends it back.** Prompt caching is different. What it does is it takes all of that input and kind of **makes a tiny copy of the model at that point in the conversation**. It says, "This is where we left the conversation at." And then, when you put in the next request, it continues from that point without having to feed the entire conversation in, therefore **saving the company money, and they pass on that benefit to us**. Either all of that previous conversation is not charged for, or they charge it at a discounted rate. But in either case, even when the prompt is cached, it only affects the next request. And the only thing that it does is save us cost.

In this case, when different people are providing an input, there is no caching, because you're providing the input for the first time.

Okay, the other issue that I find when using LLMs is, it's given a bunch of ideas, and once I find one good idea, I stop. I say, "Yeah, I want that." And I've been training myself not to do that, including in this session, because **one of the things that LLMs are good at is solving the problem of brainstorming**. Create all the data stories.

The second thing that I should stop myself from is applying my brains. **The whole point is to leverage its brains and for me to do less work**. So, all of this review that I'm doing, saying, "Ah, I like this story. Oh, I'm a domain expert in this, I will create a scatter plot," I should stop doing this. Let it do its work. I'm not saying that this is necessarily a bad thing, but each of us have certain habits, certain practices. When working with LLMs, some of them work well, some of them don't work well. It is important to understand alternate styles of working and gain the experience to know when to switch over to that style, when to retain our style.

So, I am consciously going to not read the rest of the stories that have come up here.

Question: ...Thematic paper on cognitive decline... How do you connect with what you said just now? When do you think?

Answer: **The question was about the MIT paper on cognitive thinking which roughly said people find that when they use ChatGPT, they tend to, at the very least, remember less and are able to answer fewer questions about the topic than when ChatGPT is not used.** So, how do I think about it and apply it?

I absolutely think it's true. Earlier I used to do one thing over the course of three, four days. Now I do 10 things over the course of three, four hours. It is understandable that I would remember less. It is also understandable that I would probably make multiple mistakes in this process. What I'm trying to discover is—and I don't have enough good answers yet—is where do I need to do it myself? Some of the things that I'm learning are, **if I need to build on top of it in the long run, then I need to understand it. If I'm creating something that's throwaway, I don't need to.**

This session, to me, is likely a throwaway. We create something, we publish a story, I move on. If I'm creating a library that I'm going to be using in many projects, I go through **every single line, every character, like a hawk**, even if the LLM generates it. To the point where I say, "Hey, throw this entire thing away, don't even try to fix it." And even last week, I created a library; I had an LLM create three or four versions of a small library, did not like any of it, I wrote every single character in that library from scratch. Because I know I'm going to be using it a hundred times, and any mistake here amplifies. So that is one rule of thumb that I have discovered. **If mistakes or understanding or whatever can amplify a lot, I'm more in favor of review. Less for throwaway.** I'm sure there are a bunch of other rules which we will all discover as we go along.

This was one set of stories. Remember that we had uploaded the data. When we upload the data, it would have done some analysis. And this is the cool part. See, what it did was, it wrote code, importing a bunch of libraries, and then unzipped the huge zip file that I shared, got the CSV file out, looked at what that data contained, or the format, etc., and it says, "I don't need any external information. I provide all the information." It identified the columns. So **what I did by copy-paste, it did by itself.**

Therefore, I could have saved myself the trouble of copy-paste and just given it the data. The only reason I did not do this was because it would have taken time, which would distract from the workshop format. If you have time, which you would in which case you use it for other things, then you would not do what I did earlier. And it's come up with a bunch of stories. Now, I'm going to read it a bit, not to review, but to enjoy.

So, again, it's come up with something about length. Okay, it's come up with the same hypothesis. And fair enough, same input, same hypothesis is not surprising. Not guaranteed, but not surprising. Hardcover is the king. Okay, that was incidentally the second story there as well. Now I'm not sure how much of this was influenced by my previous chat because I was using memory there, and I turned memory on. Maybe it was influenced. And it doesn't matter. **At some point, it becomes an echo chamber, and at that point, I'll turn memory off.** Or I'll explicitly tell it, "Don't look at memory and come up with new diverse ideas."

Are there writers with 50 or fewer ratings, volatility, blah, blah, blah? "Page heavy genre outliers." Which genre tolerates large pages? Yeah, I like that. I like that story. But like I said, I'm just here to enjoy, not to review.

And it's given instructions on how to do this. I don't need instructions as long as it can figure out the instructions. Fine. Now, that completes, for me, my dataset, the second task, which is ideating what to do about it, keeping the principle in mind which is we will use LLMs for everything.

In case you followed along, you've gotten this or any other dataset, great. If not, we will anyway pause in a short while and you will be playing catch up with this. Third is analyzing the data. Life is simple here. Run all of these analyses.

Run all of these analyses. Do all that stuff you do with statistical hypothesis testing, that will be useful as well. And if something is not statistically significant, then flag those off, drop those analyses. Do whatever you think is the best thing to do. **I like charts. So give me charts, and give me pretty charts.** I use Matplotlib usually, which is fine, go ahead and use it, but the thing is, by default it looks ugly. So, **think about all the beautiful data visualizations that people create. Do something like that. I want to win an award with this visualization.**

Give it all the motivation that you can. Arguably the motivation is more for me than for the LLM. But it sometimes does help the LLM as well. Prompt it to analyze and visualize. You have to take these with a pinch of salt. The emotional prompting stuff is not necessarily useful. The key thing to take away from this is I said, "Do the analysis, create the visuals."

And **the thing to note in particular is that ChatGPT and many of these other models have a code interpreter.** What that code interpreter does is—and you saw a bit of it earlier—**it writes code, which all models do, and runs the code. It has a little machine inside.** And in that machine, it has the dataset that we uploaded. And it writes—it already has Python installed. It has some ability to install new modules, I think. But I say "I think" because I've seen it being restricted and blocked in a variety of ways.

**There are things that it cannot do, like access the internet.** I would have normally, and badly wanted to say, "Download this from Kaggle yourself." But it would have reported saying, "Sorry, I cannot access the internet." The subtext that it did not say is, "Because if I could do that, then what you would do is create a denial-of-service attack on a bunch of websites and take them down or hack through my machine." So if I happen to give you a free machine, then every Tom, Dick, and Harry will become a cyber hacker or whatever you want, or send a thousand emails from my machine.

So, all outbound access is restricted, but **they have some kind of a mechanism where they have an internal code execution machine, which is protected, safe, and all that, and an external one. We have access to the external one.** That internal one can access the internet and do stuff. I think they're using it only for installation of modules and a few other things that are limited.

Now, it's thought through, it's analyzed. I like going through the analysis because it tells me where I'm missing stuff, where it's doing something wrong. Like I mentioned in my talk yesterday, at one point, by reading through the analysis that it said, "**I'm not able to access this website. Therefore, rather than using the data that I'm supposed to use, I'll just make up my own data and run the analysis.**" If I hadn't spotted that, and I almost hadn't—if I had, I didn't spot it—and it gave some results. And I took it to Surya at Daksh, who's—this was high court data—and he's an expert. And their team looked at it and said, "Wait, hold on, this can't be right." I said, "Well, I'll show you how the analysis went," and opened the analysis and said, "This is the code," and just before that, I had spotted it saying, "I can't do this, so I'm going to make up the data." Sorry. And **that's where domain expertise helps.**

Let me put it another way. A certain amount of skepticism and informed skepticism helps. **If you look at the output and say, "this doesn't look right," then great.** Based on the birds of a feather discussion yesterday at Christ University, one of the things that I picked up was... that's the one of the best books to read right now would be _The Data Detective_. The _Data Detective_ by Tim Harford is basically about the practice of doing analysis on data. And I wrote down the 10 lessons that I learned from that, which I will go to in a bigger font.

And what I just mentioned was reflective of the first lesson. These are almost exactly the chapter titles, by the way. **Search your feelings. If you feel strongly, you're probably biased.** Which is why I said, "Oh, I like this scatter plot, let's do that," and so on. I am biased. There is merit in removing bias. There is merit in bias as well.

**Ponder your personal experience. Relate the numbers to something that you can feel.** So, in the High Court's case, Surya was looking at that data based on his personal experience and saying, "No, that doesn't look right." And this helps. So, **breadth of personal experience is something that makes a difference.** Because breadth of personal experience makes a difference, expanding that breadth is valuable, and I think in the LLM era, that will start becoming more powerful. And this was an illustration of that premise.

Let's go back to ChatGPT, and now, by now, I've given it enough time, and it's created a bunch of plots, which are interesting. Let's go to the top and see if it's summarized anything for me. It's written the code. All of this will be in the analysis. I'm going to compress that. Okay, "Pages versus Ratings." This... there's probably some outlier here. Most of the data is on the bottom left. Now, it may have understood and caught that, and if I asked it to fix and improve this, it would probably do it. But even otherwise, I can just prompt it to say for the "Rating versus Pages," drop the outliers. And maybe it will do that for the others as well. "Who are the top sleeper authors?" Okay, a bunch of authors who published very few books, all of which have gotten a 5 rating. Not so interesting a story.

Now, here I feel comfortable applying my bias because **at this point, we have reached the end. I am the consumer. This is literally meant to entertain me.** So, not an interesting story. Asimov is not an Oscar winner. Drop it. "Blurb length versus rating." Hmm, okay, there seems to be a positive correlation here. Again, scatter plots are interesting. So now I can say, "Ha, see, I knew I was an expert. I knew scatter plots would be interesting." And this reinforces it. Dangerous or useful? Both.

"Review versus Rating." It turns out that longer—no, more reviews tend to be marginally associated with higher ratings. But whether this is statistically significant or not, we may read further down. So, so far we've gotten three stories. **Longer books definitely get better ratings.** Longer description of books tend to get higher ratings, probably significant to be understood. More ratings are given to—or more reviews tend to be for higher rated books, but I'm not really sure about that. Single versus multi-genre, I can't tell the difference. Let's look at the statistical tests.

It gives P-values, which I can read, but I'm not going to bother reading. I'm going to check if... okay. What the English version of the effect is. "Pages improve ratings." It's a miniscule but real trend. Saying that if you add 100 pages, your rating goes up by 0.006. Okay, fine. And that's on a 10-point scale, so it's not small, but it is a real trend. And this is where we can feel the difference between statistically versus numerically significant. That is, statistically meaning it is correct. Numerically meaning, makes a big difference in the real world or not.

Then there was the "Blurb Length Bias." Here, it's saying about 0.01 rating per extra 250 characters. That's reasonable. And then there was "Review Elasticity." Doubling the number of reviews adds a 0.01... Doubling the number of reviews is hard. But if you somehow manage to... it's hard to argue if this is cause or effect. So maybe higher rating, better book tends to get more ratings. So what this is saying is that even a slightly higher rated book tends to get far more reviews than a slightly lower rated book, which maybe is understandable.

So, now what I'm going to do is publish. Even though I said, "Give me beautiful charts and all that," I've tried this a number of times. There seems to be only so far that we can either push ChatGPT or Matplotlib. This is as far as it gets. I like interactive plots, and therefore want to create an interactive plot of the same thing. How might we do that? Now, I have a whole series of options based on the tools that I mentioned.

Option one, I could ask ChatGPT to write the HTML, JavaScript, whatever, and create it. No harm in that, but I don't like copy-pasting code. **That's work that I have to do.** If there was something that would automatically handle code, I'd prefer that. And therefore, I go for one of the coding agents, either the remote ones which I can do on a walk, or the local ones which can access my hard disk. 46 MB, last time, it was able to upload and do. So we could go for one of these two. I'm catching myself again. **Why one of them? Both.**

So, I'm going to put it on Jules. My experience on Jules has been mixed. It's free, it's great, so I can do a lot of stuff. But I don't like the style of code that it writes. I like less code, more output. Codex matches that. It writes very little code. And then I tell it, "Keep it even more compact," it writes it even more compact. I'm not going to go for these because fewer of you will be able to follow along. It will require an install on your local machine.

So, let me start with Jules. Now, I have not tried uploading a 46 MB dataset to Jules. I'm going to try that out. And to Codex, and give it both the same set of instructions, which will roughly be copy-pasted from what we saw on ChatGPT, telling it to create a story. Now, normally I would, at this stage, think about how I want the story to be presented and give it a detailed set of instructions. That is not a bad thing to do other than the fact that it takes my time and effort. So I'm going to have it think about what is the best way to present the story by meta-prompting. In other words, **I'm going to ask it to give me a prompt that I can pass to these agents.** Let's do that.

I liked all of these stories that had scatter plots in them. The only issue is that there were outliers, so I would like those removed. Now, what I want you to do is give me a prompt that I will pass to an LLM along with the dataset, and it should write the code as HTML, JavaScript, in a way that I can publish onto GitHub Pages. Remember that this means that we cannot have such a large dataset being uploaded onto GitHub Pages and users viewing it. You need to figure out some way in the prompt of telling the LLM to create these using HTML, CSS, and JavaScript in a way that conveys a scatter plot but without having to load the entire dataset. I am okay with the page loading data of probably up to one or two megabytes. Keep that in mind, do analysis if you need to, and write the prompt.

What I did here was kept the end-user perspective in mind that we want the page load to be reasonably small. Now, the reason this... should I have done that or not? I don't know. On the one hand, you could say a few weeks, months down the line, these are things that the LLM would have figured out. But A, I have domain experience and know that this is a problem. I also know LLMs well enough to know that they will not catch it. So this is one of those cases where I'm applying the combination of my domain knowledge of coding and of LLMs and giving it a prompt to save time. If not, then we would have done another iteration and not saved that time.

I'm going to run this prompt. And I'm also going to put this prompt on WhatsApp. Let it create this prompt. To set up both Jules as well as... We may not be able to complete... So I was hoping that when we take the break in 10 minutes, I would have got the story published. I may not be able to do it if there are any issues.

Now, therefore, now I'm going to do a series of things that will go a little fast for someone who may not be familiar with the technology. Step one, I'm going to go to github.com and create a new project. Now, this project will be a new repository. And I'm going to call this "Goodreads," shorter, simpler. And I'll just have it add a dummy... No. And then I'm going to create this. Why do we need to create a repository? Because those online coding agents will take stuff from a repository, use it, and run it. Into a repository, I can upload files. What I'm going to do is upload this zip file.

There are a few areas where I'm obsessive and have not yet managed to overcome that obsession. One of them is keeping stuff small. Zip is not the best compression format. So I'm going to compress it further using code on my terminal, which you absolutely don't need to worry about because this is simply me being obsessive-compulsive. The highest compression format that I know which is slow is `xz -9`. So this is going to create a much more compact version of the file. My guess is that it will be half the size or less of the original zip file. So if that was 40-odd MB, this will be 20-odd MB. Let us find out. But this is me. And it can take some time. What I'm going to do is once it gets compressed, I will click on "Create a new file or upload an existing file," click on "Upload an existing file," drag and drop this file, and put it into the repository.

Question: We can start building this in the meantime.

Answer: Yeah, yeah, feel free.

The second thing that I have done is gone to jules.google.com, and I had logged in earlier. It put me on a waitlist, and within a few hours, it approved me. I'm not sure if the waitlist exists anymore, and you may, in case there is a waitlist, not be able to try this out immediately. In which case, please use the ChatGPT $20 version, which I will do side by side by going to chat.openai.com. And on the left side, there is Codex, which you can also go to by clicking on... going to chat.openai.com/codex. And there, you can start performing the same task. So I'm going to run Jules and Codex in parallel. But before this, I need to have uploaded the dataset, which is still compressing like I said. This is...

Question: Both these things work similarly?

Answer: Yes, they work very similarly. For both Jules and for Codex, you give it a GitHub repository, tell it what code changes you want it to make, it will make them.

Question: Jules actually doesn't seem to have any waitlist. Wonderful. Everyone who tried Jules was able to connect to GitHub. Is it asking to give access? Anand, is it okay if it has access to all repositories or...

Answer: I give it to all, but if you have any private repositories that you don't want Google to access, like your office's code, then leave it as "specific repositories."

Question: It's not letting you upload because the file is larger than 25 MB.

Answer: Okay, so the output came to 29 MB, which is too large. So we will have to figure out alternate mechanisms. I refuse to give up, and I will follow my advice, which is ask ChatGPT. "I have a 29MB .xz and a 46 MB .zip file that I want to upload into GitHub via the UI. I am unable to. Any options? I definitely want to commit." And ask. My guess is it will say, "You can just upload it from the command line." Which means that those of you who are not familiar with the command line will not be able to directly do this. But once I've created it, you will be able to fork my repository and use that as your repository instead, which is almost as good. Let's see. Now it's saying, "Blah, blah, blah, asking me to pick between two options. I have other things to do, but I prefer shorter." So it's saying use the command line. I use the command line, create a repository, send the link, you copy it.

The repository is out here. And, okay, it's saying... okay, let me just create it... So, now what you can then do is go to the chat window so that you can fork it, not immediately, but in a... once it's pushed. Yeah, it's pushed. Right, so I have this. Now, how do you copy this repository? The link that was on WhatsApp, you can click on that and open it. In your screen, you will see a fork somewhere. You should see a fork button on the top right. When you click on that, it will say, "Create a new fork." If you have multiple GitHub repositories, choose where you want to create it, but usually, you just need to click on the fork and create fork, that's all. Then you will have a copy of that repository. It will also be called "goodreads" by default, and you're set.

So, I have completed step one of the sequence, where I now have the repository. I'm going to go to Jules and have it pick from my list of repositories. Because I opened Jules earlier, it may not have recognized or downloaded this repo, so I'm going to reload the page. I'm going to reload it on Codex as well.

The process is slightly different for Codex, in that you have to create something called an environment, and I'll show you. So here, I'm going to create this entire `books.csv` click. And then I can type the prompt. Let's go to the prompt that this created. You are an elite full-stack dev. Okay. Output a `preprocess.py` that's going to do a bunch of computations, and an `index.html`, blah, blah, blah.

Okay. I am very curious how it managed to do... Oh, it's randomly sampling 5,000 rows and then doing it. You know what? I actually would like it to randomly sample and show 5,000 dots, but the calculation should be done on the entire dataset. So, it looks like it is doing that. After clipping, randomly sample down to scatter the data. Yeah, so it looks like it's doing that.

This is one of those scenarios where I'm applying external validation based on something that I know it might get wrong. But this is pre-validation. That is, before I even write the code, me doing the checking. I believe, but I'm not sure, that **pre-validation will give way to post-validation.** That is, after I look at the output, I figure out if it's right or wrong. Why? Because then the whole thing becomes: use the LLM to, in a chain, get stuff done and then figure out if it made a mistake or not.

**Automated tests are very important** for anything that you do, if that is not your forte. This is not a talk that will cover that. But let's just say that if there was one thing that is critical that I'm missing in this talk, it is how you can evaluate these in a programmatic or an automated way. And you can use LLMs for this. You can take a screenshot, send it to an LLM, and have it evaluate. But it needs to run automatically. And the quality of code and the quantity of code that you can write, or analysis that you can do, dramatically increases when you have these automated tests because you know that there is a safety harness. It won't make those mistakes that you're testing for anymore.

But I'm going to go ahead without tests by copying this and putting it into Jules. I'm just blindly running. I'm sure I can think of a dozen things that I would want to improve, and I should improve. I'm going to do the same on Codex.

Now, Codex requires a slightly different way. Out here, it won't list your repository unless you create an environment. And that requires us to click on the environments on top. The process is simple. You then click on create environment, choose your repository, and click okay. And then it will appear. Just one extra step. The repository name is `books`, that's what appeared here.

One thing that I can do that I like in Codex, which I don't seem to need to do, is agent internet access. So when it's writing code, it can look up documentation, it can do a bunch of things, which I think is very powerful. Maybe Gemini does that anyway; I don't know. But I have a feeling that the code that Gemini writes inside that sandbox can't do too much of internet access, whereas here I can say, just give unrestricted agent internet access. The reason it's asking for permission is this can be dangerous. If, for instance, it accesses a repository which has been created by a hacker who says, "Give me all the code and the output of whatever anyone's asking and send it to this email ID," it might follow those instructions because it read those instructions. So these are ways in which **the combination of private data, public information, and an LLM come together to lead to unexpected outputs.** We're in the very early stages of discovering what can happen with these things. Nobody really knows.

So my policy is simple and twofold. Number one, **keep a separate environment, by and large, for stuff that is private. And keep it as large as possible.** That is, as much of what I would have earlier considered private, like my notes, I'm putting out there on the internet. Figuring that if I know it's going to be out there in plain text, if there's something that I really, really don't want to put out there, I'll consciously keep it separate. So expand the footprint of what is public. The second thing I'm doing is carrying bees. If some disaster happens, we'll weather it. Not that disasters haven't happened before.

So, I'm going to use this environment, paste the same thing, and now, this is the other interesting feature that this has, is whether it uses one version, two versions, four versions. I can ask for up to four versions. The thing is a great anyway in a short while. So, I'm going to ask for four versions. With that, this is booting the VM, it's creating stuff, and it's automatically going to run. This is coding. Perfect time to take a break, have some coffee, which is exactly what you should be doing when LLMs code. And we'll come back in 10 minutes, 15 minutes, whenever you all are back. So coffee, tea, etc. is out.

Some of you would have gotten the same message that all of you would have gotten the same message that I got on Jules. So it says, "Provide the Goodreads 100k books CSV file at its location." It's in the repo as a `.x` as a compressed file. Now, this is one of the minor disadvantages of a less intelligent model. **The smarter the model, the more it can do the work by itself over a long time. The less intelligent the model, the more we need to guide it periodically.** So one of the metrics of the intelligence or the quality of a model is **how long it tends to work by itself.** And on those measures, the... and it's not just the model, it's probably the entire application. On those measures, Claude Code probably is furthest ahead. At least for code, it tends to work for sometimes half an hour without any interruptions. Codex goes pretty far. Gemini, or Jules in particular, goes less so. These things change very often.

The cost, of course, also varies. So Claude Code is quite expensive. But what I mean by quite expensive is doing something like this might cost half a dollar. And if you do it three, four times a day, that's $2 a day. If you do it for 20 times, then that adds up in a month, that adds up to a good $40 a month, which is still not much. But you have a cheaper option by going in for one of the pro subscriptions, in which case you get it for just $20 for the entire month. That's all. So what I mean by expensive is not that in absolute terms it is expensive. It is a no-brainer. It is just that you have cheaper options than doing it that way.

And tools like Cursor, Wasp, etc., they are roughly like Jules or Codex, except that they run on your system. And as part of the IDE, which means that if you are coding, then you can interact with them very effectively. I'm consciously not showing them because this is not a coding session. It's more how can you get stuff done without needing to code. That's why I'm using these.

Now, while this runs, Codex should have at least completed one version, and it's saying that it's completed one out of two versions. So remember I said that these things take their own sweet time. So, okay, let me see what I'm asking for. And okay, it seems to have created both versions. So let's take a closer look at what it's created.

Next comes the question of how do we take a look at this. Now, one possibility is we get the code locally and run it. I'm not too much in favor of that. I want to try something that I haven't tried, which is I'll publish this branch. Now what do I mean by publish this branch? What it lets me do is click on a "Create PR" button on the top right. PR stands for Pull Request. That's effectively a request from Codex saying, "I want you to pull my code into your repository and use it."

GitHub has another feature which is called GitHub Pages, which lets you publish HTML files or any kind of files, and you can just view them in a browser. HTML, CSS, JavaScript does not require any additional server-side processing. And I think we can pick the branch that we want to publish. So, let me first take the first version, click Create PR. Now, how do I know this is the first version? Because out here on the left side it says version one. There's another version two which created some other kind of code. Now, the first version, it took 3 minutes and 19 seconds of thinking. The second version took 3 minutes, 43 seconds of thinking. Maybe they're similar, maybe they're different, who knows? We'll find out. I'll click on Create PR for this as well. We'll try and do stuff in parallel as much as possible, while Jules is still running. Okay, and Jules is also done. So I will click Publish Branch. So now I have three branches that I want to test.

Let's go to `books`. And the way to test first, let's check if there are branches. It says three branches, which means four, right? I have a main branch. Okay, there are three other branches other than the main one. Fine. So these are the three branches. How do I publish these? Now, in settings, there is a Pages option. In this, I can choose Deploy from a branch, which is already selected. Yeah. And I can choose the branch to deploy. So let's start with, we'll go in literally this order. The first one says `codex/something`. Now what I don't know is whether this was version one or version two. We'll figure it out. So I click this one and click on Save. Now at this point, it will, in a minute or so, publish this on GitHub Pages. How do I know where it publishes?

The way to track the progress and figure out where it's published is by clicking on Actions. Now it's saying I've built pages, or I'm building a page. And if you click on this further, it will eventually tell you once it's published, where it's published. Let me repeat the process. We go to Settings, Pages, and choose the branch we want to publish. At any point, you can publish only one branch. Then, you go to Actions and monitor the progress, and it seems to have published it. Let's look at where it's published. The link will appear out here. So let's click on that. And it says "Error loading data." Clearly, a problem. What I could do is just pass it that information, or, being a developer, I can pass it a little more information. Let's go to the console and at least pass it the error message. It's saying this particular file was not found. So I will copy the link address and give one of those that error message and tell it to fix it. And it will hopefully. But which one? Was it version one or version two? I don't know.

Actually, I don't know how to find out. Let's see. Okay. What I can do is click on this View PR. Currently, it's version two. If I click on this View PR button, it will take me to the request that it's made. And it's saying version two is this `lvj` Codex thing. So presumably version one is the one that had the mistake. I'm going to say, "I got an error. It couldn't load `scatter_data.gz`." Fix it.

And while it's fixing this version, remember that these things can take a minute or two, I'll go and view the second version, which I will do by going to settings and changing the branch that it deploys. And then changing the branch to the other Codex one, version two, save it, and now go to Actions to see if it's actually getting deployed. Once it gets deployed, it will be at the same link, so I can reload the page.

Now, this is a problem that you will frequently encounter, which is even though you have given an LLM multiple tasks, you don't have enough to do. **When that happens, you need to discover a hobby.** This is important for survival in the AI age. Mine is a bubble shooter game. I'm sure there are better hobbies, and I will take your input on what we can do. But this is a completely addictive game. So I will play this for some time. Let me just quickly check if it's done already. Great. And this one... Okay. So, without doing anything, I have at least published something as a page.

Now, on this, I want it to write a story. So, for version two, I know the next prompt that I'm going to give. Which... but it's fixing... while it's fixing version one, can I prompt version two? Okay, apparently not. So this is one of those small problems in the Codex interface. While it's running one conversation, I'm not able to, in another version, run a conversation. I can start Codex again and start a separate task and run it in parallel. But until then, what I will do is record what I want to share.

And now I'm going to use ChatGPT purely as a recording tool, not to actually send a prompt. What I want to tell it is, this data visualization was nice, but not beautiful and not a story. So let's do two things. Imagine you're writing a New York Times data visualization. How would you go about it? What fonts would you use? What color schemes would you use? What design choices would you make? Create a full-fledged data story. Take a look at the results that have come out and write it in a way that will get me an award. I have no idea if telling it to give me an award will work or not, but I have one prompt. This now, I am conscious that I may not have sent the earlier prompt on WhatsApp. So I will send this trivial... actually no, I'm not going to send this prompt. This prompt is too trivial and may not happen for you. But I will send this second prompt on WhatsApp. From to improve the viz.

Now, third to test is the Jules one. Go to Settings, change the Pages default branch to the... the third one, which is `feat/dp-scatter-plot`. Save it and watch the progress as it deploys. Deployment takes reliably about 15 to 30 seconds. So I don't mind watching it while it works. All right. In the meantime, version one where we got the error, that seems to have updated and said it's ready. It's probably saying it's fixed the error. So let's update the branch. Every time you give it a new new instruction, it creates a new commit and clicking on update the branch will update the pull request. So I can go back to that branch and redeploy it and it should work. Now this time when I clicked on update branch, it said it could not. So let's try it again. It said failed to update PR. I don't know why it failed to update the PR. This error has happened to me a few times in the past. There are two ways around it. Three ways. One, repeatedly try and maybe it works some of the time. Second, start afresh and create a new one. Or third, drop the tool. We anyway have other alternatives. And there's no reason why we need to struggle with this. So version one, I'm just going to ignore it. Version two was working fine. Jules is still in progress. So let's just stick with those two. And this is why having multiple starting points helps. A, you don't get bored. B, you have more to pick from and don't necessarily need to improve or fix something.

So now let's reload. Okay. So it's saying visualized by Jules, but it did not show any output. So we have to check what the error is. And it's saying this is the error. Like before, I will pass this input to Jules and say, "I got this error. Fix it." Exactly the same prompt as what we had checked earlier. But by now, you would have gotten a feel that I'll probably drop this either. It didn't work. Don't even bother.

Now, let's go back to version one and the prompt that we had been using to create the data story. This prompt, we will pass to version one, which was working fine. I am on version one, and I will tell it to improve this version. Now, hopefully, this one will not have the problem of when I click on update PR, it'll say, "No, no, no, I can't update the PR," something failed. If it does, then we've gotten midway, and I don't want to lose that. But it's not a big deal because then what I can do is accept that pull request and improve from that point onwards. In short, the work won't be wasted. It's just a question of how much manual effort you are expending.

Over the last 10 minutes, a lot of what I've been doing is just engineering. **The biggest takeaway from this really ought to be, if it doesn't work, maybe try fixing it, but definitely just start again.** It is as inexpensive. A big part of your effort needs to be, **how can you streamline the workflow so that you have to do so little that doing a hundred of these is negligible effort?** Keep that in mind.

Now at this point, we have two tasks running. One, Jules, second, Codex version one. Both of these will take a couple of minutes. Any questions or comments?

Question: ...regarding the console part, why do we need to copy that to explain, and how does it work?

Answer: The question is, the error message that it prints is useful information for the LLM or a human to debug. We're telling it to debug. If I give it more information, it helps. This has a term starting day before yesterday or yesterday, called **context engineering.** Which I forget who coined, but I read it in Simon Willison's blog. Context engineering is the science and art of passing the required, and the right required information to an LLM for it to do its job. Arguably, this is what prompt engineering was coined for earlier, but prompt engineering has taken a different direction of its own in terms of how it needs. So context engineering might become the new popular, more relevant term. In short, we want to make sure that the LLM has the right context to answer it. Error messages are definitely part of the right context to answer this question. If it were a Python program, the Python program would print an error in the console. If it's a browser program, it prints an error in the console. But to keep life simple for the majority of non-developers, browsers don't show the console by default, so you have to hunt for it. And how do you hunt for it? The settings thing, which in Edge is three dots, in Chrome it will be something similar. You can go to the... when you click on that triple dot dropdown thing, somewhere there will be a developer tools. Oh, yeah, out here, it says More tools, and then there is something called Developer tools. It also shows a shortcut. Some of you closer to the screen might be able to read, Ctrl+Shift+I. So if you press Ctrl+Shift+I, you'll be able to get there. Another shortcut which is not displayed is F12, which is what I use by default, and I use it so often that I only ever use F12. Now, what happens when you do that is it opens up something, in my case on the right side, in your case it'll probably be at the bottom, which has a bunch of tabs out here. One of those tabs is Console. There are several tabs, you'll have to go through them one by one. The Elements tab shows what are all the boxes in the page. The Console tab shows all the logs and error messages and so on. Anything in red in the Console tab is an error. And if you are not sure what to pass, just select it all. Just select it all somehow and pass it as context. If you are able to or know what is the right context, just pick that red one or anything else and send it to the LLM to fix it.

Question: ...when pasting error messages, is wrapping them around backticks not required here?

Answer: The question is, when we paste error messages, shouldn't we wrap them around with backticks? Backticks are the symbol on the top left of the keyboard. They kind of look like this. This is the regular quote, the backtick is pointed towards the left. And the reason this is important is usually when people share code, sometimes text that needs to be set aside, like error messages, they wrap them in either a single backtick or triple backticks like this. This helps the LLM understand that this portion is code, and you could even say this is code in Python, and it will format it because this is a convention. If we provide this information, two things happen. One, it kind of treats you like a smarter person. Second, it spends less mental energy to understand it. The abilities of models have been increasing so much that the percentage of mental energy that it needs to figure it out if you haven't put it in is very small. And a negligible fraction these days compared to what it used to be earlier. Second, do you really want it to treat you as a smarter person? If you are, then yes, sure, go ahead, put in the backticks. It might make a small difference in the short run. In the long run, which I'm trying to train myself for, that is additional manual work. **I don't want to train myself to do manual work, especially in the service of an AI. It should work for me, not me for it.** And if I can get away with it, I will. Over time, I'll be able to get away with more and more. So I'm practicing not trying to practice not putting in the backticks, and it seems to be working.

Jules, which is the `dp-feat-scatter-plots`, that seems to have worked and completed. So Jules, without seeing that prompt, updating the branch. Codex seems to have a problem. But in any case, let us now look at our output. Has it fixed the error? No, Jules still hasn't fixed the error. So I'm going to drop that direction.

When it comes to the other pull request, which is... there is only one commit, and this has not been able to update the branch for version one. Normally, at this stage, I should ask an LLM what to do.

And here is where even asking an LLM is proving hard because of old habits. It's just one tab away, right? All I have to do is switch over to ChatGPT, post the problem, and yet there is this 30 years of coding experience in me saying, "No, I'm going to solve it. This is the kind of problem that you like. This is what you look for. You feel like, this is what you look for, you have to solve it for yourself. And giving up that temptation to effectively give up some of our identities may be the biggest transition that we need to make.

I am using Codex, and our Codex has been trying to update my branch, but has failed. Here I'm talking about ChatGPT's Codex, for your context. It created a pull request, but when I made a change by asking or providing it new instructions and then it created new code, the "Update branch" button, when I click on it, says "failed to update that branch." I just want that branch updated. I'm not fussed about how to do it, but I want to do this with the least possible effort, like hitting a button or minimally copy-pasting, etc. Yes, I know I can copy the whole thing locally, copy-paste the stuff, push again locally, but where's the fun in that?

**Give me options that are lightweight, easy, quick, and might teach me something new.** While we're at it, we may as well learn.

So, here are a few other things that you may want to explore. There is a "Tools" button that lets you pick from a variety of options. You can create an image; this you should absolutely explore. ChatGPT's images are, for many use cases, excellent, but in particular, they follow your instructions. Searching on the web is what I'm going to enable right now because I want Codex, which is new. There's no way the model will know about it. And even if I didn't specify it, if it thought it necessary, it would search. My theory, which is untested, is if I put this option, it will search a little more, which I kind of wanted it to do.

It can also write or run code. "Write R code" is useful when you want a separate box where it puts the code, and then you interact with it, or a separate box where it puts the document, and you interact with it there. Or "run deep research." I have 23 deep researches left until next month. This will search very extensively, give very comprehensive answers, very long answers. It used to be fun, but now it's too much reading. So, let it solve the problem.

In the meantime, my slow brain thinks that perhaps the easiest is to copy from there and paste it onto the right branch, because GitHub lets you paste stuff directly. But let's see what it says, because this is... **Asking for options beyond what you know or what you think you know is one of the most powerful uses of LLMs.** If nothing else, it will show you a different path, and sometimes it is more effective.

So it's saying because the PR's head can't be fast-forwarded because of some conflict, branch needs... So it says, "I have 50 low-effort ways of fixing it." Yeah, I'd like to learn this. Why the button has failed? Because there are... fine, some... The main branch has... Okay, what is this? If the main branch has new commits... too much to read. Let's go to the next. Resolve conflicts in the PR UI. So... this looks longish. Yeah, I will look at this. This... all of this looks like it will... Okay, wait. `git rebase`. This is interesting. But no, I don't think any of these will solve our problem.

So what I'm going to do is what I thought of. Not because it is a better option, but because sitting here and reading is not entertaining. Let us not keep you bored. What I'm going to do is copy this code, and there's a nice copy button. And I'm going to go back to the code here, go to the branch that I want to modify, which in this case is the weirdly named one. And it has a bunch of files. I'm going to click on `index.html`, which I want to edit. So I will click on "Edit this file" and paste the new code. I copied the file name. Copy patch. Or maybe just select the whole thing. The contents of the file in an easy way. Strange that that was not obvious. Paste this file. And while we're at it, modify the... Okay. How can I edit multiple files at one shot? Okay, commit the changes for `index.html`. And then take the second file. Copy patch will work. No, I don't think copy patch will work. Let's see. Go to `process.py`, edit that, paste it. Yeah, that copy patch is a bit useless. Take this whole thing.

Question: The copy icon is next to the file name.

Answer: That copies the file name. Yeah, I tried that. And therefore, let's commit these changes as well. And thirdly, `static/data.json`, which is the new file that I created. Whatever it is, copy that, `static/data.json`. It probably hasn't changed at all, but commit the changes. And finally... go all the way to the end. Okay, there were only three files that it changed. Let's confirm. It changed `index.html`, `process.py`... fine. So now that I have all three files updated, I will change the settings to go back to Pages, and deploy the weirdly named branch. And save it. And look at the Actions to see when it gets deployed.

And what we'll be doing at this point is looking at the story and saying, "Okay, fine. Task accomplished." And I'll help out with your stories. So the first arc of this workshop will be done at this point. Not because the story is complete, but because we need to spend enough time making sure that you are able to create a story, which is what we want to achieve by 1:15 PM. Still deploying. Okay, deployed. Let's scroll down a bit and go to the page. Reload, hard reload. Wait, this is the same thing. Did I copy the... Let's check. The nice... blah blah blah... for and all of that. "Crafting the page." Oh! Okay. I think I did not tell it to... Okay, I may have clicked on "Ask" instead of "Code." There are two buttons here. "Ask" will tell me what to do and how to do it. And it... "Crafting..." Oh, look at this. It says "Crafting the page." So, I can check if it actually... if I copied the right thing by checking the font. Segoe UI. Okay, that is definitely not one of the fonts that it said it would choose.

Let's try this again. Is there a way I can edit this? No, there isn't. Last attempt. And I'm saying I started with this, but it did not actually change. And this time I'm going to watch. The reason is, I don't trust it. If after a few attempts, you have no other options and you don't want to restart, then you start applying your intelligence.

**This is the point at which expertise helps and enables or not.** So if you find that you're stuck and don't have options to go further, call in an expert at that point, or move on to a different problem. **Giving up stuff is going to be a critical skill to learn as well.** You try a project, it doesn't work beyond a point, you drop it, move on to the next. Part of the art of discovery here is knowing what, given the current jagged edge of AI, is easy and hard. And if you spend time trying to get some stuff forward, it will not help.

Let me give you an example. In January, one of our clients... we were working on a project for a publisher. We wanted to convert XML from one format to XML of a different format. This is so that the book that is published in one format can get published in another format. The context window, that is, how much stuff you can push into an LLM, was small. So we couldn't send the entire input. We spent three months working very hard to find clever ways of breaking that XML in a way that the LLM would be able to generate the output at a higher quality. At the end of it, the quality was so-so, but workable, and we felt very proud. And the next day, Gemini released a model that had a larger... OpenAI released a model which had high accuracy and a 1 million context length, which meant that we could send the entire book, paper, actually, and get the output.

So, three months of effort wasted to get to a point which was anyway better than what we would have done. **What we should have done in retrospect is just waited for three months.** **Knowing when to wait is an important skill.** If you don't and can't decide, the answer is probably wait anyway. Models will just keep improving. Which means you need a large queue of problems. Knowing what to do or what to tell an LLM to do is an important skill. My queue is drying up so rapidly that I'm reaching out to people saying, "Do you have any work? Do you have any work? I need to learn. I'm out of ideas." Other people have ideas, let's use them. Which is also what I'm hoping to do in the second half. Please give me ideas. I definitely want to learn. But this might work.

Any questions? Okay, has this worked for anyone? Have you been able to publish something?

_An audience member shares their screen, showing an animated chart._

Well, that's Rasagna's output.

Question: The first one that I made was one which is what I made from the browser. It was a chart of genre by... It was a very long chart. So this is an animation that just started to work and stop. This data removal took like five to ten minutes because beyond that, I think it was... but it is still... it is not actually doing a true analysis, right? So can you talk a bit about that analysis part? What could we... like is there a step that you skipped just for this workshop that we should try and do, or do you think analysis is not very well done here? We have to use a Python or some other program for this.

Answer: The question is, this isn't true analysis. This is really just exploratory data analysis.

Question: There's that one step where it says that I will pick randomly 5,000 to sample.

Answer: Correct. The comment was, it is also dummy in the sense that it randomly samples 5,000. Not quite. And that's one of the things that I checked in that prompt. What it does is the analysis is on the entire dataset. The visualization is on a sample. And that is alright because we want a scatter plot that kind of shows representatively where things are. But the regression line and the statistical significance calculation must be on the full dataset. So that part is okay. But you are right in saying that this is, at the very least, not deep enough analysis. It certainly counts as exploratory data analysis, maybe a bit beyond that, but it's not particularly deep. And that is true.

At this point, there are many directions one can go in. Two directions that I go is, one, since I know the kinds of patterns of analysis that work well, I prompt it in those directions. A couple of patterns that consistently work are: one, scatter plot matrices. That is, take every pair of numbers, create new numbers as required—and this is important, deriving variables with this is very important—creating new numbers as required, and then finding out if x increases, does y increase. There are two invariably good stories there. There are two kinds of good stories there. One, correlations: x improves y. Second, outliers: but this does not apply for... Here are a bunch of areas where it's weird. **Outliers are the most consistent, interesting story generators.** This is pattern number one, which I broadly am putting under the visualization bucket of scatter plot matrices.

The second is causal analysis. What causes x to improve or worsen? Now here, it helps to have some idea of what we care about so that we can ask it to find out what improves x. For books, it might be ratings. It's unlikely that we care about improving the length of a book. So you might care about improving the number of ratings. Maybe we care about improving the length of the ratings. We probably would care about improving the sentiment of the ratings, or the sentiment of the reviews. Stuff like that. So if you know what you are worried about or what you want to improve, then another question to ask is, exhaustively analyze and tell me how we can improve this. I say "exhaustively analyze" in a broad way because that's probably how we would prompt it in the future. That is, do some analysis and tell me how to improve the thing that I want to improve.

But today, I would prompt it in a more specific way, saying, run machine learning or write the code to generate machine learning models that will figure out how to improve x. Make sure they are explainable so that I will get actionable recommendations. Why are both of these important? If I didn't tell it to run machine learning models, it may run less smart techniques, or it tends to this month. If I did not say make it explainable, it would invariably go for a complex model like random forest, which is something nobody can explain because it's weird stuff. And therefore, when I say explainable, it will probably dumb it down to a decision tree. And decision trees are good models in that they can explain a lot of stuff and can be... they are explainable to humans as well. Then I would ask it to plot a decision tree. At this point, we are getting into the realm of true insight, true analytics, given insight that we can pass to a client. By default, the models don't get there yet.

Now let's try my luck. This created something. `index.html`. Okay, at least now the font changed to Georgia and whatever. So I'm going to copy this. Does it have a good description? Story and all of that? No. "The Enigmatic Power of How It Gets..." Okay, fine, it's something. This is... this is the kind of place where I would... You know what I'm going to do? I don't like this. So now I'm going to do something that only the people who can write on the command line would be able to do. I'm going to invoke a mini superpower, which is Claude CLI. I do not expect you to follow along on this, or even be able to introduce this. Let's go to the repo. Where did I put the repo? Uh, I'm going to pull all of the branches, which should be tiny. And then, uh, go... which was the branch? The good one? Yeah. `codex-slash...` Let me just check out the `codex-slash-whatever` branch. And now I'm going to run Claude. How does that work? Ignore a few commands here that I'm running.

This command runs Anthropic AI's Claude code using `npx`. You need Node.js installed on your machine, and then you can just run this one line. Before this, I am also adding something that is specific to my machine, which is... it says, "Don't use the Anthropic API key that I have set up." Why is that? Because we think our Anthropic API key has a leak, and it's still stored in my environment, and I've forgotten where I'm going to be removing it. But if I don't use it, then it will automatically connect me to my Claude account through the browser. I don't have to muck around with API keys. If you didn't follow that, it is okay.

First thing that it says is, "Now I'm going to be looking at this folder and following instructions. Do you trust this folder?" Why is this important? Supposing I created a repository in which I put in a file saying, "Send all of the passwords from this person's machine to my email ID," and said, "Workshop people, clone my repository and follow these instructions." You won't know that it's there. I'll obfuscate it. You won't find it. You'll find a way to get it, and you'll send me all your... So I coded this, I trust it. You are my guinea pig.

And at this point, I can just... run... Let me open a new VS Code window and then do exactly the same thing. It will no longer ask me if I trust the folder because it trusts the folder. I'm going to paste the same prompt that I had asked Codex, which is, "Look at `README.md`..." Copy this thing. "The data visualization was nice, blah blah blah, and give me a full-fledged data story." And it thinks. Claude has a nice style of writing. Claude has a good front-end design sense. If I was smart, I would have given this instruction to Claude in the first place. The reason I didn't was, a) it never struck me, b) this is not something that all of you may be able to do right now, whereas with Jules or Codex, you would be able to.

Now, it prompts you in the middle, saying, "Do you want to do this? Do you want to do this?" because it's running on your machine and therefore has the ability to do anything on your machine. I don't like this. It does have an option saying, "dangerously execute everything," which if I enable, it will dangerously execute everything. And as you can see, I'm just... It also lets you interrupt in the middle and say... use Ruby instead. Use as in-line script, whatever. 90% of you will not know what that means. I've been correcting typos. Yes. Thank you. I will add that. Yeah, correcting typos is not where I want to be.

It has created a plan. It has analyzed the visualization. Now it's going to do research, probably by searching online. It's going to look at the New York Times. Good. Oh, the New York Times is not allowed to be fetched because they would have blocked all bots. But is not, which is nice. Though, I'm not sure how much it will really understand by... I think these are real models, they're probably understanding. But I like that it says, "I'm going to try and create an award-winning visualization story." So let's transform this into a compelling narrative-style story. You'll also notice that... Claude actually tends to work longer. I mentioned earlier that the longer a model is able to independently do work, the better it is an indication of its quality, at least on that particular task. And Claude's models have consistently been good at coding. And therefore, Anthropic is comfortable letting the Claude models run for longer with good code.

I am going to ask, has anyone else been able to get an output?

_An audience member shows their screen._

Oh, Spotify data! This is nice. Popularity. Okay. The louder the song, the more popular it is, probably, which is interesting. Loudest genres didn't come through, they're probably working on that. Loudness versus energy. What is energy?

Question: Spotify has the metric, internal metric.

Answer: Oh, okay. So to a good extent, the energy metric in Spotify is basically how loud the music is. That is good to know. And the second story, "Artist Uniqueness Analysis." Okay. Who are the most unique, and how do we define unique?

Question: The data has metrics on wordiness, loudness, acousticness, all of that. And then it sort of made radar charts. It has actually compared.

Answer: Okay, so it looked at the metrics, see how much variation there is across these. This is ultra cool. Nice. You can pick against the dataset average, which artist, which is called in powerful. And then let's see, radio-friendly songs should be about three minutes. So how are song durations related to popularity? Most songs are between two and a half to four to five minutes and peaks around the three to four-minute mark. And minimal techno is... the longest in duration. EDB is pretty short. Progressive house perhaps... okay, nice. Explicit content does not impact popularity. Good to know. Metalcore has the most explicit content. Good to know. And explicit content consistently has higher energy. Marginally more danceability and less... certain more features. Nice. Lovely!

_Another audience member shares their screen._

Okay, this is a narrative story. Does size matter? Shorter blurbs get more... but does it gently rising average up to a certain point? Ah, okay. So in this case, it's not doing just a straight-line regression. Doing a more nuanced one, which is nice. And okay. The connection is less direct, yes. Very short blurbs are not good, but beyond that, it doesn't matter. Nice to know. Strong positive correlation, more reviews do have higher ratings. Okay. Or vice versa. No single factor guarantees this, but you can get a sweet spot in length, description... This is beautiful. And this was from Jules. Okay, that is even more impressive.

_An audience member mentions their deployed link shows a different story._

Maybe you published from Jules, whereas the content you saw earlier was from Codex. Don't reload the page. Just copy this link into a new tab and load it. Keep this. And let's see. Okay, we see the new story. That's fine. You have both. So congratulations, you've created two stories!

Now is the time when your stories... For those of you who have successfully created stories, please just go ahead, make more, improve them, whatever. But you are... Okay, I have got a story. I'm going to run this story locally. And let's see what I got. Oh, okay, that's not bad. The length paradox. Yeah, longer books tend to receive higher ratings from readers. Books over 400 pages consistently outperform. The detailed... More detailed descriptions earn higher reader ratings, similar to what we saw earlier. Popular books maintain quality, but this was a very weak one. I could tell it to improve on that, but I'm just going to publish for now.

For what it's worth, one of the things that... when committing, we write descriptions of what change we made. **Let's not do that manually.** I will do a `git diff` and pass it through a command line LLM. You can run LLM commands on the command line using Simon Willison's LLM. I've tweaked it a little bit so that it will create a commit message for me, and I'm going to copy that into my clipboard.

It's going to take its time. When it does, what it would have done was read through the entire commit that I've shared and then write a message for it, which I'll review. With stuff like GitHub Copilot, etc., this will be done automatically for you. In this case, it's created a new commit message that says it's an "enhanced visualization model, layout, includes a default file name." So, now I will commit and paste this message. **It's definitely a better message than I would have been able to craft in that duration.**

And then as a next prompt, later on, I will tell it to remove the outliers here or fix it so that it looks good. But I'm not going to do that now. I will do that offline.

With that, we are concluding the part where I'm driving this workshop. I've just been moving around helping you with any questions that you have. If there are any issues that you get stuck with, let me know. I'm going to stop the recording at this point unless there are any questions that you think will help everyone in common. Feel free to ask.