# Transcript - LLM Trends One of the things that we are seeing that is relatively obvious is that **the price-quality tradeoff is changing**. Here's how we are measuring this. So if you take this graph, where on the x-axis, we have the cost of each model. That's roughly how many dollars it takes to process all of the, let's say, Harry Potters put together or the King James Bible. That's about a million tokens. Then a model like GPT-4 costs about $75. A model like Gemini 1.5 Flash costs about 3.8 cents. **That's three orders of magnitude difference.** The difference between a $1,000 budget and a million-dollar budget, for what seems to be comparable quality. $30, 3.5 cents. It's just that this model is about a year and a quarter older than this model. In other words, **waiting a year and a quarter leads to a factor of 1000 reduction for a given level of quality.** Now, the implications of this are stunning. Meaning, if we started at, let's say, November, Jan 2024, when GPT-4 was released, these were the frontier models, the ones in green. Meaning there was nothing both cheaper and better than any of the green bubbles. Claude 3 Haiku popped out a bit in that it was both better and cheaper than any of the nearby models. So clearly it made sense to shift over to this kind of a model. Over time, GPT-4o mini took that spot, DeepSeek models came in and took that spot, most recently the Gemini Flash preview came in and took that spot. And we now have a frontier, which of course is constantly evolving. But the implications of this kind of a frontier are that **it's really not realistic to plan for current capability. What we want to do is plan for future capability because the pace of evolution is such that if we do plan for the present, then we are outdated by the time we finished implementing.** I'll show you an example of how that actually happened to us in a short while. A specific way that happened is the pace at which context windows were increasing. One of the problems that we had in January was we couldn't take a large XML, pass it to a model, and have it generate the revised XML. So we were working around engineering means of doing that, which took us about three months. And at the end of three months, GPT 4.1 was released, which effectively had a 1 million context window, which means that all we had to do was pass it that entire document. **Waiting for three months was arguably a better strategy.** I mean, apart from the fact that we learned a lot, that's always fun, but. And as a result, what we can now do is pass longer documents, entire documents, rather than having to worry about something like retrieval-augmented generation. And also store more memory. Effectively, every single thing that we've learned from conversations becomes part of the memory. That's enabling one of the proof of concepts that we're talking about, which is policy as code. The way this works is, supposing I take the European financial promotion guidance. Now, this is a rulebook that says that when a bank in Europe publishes something to the consumer, here is a PDF that has a list of guidelines on what's allowed, what's not allowed. It's a series of PDFs. Let's take all of those and apply them against a promotional email, a website landing page, a notification, whatever. What we're going to do is pick this as the source and ingest these rules. And it goes through document by document, looking at what are all the pieces of rules that the document has provided, taking the entire document as a chunk. And now it's crafted a set of 39 distinct rules, which, for example, let's take one from the marketing side. Avoid excessive social media promotion contacts. The rule that it's identified is firms must not bombard consumers on social media with repeated financial promotion. Another is that firms should regularly test and monitor them, each with varying levels of priority with a rationale for these. So, A, **we can extract rules directly from a rulebook, and B, we can start applying these**. So let's take the same set of documents that we've identified. I'm going to show you what one of these looks like. So this is a promotional email for Life Shield insurance that says, "Dear customer, blah, blah, blah, why choose Life Shield?" and so on. Let's see how this does against the rules that we've identified. We'll validate them. So now it's going through each one of these rules, evaluating them one by one. So the first document's been evaluated, the second document is being evaluated, it will take its time. But some of these have failed. So for instance, promotions must provide a balanced view of benefits and risks. So this does not provide a view of, it doesn't emphasize... it talks about the good, but not the bad. This one does not warn about the risks related to insurance. And some of these are not applicable, some of these are more applicable than others. Effectively, **in one shot, we have converted documents that serve as policy guidelines into code, directly**, which of course is reviewable. And that's something that you can start applying to, for instance, author guidelines, that's something you can start applying to internal policy documents, email conversations with customer support that serve as a, let's say, common law kind of a thread that you can then apply to future guidances. There are two levels here. One is the documents serve as a base for rules. The rules, which themselves are editable, serve as a base for validations, bypassing or eliminating the need for code generation as a bottleneck and moving the process away from a development cycle into purely a subject matter expert-based cycle. I promised to pause periodically. Let me pause here. That's one of the trends that we're seeing. Any comments, thoughts, any directions you want to move in? --- **Question**: Is this the approach that's happening with the QA framework? It looks very... it sounds very consistent with what we're trying to do there. **Answer**: Yes. For the QA framework, we take the guidelines and we bring that into individual prompts. And each one of those prompts are again validated, tested right there into that framework. And then it gets into production. **Answer**: I think with this, what you can do is probably take that guideline, the prompts would auto-generate. Right, so that's the next level of it. Or let's say you have a style sheet for copyediting, you could take that, load it in, you've got your set of copyediting guidelines which are already done. --- **Question**: Follow up question. This seems like it's probably computationally expensive. Is that true or not true? **Answer**: How does it matter? --- **Question**: It's a one-time thing, but if we were to run it at scale and apply it across multiple different use cases, is it viable from a business operations perspective to utilize this approach more broadly? Because there are many things that we do. **Answer**: Let's take two directions to it. A, it is. B, it is not. **If it is not, let's wait for a year and a quarter for it to become a thousand times cheaper.** B, it is. And the reason I'm saying it is, is because we already have models that for a million tokens cost three and a half cents, and they're not bad. They're not bad at all. Third, there are far more efficient ways of doing it. So the algorithmic improvements keep going. So today we see this as practical in many situations, far more than it was six months ago. But if there are some where it's not, we can figure that out. It will be in six months. **Answer**: If you're looking at running that through at scale, you could consider doing that for lightweight checks across the board at submission upstream. If you're looking at revisiting your guideline to a probably a lighter weight guideline type for submission, you could run it at scale across the board without any expense. --- Okay, let's move on. And if anything else strikes you as we go along, please, we can change course at any point. But the next trend that I see is **the amount of attention LLMs are paying to the document is increasing.** That completely unreadable chart was the GPT 4.1 evaluation of a needle in a haystack. That is, given a document of one million tokens long, can it identify something from somewhere tiny in the middle that was just obscurely mentioned? And earlier models failed at that. Starting from GPT 4.1, they are much, much better at it. What are the implications? One, we are now able to write longer pieces of code, which require that level of attention to detail. We are now able to write structured output, JSON schema, XML, which require that level of attention to detail. We are able to self-correct because the models are able to look back, think, and find that tiny little error in the middle of what it generated. And therefore, we are able to have far stronger reasoning as part of the approach. Now let me give you an example of how this translates into action by showing you three, maybe four examples. One is a vibe-coded application. Topic Trends is an application that an intern vibe-coded in about two hours. The broad mandate that I had given this intern was, build me an application that will take papers from arXiv on a specific topic. And KG suggested quantitative biology, another colleague suggested portfolio management, and so on. So given these papers, have it identify a set of topics, but the user should be able to pick their own topics. And then against any of these sets of topics, automatically classify the journals and figure out the trend so that the users can figure out whether certain topics are trending more, trending up, trending down, whatever. Code, auto-generated. Number of iterations, two. And this is the result. What we see is that there are... the majority of the topics were around... what is this? Maroon. Stochastic and mathematical models in biology, but that's trending downwards. Still the top at the moment. Something like global bio-image data sharing, which is the orange one, that seems to be trending up a fair bit. And so on. We also get to see if I want to be a little more confident, not let's say at... right now we are matching at at least 30% confidence level. Now if I said I want to be even more selective, I want to be let's say 50% confident, then what we're finding is that there was this little blip, but leaving that aside, one of the more popular topics is still the maroon, stochastic models in mathematical biology, and global bio-image data sharing still continues to be something that rises. And if I said, look, even a rough match is good enough for me, we get to see the levels of confidence. Now, while this in itself is useful to identify, the more important part is that **because the models are able to generate code at higher levels of confidence, we are able to generate code dynamically, on the fly. And that's enabling a large number of new use cases.** For example, let's take data schema validation. We've got a data set and I want to understand what that data set is about. This is a classic data engineering problem. Roughly a third of a data engineer's time goes into processing messy data. We need SMEs to tag the column names in the data sets. They don't have time. We need to create data quality rules, figure out the rules to move them from place to place. What if we could take a data set—and I'm going to take a pharmaceutical data set as an example—and this is effectively dynamically generating the schema for each one of the columns in the data set. For example, on the patients table and the clinicians table and the drugs table, we have... The input is just the underlying Excel file that was uploaded as a sample. It knows the column names and a few columns, that's it. It is inferring from that that there is a column called ETHFL, which is probably the ethnicity flag or code, that the PT type is probably the patient type or category. Some of these columns have personally identifiable information, so you want to be careful with them. These columns have more details such as, let's take the ethnicity flag. It seems to be, say, Asian, Latin American, African, etc. Patient type flag seems to have these kinds of values, and we can infer the entity relationship diagram based on the relationships between these dynamically. And automatically generate the quality rules, which will take its own sweet time. But ultimately, it's going to give us a dump saying, run this check, these are the rules that I have inferred about what to check on each one of these tables, fields, etc. And if there are any mismatches, it will flag those off. Effectively, **the first iteration of where the data engineer goes through along with an SME the database, identifies the schema, etc., now gets automated into a series of tests**, which they can then use as a starting point to iterate. It's a second example where dynamic code generation is becoming feasible. Let me show you an even more tighter example of this, and you'll see this in action. What if we could replace a data analyst, a data scientist? Let's say, I'm going to take a pharmaceutical example again. This is... We'll look at craft beer. Yeah, you could take craft beer. Did I have craft beer? I have no idea what the data set is, but let's do that. We have an SME in the room for it. Okay. So here's craft beer data. And let's assume that this is the context. We want to generate insights for brewers and distributors, something to do with consumer taste preferences, market trends, regional variation, blah, blah, blah. And if you have any other hypothesis or any other problem area that you might want to give to a virtual analyst, we could just pick that. But taking this as an example, step one is it crafts a series of hypotheses the way an analyst would. And because LLMs are as good as a fresh intern at crafting hypotheses, given the data schema, it's able to do a reasonably good job. But the nice part is, because they're good at code generation, we can check if, for instance, do consumers show a strong preference for American IPA and American Pale Ale styles, given their high representation in the data set? Okay, test it. Now, dynamic code generation, it's writing code to test the hypothesis, it's writing code to evaluate the statistical significance of that, and running the code in a sandbox. Spinning off a container and evaluating it is fairly straightforward these days, but the code did not work. So fine, let's move on. We don't even need to wait for this hypothesis because we can always run this in another iteration. Second hypothesis, and it finds that there's no evidence that more bitterness is more popular among the craft consumers. Now that may be because the data is too small, it may be because it's not true, but at least there's no evidence. So test them all. I don't even want to have to sit and inspect each one of these. You've come up with a bunch of hypotheses, you've generated the code for all of these, you're evaluating those, and some of them are true. So for instance, it's finding that the high uniqueness in beer names indicates a consumer interest in novel and distinct products. Okay, that's useful. Regional variation has an impact, serving sizes has an impact. Yeah, fine, but I don't want to read all of this. Synthesize this. And that takes these results. This is something that LLMs, we know are good at. So the summary is ultimately that **focusing on unique or creative beer names is a good idea. You want to tailor it to regional marketing, not global marketing. A 14-ounce serving size seems to be, as a core offering, something that works well.** Other larger formats will satisfy consumer demands across, but this is a size that you certainly want to maintain. Now, this is... I have no idea what this means, by the way. So I'll look to the experts to comment on it. But **the crux of it is, a data scientist is now at the stage where their current tasks are automatable** because we're finding that code generation is automatable, because among other things, the ability to write code robustly is proving better. Let me pause before I go on to my next theme. Any comments, thoughts, questions? --- **Question**: So data science as a career is dead? **Answer**: That was... my last talk was titled, "Rest in Peace, Data Scientists." It didn't go too well. But I think **it's the tasks that are dying, not the role.** You need to evolve to higher level roles. Exactly. --- **Question**: ...a reliable, I think that type of conclusions, I think is, maybe, I wouldn't say arbitrary, but may not be a true reflection of what's working, what's not working in terms of marketing right now. **Answer**: True. When that improves, it seems to improve the reliability of many kinds of output. Pass an LLM a data set and ask it to suggest a few numbers, we have no idea if it's hallucinating. Have it generate code to come out with the numbers. We know that if it's wrong, it's wrong in some massive way. The code is right. And if the algorithm was completely off and it misunderstood, that's fine. If there are syntax errors, it's able to spot it because it's not even able to run. So by passing it through a deterministic pipeline, which is an interpreter or a compiler, we narrow the field of errors dramatically. But this is not the only way in which that field of errors can be narrowed. What we're seeing is in any case that **LLMs are able to think for a lot longer without errors.** The trend has been that from, let's say, about a minute of independent thought in early 2022, which is what GPT 3.5 was doing, around 2025 beginning with Claude 3.7 Sonnet, running autonomously for an hour is possible. Which is roughly saying that an entity with the level of intelligence of a college graduate is able to work independently for an hour. Now this has gone up with the newer GPT-5 models to as long as 10 hours, whether it's Claude or any others. Now that's enabling a bunch of new possibilities. Autonomy is clearly one, but more interestingly, creativity because we can let them ramble a fair bit and still come to something that is useful. One of the things that I'm personally using it for is to automate innovation. I take notes of all kinds, including extracting them from my emails, from my conversations, from my code, all kinds of things. So this application, the Ideator... here's how it works. It picks random notes. So for instance, something that I noted on 31st August was how educators are using Claude. I could pick something else at random. Let me pick something related to LLMs. So as of 11th August, the OpenAI chat completions API has a certain set of capabilities. Let me pick something else from my... oblique strategies, or let's say core concepts, discussion, home gardening, persuasion. Yeah, okay. Let's take persuasion. These are completely distinct ideas. But what if we could combine OpenAI chat completions with the core concepts of persuasion and come up with a research paper about a topic that has not been covered so far? And create an idea. This just generates a prompt, sending it to ChatGPT, and the prompt will appear in a short while. Let's run it. But this is a template that says, you are a radical concept synthesizer that astounds even experts. Generate big, useful, non-obvious ideas about a research paper, blah, blah, blah. And you'll be given two concepts, however many concepts. First concept is here, second concept is here. And we give it instructions on how to generate a series of candidate ideas using multiple mental models, how to score it—we are looking for novelty and utility, but that may vary—and how to evaluate, pick the top score, and how to communicate the output. Generate, mention the insight, explain how to build, test, why it's surprising, critique yourself, and write it in this style. This is going to take approximately five minutes, and I'm not going to wait for it, we'll move on, but we'll come back. What I've found is, and I've done this about 17 or 18 times in the last one month, **without exception, every single one of those ideas, at least the top idea that it has picked, or the second or third, between the top three, has been something that I could take, create a demo out of, create something that is marketable, whatever, without exception.** So now the problem that I used to have earlier, as someone who leads an innovation team, was how do I get new ideas? **Now my problem is execution capacity. Ideas are not the constraint.** And if possible, how can I translate that ideation to execution without me being the bottleneck? Right now, I am the bottleneck, having to understand this, interpret this, and so on. So I'm trying to templatize all of that. In short, because we now have the ability for it to think longer—and we'll come back and see what this one generated—even creativity is something that we are able to automate. But the other thing that we often worry about is hallucination. What does it take to reduce the kind of errors that we have, and how is this performing? **Research is now showing that LLMs are better judges of LLMs than the average human** in many areas. Subject matter experts still are better, but the number of or the percentage of subject matter experts that an LLM is able to beat is consistently increasing. In maths and physics and in code, it is able to beat all but maybe the top 0.001%. In areas where it's got less training, it's still able to beat 90%, certainly that includes the general public. Which means that **our ability to scale validation is a whole lot higher.** Which means, clearly, better quality of not just LLM-generated output, but human-generated output as well. Validation is a no-brainer now. And at scale because the cost is small and falling. There's just no question about that. And allows us to have continuous improvement because the role of the subject matter expert now is not as much to validate at scale, but allow the LLMs to say, "Oh, here's one thing that I'm not particularly confident about," I'm putting up my hand. You tell me, I will incorporate that, take that feedback, and move back in. How have we been using that? Let's take a couple of ways. One is a consensus mechanism. So we were looking at classification of chat messages. Supposing a chat message came in saying, "Help with adding some items," or "When will I receive my order?" Is this being classified correctly? Now, we expected the model to put this into the delivery period bucket, but this particular model placed it into the track order bucket. The model happened to be GPT 4.1 mini, not such a great one. But Nova-light did manage to get it right. So, different models on different inputs have different accuracies. So the ones on the right are where all the models are consistently getting it right. The ones on the left are where the models are failing. And there are some, like "help registering," where all the models are getting it wrong, which of course begs the question, did we frame the question or the categorization right in the first place? But the point is that these models are off. How off are they? It turns out that if I picked a model at random, on average, they make mistakes of about 14%. Fair enough. What if I said I'm going to use two models and only if they both agree do I let it pass? Two things happen. **The error rate falls dramatically from 14% to 3.7%.** Fantastic. My manual effort increases because I now have to, where they disagree, I have to check manually. That increases from zero to 12.6%. Not bad. The beauty of this is that I could double check, I could triple check, I could quadruple, quintuple check. The cost is negligible for these models. And this is, remember, picking random models. So if I randomly pick two models, then at quintuple checking, the error rate is 0.7%. I get 99.3% accuracy. But the manual effort is 28%. That's okay. What that means is **I save 72% cost for practically no deterioration in quality, in fact, arguably improvement over human quality.** And if we pick the models carefully, now some models are not very correlated. The ones in blue, for instance, are not correlated between each other. Let's take this column: Gemma 3 versus Qwen 2. The correlation is pretty low, which means that if one says something, the other is likely to say something else. And it's very unlikely that they will disagree on the same wrong answer. If they disagree, it's more likely to be genuinely, they are not clear or the question is a hard one. So if I pick models like this, I don't even need to quintuple check. Quite often, I can just double check or triple check and get to that same kind of quality. So multiple LLMs as a judge is a great mechanism. We've put this to fairly good use in optimizing prompts. So for instance, one of the things that a lot of people ask us is, you know what, does adding "think step-by-step" help? That's a reasoning prompt. Does it help if I say, "You are the world's best expert in a particular topic," and then have it generate the rest of the result? Does it help if you say something emotional like, "I'm totally overwhelmed, my career depends on it, this is really important"? Does it help if you bully it? And we've had memes on all of these. All we have to do is test these out. And the testing frameworks are now becoming a whole lot easier, including LLM as a judge. And what we find is that **only one of these prompts consistently works across models: "Think step-by-step,"** which is also why that's part of the reasoning process. Emotional prompting, shaming, all of these actually consistently tend to make the models worse on average. Some models do better, but the majority of models do worse. Now, if we are able to therefore apply these kinds—and some of the problems that we gave it were, for instance, just mathematical problems. Can you solve a mental math problem as an example? But if we are able to automatically evaluate using LLMs as a judge and multiple LLMs as a judge, then can we move to a mechanism where we dynamically improve prompts? That's something we tried with... let me take a pharmaceutical example. This was a case where a clinical trial result... Let me show you what that looked like. So our client gave us something like this data. The input is... going to make it bigger. The input is roughly a clinical trial protocol. "Following administration of the investigational antibody, blah, blah, blah." And this is something that a patient's not going to understand. What we want to do is take this and instead say, "After you receive the study medicine through an IV, we will watch you closely for 4 hours and we'll check your blood pressure, heart rate..." English. We want to move from A to B. So let's copy that, paste it out here. And firstly, have an LLM generate the prompt automatically, which it does. It says, "You are an expert medical communicator, blah, blah, blah," and it explains how to create this summary. Then, have it generate the output against the inputs. So one by one, it crafts the output. Now, let's see how good the output is. We're evaluating it on two things. One, does it miss anything? Sorry, does it have any extra content? And does it have all the relevant content? Turns out that on the evaluation, on the first one, it seems to miss all of the criteria. The generated output introduces some additional detail about the strength of the medicine, whereas what we want is not that much detail. So it seems to be consistently trying to be more precise or more technical than we wanted it to be, but it doesn't seem to be introducing extra content. Fine. Now it has a score, 16 point whatever. Let's have it revise the prompt. We tell the LLM, "Look, we gave you this input, you generated this output. Improve yourself." And these are the corrections that it's made to its own prompt. Start again. Let's apply this prompt and have it generate a series of outputs because it's learned from itself. And then evaluate that output, which it will... this is going to take its own sweet time... and come up with a series of scores. It's still missing on the first one, but over time it will generate the rest. Hopefully, the result of this is going to be a higher score when it's finished evaluating. And it's paused for some reason. But the important thing is, this is automated end-to-end prompt evaluation in a cycle. And this, something like this is possible because the quality of LLM evaluation is matching human evaluation. In many cases, in fact, we've created synthetic persona, that is generated persona for surveys, where rather than running a survey against people, which can be too expensive, run surveys against synthetic audiences, and they tend to have something along the lines of 80% match with the responses that a human would provide. I could go on, but let me pause here. Any comments, questions around LLM evaluations or dynamic code generation? --- Okay, that's a clear signal that I'm going to skip the last bit. We'll probably go back and take a look at what it generated. And yeah, let's just do that and end with that. So the idea... a bunch of candidate ideas, but **"Rhetoric as Inference Schedule."** The paper that it's suggesting that we write is, **"persuasion isn't about what the model says, it's also about when and how it thinks."** So scheduling inference time knobs—verbosity, reasoning effort, etc.—can lift conversation and build trust. It's saying, build a middleware that tags each turn with a rhetoric phrase. Okay, put in a low verbosity, reasoning effort medium, etc. So tune the parameters. And... what exactly is this saying? Okay. Explain this in simple language. And let's get a quick response. I've had this problem. It usually tells me things that are way beyond. Instead of keeping an AI's settings fixed, we change them step by step to match how persuasion works. In other words, create a theory of persuasion based on experimental validation. Have a person interact with an LLM, have it dynamically change the verbosity, reasoning effort, and all of those parameters, and figure out which of those settings really changes persuasion. Interesting. Shit, I'm going to do this. But yeah, this is how my queue builds up. There are several trends out there. You can see the rest of the slides at any point. But **the one takeaway that I have from all of this is really what I mentioned right up front, that things are moving so fast and in such big leaps that, like compound interest, it's not something we can easily absorb.** **Humans are not geared for acceleration.** And it takes a certain amount of systematic effort. But **one of the first things, therefore, that I plan for is the future.** A few leaps ahead. And let's not worry about cost, let's not worry about feasibility. If a prototype shows moderate potential, there is a reasonable chance that it might be productionizable. We need a portfolio because some of these will fail, there is a jagged edge. But it's really hard to predict which of these will succeed wildly and which of these are going to fail. So, **portfolios over process is where I've tended to land on this.** But that's what it is. Thanks for listening. Thank you.