# Goodbye MBA, Hello AI Transcript Summary - [01:22](https://youtu.be/sSyBUSuLduQ?t=01m22s) LLMs are the biggest technological inflection since the PC / Internet / electricity and you’re starting your MBA right in the middle of it - [02:03](https://youtu.be/sSyBUSuLduQ?t=02m03s) Much of today’s MBA syllabus will age out fast; adaptability is now a core competency - [03:54](https://youtu.be/sSyBUSuLduQ?t=03m54s) LLMs have a “jagged-edge” of capability: they ace some tasks yet flub two-digit multiplication and hallucinate facts, so managers must anticipate and contain failure - [07:02](https://youtu.be/sSyBUSuLduQ?t=07m02s) Classic management tools—redundancy, cross-checks, human escalation—still work: using 2-5 diverse models cut classification errors from 14 % to < 1 % while saving 72 % of effort - [09:46](https://youtu.be/sSyBUSuLduQ?t=09m46s) Model quality-per-dollar is skyrocketing; the performance frontier keeps shifting monthly, driven partly by models that improve their own research - [14:21](https://youtu.be/sSyBUSuLduQ?t=14m21s) LLMs already deliver hours of analyst-level work in minutes (e.g., end-to-end ML pipeline in 15 min), so some entry-level roles will compress or vanish - [23:01](https://youtu.be/sSyBUSuLduQ?t=23m01s) High-value knowledge jobs—strategy, finance, contract risk, market research—are being refactored by AI - [45:22](https://youtu.be/sSyBUSuLduQ?t=45m22s) World-Economic-Forum skill map: double-down on creative thinking, AI/data literacy, cyber-security and human-centric leadership; de-emphasize rote coding, clerical work, low-level analytics - [49:17](https://youtu.be/sSyBUSuLduQ?t=49m17s) How to study with LLMs: use them as Socratic sparring partners, run “draft-critique-rewrite” sprints, curate a personal prompt library, rehearse interviews/cases, and offload grunt work - [52:59](https://youtu.be/sSyBUSuLduQ?t=52m59s) Imagine your 2027 where “manages AI teammates” is normal. Pair uniquely human strengths with AI leverage—and keep it fun despite the chaos Funny bits - [01:08](https://youtu.be/sSyBUSuLduQ?t=01m08s) “It’s awkward when there’s a pause and you don’t know if you’re supposed to clap… I'll give you a signal when to clap. It will be right towards the end.” - [12:22](https://youtu.be/sSyBUSuLduQ?t=12m22s) Grok-4: “Apparently it basically repeats whatever Elon Musk says.” - [20:02](https://youtu.be/sSyBUSuLduQ?t=20m02s) To the lone student without a phone: “I love you… the rest of you, scan the QR code.” - [39:07](https://youtu.be/sSyBUSuLduQ?t=39m09s) “At least for the next few years, use AI. Don't talk about AI.” - [39:48](https://youtu.be/sSyBUSuLduQ?t=39m48s) “Don’t obviously tell your boss; within a few minutes, wait a few days… play Minesweeper—or Minecraft.” - [50:04](https://youtu.be/sSyBUSuLduQ?t=49m52s) “Please sleep. Tell the LLM to do the assignments… This is going on record, but I’m not faculty, so I can say what I want.” [00:01](https://youtu.be/sSyBUSuLduQ?t=00m01s) Okay, can we settle down please? Thank you all. On behalf of the Department of Management Studies, it gives me immense pleasure to welcome our keynote speaker, Mr. Anand. [00:23](https://youtu.be/sSyBUSuLduQ?t=00m23s) So Mr. Anand, as you can see, is an LLM psychologist. I think you will tell us why you call yourself that way, even though you say it's not an official title. Okay. And he just calls himself that and he has he co-founded Gramener, a data visualization which Strive acquired. He's an alumnus of IIT Madras, a regular TEDx speaker, and the only person who has hands transcribed every Calvin and Hobbes comic strip. I went to your blog yesterday and I read all those Calvin and Hobbes, yeah. Because Calvin and Hobbes is a favorite for me also. So we are looking forward to his talk and welcome once again. Thank you. --- [01:08](https://youtu.be/sSyBUSuLduQ?t=01m08s) It's awkward when there is a pause and you don't really know if you're supposed to clap or not supposed to clap. Generally it's good if somebody gives a signal. I'll give you a signal when to clap. It will be right towards the end. [01:22](https://youtu.be/sSyBUSuLduQ?t=01m22s) You've all picked a very interesting time to start your MBA. Arguably, you haven't really picked it, it's been picked for you. But this is one of those times where so many people are concerned if their let alone job, if their role itself will exist a few years down the line, thanks to AI. Now, people have been calling AI, well, specifically large language models, ChatGPT and the likes, the biggest technological thing since maybe the PC, maybe the internet, maybe even electricity, who knows? It's certainly had high impact. How high is questionable. But high is scarcely questionable. [02:03](https://youtu.be/sSyBUSuLduQ?t=02m03s) And what that means is a fair bit of the stuff that you are going to learn in the next two years is going to be completely outdated. Some of it is already completely outdated. And some of it is definitely going to be outdated. The faculty knows this, you know this, the recruiters know this. What we don't know is which parts. It's a big open question. And this time you have chosen your MBA. [02:28](https://youtu.be/sSyBUSuLduQ?t=02m28s) Welcome. [02:31](https://youtu.be/sSyBUSuLduQ?t=02m31s) But see the thing is LLMs can fail. They can fail in weird ways. If you ask an LLM to multiply two numbers, and I did, I said, "Tell me, what is 12 x 12? What is 123 x 456?" A bunch of fairly simple problems, simple for a calculator, even for mental mathematics, at least the first few problems many of us can do in our heads. And different models scored differently. So for example, let's take one of the weaker models, LLaMA 2, which is a slightly old model. When asked for the two-digit multiplication, it did not get it right at all. Three-digit multiplications completely failed. [03:16](https://youtu.be/sSyBUSuLduQ?t=03m16s) Models like LLaMA 3, which is a more recent model, that managed to get it consistently right. That is, the two-digit multiplication it managed to do really well. The three-digit multiplication it goofed up a few times. The four-digit multiplications it was not able to do. Now, you would expect that a neural network which is vastly superior to what you would have in your pocket calculator would be able to do this sort of a thing, right? It does not. LLMs currently are bad at mathematics, simple mathematics. [03:54](https://youtu.be/sSyBUSuLduQ?t=03m54s) And this is often surprising to people because you would not necessarily expect that. And this is not the only way in which it is surprising. This nature of surprise is what many people are calling the jagged edge. It's like a sharp object that can cut you in ways you're not aware of. In some areas, it is doing a fantastic job. In some areas, it is doing a very poor job. One of those problems is or one such problem is what they call hallucinations. [04:24](https://youtu.be/sSyBUSuLduQ?t=04m24s) Hallucinations are where an LLM gives you an answer, but the answer is wrong. Now there may be many reasons why the answer is wrong, but the fact that it is telling you that this is the answer means that it at least believes that it that is the correct answer. LLMs don't yet seem to be capable of intentionally lying in direct ways, and picking my words very carefully. Currently, intentionally, direct ways. All of this is changing. Everything is changing with LLMs. But at least they're not lying as much as they're just hallucinating. Arguably, we make the same kinds of mistakes. When a human makes it, we call it a mistake. Humans, to err is human, literally. [05:10](https://youtu.be/sSyBUSuLduQ?t=05m10s) And people are often surprised by it. They say it's a machine. It should not make a mistake. Machines are reproducible. But LLMs, AI in general, and I'm going to use the word LLMs, LLMs stands for Large Language Models. I'm going to use the word LLMs as a rough proxy for AI these days. They seem to make mistakes like humans. Which is interesting for you because what you learn in MBA is, among other things, to manage problems. [05:43](https://youtu.be/sSyBUSuLduQ?t=05m43s) One of my school seniors who told me, "Anand, there are three groups of people. There are the scientists. Their job is to eliminate problems. And then there are the engineers. The job of the engineers is to reduce the problems. Then there are the managers. Their job is to manage their problems. Problems will still be there, they just have to keep it bounded. You're on the path to that third category where all the stuff that everyone else has given up on is on your head. And we actually have several ways of dealing with fragile people." [06:21](https://youtu.be/sSyBUSuLduQ?t=06m21s) So for instance, let's take how you can solve the problem of hallucination. It's surprisingly, there are many surprisingly simple ways in which one can do that. For example, what I did was, had different models classify inputs, like supposing there's a chat message that says, "I don't know what I have to do to find the invoice." Now, which category of request does it belong to? Should it go to the invoicing department? Should it go to the, let's say, new connections department? When will I receive my order? Should that go to the invoice department? Likely not. It should go to the order processing department and so on. [07:02](https://youtu.be/sSyBUSuLduQ?t=07m02s) So, is an LLM correctly able to categorize which department a chat message should go to? Was the question. And different models scored differently. So 94% score for GPT-4.1 Mini, whereas a mere 57% score for LLaMA 3, and so on. And on average, across all the models, the error rate was about 14% when I ran it. Meaning that one in seven times it got it wrong. Six in seven times it got it right. Humans can do better. [07:38](https://youtu.be/sSyBUSuLduQ?t=07m38s) But the interesting thing is when I asked two models to make the same calculation or the same prediction, they also on average got it wrong 14%, but they didn't make the same mistakes. They made different mistakes. And the correlation between the mistakes when I plotted, it was not very highly correlated. The yellow areas are where the models perfectly agreed. Models would generally agree with themselves, that's okay. But let's take Gemini 2.5 Flash Preview, some model, versus Claude 3.5 Haiku. The correlation was not that high, meaning that this model would say, "I'll send it to the invoicing department," that model will say, "I'll send it to the budgeting department." [08:21](https://youtu.be/sSyBUSuLduQ?t=08m21s) If the models were both in agreement, you say, "Okay, I have a little more confidence." If the models are in disagreement, let's do a cross-check. And that cross-check involves human effort. It's not automation. But at least I've reduced the amount of cross-checking. So we calculated, supposing you have two models, then how often do you need to cross-check? And the answer was only about 12% of the time. But the error rate reduced to 3.7%. Meaning only in 3.7% of the time, did both models get it wrong? [08:55](https://youtu.be/sSyBUSuLduQ?t=08m55s) If you have three models, triple check, then the error rate falls to 2%, 1.5%, 0.7% for five models. Now, the thing is these models are cheap. So crazily cheap that I can have 10 of them run at no incremental cost to a human. If even one of them disagrees, I will need to send it to a human. It is not 100%. That happens only for maybe about 28% of the cases. Which means that I've saved 72% of the effort for 99% quality. That's great. And this is how we work anyway. Have a bunch of people do it. If it is ultra critical, you have a fail-safe, you have an extra fail-safe, and so on. This is one of the classic ways of dealing with unreliable systems. And some of those principles that you're going to be learning are still going to be useful. [09:46](https://youtu.be/sSyBUSuLduQ?t=09m46s) Some people say, "Hold on, all that is very nice, but the LLMs are getting so much smarter rapidly. Today they're hallucinating. Will they continue to hallucinate?" In fact, they're getting smarter so rapidly, to give you a sense of how rapidly they are getting smarter, let me show you this graph. The x-axis is the cost of models. Each of these bubbles represents one model. And the y-axis represents the quality of the model. Let me explain what I mean by cost first. Cost is how many dollars it costs to process 1 million tokens, which is roughly take the entire King James Bible, or all the seven Harry Potters. That's about a million tokens. You send it to an LLM and say, "Read it, understand it, and be able to answer a question from this." That is X dollars. Some models, like GPT 4.5 Preview, cost as much as $75 for that. Some models, like Amazon Nova Micro, cost 3.5 cents for that. Huge difference in the cost. [10:53](https://youtu.be/sSyBUSuLduQ?t=10m53s) Quality, how do you rate quality? Well, what people did was they went to or they created a site called LLM Arena, where you can ask something, which is like, "When will we have AGI?" which stands for Artificial General Intelligence. And they have two models. They won't tell you which models. They'll have two models answer the same question. And then whoever is asking the question can decide, "I like model A or I like model B." So one of these wins the match. And just like you have an ELO score in chess matches, there are these rankings. They create the rankings for these models. [11:34](https://youtu.be/sSyBUSuLduQ?t=11m34s) So, taking a quick look at this. This is giving me some long answer, blah, blah, blah. Okay, saying key considerations. This is also giving me some long answer. Expected, blah, blah, blah. At the end, is it something? Okay, 2040 to 2070 is the consensus. Early signs in 2030. Okay, I kind of like the left model. I don't know why. Now, people have done this, millions of people, currently it's about 3.5 million people or so. And it then, after I submit, it should tell me which model. Okay, yeah. Oh yeah, they're raising funding. So they'll start doing stuff like this. Okay, so I've basically said, "Qwen 3 is better than the newly released ultra expensive Grok 4," which apparently I've heard basically repeats whatever Elon Musk says. [12:22](https://youtu.be/sSyBUSuLduQ?t=12m22s) But, I have voted for Qwen 3. And these votes all aggregate and go towards this pricing model. Now, on this axis, in May '23, there were only about three models: Claude 1, Claude Instant 1, GPT 3.5 Turbo. Which had a certain cost, which had a certain quality. Over time, more models emerged, more models emerged, more models emerged. And the interesting thing is, as models emerged at different prices and quality, some models are clearly better. Better in the sense that there is no model that is both cheaper and higher quality. [12:58](https://youtu.be/sSyBUSuLduQ?t=12m58s) So let's take for instance, Gemma 2. In June last year, there was no model that was both cheaper and better than Gemma 3. And a model like, let's say, Qwen 2, there's no reason for me to use. Why? Because Gemma 3 is both to the left, therefore cheaper, and above, meaning it's better. So why would I ever bother using any one of these gray models? And red models are the other end, the utter worst of these models. So this frontier kept improving, is continuing to improve. There was a huge buzz when DeepSee V3 was released and DeepSee Car1 was released, they were moving the frontier. And then Gemini started moving the frontier to the point where now all the models are Gemini models. And now people are saying, "No, no, no, this quality metric is not necessarily the right thing." I'm one of those people because the Gemini models are not currently, in my opinion, uniformly the best models. They are good models, not the best models. [13:51](https://youtu.be/sSyBUSuLduQ?t=13m51s) But what is clear is that across the years, the quality and the quality per dollar both have improved dramatically, are continuing to improve dramatically. There is no sign of this slowing down. Scaling laws are what people are calling them. There are many reasons why these are happening, I'm not even sure I understand why, but the short answer is there seems to be a long way to go. And one of the reasons there seems to be a long way to go is that LLMs themselves are helping in LLM research. [14:21](https://youtu.be/sSyBUSuLduQ?t=14m21s) If that is the case, another thing that we're noticing is that the amount of time that LLMs are able to work independently seems to be growing. When GPT-3 was being used, which was around 2021-ish, it was able to do stuff that was roughly at the level of you ask it a question, it will give you an answer. And that is something that a human might take maybe a few seconds for. GPT 3.5 took that under a minute. Now with models like O1, Claude 3.7 Sonnet, which you'll see on claude.ai, they're able to do work that roughly takes an hour for a human. That is something. [15:08](https://youtu.be/sSyBUSuLduQ?t=15m08s) And this is another rough way of measuring how good Large Language Models are. You give it a problem, you see how long it would take a competent human to solve the same problem, and can it do that with reasonable reliability independently? I was just telling Professor Rahul a short while ago that I was at a Microsoft event, and one of their clients had a problem. The problem was that the floor needs to be tiled in buildings. And they have to pick the right tile. For this, they said, "We want to build a machine learning model that will take constraints like what is the width and the height of the floor, how much of weight can the floor support, what is the height of each of these tiles so that you can put electrical conduits and other stuff through it," several other parameters. And they said, "We have run thousands of tests, millions of tests perhaps. And we have the data using which it is possible to pick the optimal tile given these inputs. We want to create a machine learning model for this. We've been trying this for quite some time. We have not been able to." "Anand, you, whatever, LLM psychologist, whatever that means. Can you come over and see if you can help with this model?" Now, earlier I used to call myself a data scientist. [16:27](https://youtu.be/sSyBUSuLduQ?t=16m27s) So my job was building models like this, which is what they called me for. What I did was, took their data, uploaded it into ChatGPT. I have the $20 per month account, which is the Pro account. There's also a Plus account. The Pro account is the $200 one. The Plus account gives you enough access to one of the newer models, O3, which is a very good model. And I, after uploading the dataset, look. And when I said said, I mean literally said, spoke. The mic button is there. So I clicked it and said, "Look, I want you to build me a machine learning model. What it should do is take this dataset and figure out what is the best tile to use based on these parameters." It took me about 30 seconds, maybe 60 seconds to explain the problem to it. [17:20](https://youtu.be/sSyBUSuLduQ?t=17m20s) And I said, "Now write me a Python program, build the model. Look, remember that I'm feeding you not the full dataset, only about some 5,000 rows or 10,000 rows, because confidentiality and all that. So you write the program on this in a way that it doesn't overfit." What that means is, "Don't try to program that gets perfect optimization because you may optimize for this 5,000 rows, but I have 100,000 rows for which if you try and optimize too much, it won't work. So get a good explainable model for me." In fact, when I corrected myself and said, "Don't give me one model, give me four or five models, and then compare the accuracies for all of these models. Tell me which one is the best." And then I will pick. [17:59](https://youtu.be/sSyBUSuLduQ?t=17m59s) Then turned around and these two data scientists were looking. The first expression was, "Are you mad?" And later on, it was like, "Are you God?" Six minutes. Or less than five minutes, something like that. It worked independently, did the job that they were trying to do for a few weeks, created a model, ran the program, it wrote a Python program, it ran the program, created the accuracy results, found that the decision tree model was pretty good, but only slightly behind another model called SVM, which was slightly behind another model called random forest and so on. And it said, "For all of these, I'm able to give you about 89 to 94% accuracy, which is pretty solid." Then I said, "Now download these pickle files," that is, "Now let me download the model." It let me download the model. "Now build me an application that will allow me to send the inputs to the model. Build me a web application so that I can put in the same inputs and it will give the user an output." It did. [19:03](https://youtu.be/sSyBUSuLduQ?t=19m03s) The entire thing done in 15 minutes. This is a project that would have normally taken about six to eight weeks for us a pilot, and would have charged at least $50,000 to $100,000. Model does it. And this is what we measure as the ability for models to think independently. And clearly they can start replacing some kinds of people at least. We are in the early stages, and as you can see from the line, this is going forward. [19:32](https://youtu.be/sSyBUSuLduQ?t=19m32s) In this context, I'd like you to take out your phones and scan this QR code, which will take you to a Google Form. And I'd love for you to fill this out. If you don't have phones, don't worry about it. Later on, borrow it from somebody. I'm curious, does anyone not have a phone that can scan this? If so, can you raise your hand? I love you. Anyone else? [20:02](https://youtu.be/sSyBUSuLduQ?t=20m02s) Okay, one. No, great. So I'm just coming out of a Vipassana program where for 10 days I was not carrying my phone and that is the first time that I'm not been doing that for ever since 2005 or so. But I'm very impressed. Congratulations. But for the rest of you, the question that this will ask is simply, "What skills do you think will make AI less important," or sorry, "Because of AI, what skills will become less important, and what skills will become more important?" And you're welcome to plan this one year in the future, five years in the future, 10 years in the future. Type one entry per line. I've set it to a minimum of 100 characters, sorry about that, that means you have to type a lot. I'll be talking for some time, sharing other useless stuff. You can type your answers in the meantime. [20:51](https://youtu.be/sSyBUSuLduQ?t=20m51s) And the reason I'm having you do this is, this filling out the survey is more useful than my talk. I'll talk about 100 things. It's when you think, and say, "Yeah, I think this is less important, this is more important," that you will place emphasis on certain areas and deemphasize certain areas. Sometimes it doesn't matter what you pick. What matters is that you have a starting point that you can improve on. So, give this a shot, fill this out. And I'm going to be feeling, I'm not saying anything important until you finish the survey for a while, so don't worry about me talking. But think about what it is that you think AI can do and AI can't do. What are things that have been going on for centuries that AI is unable to replace, maybe that will continue. What are new things that will come out because of AI? If we had aliens on the planet, what are things that we might be able to do or might want to do for which new skills will start becoming important? These are all things you may want to consider while you fill out the survey. [22:00](https://youtu.be/sSyBUSuLduQ?t=22m00s) And I'm going to continue. In a while. Just locate my slides again. And keep going. We will come back to what you've said. I'm going to be doing a little exercise based on what you've submitted. So if I have enough responses, I will be able to show you how an LLM can take over at least one kind of job, which is that of a market researcher. [22:30](https://youtu.be/sSyBUSuLduQ?t=22m30s) So what I'm doing right now is a market research of sorts. I'm asking you a couple of questions. You're filling out a form. A market research analyst will probably take that and summarize it, synthesize it, do some segmentation and so on. I'll show you how an LLM can actually do that live with this survey. So, do continue filling it in. And in the process, what I'm also going to show you in the next maybe 10 minutes or so is what are some of the other jobs that are being disrupted, have been disrupted, will soon be disrupted, and how? [23:01](https://youtu.be/sSyBUSuLduQ?t=23m01s) I've got lots of examples. If anyone has a specific preference, let me know. But I mean, by saying, "Anand, can you show me this particular example?" But otherwise, I'm going to almost randomly pick a few of these limiting myself to the next 10 minutes and show you how this sort of a thing can be automated. Any picks from anyone? [23:22](https://youtu.be/sSyBUSuLduQ?t=23m22s) Strategy? Ah, okay, cool. Let's do that. So one of the things that we do as part of strategy is to try and figure out what direction we should go in. One of our clients, they came to us and said, "See, we have medical data. Patients across different locations. We know some of their diseases, we know what medications they're on and so on. And what we are doing with these patients is, they are currently taking injections for diabetes, insulin, some injectable. Let's assume it's insulin. And we want them to switch over to pills. Why? Because we have developed a pill, we believe it is just as effective." "Now who should we target?" This is a classic product strategy, marketing strategy kind of question. [24:27](https://youtu.be/sSyBUSuLduQ?t=24m27s) Now the person asking this question says, normally, to a bunch of MBAs, senior managers, whoever, "This is the problem, go solve it." What they in turn do is do a certain amount of market research, product research, use their domain expertise, find domain experts, ask them a bunch of questions, and then say, "Come up with a set of hypotheses on what segments I should target to go after pills instead of injectables." Once they have that, they will then pass it to a bunch of analysts, data scientists, clinicians, statistical experts, who will come together and test each of these hypotheses with data. Then it will come back to the person who's coordinating this, who will say, "Okay, this hypothesis is validated, this hypothesis is validated," putting it all together and finally coming up with a recommendation. [25:25](https://youtu.be/sSyBUSuLduQ?t=25m25s) Let's see the model do that. Step one. It has now generated six hypotheses. And it looked, it went too fast, so I just add a space here and ask it to generate the hypotheses live. Now it's going to generate a different set of hypotheses. All I did was add a little space so that when it runs again, it will come up with randomly different output. And here are some examples. It's saying that behavioral factors such as patients expressing feelings of being overwhelmed or having a history of discontinuation on injectables, they are the people you should target for pills. Makes sense. The guy's been stopping medicines, yes. Now, it's producing these hypotheses not randomly, but based on the data that I've provided. Looking at the data, saying, "Oh, you have details about the history of discontinuity," and that would have been there, adherence status, poor, whatever, overwhelmed. So we have these details. So it's looking at the data and based on this saying, "Not only are these useful hypotheses for your objective, these are answerable hypotheses for your objective." [26:30](https://youtu.be/sSyBUSuLduQ?t=26m30s) Next, let's test these. So I will click on test. What's happening here is an LLM is writing a Python program which will take that data as input, test if that hypothesis is in fact true, check the statistical significance, and come up with the result. So it's written the program. Step three, it is now running the program. It has now run the program and come up with the result. That is step four, which is it's saying, "The behavioral indicators reliably predict the readiness to switch to oral medication." And we can look at the analysis carefully, see what exactly the results are if we run it ourselves. But what's clear is that we have a reproducibly testable way of validating this conclusion, which is exactly what we would get from a human anyway. [27:19](https://youtu.be/sSyBUSuLduQ?t=27m19s) Some analyst says, "Yes, yes, this is true." Then I want to say, "Show me your code. Make sure that it works properly." That's exactly what I can say with the LLM. I can do this for all of the hypotheses. It is now writing the code for every single one of those. It is going to run the code for every single one of those. It will get the results. It will then summarize or synthesize the result. Now it's saying that low income was one hypothesis. Makes no difference. Whether they are low income or high income. Active discussion of injectable treatment challenges significantly increases. Okay, those who had routine or follow-up visits, those are people you should be targeting. Okay, very good. Let's expose analysis. So now there are a few where there seems to be a hypothesis that is validated, a few where it is not. Let us summarize these results. [28:09](https://youtu.be/sSyBUSuLduQ?t=28m09s) And now it is giving me roughly the equivalent of a deck. Saying that, "Look, here's an exact summary. Focus on patients who are struggling with injectable therapies, who are exhibiting signs of being overwhelmed, or discontinuation history. These are your top candidates. Have your healthcare provider actually actively discuss injectable treatments and challenges," and so on. And if you wanted a slide deck, you could take this format, copy-paste it into a slide deck, and your presentation is done. This process took five minutes. Me slowly explaining this to you. Five minutes to do roughly the kind of work that when I was at the Boston Consulting Group, would have taken us five weeks working about 14 hours a day as a team of a project leader plus two consultants plus two associates. This may not be perfect, far from it. But you remember how fast LLMs are improving. [29:08](https://youtu.be/sSyBUSuLduQ?t=29m08s) So my former job as a BCG consultant is gone. Definitely by the time you get out of it. Of course, the job will remain. What I was doing is gone. [29:20](https://youtu.be/sSyBUSuLduQ?t=29m20s) What other job would you like to look at? What other job would you like to see me vanish? Financial analysis. Let's take a look at financial analysis. This is a fun one. Actually we are doing a project right now. Let me talk about this. This is for a private equity fund. They get reports from all of the private equity companies that they invested in. These companies have very long, detailed performance reports. How is their operations doing? How is their finance? How is their segment-wise performance? Financial reports, market reports, all kinds of things. A typical presentation is about 400 pages. Hard for a human to read all of these. A nightmare to do this at scale, which is what a private equity firm does. So they have analysts, and that's their job. [30:19](https://youtu.be/sSyBUSuLduQ?t=30m19s) This project is in its current stages. When that happens, the role of those analysts is going to be a very interesting one. We have a few analysts working with us and they are looking at the application as it's evolving, with great interest. But let me show you a slightly different application. Supposing we took credit research reports from several public credit research sources and fed it to large language models and said, "You have this as a source, you can refer any one of these." Now, what does a financial analyst do? Financial analyst has someone from their management team asking a question, which they then do research and come up with answers for. An example of such a question, and you're welcome to suggest any other question, but I'm going to pick one of the questions here. Let's say, "How is consumer spending linked to credit market conditions?" [31:16](https://youtu.be/sSyBUSuLduQ?t=31m16s) Let me pick that as a question. Now what it's done is read through a whole bunch of documents. This 101592603.pdf page 219 has some details about the industry credit outlook in 2024, which it has read through and found it to be relevant. Similarly, here are a bunch of other documents. Some documents are less relevant, only 80% relevant. Some documents are more relevant, 99% relevant. And having read through all of these, the summary is that consumer spending is fairly closely linked to the credit market conditions. Tightening the credit environment and high debt costs can constrain the household borrowing and the disposable income. So you can have lower consumer confidence. And what we're seeing is elevated interest rates and refinancing challenges, blah, blah, blah. We already have reducing household savings. [32:10](https://youtu.be/sSyBUSuLduQ?t=32m10s) For a person who is a financial analyst, this is interesting. This is exactly the kind of stuff that they would share. But I'm not a financial analyst. So, you may find this more relevant than me. I'm someone who posts on LinkedIn for fame. So, let me get this as a LinkedIn post. And in a funny language. And get this result. "Secret sauce behind consumer spending," with emojis and all. "It's all about the credit market." Actually, why am I doing this? Read aloud? Read aloud is not there? Okay, now I'm only enough. Okay. So, I will read it aloud. "So when the credit conditions are tight and the borrowing costs skyrocket, consumers tend to tighten their wallets. That makes sense. High debt costs and limited financing options make it harder. When credit is cheap, then consumers spend more, boosting the economy. So if the credit market is feeling grumpy, our shopping cart might feel lighter." Ah, okay. Fine. Good. So I should know whether the credit market is good or bad to figure out if my retail expenses are likely to go up or not. Makes sense. [33:17](https://youtu.be/sSyBUSuLduQ?t=33m17s) And we can then ask deeper questions. So, "How is the credit market in 2025?" So, "How are the credit market conditions in 2025?" Now, the thing about this is normally I would have a financial analyst who will come up with a very serious report. And then I will have to get one business analyst to come and say, "Please explain this to me in English," so that I can understand it. And then they will translate it. Now I have the translator and the financial analyst for the price of about 0.2 cents. How much will they pay you as a financial analyst without the translation capability? [33:54](https://youtu.be/sSyBUSuLduQ?t=33m54s) Stuff like this is starting to happen. Not starting to happen. Stuff has already happened. What other job shall we vanish? [34:05](https://youtu.be/sSyBUSuLduQ?t=34m05s) Contract risk. Risk management. That's a nice one. Actually, they're all nice. Contract analysis. So we were working with authors. Authors sign contracts with publishers. There are a whole bunch of clauses, like, "You should complete the book. If you don't complete the book, then we have the right to take over whatever you've completed, give it to another author." During that time, "You should not get involved in any scandal that will make sure that the book sales will go down," blah, blah, blah. Lots of conditions. And we want to make sure that all the contracts have all of the conditions covered. Now, every department will have its own style of writing contracts. Every author will have their own style. The publisher would have acquired another publisher. That publisher would have had historical contracts. They will continue. It's a messy business. [34:57](https://youtu.be/sSyBUSuLduQ?t=34m57s) Contracts are extremely interesting to read as you know from every legal document. Fascinating English. So, what if we could create a wheel like this where for each of these contracts on the right side, we could look at what are all the risks? Is there a force majeure clause, which basically says, "If there is an act of God, then are we still okay?" Is there intellectual property covered? Is there a non-compete agreement? So it turns out that this particular contract too does not have hidden or additional costs covered as part of the contract. There is no confidentiality to protect sensitive information. There are no there's no risk coverage of disparaging statements and reputational risk. So the author is free to say the publisher is a terrible publisher. All of these things are not covered. Or here's another contract where there are other bunch of risks, and so on. [35:47](https://youtu.be/sSyBUSuLduQ?t=35m47s) How does one come up with stuff like this? And here's the same thing, by the way, in slightly easier readability. So it says, "Contract 1 does not cover breach of contract, but contract 1 does cover deadlines and delays because in section 2.2, it says the author shall deliver in six months." It covers copyright ownership. "The author has the ownership of the copyright." It doesn't matter who has ownership, as long as the clause is covered, you're fine. So not only do we know that the clause is covered, we know where the clause is covered, if it is covered. Now of course, if it says the clause is not covered, maybe the LLM is hallucinating. So we double-check. We have a bunch of other techniques. But we can solve this problem. [36:25](https://youtu.be/sSyBUSuLduQ?t=36m25s) Now, how does an LLM do this? Turns out, it's actually not very hard. All you have to do is paste a contract, let's say contract 1 here. You give it all the terms that you want to check for and say, "Is the term there? Is the term not there? If it is not there, then mention it. If it is there, mention where it is covered." So let me add apart from breach of contract, let's say, multi-country jurisdiction, which basically means I can sue you in any court I want. And click on analyze. So now what it's doing is going through each of these terms, checking if that is part of the contract or not. And multi-country jurisdiction, it did not find. Breach of contract, it found. Specifically, in the termination section, this is the piece of text that was there. These it did not find. These it found. Put in any piece of text, it will do the analysis, get the result. [37:20](https://youtu.be/sSyBUSuLduQ?t=37m20s) The power of LLMs has been amplified by the fact that it is a programming interface. You can program it to do what you want. What I mean by program is, tell it it will follow your instructions. And the programming language seems to be English. Which means that anyone can program. And that's powerful. Which means one more job that's at risk is a programmer's job. You can have it write a program, run the program, deploy the program, all at one shot. And if you want to see that, time permitting, I can show you that as well. [37:54](https://youtu.be/sSyBUSuLduQ?t=37m54s) But that is probably all I will have time for to show you. Keep in mind that I can pick any one of these and more. These were just all I had space for on the page. And these jobs are going to vanish. [38:11](https://youtu.be/sSyBUSuLduQ?t=38m11s) Let's talk about the skills. I'm actually going to show you one more job that is going to vanish. Let's take that form that you filled. This form which is the skills form, let's see how many using LLMs for learning. This is probably the one. LLM impact on skills. We have 31 responses, which is great. And I'm going to create a new spreadsheet. And we have a whole bunch of these skills. So, I'm going to take all of these skills and do, so this is the first part, which is what skills will become less important. Don't worry if you can't read it. I'm not even going to try and read it. I'm now the modern AI-enabled market researcher who basically hates doing work. [39:07](https://youtu.be/sSyBUSuLduQ?t=39m07s) The idea is, one more thing, at least for the next few years, the general trend will be, trend is, use AI, don't talk about AI. One, people will say, "Oh, AI did it. You didn't do it." They won't pay you as much or give you as many marks, which may matter more in the next few years. They will dismiss your work even if the AI can do a better job than you can. So over the next few years, the premise is, get the AI to do all the grunt work. Validate it. I'll talk a little bit about how you can make sure that the AI is actually doing good work. But please take credit. At least take credit. [39:48](https://youtu.be/sSyBUSuLduQ?t=39m48s) And now let's paste all of this into an application that I built, which actually I should just modify a little bit. Remove all the quotes. Sort, then sending. Okay, yeah, copy all of this. What I've done now is, put in all of the things that people say will become less important, and cluster these. Now what this clustering does is, takes all of the things that are very similar to each other and groups them. Now I have here a reasonably large cluster of half a dozen things. Coding. Lots of people are saying coding is going to be less important. Content generation is going to be less important. What are these? Data analysis is going to be less important. Yeah, makes perfect sense. Now this slider basically lets me decide the degree of similarity. So, that's fine. But now I again I'm lazy, right? So I'm not going to sit and read through all of this content. I just want to entertain myself, so I wanted to see what those were, but I wanted it to cluster. [40:47](https://youtu.be/sSyBUSuLduQ?t=40m47s) So, "You find the topics. Give me maybe a dozen topics and use a nice model to name them." And find the topics. So what it's done is clustered them. It's saying there's a cluster of fact-checking, editing, coding skills, blah, blah, blah. 11 skills that seem to have come together. And it says, "I'll call them creative content skills." Data entry tasks is another cluster that seems to have come together. Graphic and web design, and so on. So if I look at this, the list of clusters that people feel, or the list of things that people feel are going to be less important, are that. I've done my market research. I have done the market research. I'm going to take full credit for this. And also be patient. Don't obviously tell your boss. Within a few minutes, wait a few days. Play Minesweeper in the meantime, or Minecraft, which is even better. [41:45](https://youtu.be/sSyBUSuLduQ?t=41m45s) The flexibility of this is incredible. So at that point, your boss will say, "Look, you've given me skills. I don't want skills. I want job descriptions. I want to know what kind of roles to hire." Go back, take a few days, do some research. But then what you say is, these clusters should represent job roles, not skills. And then find the topics again. So it will do the clustering, naming the topic. Content creation roles, data entry clerks, graphics and web designers, customer service representatives, traffic control managers. Okay. That's more useful. So now you've provided me as input your thoughts on what kinds of roles will become less important. [42:36](https://youtu.be/sSyBUSuLduQ?t=42m36s) Let's look at what kinds of roles will become more important. Sort. Sort. Paste somewhere. And cluster again. I'm still going to manually look at it because it's always interesting to get a feel for the data. But there seems to be one big cluster. Before I select it, any guesses on what this big cluster that everybody seems to agree about is? There was. Yeah. Probably an easy one. I'm not so sure. I'll come to why. Or rather, the term seems to be moving towards context engineering. Prompt engineering seems to have the connotation that you say, begin with, "You are an expert data processor," or tell the LLM, "This is very important for my career." Emotionally blackmail it so that it will have a higher accuracy, stuff like that. These used to work. The models have become smarter. Not only do they not work anymore, in some cases, they actually make the LLMs perform marginally worse. How do I know this? I ran a test across several models, several data points. Looked at the quality. Found that for newer models, it's actually worse. [44:00](https://youtu.be/sSyBUSuLduQ?t=44m00s) So, prompt engineering, or rather, the prompt engineering techniques that were actually true a year ago, are no longer true. A big part of prompt engineering was and is pseudoscience. Meaning, nice term, everybody can say, "I'm a prompt engineer." You don't need a degree for prompt engineering. But over time, the distrust for that has been building and will continue to build. So the new term that has emerged is figuring out what information do you need to give to an LLM for it to do a better job. And that seems to be gathering momentum as context engineering. It may not pick up as a term, but something to keep in mind. [44:41](https://youtu.be/sSyBUSuLduQ?t=44m41s) Let's take a look at the topics. Again, I want a dozen roles. So there's going to be an AI operations specialist who will know how to use AI better. Okay. An emotional intelligence coach, ethics and compliance officer. Yeah. This I completely agree with. Compliance in particular. Prompt engineer, maybe not. Education and counseling professional. Okay. Software developer. Interesting. This was coding on the other side as well. Customer support specialist, LLM researcher. That's roughly what I do actually. Marketing and business analyst. That's roughly what the LLM was doing a short while ago. What is what the LLM is doing right now. Not so sure. And so on. [45:22](https://youtu.be/sSyBUSuLduQ?t=45m22s) What does research say? So there is this report by the World Economic Forum, where they were looking at which are the skills. See if I can right click on this, open image in a new tab, and zoom it out. X-axis, what are the skills that are important today? Right side, more important today. Y-axis, what skills will be important in the next five years? Top is more important in the future. Top right or think of it as a diagonal. If it's on the diagonal, out here, it's important now, it will stay important. Bottom left diagonal, if it's it's not important now, it's going to stay less important. [46:10](https://youtu.be/sSyBUSuLduQ?t=46m10s) Let's look at the top left, emerging skills. Networking and cyber security, not networking. Very interesting. And I completely agree. Cyber security is not an area I know anything about. It's an area I hate. And clearly seeing potential for it. And I'm going to have to just sit down and start learning about it because people are concerned about what LLMs are going to be doing with all of the data that they have. Or the person who runs the LLM, what are they going to be able to do with that kind of data? Environmental stewardship. Not sure. Design and user experience, just barely picking it. Now we don't know if these are right or wrong, but this report, which seems to be fairly well respected, says, "Here are a few skills that seem to be more important in the future." But clearly AI and big data, technological literacy, creative thinking. These are important and will continue to be more important. Analytical thinking as well. But note that anything that's kind of towards the left of the diagonal is going to be more important than people think it today. Anything that's on the right or bottom of the diagonal, bottom right of the diagonal, like analytical thinking, it's going to be less important than people think it is today. But still important. [47:29](https://youtu.be/sSyBUSuLduQ?t=47m29s) Take a close look. What is clear is, stuff like grunt work will go away. Which is a good which is pretty good news for you, right? You don't have to do the grunt work. But what that means is, the stuff that is left behind beyond grunt work is what remains for you. Now, grunt work is actually very easy on the mind. You can just do it in a routine way, day in and day out. You want to chill, you just do it. Somebody said, "Look, here is broadly the outline of the presentation that you have to make. Can you convert this to PowerPoint?" Happily, I can do that for three, four hours. Don't have to apply my mind. I'm doing something that is productive. Now that gets taken away from me. Because that guy will sit and do it in an LLM. [48:12](https://youtu.be/sSyBUSuLduQ?t=48m12s) He's my team member. Now if I ask him for work, he will actually ask me to do something that requires me to apply my brains. I don't know if I have any. If I do, I'm too tired to apply them. I don't like work. Challenges of the new era. You're going to have to face it. But in the middle of this, you also sit and think, "In this particular course, should I be focusing more on X versus Y?" Keep this as a guide. And I don't mean this chart, but rather what you had put in as your survey entry. That will give you a sense of what you believe. And keep correcting that belief, test the hypothesis, see if it is valid, if not, improve it. On where you might want to place more attention, and where you might want to place less attention. The faculty will be evolving with you. Nobody really knows what's going to be more and less important. So, they'll be making changes as well to the content as it evolves. Work with them or don't work with them. Work independently. However, but at least know that the world is changing in a way that you need to take some decisions. [49:17](https://youtu.be/sSyBUSuLduQ?t=49m17s) And learn using LLMs. This is a critical skill that will apply irrespective of the kind of education that you're going to have. The obvious use of an LLM is, you get a problem, you feed it the problem, take the result, give it. This you've already done. I don't need to teach you to do it. Everybody will tell you it's a bad thing to do. It is not a terrible thing. It may be a bad thing, but there are worse things. So okay, you have 10 assignments. You don't have time to do all 10. At 3:00 AM, you're not going to sit and apply your brains. You need sleep. Please sleep. Tell the LLM to do the assignments, submit it. This is going on record, but I'm not faculty here, so I can say what I want. [50:04](https://youtu.be/sSyBUSuLduQ?t=50m04s) But when you can, and as often as you can, to make sure that you do something more than the LLM, and are actually employable, try out some of these. Use the LLM to challenge you. Use it as a sparring partner. What do I mean by a sparring partner? Upload all the course content. Upload all the lecture notes. Have it quiz you. Have it challenge you. See if it can ask a question to which you answer it. And it can find all the flaws in your answer. Try and beat it. See if you can find the flaws in its answer. Beat the LLM at its own game because it's like playing a chess match with a computer, that's how you improve your skill. Play chess matches with it, your skill will improve. [50:50](https://youtu.be/sSyBUSuLduQ?t=50m50s) Sprint. Let it create the output. Treat it as a draft. Send it to another LLM and have that critique. Read that critique. Tell it what are the hidden assumptions we are making. See, this is how you will be reviewing LLMs work in the future. So this is a good practice. Reviewing work, whether of humans or of LLMs, is good practice. And then once you have that, write your own version. And I don't mean necessarily write fully, but incorporate, absorb, in the process it's flowing through your head, so you improve. Use this as a sequence, as a sprint sequence to accelerate your work. [51:25](https://youtu.be/sSyBUSuLduQ?t=51m25s) Third, if you find that you've written a prompt that has given some fantastic output, save it. Over time, you will gather a prompt library that is useful. It'll get outdated, but you will build the habit. Rehearse stuff. You're going to present something, you're going to be delivering a lecture, you're going to be submitting an assignment. Have it quiz you on the output. Imagine that you are making an investor pitch. Have it quiz you like a CFO, like a regulator, and so on. And give feedback on your output. And when I say, "Give feedback," I mean literally speak to it. Take a mic, talk to it. It talks back. And you can do this while you're walking. You don't have to do this while sitting in front of a computer. Use your phone. This morning, I was walking, spent about an hour and a half on the road. And during that time, 80% of what I was doing was talking to ChatGPT. Making some changes to this presentation, which it did from my phone. Have it writing a program, which it deployed. A bunch of other things that I was planning. I'm planning to write a comic storybook. It gave me the script for the comic storybook. All of that you can do just through voice. Practice. [52:34](https://youtu.be/sSyBUSuLduQ?t=52m34s) And at the same time, I'm telling you to do more of this, more of this, more of this. What should you do less of? Do less of grunt work. If it can be done by the LLM, at least learn how the LLM will do it. Try it. Use it, don't use it, that's your choice. Definitely review it, that is helpful, unless you absolutely are short of time. But don't do grunt work. [52:59](https://youtu.be/sSyBUSuLduQ?t=52m59s) Think about what your CV is going to look like in 2027. What is it that is going to get you a job when LLMs can do a significant portion of many jobs that people are doing today? Keep that at the back of your head and work on it. I wish you all the very best. You have a very few, very interesting years and a few very interesting decades ahead of you. Make the best use of it, but have fun. Life is very chaotic. Have fun. Thank you. [53:44](https://youtu.be/sSyBUSuLduQ?t=53m44s) The trick is pausing after saying thank you so that you sort of get to know when to clap. And this is the time when you ask questions. I think we have time for. [55:16](https://youtu.be/sSyBUSuLduQ?t=55m16s) **Question**: How do you think roles with regard to people will change? People management roles, how will they change? [55:24](https://youtu.be/sSyBUSuLduQ?t=55m24s) **Answer**: I don't know. Probably not much. What a lot of the research that I'm reading says is that networking will become more important in the era of LLMs. Empathy will become more important. But I'm also hearing the opposite, which is empathy will become less important. Why? Empathy will become more important because AI everywhere, so people value the human connection. Fair. People are saying, AI is so good at empathy that you have a higher barrier to fight against. And empathy will actually become a sort of a commodity because AI is shockingly empathetic. They will let you talk for however long you want. They will say you are fantastic. How much more empathy can you give than that? So, could go either way. [55:16](https://youtu.be/sSyBUSuLduQ?t=55m16s) They say, "Therefore, the ability to provide feedback to a person is important." LLMs are giving me better feedback than before. I don't know. I'm not sure. Are there any skills that I've seen, that I feel, will help me survive? No. But at least to keep up and compete with LLMs, I'm learning to be nicer to people. [55:51](https://youtu.be/sSyBUSuLduQ?t=55m51s) I have a list though, which I'm, while whoever wants to ask their next question, thinks of their next question, let me see if I can find. Yeah. Here is my current list of things I should probably learn because they might grow in importance. And negotiation is on that list. Change management is definitely on that list for the next five years. Learning about local cultures might end up being there. Remote working seems to be a reasonably important one. Is ethics and AI dilemmas, yeah. Very important. But I don't have a problem if AI is, if AI destroys humanity and takes over the world. So, I'm just adjusting to that. But yeah, this is my list. [56:37](https://youtu.be/sSyBUSuLduQ?t=56m37s) **Question**: Do you think there'll be a line between truth and lie in the future? [56:38](https://youtu.be/sSyBUSuLduQ?t=56m38s) **Answer**: Yeah. Is there a line between truth and lie now? Meaning or in the past? [56:44](https://youtu.be/sSyBUSuLduQ?t=56m44s) **Question**: If we ask it to give a positive feedback, it is giving us positive feedback. And if we are asking it to show its flaws, we are getting negative feedback also. [56:44](https://youtu.be/sSyBUSuLduQ?t=56m44s) **Answer**: Yeah. So basically, if you ask it for good positive feedback, you'll get positive feedback. If you ask it for negative feedback, it'll give you negative feedback. That's an excellent question. Was this a truth or was this a lie? I don't know either. Certainly is a good question. It is certainly also out of character for me to say, "Oh, this is an excellent question." I may have never done this before. If with humans, we can't tell the difference, and humans have existed for millennia. With LLMs, we can't tell the difference. And that may be because LLMs are like humans. That may be the nature of truth itself, that truth and lie are subjective. So, short answer, I don't know if it is any different. I don't know what truth and lie are at some level. [57:44](https://youtu.be/sSyBUSuLduQ?t=57m44s) **Question**: How do I feel about the environmental ramifications of LLMs, meaning LLMs are consuming too much? [57:44](https://youtu.be/sSyBUSuLduQ?t=57m44s) **Answer**: Or, how you feel about it? Okay. So, the energy cost of LLMs, let's put it this way, one chat is, I think, approximately equal to one mug of hot water in a bath. No, or less than that. One spoon, not mug. One spoon of hot water, whether you are to drink it or to take it and so on. That's a quantitative comparison. Now, if instead of asking an LLM, I were to do the work myself, then I'm likely, even as a human, to burn more carbon dioxide, just by breathing, let alone everything else. You know, using a computer, using the AC, using all of the infrastructure that society has to provide. And therefore, end up increasing the environmental cost of the same task. [58:58](https://youtu.be/sSyBUSuLduQ?t=58m58s) So I don't think there is any doubt that for a given task, using any kind of automation, and certainly a powerful automation like an LLM, is more efficient. What people are saying, however, is, because LLMs are so cheap, what we will do is, earlier we would have thought a lot about a human doing this task or using some other mechanism. Now it is so cheap, we will do a lot more. And therefore, there will be a lot more energy that is burnt. Yes, more energy will be consumed. Hopefully, that more energy is going to be consumed to save energy in other places. Will it, will it not? Not sure. But going by historical trends, we know that people make several mistakes, and people are energy conscious in aggregate. So overall, we have become more energy efficient in the last 20, 30 years. So, perhaps the impact of LLMs will likely be that there will be less we will become more energy efficient, not necessarily less energy consuming in the short run. Certainly not uniformly so. But the net benefit will certainly be more energy efficiency in aggregate, is my guess. Yeah. [59:51](https://youtu.be/sSyBUSuLduQ?t=59m51s) **Question**: Sir, I have a small comment. As teachers, how do we prepare to correct LLM generated assignments? [59:51](https://youtu.be/sSyBUSuLduQ?t=59m51s) **Answer**: That was the slide and the appendix. Which I knew I wouldn't have time for. You encourage them to use LLMs. We need to prepare ourselves, right? Totally. And it is a fascinating topic that I'd love to talk for another half an hour or more.