# Transcript

**Host**: [00:00] Welcome back, folks. We have the next talk ready with us. You've checked the topic on the screen for the talk, which is Finding Actor Lookalikes with Multimodal LLMs. And to do this talk, we have with us Anand S. He's an **LLM psychologist at Straive** and an amazing fact about him is that he's an amazing **Calvin and Hobbes fan**, so definitely something that you can check out with him a little later offline. But all in all, can we start with a huge round of applause for Anand?

**Anand**: [00:33] Quick quiz. Who would you like to find the closest lookalike for, at least in Bollywood? Raise your hands for Ranveer Singh. One, two hands, three. Deepika Padukone? Okay, double hands. Shah Rukh Khan? Wow, he beats Deepika Padukone! I did not expect this. Alia Bhatt? Okay, so we want Shah Rukh out there. Me, I was gunning for **Irrfan Khan**, actually. Who can replace—I mean, no one can, that's another story—but I mean, facially, who might be the closest?

[01:14] So I was very curious about this and I thought, okay, chalo, and this was 2021, and I thought maybe we can use AI to solve this problem. There are a couple of issues though. Around the time when LLMs came in, they weren't anywhere near as reliable. I started asking them for instance, "Can you guess a random number?" Here is how they perform. Let's take **Claude 3 Haiku**. When I asked it to guess a random number, I expected it to guess a random number uniformly. No, I didn't actually, I knew it wouldn't. **But what I did not expect was that the number that it would guess almost consistently every time is 42.** We know why. The answer to the ultimate question of life, the universe, and everything. So clearly, Claude is trained on a lot of science fiction, and which is fair.

[02:08] OpenAI GPT-3.5 Turbo in particular was picking 47 over anything else, though 42 was also fairly popular. It liked 57, it liked 37, it liked 71, 72, 73, 74—all the 70s. You kind of get where this is going. If you want me to guess a random number, I will guess a very prime number, very unusual number, exactly the way a human guesses. If you've seen the Veritasium video on how humans guess random numbers, **37 is the most often guessed random number**, far from random. And you also see that it doesn't guess any single-digit number, doesn't guess any multiples of 10, doesn't get any repeated digits—11, 22, 33—all no. No 100. **Exactly the same kinds of biases that a human exhibits.** And I'm supposed to use this in a reliable way? That was part of my problem.

[02:59] But the other part of my problem was the cost. If I had to run an LLM on a GPU, even now I'm looking at burning something around the order of $5,000 in two months. At that time, it was way higher than that. I talked to my co-founder and said, "Yaar thoda [English translation: friend, a little bit] I want to play around with actor lookalikes," and he said, "Anand, we're in the middle of a fundraising initiative. $5,000, two months of your time to find out who's the actor closest to Irrfan Khan? Don't you think we have better things to do?" I said, "Okay, fine."

[03:31] And I waited, and things have changed a little bit. Actually, two things have changed. First, we actually got acquired, which helps. And secondly, **we are seeing that the price of LLMs is going down dramatically.** So let's take a look at this graph. The x-axis is the cost per million tokens. And so there are some models that cost as much as $30 per million tokens—that's the GPT-4 models. There are some that cost as little as, let's say, $1 per million tokens—that's BPLX-70B Online, and goes on on a logarithmic scale to something much lower. So there are expensive and there are inexpensive models. And on the y-axis is the Elo score. For those who may not be aware, LMSYS has a crowdsourced setup where you can ask any question to two chatbots, and they don't tell you which chatbot it is. You then say A is better or B is better or both are good or both are bad, and it takes them in like a chess match and calculates the Elo score like grandmasters have. And this is the current Elo score. **o1-preview is right up there on top**, having come up fairly recently, followed by ChatGPT-4o Latest, followed by o1-mini, and so on.

[04:52] So that is what we have on the y-axis, which effectively indicates the quality of the model. And this is something that has been changing over time. So in August '23, this is what it looked like. There were a bunch of GPT-3.5 models, there was also Claude 1, there was another Claude model, and that was the state of the art at that time. Over time, a few things happened. Firstly, you'll see that the frontier gently improved, that is models started becoming both better—that is moving up—and cheaper—moving to the left. Which means that there are a set of models that are **Pareto optimal**. Let us take for instance, OpenChat 3.5. Now I would never use, let's say, Qwen 1.5-7B in Feb '24 because I have at least this model which is both cheaper and faster. So why would I? In short, I would never pick a model that is not in green. In other words, something that has nothing both above it and to the left of it.

[05:46] This evolved. So Claude 3.5 Haiku came in and that made a big jump, it shifted the lever a fair bit, so I had all of our teams move to Claude 3.5 Haiku. And then Llama 3 caught up, which was a great move. And then Gemini 1.5 Flash came in, again moving the needle a fair bit. So we said, "Okay, now let's move it to this one, that was great." And then a whole bunch of other models caught up. And then GPT-4o Mini came in a few months ago and changed the whole equation again, making it much cheaper. And then Gemini Flash moved the needle again, making it half the price of GPT-4o Mini. Crazy. **The speed at which the cost has come down, and the impact of this cost to give you a feel for it: $30 per million tokens is the bunch of red boxes that you see on the right side. GPT-4 Turbo is one-third the cost of that. Claude 3 Haiku at 25 cents is 1/40th the cost of that. And Gemini 1.5 Flash at 7.5 cents is minuscule. This is roughly the difference from bottom-left to top-right between a $10,000 budget versus a $4 million dollar budget.**

[07:01] With this kind of cost change, I had no problem going back to Naveen and saying, "Yaar [English translation: friend], this I can fund out of my own pocket, I don't really need you to give me any kind of approval." But another thing that we realized was **embeddings are even more incredible**. For those of you who may not be aware of embeddings, think of them this way: It's take a bunch of text, you can convert it to a set of numbers which have an interesting property that two numbers that are close to each other are similar in meaning. But just as importantly, these have the property of extremely low cost—about 1/10th of Claude 3 Haiku, which in itself is a very cheap model. Two cents is what text-embedding-3-small costs, and you can run this on a Colab notebook perfectly fine with models that are of even better quality.

[07:48] So practically free. But the other advantage that I get is that **they are quantitative** in some sense. Firstly, if I ask "What is the embedding of a piece of text?" by and large it comes back with the same response. I say "by and large" because I thought it always came back with the same response. Heck, I teach a course at IIT Madras, I set an exam paper assuming that it would always come back with the same response, and I found that roughly in about 2% of the cases it does not, and therefore I had to give the students full marks for getting the wrong answer. But that's a different story. They by and large come back with the same answer, so experiments are mostly reproducible unless somebody changes the model behind the scenes. And therefore the distances between two words in terms of similarity is also reliable. Accurate? Interpretable? That's a second-order question, but it seems to by and large tie with intuition, and the reproducibility made a big difference. So I started using embeddings.

[08:47] Now, what exactly do embeddings look like? Let's see. What I'm going to do is build a small app that is going to take a piece of text and create an embedding. Now, this is the kind of thing that you would use an AI code editor for. I'm just going to paste this text and run it. So I'm using Claude 3.5 Sonnet to write a piece of code. This is something that you can do on Claude.ai as an artifact, or ChatGPT, or wherever. It would write the code and run it, except in my case I find it a little more convenient where after it writes the code, it also shows me the output. And then I can type a phrase like, let us say, "apple". And it would actually call the API, generate the embedding, and you can see that this embedding is a whole bunch of numbers. Lots and lots and lots of numbers, about 1,530 odd in this particular case. And these numbers are what constitute the meaning of this—of this phrase, "apple". And it could be a longer phrase: "Apple has a pie." And that would give me an embedding.

[10:01] Now the thing about these is that the phrase, let's say, "apple" and the phrase, maybe let's say, "orange" are likely to have embeddings that are similar to each other. Now what do we mean by "similar"? Think of it as if you plot them as a point in a multidimensional space and calculate the distance between them—you probably calculate it as a **cosine similarity**, that's one of the ways of calculating the distance—it would come out to be a reasonably small number because apple and orange are likely to be close to each other.

[10:32] A quick note on this, by the way. I'm fairly sure the majority of you are moving towards LLM-based coding. **Me, I have moved entirely towards LLM-based coding.** I just stopped coding a long time ago. Now I've stopped even reviewing half of the code. This particular deck in its entirety was generated using an LLM. Beyond that, the only thing that I did was the last 15 minutes of "Oh, maybe we should do this, maybe we should do that." That is purely obsessive-compulsive disorder. The deck was perfectly fine working as a reveal.js script, and you'll find the prompt in the GitHub repository. If you haven't switched over to AI coding, please do. If you aren't using AI to generate large swaths of code, then please shift to that as opposed to, let's say, just completions.

[11:22] But let's move back to embeddings, which are the fun part of it. What we're going to do is do a small exercise where we're going to take a bunch of movie quotes and cluster them. The quotes look like this. I have, let's say, from *Beetlejuice*: "Beetlejuice, Beetlejuice, Beetlejuice!" or "It's showtime!" or—let's pick something that I kind of recognize over here—ah, "Bond, James Bond," "Martini, shaken, not stirred." There's a whole bunch of these, about 150 of these quotes. Let's see if we can take embeddings, find out which quotes are similar to each other. I'm going to copy that and I'm going to paste this in a small app that I built that—let's reload this, paste it, and just say "Cluster".

[12:07] What it's doing here is taking all of those embeddings for the 150 quotes that I have and plotting each one of those as a point in that multidimensional space, calculating the distance between every pair of quotes and then seeing which ones are close to each other. So if I had to find the—the quotes that are most close to each other, these two: which are "It's showtime!" and "It's clobbering time!" Yeah, very similar, absolutely. These two: "They're here!" and "They're back!" Okay, that's also fairly similar, fair enough. And we can then move down the list. What else is similar? What else is similar? So there's all of these quotes which seem to be in the same bucket. Yeah, it's "Oh, John," "No, John," "They're here." Yeah, you can kind of see how it leads to a progression of similarity.

[12:53] And then there are other clusters that form. So it's almost like there are a whole bunch of quotes, they have their own network structure, they have their own clusters, and there are some that eventually get left out. So this: "Machete don't text" and "There can be only one." These are practically outliers. At least in that 150, there was no quote that was close enough to these compared to the others. So:
1. Outlier detection now becomes a whole lot easier.
2. Cluster detection becomes a whole lot easier.

[13:25] Hold on. If we can do cluster detection, then maybe we can get a sense of **what are the different types of movie quotes in this list of movie quotes without actually having to sit and read and classify all of these, right?** And that is actually pretty straightforward. What we can do is do a **K-means clustering**, and let's say I want maybe 15 topics, and I'm going to ask, say, GPT-4o Mini to then name these clusters. So what it's doing here is creating a whole bunch of clusters, about 15 of them as I had requested, and for each of these it's giving a name. Now I could have chosen maybe 16 clusters, 14 clusters, however many clusters—it's entirely arbitrary and up to me as you would expect K-means to be. And the names that it gives are by reading some of these and saying, "Okay, it looks like these are inspirational life lessons," or "Apropos, tomorrow's another day," "My mama always said," that's *Forrest Gump*, "There can be only one," that's probably not inspirational. And it's also saying, "Okay, yeah, maybe this is a bit outside the current cluster, I may not have put it in the right place." But there are philosophical reflections, classic movie references, heroic phrases, high-energy showdowns, intense outbursts.

[14:34] In short, with embeddings and almost entirely with embeddings—which if you remember is practically zero cost and highly reproducible—**we are able to take text, automatically figure out what groups they belong to, create a taxonomy out of it, and if you want, later on change that taxonomy and re-validate and say, 'Now put them against this particular taxonomy,'** which is obviously fairly powerful.

[15:00] How does this work? The Jupyter Notebook which is linked from the GitHub repo is something that I'm going to very briefly take you through. So there are a couple of functions that are of relevance. The first is the clustering part. This is the thing that figures out how to create this cluster—well, not exactly, but yeah, it's probably closer to the thing that figures out which buckets to put each of these quotes into. How do we do this? What we do is take the embeddings of each one of those, share that as docs, and for each of those docs, we get the embeddings in a cached way—there's no point hitting the LLM multiple times for speed perspective—and then fit the model using K-means as an algorithm. Why **bisecting K-means**? I don't know, I just read that it was slightly better. It apparently looks at the data a little bit before it does the partitioning and therefore tends to get slightly better results. But by and large, this is just K-means. And then we calculate also the distance for each of those quotes against the center of its cluster and see how close is it, how far away is it, which will also give us a sense of do each of these quotes really belong in the cluster. So some of these like "Smoking" is probably slightly better off outside the cluster, this one also probably slightly better off outside the cluster, but many of the others are, yeah, positive and therefore belong in the cluster. **In short, embeddings plus K-means is all it takes to put any bunch of text into buckets.**

[16:42] The second thing on top of that is the naming, which is straightforward. All we do is take some of these examples and send it to an LLM saying, "Here are clusters of documents. Now give me two to four-word topic names for each of these clusters and capture the spirit of each cluster." Or say, "Give it to me in Hindi, for example." And if it was good enough, then it would give you the topic names in Hindi. In other words, the prompting part of it is all yours. [English translation: It's all your property.]

[17:14] Okay, that was interesting. The other thing that you can do with all of these is—let's flip it back into English because I read that better. Given a set of topics—so let's say we have here on the top right all of the topics that it had identified: Iconic Movie Quotes, Memorable Catchphrases, Classic One-liners. I don't want so many. Let's say I want to get rid of Iconic Movie Quotes and anything that are generic. I want only specific stuff. And taking these, now if I said, "These are my topics. I don't care what you think are the topics. Now you start putting these directly against those particular topics, classify them in other words." So what it does is takes the embeddings for all of the stuff on the right, all of the stuff on the left, creates a matrix, and you have the first one—which oh, looks a bit weird, so let's ignore that. We have the first one which is *Beetlejuice, Beetlejuice, Beetlejuice* falling under Creepy Movie Moments, probably. Everything else has slightly lower similarity. "It's showtime!" very strongly falls under High-energy Showdowns.

[18:25] So now I can pick my taxonomy, figure out which one of these it's most similar to, and solve the problem. And all it takes is actually a very efficient operation which in this case is the **dot product**. For most of these embeddings, they are structured in a way that if you take the dot product of the matrix of the embeddings of the documents and the matrix of the embeddings of the topics, you get the answer. In case you're not familiar with what a dot product is, just think of it as one function which is surprisingly fast even without a GPU. That will give you the distance between any two sets of—for any set of documents and any set of topics, which obviously means that we can now solve the topic modeling problem in an entirely automated way.

[19:12] But the best part of it is we can apply this to images as well. **Images also can have embeddings the same way that text does**, and one of the models that I've seen that works with image embeddings is Google's Multimodal Embeddings API. I'm using this because this happened to be the first one that I found and it worked well enough, so I've never really dug deeper into this. There are several other **multimodal embedding models**. I don't know what they are and I'd love your inputs on suggestions on which ones are better. The way this works is you send a request that passes—it sends a text or an image, you can even send it a video, and it will embed all of these. And the beauty of this embedding...

---

**Anand**: [19:59] ...of this embedding is that it embeds them all in the same space, which means that the distance between the photo of an apple and the picture of an apple is actually very low.

[20:14] Now a whole bunch of things start becoming possible. Meaning, I can take the picture of one actor and the picture of another actor and I can see how close they are. I can take the picture of an actor and I can take the word, let's say, "smile" and see how close the actor is to "smile," and so on.

[20:32] So let's explore this. We put together a small prototype that explores images, and I'm going to take Bollywood actors to begin with. So we have a hundred Bollywood actors. The x-axis is the word "smile," or the embedding of the word "smile"—whatever, it's the word "smile." The y-axis is the word "frown." And each of these photos are positioned based on their similarity to those two words. So what this is saying is this picture of Tom Holland is the closest to a smiling picture; Tom Holland's smiling the most. And this picture of Meg Foster is smiling the least. Clint Eastwood is totally frowning. Yeah, we get that. And the person who's probably smiling the least is Anya Taylor-Joy, maybe—who's this? Claudia? Claudia Jesse.

[21:27] We can start looking at who looks most African, who looks most Chinese. So according to these photos, Jackie Chan's face is the highest in terms of similarity to Chinese, whereas Zendaya's is probably the lowest. Morgan Freeman's is the closest to African, and you can see the African Americans jumping to the right side; you see Eugene jumping to the left side. And we can also start looking—and you can pick pretty much any word and start looking at the similarity.

[22:01] How does something like this get used in practice? We had a client who's a logistics provider, and they said, "We get a whole bunch of photos of drivers who are applying for our fleet. Sometimes there are duplicates. Sometimes they don't send their photos—in fact, a whole bunch of people send the logo of our company as their photo. Sometimes they're wearing tuxedos, not a driver's uniform. We also don't want them wearing that high-visibility vest; we want them against a plain background and so on. So can you help build a model that will solve this problem?" We said, "Yeah, sure, whatever. We will do training, we will do computer vision, blah, blah, blah," and we budgeted for something like a six-month exercise. They said, "No, no, no, it has to be a pilot; we have to do it in six weeks." We looked at our hands and said, "Okay, maybe we can try with LLMs." They said, "Okay, fine."

[22:50] **The sixth day, the project was done.** All we did was this: took the embeddings, and we found that if we take the word, let's say, "high-visibility vest," it has a good match. "Plain background," it has a good match. Now neither the client nor we knew what to do with the remaining five weeks. We said, "Can you do this enhancement? Can you do that enhancement?" We literally had to tell them, "Boss, these are once-in-a-lifetime occasions when the project actually gets done before the budgeted time. Let the team have a holiday." And it's surprising—I don't think the world has gotten used to the scenario where **we don't have to train models anymore to get to the result that we want.** The pre-training and some amount of prompt engineering works well enough.

[23:35] So let's start answering the question that we want, which is: which actors are most similar to each other? Who's the lookalike for Shah Rukh Khan? Out here, what we have is on the right-hand side the most similar photos and on the left-hand side the rest of the photos. So as I move the slider to the right, you'll find that only the most similar photos start coming. And the three most similar photos are two photos of Riteish Deshmukh and one photo of Shreyas Talpade.

[24:08] Right? I had once done a search of Riteish Deshmukh and Shreyas Talpade, and I wasn't the only one who thought that these two looked like brothers—very similar faces. You can start looking for outliers like before. Who has the face that looks most different from the rest of Bollywood? And if we move this little bit... I have to move the slider just the right amount to get a few people to come up over here. Yeah, okay. So three people come out as completely unusual Bollywood faces: Manoj Bajpayee, Tusshar Kapoor, and Kunal Khemu. Very atypical Bollywood faces.

[24:54] But let's answer your question. Who looks most like Shah Rukh Khan—or at least in those photos? If we, apart from Shah Rukh Khan himself, look at photos, this photo of Varun Dhawan seems to match Shah Rukh the most. I don't know if you'd agree, but at least in this set of photos, that's the closest. So now as a casting director, I have at least one backup.

[25:24] Who looks most like Deepika Padukone? Any guesses? Who looks most like Deepika Padukone? [Audience replies] Who said that, and who is that? [Laughs] Let's find out. Deepika... okay, a whole bunch of people at this level of similarity. But two people stand out: Priyanka Chopra—yeah, I kind of felt that they look similar—and at least one picture of Deepika looks like herself, so maybe as a second backup.

[25:56] Who looks like Ranveer? At the moment, no one at this level of similarity, so we'll have to dig a little deeper, but we find... okay, Vicky Kaushal is not too different from this photo of Ranveer. Yeah, similar beard, similar look, maybe, maybe. And Alia Bhatt... whole bunch of people that seem close enough to Alia, but the closest on this in terms of pictures is this picture of Katrina Kaif, who might be a decent lookalike.

[26:22] But to answer the question that I had almost two years ago: who could replace Irrfan Khan? And I think one answer clearly matches: Nana Patekar. Maybe. Yeah, even as an actor, probably. It's a loss that we no longer have Irrfan Khan, but his legacy will live on.

[26:50] But the message that I wanted to leave you with is primarily that **embeddings are really cheap. They are quantitative and therefore reproducible, perhaps even reliable, and they are multimodal.** Just use them a bit more. Do try it, do let me know how it works. My email ID is out there. Thank you very much. [Applause]

[27:22] **Host**: [27:30] Hey, that was a really fun talk. It's great to see you over the years at PyCon. The one question that I had was: using an image embedding model for this that also embeds text, so it's the same model—so I'm curious why it wouldn't rank spouses in similarity as well? Like, why is it just visual? Because if you think about an LLM, it might make connections at the visual level but also at the text level of, like, "This is this actor," and then the spouse would maybe rank similar. Or is it more trained like a diffusion model where it's just the reconstruction of the image that is the score that it's sort of training on?

[28:13] **Anand**: [28:13] I can give you a long answer to that, but I'll give you a summary of the answer. I have no clue. I have no idea how they're trained. I haven't even explored what similarities work and don't work. Great question. Thank you. I'm going to try that out next. I explicitly asked for photos, but I think the question is why would Deepika and Ranveer, who are often mentioned together because they are husband and wife, not be similar in the image embedding space as well? And I'm guessing it hasn't been trained that way, but I know nothing about how it actually has been trained. It's worth exploring if there is a site—and how shall I put it? Two people looking image-wise roughly equally similar, but one pair is husband and wife and another pair is not—are they going to come closer together? If so, how much closer? Worth exploring.

[29:11] **Audience Member**: [29:11] Sorry, another one which is: what data visualization... like, I saw LLM Foundry, but I couldn't find anything online.

[29:20] **Anand**: [29:20] LLM Foundry is something that we built. So basically, large organizations which are slightly bureaucratic—which is good for controlling my blood pressure—don't allow ChatGPT, Stack Overflow, Google, GitHub, and the likes. I don't know why there is an internal network, but coming back to that part, so we said, "Let's build something that is auditable, where you can decide which of these high-class citizens will have access and which of these low-class citizens will not." We built a router internally so that people could have access to ChatGPT. Because it's an internal router, you don't get access to it by default. We give access and publicizing it less because the less I talk about it, the less everybody will come.

[30:09] **Audience Member**: [30:09] And the visualizations, like Plotly, or like... the visualizations were really cool. Are they done in Plotly?

[30:14] **Anand**: [30:14] **All the visualizations are from some LLM where I just said, 'Do this, this is what I want,' and it generated it. I no longer code.**

[30:24] **Audience Member**: [30:24] Pretty cool.

[30:25] **Anand**: [30:25] Thank you very much.

[30:28] **Host**: [30:28] Thank you so much, Anand. Just as a token of appreciation, we have a small goodie for you. [Applause]
