# Transcript

**Nathaniel**: [00:07] I can announce you in?

**Anand**: [00:08] Oh, yeah, sure, sure.

**Nathaniel**: [00:18] Okay, guys, good afternoon, everyone. And thank you for joining us for another computer vision workshop. So, today we have Mr. Anand from Straive to actually go through with us what is possible with computer vision. So, actually, a big focus of today is not—

Hi, welcome. So, we'll be starting shortly. And the big focus of today's workshop is actually exploring what is possible with computer vision, with our agents, and just seeing what we can build together. Because a lot of the time that we learn in the classroom is very limited by just normal projects, just very normal learning, the things like the fundamentals. These are stuff that is already expected in your projects. What can you do with what it is to bring one step ahead? And today we have Mr. Anand to bring us to it. So, a big round of applause.

**Anand**: [01:23] Since this is a workshop, should—

[silence]

Sorry, that took me longer than planned. But because this is a workshop and not a classroom, **you're going to be doing the work, not me.**

So, let's start with a simple survey. You can see the QR code. You're welcome to scan that or you can just go to forms.s-anand.net. That's easy enough to type. You may want to open this on your laptops because that might make it easier going forward. But this is the link for all of today's material, and we'll be going through questions on the way.

Why questions? Two reasons. What I find is, when I was in high school, I was reading a whole bunch of books. I would read a book, and I'd find that it didn't teach me anything new. It made sense, so, "Yeah, I could have figured it out by myself," right?

What I realized later was that there are a hundred ways of saying things that make sense. Some of them are correct. Some of them are wrong. And I didn't know enough to be able to differentiate. So, one of the things that I've learned, which people have learned several times over the millennia, is **before you try and learn something, write down what you think you know, and then go ahead, test it out.** If it's the same thing that you thought, then, well, you got it right. You've learned something. If it's different, then you've concretely learned something. That's what we're going to be practicing today. We're going to be writing down what we think. And as we progress, let's see if that's in line with what we thought.

But the first question has nothing to do with what we'll be learning today. I'm just curious. What are you planning to do after you graduate? I ask this to a whole bunch of people, and I think our choices determine a big part of what we need to learn. If you're planning to study, then the kinds of things that you learn, or need to learn, from a computer vision perspective, are a lot more theoretical because you need the foundations to be able to go deeper. Whereas, if you're planning to get into the industry, then really you need enough to be able to clear an interview. Whereas, if you're planning to start a startup, then you need to get the job done, and frankly doesn't matter how you get the job done. Or if you're planning to take a break, you probably shouldn't be studying anything related to this. Or there could be a bunch of other answers. I'm curious to hear what you have.

But this is the question that I'm going to focus a little more on. So, I'm someone who hires data scientists. Now, today we are at that stage where an agent can do a lot of things. It's pretty smart. Now, on the margin, an agent costs less than a fresh data science student. So, the question then becomes, what should I hire a data scientist for? Oh, well, let's also ask the question, why am I hiring a data scientist? Part of the reason, frankly, is because I have a budget. And if I don't spend the budget, I won't get it next year. So, I'm going to hire data scientists anyway. That is the reality. In many ways organizations don't move based on necessities, they move based on supply and demand, and there's a lot of supply of politics, there's a lot of demand for politics, budgets. All of these things make a difference.

But since I do have the budget and I am going to hire, I'm curious what you think, and I will be sharing what I think at the end. But what would I hire for? Would I say, "Oh, what's this person really done? Can I measure that, and that's what really counts?" Or am I going to ask for, "Show me some of the work that you've done," the output, not a CV that says, "This is what I've done," but actual outputs, at least a portfolio? Or am I simply going to say, "Is this the smartest person in the class based on their grades?" Am I going to say, "How well is this person able to put their point of view across? Are they able to persuade someone?" That's something an agent can't perhaps do as well as a person, as a human. Or maybe it can, who knows? Or something else. Take a guess. And the reason I'm saying take a guess is, once you have this as a point of view, then over the course of this session and over time, you will get a sense of whether you've learned something along this direction that changes your mind or not.

But with that, let's start. Okay, we've got about a dozen responses on these. Fill this out. This is where you'll be seeing all of the content as we go along. And I have a third question for you. Can an LLM read binary files? What I mean by that is, you pass it English, or German, or French, or Chinese, or Japanese, or whatever, it obviously understands the language. If I pass it a BMP file, if I pass it a JPEG file, can it recognize it as is?

Now, I'm not talking about vision models. Now, you know that if you upload an image into ChatGPT, it can recognize it, but the way it does it is a different circuitry. That's the vision part of it. It reads the binary as is. What I'm saying is, supposing I just took the raw bytes, the pixels, in a small image and gave the pixels as it is. So, let's say we take the digits and you draw the digits on a grid or a spreadsheet with zeros and ones, and you encode it, which happens to be, I think, the PBM file format. You just gave it the PBM file format as zeros and ones, said, "What number is this?" Can it, in its mind, reshape the whole thing and see what the results are?

I'm going to wait till we have the answers for this one because that will shape the next set of questions. I'm actually going to look at the answers for the rest as we go along. Okay, thirteen responses. Interesting. Most of you plan to get a job. Some of you plan to start a startup. Not that many plan to study. Okay. That's useful to know.

And no one thinks your grades count. Wow, okay. I did not expect that. The majority of you think somebody would be looking at your portfolio of work, not your actual outcomes. So, as long as you can show good stuff, it doesn't matter whether you've gotten stuff done or not, okay. And your interpersonal skills are on par with actual outcomes, but not so important. Okay, "all of the above" and "how one uses agents", that's another opinion, fair enough. That's useful to know.

We've got a dozen responses here. The majority of you think, about two-thirds of you think, no, it does not speak Base64 natively. You actually need special circuitry to be able to read images. That's exactly what I thought, and told one of my colleagues, "Look, what I want you to do is experiment and figure out." That's my theory. The reason I thought that was because when I tried—yeah, I mean, it just seemed like, how is it going to see?

So, what my colleague, Atharva, did was he took these pictures: the digit 1, 7, a cross, a circle, etc., and encoded it in a variety of different formats. So, he said, "I'm going to encode it just like a text grid." This is literally the text that went in. "I'm going to encode it as PBM. I'm going to encode it as PGM. I'm going to encode it as BMP." This is Base64 encoded. This is PNG, again, Base64 encoded. And passed all of these to different models and asked them many times, "What do you see?"

If our results are correct, that is if the majority of us are right, then it should have gotten the majority of this wrong. Maybe it would have gotten a few right by guessing, something that is sparse it kind of would have gotten right. But this is the actual result. Let's take the first column, the text grid. So, when given the input in this format, so 7 was represented as zeros and ones in a grid, no model makes a mistake today, at least among the ones that we looked at. And these are not—well, Llama is among the weakest of the models that we have right now. And GPT-4o is not even the smartest. We have better models. And Sonnet 3.5, again, isn't among the best models. Opus, Fable, these are even better models. But these models have just been given these pieces of text, and they can read the text grid. They can read the PBM, mostly the PGM, the BMP, and even the PNG.

Now, PNG is interesting because it's double encoded. BMP is a kind of simple file format. So, in BMP what we're doing is taking the brightness of red, yellow, green and just putting that as simple numbers. Whereas, in PNG we are compressing it sort of like a zip file and then converting it to Base64. And these models, almost like they read English, they read Base64. They read the PNG compression almost like they read English, which means that if I tossed it an EXE file, they don't need to run it someday. Maybe they can just look at it and say, "Oh, it looks like you've uploaded Instagram or you've uploaded a browser here."

In fact, they are able to edit model weights. What does that mean? Now, a model is a neural network, an LLM is a neural network, and it comprises of weights. How many people have—okay. How many of you understand how an LLM works? And just to make sure the counts are right, how many of you do not understand how an LLM works? Okay. How many of you have not raised your hand so far? Yeah, no, I have tried this a number of times and I find that the better approach is what I'm going to try right now. Everybody raise your hands. Okay, now I can easily see who's not raising. Okay, great. If you understand LLMs well, drop your hand. Okay, good. Now I get the answer that I need, fine.

Let's do it in two steps. An LLM, which is a large language model, is really a neural network. So, the way that works is, think of it as you've got a bunch of neurons. So, that's a neuron, that's a neuron, that's a neuron. Actually, I should probably just use a Google Sheet to draw this, and that would be easier. And I said this is a workshop. We'll be getting to the you doing the work part in a short while. But let's zoom in a bit.

And supposing we said that's a neuron, that's a neuron, that's a neuron, a neuron is simply a function. That's a neuron. That's another neuron. And then we have this neuron. So, what happens is you have a bunch of inputs. So, let's say 1, 10, series of numbers: 30, -40, 50, 20, 10. Now, each one of these cells gets inputs from all of these numbers, but with different weightages. So, this neuron may actually only look at this one, this one, 50% of this, 20% of this, and so on. And it adds them up, rounds it off a little bit, and sends it onwards to the next one. Now, the next neuron may be looking at the previous set of inputs, and it may say, "I'll take 30% of this, 20% of this, 40% of this." And that is this one. This neuron takes different weightages, and so on.

Now, why am I saying neuron? Because that's actually how the neurons in our brains roughly work. There's just billions of little cells that take electrical signals from nearby neurons, and they have higher weightages for some of the nearby ones, lower weightages for some of the further-away ones. And for some miracle that we don't really know how to explain well, it looks like these neurons are able to function just like brains, yeah.

So, that's kind of how they work, but that doesn't really help us. I mean, okay, so what? Well, it turns out that because that's how our brains work, we, once we're shown that here's something that works, we can copy it. And now that we have enough computing power, enough GPUs, and so on, what we're finding is that we are actually able to train a brain, like we've trained a child, on all kinds of things. And for whatever reason, that works.

Specifically, how exactly do LLMs generate output? Let me show that. At any point these neurons are generating a bunch of outputs. These outputs are mapped into little letters, words. It's saying, "Okay, this is the word that I should be speaking next." It's almost like, look, take all the words that we've passed it so far, pass it through the neural network. So, it's almost like that's the number for the second word, that's a number for the third word, that's a number for the fourth word, and so on. You take all of these inputs and you pass it, and it then says, "Okay, now the output is going to be the next word." So, let's say it predicts word number 67 is the next word. So, then you take that, put it back in, try and get the next word, and so on.

So, at any point it's just predicting the next word, except that the good part is it doesn't just say, "This is the word that I want to get to." It predicts what the next word might be. So, for example, suppose we ask, "In what episode of Friends did Joey eat too many marshmallows?" So, it begins by saying, "Okay, maybe I should say, 'Joe'. There's a 70% chance I feel like saying, 'Joe'. There's a 28% chance I should start with, 'In'. There's a 1.7% chance I should start with, 'The'," and so on. But once it's said "Joe", or once it's picked, and it's picking randomly, then the next word it's very confident should be "ey". So, after it said "Joe", "ey" follows. When it says, "Joey eating too many marshmallows is featured in season," the next word that it wants to say is "three", but it also feels maybe it was in season seven. It needs to think a little. Or when it says "episode 20", it says, "45% chance I'll go with episode 20, but there's as much as a 10% chance it might have been in episode 21." So, it's thinking somewhat probabilistically.

Now, why is this important to know? It isn't, frankly. Trivia. But what surprised me was the fact that even though I had assumed that these models, they're just learning words, they're just processing this—I mean, how would they have learned how to speak PNG? And yet they seem to have learned how to speak PNG. Maybe because people took, I don't know, all of the images on Google Images and all of the videos on YouTube and so on, and some of them were annotated and it looked at this, just studied it long enough that it can speak a completely new language.

Let's take another question then, which is, scrolling down, can they count? Yes, please.

**Audience Member 1**: [19:32] Yeah, so the question I have is, like, when it's prompted to decode the image, are they just looking at the image, then they will know whether it's [inaudible]?

**Anand**: [19:41] So, the question that was posed was: here is a bunch of text, zeros, ones, etc., what is it? And I think it may have been given options. You have the link, so you should just check out what the exact prompt was. I think it may have been a classification question, or it may have been just, "Tell me what the number or digit is." Yeah, the prompts are all available somewhere, so you should be able to figure that out. But in short, to your question, the question that was asked is, "What is this piece of text?"

Yeah, now that you mention it, I am curious where the actual prompt is. Okay, yeah, no, well, the repository is there. Oh, and, okay, wait, hold on. Here, every raw reply. That's what I wanted to see. View the raw file. That's about 7MB. And it's a JSONL file. So, it says, "The following is a Base64 encoded PNG file. Image is so and so. Do not use any tools. Reconstruct the pixel grid, blah, blah, blah. Tell me what the image shows, and then give your confidence." So, it's not even being asked to choose from different options. It is literally being asked, "What does the image show you?" That's useful to know.

Another question was there on number four. Do you think an LLM-based model can recognize what's happening in a video? Now, that's a tougher, trickier one, but what's your guess? And keep in mind that, again, I'm saying model. I'm not saying this is something that's been given a tool that it can run ffmpeg on or whatever. But when I say video, stream of images is a perfectly valid interpretation of video. And my question is today, not in the future. I'm sure in the future things will go in strange ways.

Let's give everyone a few minutes. Okay, we have 10 responses. I'll wait till we have maybe a—yeah, okay, no, 10 responses is large enough. Let's see. And now the majority of us feel, yes, it can. Which is kind of interesting, right? Short while ago we thought it can't even recognize an image. Now we are leaning more towards, "Okay, yeah, from an image a video is not such a big deal." Yeah, quite possibly, let's take a look. And that would lead us to—let me show you what one of my colleagues was doing with an NFL broadcast. How many of you have heard of JEV, the model called JEV? Okay, see a few nods. These are newer models, and since many of you may not be familiar with these, think of them as fast, cheap models that do a little bit less than what the the major models do.

So, okay, you can't hear it and doesn't even matter that you hear it, but here is the video of a match, and as the match progresses, and I'll, yeah, just turn off the sound, yeah, the model is able to detect what's happening at any point and start putting that information out there on the top right. In other words, it's literally seeing the video. Another example of that is a rocket launch analysis where the model was looking at—okay, yeah, I'll just play it and maybe going to jump in into a certain spot. Hmm. Sorry, I should have—

Did it not—okay, yeah. On the right side it's continuously calibrating what it can see, whether the telescope is free, what phase it's in, just updating in real time what information is being showed on the video. All of this is real-time analysis and is the kind of thing that is possible today. So, think about it. We don't really need a sports broadcaster. Just play an event and the commentary can be automated, just like we have automated captions today. Same kind of description can happen for video as well.

Or that can happen post facto. Another one of my colleagues was analyzing a—let's see. Let's take a Super Bowl commercial and analyze this. So, what he did was took a bunch of videos, looked at at each point in the content what was, which brand was being advertised, how much of a specific brand is being shown at a certain point in time. Is there something that has a negative connotation that comes in in the middle of the advertisement, so the marketer would probably not want to show it? Take the whole thing, analyze it at one shot, it becomes easy.

The reason I'm showing you this is to give you a sense of the kinds of things people used to do manually or found that it was not even possible or realistic to do manually. But now with LLMs it starts becoming possible. And **part of this sort of a thing is what you would be expected to do.** Except you may say, "Wait, hold on, I don't know how to do it." Yeah, my colleague didn't know how to do it either. I just said, "Do it." He told his agent, "Do it," and it did it, and he asked me, "Does this look right?" I said, "Change this, change that." **That's your role, really: knowing what's possible, when it doesn't work figuring out whether you can make it happen, or figuring out if you should just wait for some time before the models and the systems get smarter.**

But that begs a different kind of question, and let me pose a question to you. You may be qualified to answer it, you may not know the answer, that's okay, but what's your guess on—what I've shown you so far are largely large language models. That's GPT-4o, or Claude Sonnet, or whatever. Now, you may have heard of some specialized models, YOLO being an example of a computer vision model, and you may be building your own custom models. What do you think? Given the sheer size, smartness, etc., of these really powerful models, do any of the models that used to exist, the specialized models that used to exist say a year ago, two years ago, or even now people are building them, do you think they would beat the LLMs on vision tasks? Just take a guess.

Okay, seven responses so far. I'll wait, okay, I'll wait till we have at least 10 of us give a yes or no. Okay, we have 10, cool. And the answer is, okay, 13. Hmm, oh, this is close. Okay, but on the margin we seem to think that, no, specialized models are probably not needed and LLM—okay, sorry, no. Do you think LLMs can count better? Okay, on the margin we think, no, LLMs can't, for instance, count better than specialized models. So, there's still a room for specialized models, but it's not a clear, decisive, "Yeah, we all think this way" kind of a thing.

Well, as of now, let's see what we found in at least one scenario. We ran a benchmark and—okay. The benchmark seems to have vanished.

So, the—what one of my colleagues did was he took a bunch of stamps. He placed all of these stamps on his desk, took a picture. He also, I think he had a bookshelf somewhere, and yeah, so he asked different models to count the number of books that he had on this bookshelf. He had a huge number of bottle caps, took a picture of those, asked it to count the number of bottle caps, etc., and tried to see what models were doing well or not so well on each of these. So, when he did it, for instance, okay, now the quality of the models is basically, I mean, the bigger the bar is, the better the model is.

Now, what's interesting is that you see specialized models on both ends. CountGD, Countix, these are specialized counting models, not doing so well. YOLO is a—YOLO-World v5, for instance, is, again, a specialized image model doing extremely well. DeepSeek-VL, Claude Haiku, Sonnet 3.5, these weren't doing as well in terms of the counts, but doing pretty decently.

**I think we are at that stage where there is still a place for specialized models.** The specialized models aren't improving as fast as the generalized models. But the good part is these days the way

---

**Anand**: [29:59] ...these days the way in which people are improving the specialized models is by telling a generalized model, they just go to ChatGPT and say, "Look, somebody built this model. Here is my problem. Can you improve the model?" That's it, really.

[30:13] That seems to be increasingly the way in which specialized models are improving. But I think the majority of you are correct that **there is a role for specialized models in our current scenario.**

[30:27] Here's a question that you should now think about then. So, given this, do we need to learn, or what models do we need to learn and use and optimize? Should we go for the specialized models, or maybe we just have coding agents do it for us? If you had to choose, Option A: I am going to just let the coding agents figure out what I should do. Or, as someone who wants to explore computer vision models, let's say I need to identify what's the state of the models and what is the model to start building and optimizing from, and learn the techniques by which I can optimize the models. What would you pick? Again, there is no right or wrong answer on this one. I'm just curious to see what you think.

[31:24] Okay, we have 10 responses. 11. Let's start seeing the results. Okay, majority of you, and this is a clear majority, think, in most cases we need to identify and optimize the models ourselves. About two-thirds of you think that we should really be—we should know the models, we should optimize the models. Like I said, there is no right or wrong answer.

[31:49] Let me share some of the things that our team was working on, and this is from people who do not know how to build models. The first person who does not know how to build a model is me. I have never built a computer vision model ever. And here I am, conducting a computer vision workshop right here, but—

[32:17] One of the things I was doing was analyzing my push-ups. I took a video of my push-ups and asked ChatGPT a few days ago, and I'll show you the prompt. Look, how good are my push-ups? Here's what it showed me. It said, "Okay, that's me. Those are my bones or whatever. That's me checking the camera." And I start doing what, well, one might call push-ups, and it's doing an evaluation. And it gave me a very generous 71. I realized that it was just the top half of my body that was moving. My hip was barely budging. These aren't push-ups. Yeah, exactly. I mean, there's probably a name for these things.

[33:09] So, I saw this, I mean I didn't see the actual lines at that point. I just saw the video and I said, "Okay, yeah, I should improve." And a week later, this is what I got to. And as you can see from the score on top, I was doing a little better at this stage. The pike, in particular, it started off a bit weird, but this is not getting into the red very often.

[33:38] Now, this is cool. I've seen these really fancy specialized apps, equipments, etc., where live they take a video of you and immediately give you feedback. This could be done live. You know what? In this workshop, I'd love for you to do this live. It can be done. And how do you do it? I don't know. How did this do it? I don't know. I'll show you the prompts that I used.

[34:16] And all I said was, "Look," this was to Claude, not ChatGPT, okay? And I said, "I have videos of me doing push-ups. I want to identify the quality of my push-ups, and, yeah, you figure out what is the library, tool, or technique that should be used. I want a single webpage that gives me a score. Now, I don't even know what I mean by a good push-up. So, you figure it out. Maybe it's about how straight my back is, how low I go, how straight my arms are." I really didn't know what I should be asking for, so I just tossed in a few things that came to my head and said, "Look, it'll be great if you could draw some lines showing my bones or whatever, put in some metrics, and in order to do this we need to figure out the best approach." And I also said, "Give me a comparison of the different approaches you used. So you might have used different libraries, you might have used different techniques. Fine. Tell me what that comparison looks like." And, yeah, it would have been fine except that my system just heated up too much, so it crashed, it rebooted, crashed again, blah, blah, blah. The rest of it was just doing that.

[35:26] But now, here's the thing. It also compared the libraries and approaches. Yeah, here we go. So, it said there is something called RTMpose. That's a specialized library that detects poses. Had never heard of it before. There's something called ViTPose, again, specialized library. There's a YOLO variant for poses, again, specialized. Google has something called MediaPipe, which is somewhat generalized, but has a pose component to it. And then there's Gemini 4 Argon, which is a general-purpose model. It picked Gemini Argon, I didn't tell it to. And it also used Gemini 3.8 Flash. And it said, "Look, overall, this is the ranking. RTMpose, if you had to pick only one library, that's the library you should go for your push-ups." Maybe if I, if I tried pull-ups it would have picked a different library, who knows? Or maybe the difference is marginal. But it also said, "Look, what I can do is take the median of these four, and that actually seems to be an even better mechanism."

[36:30] So, what I learned was three things. One: **I didn't really need to know what library to use or what technique to use. It can figure that out.** Second: **it seems to be able to create a benchmark and verify it on my behalf.** And when I look at the result, it looks good enough, meaning I don't know more than it to comment on the process. And third: **it seems to have done an optimization, meaning it's picked multiple libraries, figured out a composite of a technique that combines multiple libraries, and come up with something that's better than any of those individually.** So, it's able to do the optimization as well, which I thought was pretty cool.

[37:11] Something else that another colleague of mine built. This was a pretty interesting one. Let's see. Start the webcam. Opening camera for whatever—uh, okay. Yeah. So, what I think I can do is sculpt this thing based on my hands. I'm just creating pottery by moving my hands in and out. Let's shape this a little bit. Okay, let's pull this out a little, push this out.

[37:57] Yeah. How is this happening? You see, when I place my hand out here, little dots form near the top left, right? Let me make it a bit bigger. Yeah. So, what's happening is when I place my hands like this, he's used, this case, the library happens to be MediaPipe, but he didn't even know about MediaPipe when he built this, and then it's able to detect that I have formed a fist. And as I move it in or out, it shapes the pot at that particular location. When I move my hands up, it moves the slider up, and now I'm able to move this in or out. Okay. Maybe that didn't work. Yeah, now it's working. Which was interesting.

[38:49] Again, identification was his, or its, the agent's. The conceptualization was his. That's the role the human was playing. In fact, this is something that we can even play collaboratively. Let's try this. Could you scan this QR code, and preferably on your mobile phones, and turn on your cameras. When you do that, you should be able to place your hands once you enable your camera, and, yeah, move stuff around. Frankly, I don't know how this game works. Oh, oh, I see. You have to—oh, it's shaking it, is it? Okay, I haven't even played this game so far. So, between gestures of various kinds, okay, yeah. That happens to be an ATP synthase molecule. And what you're doing is, effectively, you were collaborating to generate more protons based on gestures.

[40:10] Now, your phone is a—I stopped the game by the way, so. Your phone is a remarkable device. It has a camera, it has a microphone, it has an accelerometer, it has a gyroscope, it has, I'm sure I'm, well, Wi-Fi of course, and I'm sure I'm missing a whole bunch of sensors. But what you can do is combine all of these, and just with a browser start creating all kinds of special effects. So, I asked him, "Hey, how did you do this? What libraries did you use?" He said, "I have no idea." He said, "I'll have to look at the code. I'll check and get back to you." The Anthropic team had similar answers. They said, "We don't really understand how Claude Code works. No single person actually understands it. If we need something, we just ask it to create an explainer for Claude Code and it explains it."

[41:07] If that's the case, then here's my question to you: agents seem to be doing data science quite well, and they're constantly improving. If so, what do you think you should learn, and why? I've given four options, but you're very welcome to pick "Other" and share your thoughts. We will be getting into the doing stuff in a short while. Yeah, pretty much, probably on time.

[41:53] Okay, nine responses. I'll give you a bit of time. Let's, okay, 11, it's close enough. So, we're split between verification and finding problems. No one now seems to think that doing data science is something that we need to worry about. The agents can probably do the data science. Last year I delivered a talk, the title was "Rest in Peace, Data Scientists". What we've been learning so far as data science is something agents can do. That is not what we should be focusing on. So, I'm glad that most of you think similarly.

[42:35] Between verification and finding good problems, it's a toss-up between us. Execution is a possibility so that we can catch up and not be left behind, that's a fair point. And conveying human needs and emotions logically, yes, agreed, that's a fair point as well. Aligned with all of these. On these, there's no right or wrong answer, I think it depends on the scenario, but I'll share my point of view on all of these questions in a while.

[43:03] Let's therefore jump to doing something. In this session, and from now on it's you doing stuff. I showed you a bunch of things. All of what I showed you was done by people who know nothing about computer vision, who know nothing about, well, in some cases who know nothing about programming. They just went to some agent and said, "Do this for me." Some of them were able to do it, some of them were not. **A big part of what enables it is, are you giving the agent the right kind of tools?** And in today's workshop, we'll be roaming around, pick something ambitious. What I want to make sure you do is, if you can leave this workshop saying, "Damn, I didn't even know this was possible, and you've created it," that would be a win. Even better, you get to the point where no one in this room is able to figure out how to solve a specific problem that you've gotten, in which case you will reach the limits of what even agents can do.

[44:08] Here are some ideas on what you might build, but the other option is very much available to you. Just pick one, you don't have to limit yourself to this, but for now just make a choice. One possibility, this is perhaps the simplest, is build a sign language tutorial. So, it's a browser-based application. I would like all of you to **build an application that you can deploy on GitHub Pages, or as a static HTML file.** And if you don't know what that is, it doesn't matter, just tell it to deploy it as a static, a webpage, that's it. That way, you build a portfolio.

[44:45] And let me go back to the answer to one question: what do you think a data science recruiter looks for when hiring, right? At least speaking for myself, I agree with the majority of you. I'm going to look at your portfolio of work. I'm not going to look at your grades. The pace at which things are changing, colleges have not yet caught up. Your grades are a reflection of other things. If I were looking for a Master's, I mean if I were admitting students for a Master's degree, I'd look at your grades, but not for a job. Your interpersonal skills do matter and I will be looking at it, but what I find is that there is an interesting negative correlation between interpersonal skills and the ability to get stuff done. I think that's because people clear interviews either because they can do good stuff, or because they can talk well. And, yes, they are independent, but in reality what happens is that the sum of the two scores is what ends up promoting somebody, so there's a negative correlation usually. So, I deprioritize this. And actual outcomes would be great. Now, the problem is, I can't verify most of those actual outcomes. Somebody says, "I saved a client," or, "I was doing an internship and we saved them $85 million." Who am I going to call? I can see the portfolio of work.

[46:14] So, in this workshop, I'd like you to build one small component of your portfolio. Make sure that this becomes part of your portfolio, and therefore plan accordingly. So, you could just build a simple sign language tutorial, that's just to give you an idea. I move my hands and it says, "Oh, you made the A sign. Here's how you make the B sign." Okay, so I make the B sign. And then I do something in sign language, it literally translates what I'm saying into English. That would be cool. Or a gesture-based musical instrument. What if I could play a keyboard virtually with my fingers, or moving my hands up or down, or with my head, or you're very welcome to make it so that you move your phone and it plays a musical instrument. Or can you convert your webcam into a video studio? Could you have, I don't know, Barack Obama walking behind you? Could you place yourself inside an aquarium? Could you display a tab, a browser tab, behind you? Could you create a data visualization of how many times you blink and the visualization appears in front of you? And join Google Meet with that. Yeah, that's your new camera. Why not? Or anything else, absolutely anything.

[47:36] So, pick something, just toss in a choice. What I will be doing as we go along, and I'll unhide this, is asking that you share your links. There may be multiple links, and I'll make this editable so you can change your input at any point. Through both of these, you're very welcome to correct your answers and share the links of what you've built.

[48:11] Any questions?

[48:23] Cool. Us and the volunteers will just be floating around. I'm also going to build something. Why should you have all the fun?

[WORKSHOP-BREAK]

[48:37] But here's the thing. Firstly, I was quite surprised how many of you started using gesture-based controls. Quick check, how many—okay, first, can everybody raise their hands? Everyone, yeah, hands up. Yes, just, just raise your hand. Now, if you've already used gesture-based controls before, lower your hand. If you've used gesture-based controls before, lower your hand. Okay, the majority of you have not used gesture-based controls, and it's the first time that you're using that. Okay, that that's particularly impressive, because I did not think that's what most of you would go for.

[49:16] But there's the thing, right? In what, less than an hour, at least well more than half a dozen of you, you've created something that works, in many cases works well, that's based on gestures, through your camera, and have deployed it for the entire world to be able to use your application. Clearly powerful.

[49:41] Now, what that means is, if you can do that in less than an hour, anyone can do that in less than an hour. **That is not your differentiation.** The differentiation is, what is it that we failed today? So, I wasn't able to use some of these gestures. I was struggling. Some of them worked well, and you were iterating to see if it was working well for you, not working well for you. That is something that maybe everyone can't do. Some of you thought of some really creative uses. Maybe that is something that others are less likely to be able to do.

[50:15] Play around with it. The rule of thumb that I follow is: **delegate everything to agents. Let them do what they can do well. What they can't do well is the space we play in.** See what others are able to do. What most people are doing well with agents, again, is not your differentiator. What they're not even thinking of is probably where your space lies. And if you create a portfolio that matches something that comes naturally, you'll find it much easier to go back to someone and say, "This is who I am. This is what I can do."

[50:52] So, with that in mind, I promised I'd share my opinion on all of these questions. So, let me do that. First question is not relevant to me, but most of you are planning to get to work. Remember that you will be competing both with agents, and therefore there's no point learning what agents can do, as well as with people who will be using agents. So, yes, you do need to know how to use agents as well, but what is it that you can do that's different matters. And I completely agree with you, I would evaluate a portfolio of work. Today the session was intended that you would be able to come up with a portfolio of, at least one element in your portfolio, maybe more, and take that forward. But keep in mind this is just one thing that you can do in one hour. There's lots that you should be doing, and therefore if you create 100 things, pick what is really suitable for the person you are speaking with and craft custom portfolios, effectively personalizing it to the person you're meeting. That's what gives it power.

[51:54] Yeah, these I think we know: agents can speak many languages, but given the direction that they're going in, just assume that almost anything that is possible to understand, they will understand. That's what reinforcement learning is all about. If something can be taught, put another way, if you give it an input and any system of intelligence is able to figure out if that input is correct or wrong, they will learn it. Therefore, the two areas where AI will probably not learn fast enough are where it's impossible to figure something out in a short duration of time—how do stars grow? How do plants grow? How do humans or businesses react to strategy? These things take years to experiment. Or where there's no such thing as a right answer—will this painting be appreciated by people? Today maybe the answer is yes, tomorrow tastes may change. So, these are the things where there is a differentiation.

[52:57] And given that agents can pretty much do all of the stuff that can be learned, delegate maximally, focus on what's left. My opinion on this: doing data science, not worth it. Agents can do that. Verification: I'm not as convinced as you are. I used to think that we would need to do that, but here's, there are two problems with that: work will pile up, agents will keep building stuff, it's humanly impossible to verify. And if verification can be objectively done, agents will take over a decent portion of that. What seems to be emerging is governance. How do I set up stuff so that agents can do the verification? That's emerging. Finding problems: yeah, that, I think so. Agents can do a good job of that, but finding creative problems, niche problems, on the edge, I think there is merit there. Execution: definitely. That's the big problem. ChatGPT can tell you, here's how you should do your exercise. You've got to do it. Here's a diet plan you should follow. You've got to follow it. It is the harder part, and if you can get stuff done, that may matter more than anything else. In my list, this would be right on top, of course the others you had shared.

[54:20] What would I take away from this workshop if I were you? Actually, I don't know, and that is, in fact, therefore going to be our final question: what would you take away? What have you learned? Let me go back to the form and I'll paste the QR code in case you need it. Zoom in. But there's one final question that I would like all of you to answer: **what is something that you learned today?** It may be something that you didn't even know was possible, or maybe you thought it's possible, but it's actually not. Maybe something that you learned, but now you start wondering, is this even useful? Maybe some limitation that models have that you could build something around, anything. Just share what you've learned. We'll give it a few minutes, and before we wrap up we will have a group picture in a short while, but just wait for the responses to get up to maybe a dozen or so, and then we can wrap up.
