WEBVTT 00:00.000 --> 00:04.000 Anand: Okay, why don't we dive 00:04.000 --> 00:08.000 Anand: in? Uh, I guess we you've seen the uh 00:08.000 --> 00:12.000 Anand: latest GA sorry latest project 00:12.000 --> 00:16.000 Anand: one question. Uh, but all uh before we 00:16.000 --> 00:20.000 Anand: dive into anything else, anything that we need to discuss about uh 00:20.000 --> 00:24.000 There was some issue. So I think 00:24.000 --> 00:28.000 Mayen has pointed out 00:28.000 --> 00:32.000 Anand: I uh think my PR they have made some tweaks to 00:32.000 --> 00:36.000 Anand: pushed and I am able to uh submit for question 00:36.000 --> 00:40.000 Anand: two but man are you able to okay thumbs 00:40.000 --> 00:44.000 Mayank: Yes. 00:52.000 --> 00:56.000 Anand: resolved okay I thought I'll give you the 00:56.000 --> 01:00.000 Anand: philosophy behind um the 01:00.000 --> 01:04.000 Anand: questions that are emerging in B1 01:04.000 --> 01:08.000 Anand: and then we can go on to G83 01:08.000 --> 01:12.000 Anand: maybe. 01:12.000 --> 01:16.000 Anand: Uh yeah, so 01:16.000 --> 01:20.000 Anand: the prediction market is as we discussed 01:20.000 --> 01:24.000 Anand: last week and let's see what the current 01:24.000 --> 01:28.000 Anand: state of uh the predictions 01:28.000 --> 01:32.000 Anand: are. Huh. The There you go. 01:32.000 --> 01:36.000 Anand: Okay. I believe there were about uh 200 01:36.000 --> 01:40.000 Anand: to 300 uh predictions 01:40.000 --> 01:44.000 Anand: or trades made so far and uh 01:44.000 --> 01:48.000 Anand: most people seem to think $10 to 01:48.000 --> 01:52.000 Anand: $20 is where it'll stay. Of 01:52.000 --> 01:56.000 Anand: course, a lot will depend on the 01:56.000 --> 02:00.000 Anand: questions that will be asked for 02:00.000 --> 02:04.000 Anand: what gas will be there and how many people are active 02:04.000 --> 02:08.000 Anand: etc. But uh the published 02:08.000 --> 02:12.000 Anand: reasons from the class they are flowing 02:12.000 --> 02:16.000 Anand: in. So this is a good 02:16.000 --> 02:20.000 Anand: view to understand what we collectively think. And uh 02:20.000 --> 02:24.000 Anand: the skill that we want people to learn is 02:24.000 --> 02:28.000 Anand: leveraging the wisdom of the crowds. Being able to 02:28.000 --> 02:32.000 Anand: influence using data that is not 02:32.000 --> 02:36.000 Anand: available to an agent like you know where the course is going 02:36.000 --> 02:40.000 Anand: based on discussions with others which an agent will 02:40.000 --> 02:44.000 Anand: not be able to figure out things like that. So what does it 02:44.000 --> 02:48.000 Anand: take to use real life data 02:48.000 --> 02:52.000 Anand: and feed it to an agent so that it can guide you on the right 02:52.000 --> 02:56.000 Anand: kinds of trades to make or you make the trades yourself 02:56.000 --> 03:00.000 Anand: and B are you in a position to influence these 03:00.000 --> 03:04.000 Anand: that's another element second 03:04.000 --> 03:08.000 Anand: one I will very briefly go through the uh skill 03:08.000 --> 03:12.000 Anand: that we want uh people to learn 03:12.000 --> 03:16.000 Anand: This may be the most 03:16.000 --> 03:20.000 Anand: important uh at least of the 03:20.000 --> 03:24.000 Anand: three uh P1 questions and perhaps of many questions that we've 03:24.000 --> 03:28.000 Anand: designed so far. Uh the idea 03:28.000 --> 03:32.000 Anand: is agents can do anything. How do we figure 03:32.000 --> 03:36.000 Anand: out what is useful? The entire specification 03:36.000 --> 03:40.000 Anand: problem is what we are trying to teach here. Put another way we 03:40.000 --> 03:44.000 Anand: saying people uh promise to create 03:44.000 --> 03:48.000 Anand: something. That's what you'll be uh declaring in your 03:48.000 --> 03:52.000 Anand: recording. Give it a week, think about it, make a couple of corrections 03:52.000 --> 03:56.000 Anand: if you want. You have up to uh two revisions 03:56.000 --> 04:00.000 Anand: and by uh what's 04:00.000 --> 04:04.000 Anand: the end date? Wednesday, I think. Yeah. Uh 04:04.000 --> 04:08.000 Anand: by Wednesday, end of day, uh make 04:08.000 --> 04:12.000 Anand: sure that you have committed to what you will build. And when I say build, I mean it could 04:12.000 --> 04:16.000 Anand: be uh any service offering also like look 04:16.000 --> 04:20.000 Anand: you ask me one question I will give you one answer and that's the 04:20.000 --> 04:24.000 Anand: offering that you publish and then we'll create 04:24.000 --> 04:28.000 Anand: a small marketplace nothing major in the marketplace it 04:28.000 --> 04:32.000 Anand: simply will be a U credits. distribution 04:32.000 --> 04:36.000 Anand: mechanism. Anyone can buy 04:36.000 --> 04:40.000 Anand: services or product from others by paying 04:40.000 --> 04:44.000 Anand: credits at a price that the seller and buyer agree 04:44.000 --> 04:48.000 Anand: on and that gets transferred from the 04:48.000 --> 04:52.000 Anand: buyer to the seller. This 04:52.000 --> 04:56.000 Anand: is really uh uh aiming 04:56.000 --> 05:00.000 Anand: at two things. one are you able to create something that 05:00.000 --> 05:04.000 Anand: is useful and it's when people buy your offering 05:04.000 --> 05:08.000 Anand: that your score goes up at the 05:08.000 --> 05:12.000 Anand: same time it is to leverage what people are offering if a really 05:12.000 --> 05:16.000 Anand: smart student says uh I will give you one human 05:16.000 --> 05:20.000 Anand: answer or somebody with a unique piece of hardware says I'll let 05:20.000 --> 05:24.000 Anand: you know run a game on it or play a game or somebody 05:24.000 --> 05:28.000 Anand: really good with interview skills is able to interview you, 05:28.000 --> 05:32.000 Anand: whatever. Effectively, you're getting through 05:32.000 --> 05:36.000 Anand: credits something almost the equivalent of cash to be 05:36.000 --> 05:40.000 Anand: able to pay for valuable services that the students may offer. 05:40.000 --> 05:44.000 Anand: So, leveraging that to your benefit is the other 05:44.000 --> 05:48.000 Anand: skill that we're hoping people will learn. 05:48.000 --> 05:52.000 Anand: Any questions on uh human 05:52.000 --> 05:56.000 Anand: exchange? 05:56.000 --> 06:00.000 JK: this during our discussion but I think uh one point of question would be uh 06:00.000 --> 06:04.000 JK: the human exchange part uh I mean we discussed the the 06:04.000 --> 06:08.000 JK: outline format but should everything be tied to uh 06:08.000 --> 06:12.000 JK: anchored around TDS was one question I think we we were discussing that 06:12.000 --> 06:16.000 JK: day that It needn't be right. I uh it could be practically 06:16.000 --> 06:20.000 Anand: Absolutely. That is 06:20.000 --> 06:24.000 Anand: a good point. I will add that as a 06:24.000 --> 06:28.000 Anand: clarification. Uh that 06:28.000 --> 06:32.000 Anand: uh in the TDS uh human exchange 06:32.000 --> 06:36.000 Anand: question uh clarify the following. One, 06:36.000 --> 06:40.000 Anand: the service does not necessarily have to be limited to 06:40.000 --> 06:44.000 Anand: TDS. It can be about 06:44.000 --> 06:48.000 Anand: anything. I'll 06:48.000 --> 06:52.000 Anand: add okay 06:52.000 --> 06:56.000 Anand: it should be legally permissible 06:56.000 --> 07:00.000 Anand: as per ITM as well 07:00.000 --> 07:04.000 Anand: as the government 07:04.000 --> 07:08.000 Anand: laws. 07:08.000 --> 07:12.000 Anand: Fine. We'll clarify 07:12.000 --> 07:16.000 Carlton: So, no Bluetooth peer-to-peer communications, is 07:16.000 --> 07:20.000 Anand: no, that's Wait, wait. Who's running 07:20.000 --> 07:24.000 Carlton: it? 07:24.000 --> 07:28.000 Anand: that? 07:28.000 --> 07:32.000 Anand: Wait, no. Hold on. I was actually thinking of doing some Bluetooth peer-to-peer ing water 07:32.000 --> 07:36.000 Carlton: Yeah. 07:36.000 --> 07:40.000 JK: app uh which uh in app store uh which 07:40.000 --> 07:44.000 Carlton: So 07:44.000 --> 07:48.000 Carlton: yeah, because they cannot monitor that. So because it works completely 07:48.000 --> 07:52.000 Carlton: offline, they are not able to monitor coms. So that's why they abandoned both in the 07:52.000 --> 07:56.000 Carlton: Google Play Store as well as in the Apple App 07:56.000 --> 08:00.000 Carlton: Store. No, 08:00.000 --> 08:04.000 Carlton: not not yet. Anyways, 08:04.000 --> 08:08.000 Anand: mind. 08:08.000 --> 08:12.000 Anand: Cool. Anything else 08:12.000 --> 08:16.000 Anand: on this 08:16.000 --> 08:20.000 Anand: question? Okay. Uh yeah, 08:20.000 --> 08:24.000 JK: Priy. 08:24.000 --> 08:28.000 Anand: breathing. 08:28.000 --> 08:32.000 Preethi: had created. Sir, is there a minimum maximum limit to the services 08:32.000 --> 08:36.000 Preethi: being offered like like only one service is uh is to 08:36.000 --> 08:40.000 Anand: I do remember your question and 08:40.000 --> 08:44.000 Anand: I'm trying to see where I had that open but 08:44.000 --> 08:48.000 Anand: uh so there are two parts to this 08:48.000 --> 08:52.000 Anand: one uh is can multiple students uh 08:52.000 --> 08:56.000 Anand: buy or uh exchange the service. The answer is 08:56.000 --> 09:00.000 Anand: yes. Can the same student make a repeat purchase 09:00.000 --> 09:04.000 Anand: so to speak? The answer is yes. All we're saying is 09:04.000 --> 09:08.000 Anand: everybody will be get getting a certain number of 09:08.000 --> 09:12.000 Anand: credits you between you. So what you can do for 09:12.000 --> 09:16.000 Anand: instance is say today I have decided 09:16.000 --> 09:20.000 Anand: that I will offer this service 09:20.000 --> 09:24.000 Anand: to one of my friends for one 09:24.000 --> 09:28.000 Anand: credit and you register that both of 09:28.000 --> 09:32.000 Anand: you sign off saying I have delivered the service and 09:32.000 --> 09:36.000 Anand: received one credit and vice versa I have received the service and 09:36.000 --> 09:40.000 Anand: uh I'm delivering one credit that transaction happens 09:40.000 --> 09:44.000 Anand: on a portal that I will uh design. You can offer at a different 09:44.000 --> 09:48.000 Anand: price to someone else. You can do 09:48.000 --> 09:52.000 Anand: whatever you want, however you want. It 09:52.000 --> 09:56.000 Anand: is just saying that this particular 09:56.000 --> 10:00.000 Anand: service is available for people to take advantage 10:00.000 --> 10:04.000 Anand: of. Keeping it open that way. So short answer to your question Pi is 10:04.000 --> 10:08.000 Anand: yes. 10:08.000 --> 10:12.000 JK: service still belongs to it. So you can decide whomever you want 10:12.000 --> 10:16.000 JK: to and whatever the rate 10:16.000 --> 10:20.000 Anand: Okay. 10:20.000 --> 10:24.000 JK: Yeah. 10:24.000 --> 10:28.000 Anand: Yes. 10:28.000 --> 10:32.000 Preethi: just uh lending a year for 10 minutes or something of that sort. That's So that should 10:32.000 --> 10:36.000 Preethi: be agreeable. That should be uh like I should 10:36.000 --> 10:40.000 Anand: No, no, you can uh that's going to be 10:40.000 --> 10:44.000 Anand: tricky before Wednesday. No. Uh you estimate what will 10:44.000 --> 10:48.000 Anand: have high demand and that you can fulfill and you 10:48.000 --> 10:52.000 Anand: offer it. Um multiple services 10:52.000 --> 10:56.000 Anand: might end up being confusing but 10:56.000 --> 11:00.000 Anand: also broadens your scope. You can make the trade-off. Nothing from 11:00.000 --> 11:04.000 Anand: our side that stops you other than 11:04.000 --> 11:08.000 Preethi: a single recording for any number of services like one or two or three or 11:08.000 --> 11:12.000 Anand: legality. 100 if you think people will even listen to 11:12.000 --> 11:16.000 Preethi: right no I'm saying everything to be mentioned in that one 11:16.000 --> 11:20.000 Anand: it. Oh yes. uh what will be published is the 11:20.000 --> 11:24.000 Anand: recording and a transcript of that 11:24.000 --> 11:28.000 Anand: recording and probably maybe an 11:28.000 --> 11:32.000 Anand: automatically extracted title that summarizes it. Keep in mind there are 11:32.000 --> 11:36.000 Anand: going to be hundreds of services on a 11:36.000 --> 11:40.000 Anand: portal. Will people immediately 11:40.000 --> 11:44.000 Anand: understand it based on an automated summary that comes out out of what 11:44.000 --> 11:48.000 Anand: you said is something you have to 11:48.000 --> 11:52.000 Anand: decide. 11:52.000 --> 11:56.000 Carlton: So I mean that brings another 11:56.000 --> 12:00.000 Carlton: question. Uh when we when you design the 12:00.000 --> 12:04.000 Carlton: marketplace uh how does it decide what goes on 12:04.000 --> 12:08.000 Carlton: top? Is it is there some algorithm? Is there 12:08.000 --> 12:12.000 Carlton: alphabetical? Because that will also make a difference in how you pitch it, 12:12.000 --> 12:16.000 Anand: very true. I think random will be the most 12:16.000 --> 12:20.000 Anand: uh how should I put it? Fair. However, we will 12:20.000 --> 12:24.000 Carlton: right? 12:24.000 --> 12:28.000 Anand: most recently purchased, maximum credits uh 12:28.000 --> 12:32.000 Anand: exchanged, maybe a few other reasonable marketplace kind 12:32.000 --> 12:36.000 Carlton: Okay. So, there's 12:36.000 --> 12:40.000 Carlton: no SEO type capabilities 12:40.000 --> 12:44.000 Carlton: in 12:44.000 --> 12:48.000 Anand: uh more aimed at humans than the marketplace 12:48.000 --> 12:52.000 Carlton: the 12:52.000 --> 12:56.000 Carlton: Okay. 12:56.000 --> 13:00.000 JK: Majay Krishnan uh who is asking whether the services are 13:00.000 --> 13:04.000 Anand: content. Yeah, I know 13:04.000 --> 13:08.000 Anand: you've answered that already, which 13:08.000 --> 13:12.000 JK: is you get you get to purchase credit credits and you get some 13:12.000 --> 13:16.000 Anand: is 13:16.000 --> 13:20.000 Anand: Yeah. And any other questions on this one, please just toss it out here. Let's 13:20.000 --> 13:24.000 Anand: go to the third one which is air inc. Uh 13:24.000 --> 13:28.000 Anand: the idea is that each of us will 13:28.000 --> 13:32.000 Anand: uh on our mobile phones this won't work on uh 13:32.000 --> 13:36.000 Anand: the uh on 13:36.000 --> 13:40.000 Anand: most PCs that I know of. So you 13:40.000 --> 13:44.000 Anand: log in on your phone and 13:44.000 --> 13:48.000 Anand: then it will ask you to draw. Let me turn 13:48.000 --> 13:52.000 Anand: off my blur and see if 13:52.000 --> 13:56.000 Anand: you are able to 13:56.000 --> 14:00.000 Anand: uh Okay, if this is clearly 14:00.000 --> 14:04.000 Anand: visible, right? Okay. But for 14:04.000 --> 14:08.000 Carlton: You 14:08.000 --> 14:12.000 Carlton: can just log in through your phone and share your phone screen 14:12.000 --> 14:16.000 Anand: I'm logging through my phone and share my Okay, I 14:16.000 --> 14:20.000 Carlton: maybe. 14:20.000 --> 14:24.000 Anand: So, I will do 14:24.000 --> 14:28.000 Anand: that before our next session. But 14:28.000 --> 14:32.000 Anand: uh once I log in uh there should be 14:32.000 --> 14:36.000 Anand: a little dot that you can see that 14:36.000 --> 14:40.000 Anand: moves around. That dot I don't it's right now at the bottom of 14:40.000 --> 14:44.000 Anand: my phone because I' I'm holding it vertically. But as 14:44.000 --> 14:48.000 Anand: I tilt it will move. And we're supposed to place the phone flat 14:48.000 --> 14:52.000 Anand: like this. Follow a set of instructions for calibration. 14:52.000 --> 14:56.000 Anand: And once you've calibrated it you will be able to draw 14:56.000 --> 15:00.000 Anand: shapes. uh like for instance if I do 15:00.000 --> 15:04.000 Anand: a hold to draw it's 15:04.000 --> 15:08.000 Anand: drawing on my screen some shapes just because I'm tilting 15:08.000 --> 15:12.000 Anand: it it's moderately intuitive once you get a hang of it 15:12.000 --> 15:16.000 Anand: you'll be asked to draw three words 15:16.000 --> 15:20.000 Anand: random letters and submit it that is the first part 15:20.000 --> 15:24.000 Anand: of the assignment for this uh 15:24.000 --> 15:28.000 Anand: random set of I mean And uh once you do that 15:28.000 --> 15:32.000 Anand: you submit it for now. That's it. The date 15:32.000 --> 15:36.000 Anand: is next Monday end of day is when this uh 15:36.000 --> 15:40.000 Anand: gets submitted. After that we will 15:40.000 --> 15:44.000 Anand: be asking students 15:44.000 --> 15:48.000 Anand: given the underlying tilt data uh 15:48.000 --> 15:52.000 Anand: etc. basically the measurements of how 15:52.000 --> 15:56.000 Anand: people drew it. Can we figure out the word that people 15:56.000 --> 16:00.000 Anand: drew? That will be the machine learning exercise that we will end up 16:00.000 --> 16:04.000 Anand: having to solve. For now, this 16:04.000 --> 16:08.000 Anand: week it's just draw the stuff that's going to take 16:08.000 --> 16:12.000 Anand: maybe 5 minutes. And yeah, don't use 16:12.000 --> 16:16.000 Anand: agents for this. uh we will check if it is agent drawn 16:16.000 --> 16:20.000 Anand: and that's easy enough uh to do. uh 16:20.000 --> 16:24.000 Anand: a there's no point. It costs you too much credit compared to the 16:24.000 --> 16:28.000 Anand: effort that uh you would do and uh 16:28.000 --> 16:32.000 Carlton: Uh, I've got one 16:32.000 --> 16:36.000 Carlton: question with regards to how the words are drawn. So 16:36.000 --> 16:40.000 Carlton: presuming there there will be several characters in a word, will there be a way to 16:40.000 --> 16:44.000 Carlton: pause after writing a character? Is 16:44.000 --> 16:48.000 Anand: have to hold and move to draw. You release it, it will stop 16:48.000 --> 16:52.000 Carlton: that 16:52.000 --> 16:56.000 Anand: easy. One of the few things that I actually 16:56.000 --> 17:00.000 Anand: tested extensively that quite easy. 17:00.000 --> 17:04.000 Anand: Little bit 17:04.000 --> 17:08.000 Anand: of physical skill about as much as balancing a little bit of water in your 17:08.000 --> 17:12.000 Anand: hand and no more than 17:12.000 --> 17:16.000 Anand: that. Yeah. So those are project one questions. 17:16.000 --> 17:20.000 Anand: Um any any questions on the 17:20.000 --> 17:24.000 Anand: motion game I think he says the screen seems to be 17:24.000 --> 17:28.000 Anand: frozen. Is my screen frozen for anyone else? 17:28.000 --> 17:32.000 Anand: Yeah, I had to stop the screen 17:32.000 --> 17:36.000 JK: think so. We were not sharing. I mean it it will the whatever 17:36.000 --> 17:40.000 JK: we have discussed till now will not work on a PC like what Anan said. So that's why you 17:40.000 --> 17:44.000 JK: saw the screen blank while was 17:44.000 --> 17:48.000 Anand: share. Okay, there are no 17:48.000 --> 17:52.000 Anand: questions on uh project on 17:52.000 --> 17:56.000 Anand: question three and of course maybe more questions will emerge as we go along. 17:56.000 --> 18:00.000 Anand: Let's talk about GA3. Um, so one of the 18:00.000 --> 18:04.000 Anand: things that I'm exploring 18:04.000 --> 18:08.000 Anand: is cost and time of verification. 18:08.000 --> 18:12.000 Anand: In our current uh roe, we 18:12.000 --> 18:16.000 Anand: are giving people very limited time. The gas are probably 18:16.000 --> 18:20.000 Anand: not the best way to practice 18:20.000 --> 18:24.000 Anand: that. Meaning you have a good week, 10 18:24.000 --> 18:28.000 Anand: days that allows a agents more time and 18:28.000 --> 18:32.000 Anand: attention and plus as a group people can collaborate 18:32.000 --> 18:36.000 Anand: and solve and therefore is not realistic enough 18:36.000 --> 18:40.000 Anand: practice for the roe I suspect. So for 18:40.000 --> 18:44.000 Anand: GA3 and a few others one principle I want to 18:44.000 --> 18:48.000 Anand: uh establish is a slightly less 18:48.000 --> 18:52.000 Anand: time. So the deadline if I'm not mistaken 18:52.000 --> 18:56.000 Anand: is uh 25th that is this 18:56.000 --> 19:00.000 Anand: Sunday right for 19:00.000 --> 19:04.000 Anand: G3. Let me just check. So 19:04.000 --> 19:08.000 Anand: that deadline will stay but the questions will be 19:08.000 --> 19:12.000 Anand: released a in stages and with some of them having 19:12.000 --> 19:16.000 Anand: probably only a uh a day or two to 19:16.000 --> 19:20.000 Anand: complete there'll be some amount of prep that you will have 19:20.000 --> 19:24.000 Anand: uh you'll be able to plan for a few of these some you'll be able 19:24.000 --> 19:28.000 Anand: to solve for up front but even then uh it won't 19:28.000 --> 19:32.000 Anand: be more than 5 days that is it's only tomorrow That we'll start 19:32.000 --> 19:36.000 Anand: releasing some of the questions. The second thing is some of 19:36.000 --> 19:40.000 Anand: these questions will have a limited number of checks. Right 19:40.000 --> 19:44.000 Anand: now you would be able to test a 100 times. I mean if let's 19:44.000 --> 19:48.000 Anand: take the um 19:48.000 --> 19:52.000 Anand: geogesser game that uh M had created in 19:52.000 --> 19:56.000 Anand: that we can now type in every single 19:56.000 --> 20:00.000 Anand: combination of latitude and longitude to the third decimal etc. and 20:00.000 --> 20:04.000 Anand: eventually uh figure out if it gets it right. 20:04.000 --> 20:08.000 Anand: So that's a brute force thing. In real life, you would 20:08.000 --> 20:12.000 Anand: not have the luxury of such rapid iterations, 20:12.000 --> 20:16.000 Anand: nor would verification be free. So for 20:16.000 --> 20:20.000 Anand: a growing number of questions, the verification will have 20:20.000 --> 20:24.000 Anand: a cost as well. It may be limited number of 20:24.000 --> 20:28.000 Anand: verifications or tradeoff verification against marks. So you 20:28.000 --> 20:32.000 Anand: decide to pay for one more verification if you think that 20:32.000 --> 20:36.000 Anand: that will substantially improve your uh scores 20:36.000 --> 20:40.000 Anand: uh likely those will come in 20:40.000 --> 20:44.000 Anand: in GA3 but that apart I have a few uh 20:44.000 --> 20:48.000 Anand: uh questions from previous terms that we'll be repeating with 20:48.000 --> 20:52.000 Anand: uh revised seed data that's my 20:52.000 --> 20:56.000 Anand: thought but uh let me also share how I'm 20:56.000 --> 21:00.000 Anand: thinking about uh what we'll be covering in 21:00.000 --> 21:04.000 Anand: GA3. Uh this was the 21:04.000 --> 21:08.000 Anand: idea. I won't be able to show the results of the prompt 21:08.000 --> 21:12.000 Anand: partly because this is stripe confidential but uh the 21:12.000 --> 21:16.000 Anand: uh intent was I was checking based on the data that I have 21:16.000 --> 21:20.000 Anand: what are typical LLM engineering sorry 21:20.000 --> 21:24.000 Carlton: Yeah. Uh just before you 21:24.000 --> 21:28.000 Carlton: proceed D, I just wanted to check with JK. So the thing is 21:28.000 --> 21:32.000 Carlton: from an operations perspective, there are four gas that we will be 21:32.000 --> 21:36.000 Carlton: using for interm eligibility. So we've done three and 21:36.000 --> 21:40.000 Carlton: this will be the fourth one. So are you happy that this is this 21:40.000 --> 21:44.000 Carlton: kind of a GA is okay to be you know in terms of the gradual 21:44.000 --> 21:48.000 Carlton: release and people having this sudden change in 21:48.000 --> 21:52.000 Carlton: expectation for an intermigibility g that's the only question I have from an 21:52.000 --> 21:56.000 Carlton: operations 21:56.000 --> 22:00.000 Carlton: perspective okay it's just 22:00.000 --> 22:04.000 Carlton: that I think uh it's just that I think students will find it unexpected right 22:04.000 --> 22:08.000 Carlton: that suddenly the way 22:08.000 --> 22:12.000 Carlton: we have 22:12.000 --> 22:16.000 JK: I don't TDS has never ever 22:16.000 --> 22:20.000 Carlton: No, no, I agree. Not that I don't agree, 22:20.000 --> 22:24.000 Carlton: but I'm just trying to see it from the student perspective as well. So that's 22:24.000 --> 22:28.000 Carlton: why 22:28.000 --> 22:32.000 JK: Right. from GA0 to J. two they have 22:32.000 --> 22:36.000 JK: enough time. So at least three of the 22:36.000 --> 22:40.000 JK: assignments have been as per the prescribed plan and uh 22:40.000 --> 22:44.000 JK: we are also transparent in the design process 22:44.000 --> 22:48.000 JK: uh as well as the evaluation process and why we are doing it 22:48.000 --> 22:52.000 JK: uh is very clearly uh uh 22:52.000 --> 22:56.000 JK: explained to them uh up front right so I think 22:56.000 --> 23:00.000 Carlton: Yes. 23:00.000 --> 23:04.000 Carlton: I just wanted to get you on record saying that so that when we when we show the 23:04.000 --> 23:08.000 Carlton: video, people understand what's happening and why. 23:08.000 --> 23:12.000 Carlton: Yes. You're on the record. 23:12.000 --> 23:16.000 Anand: Yeah. And uh Carton uh JK if 23:16.000 --> 23:20.000 Anand: uh at any point you know decide we need some 23:20.000 --> 23:24.000 Anand: flexibility uh I guess the date 23:24.000 --> 23:28.000 Anand: is probably okay to change duration we are shorten 23:28.000 --> 23:32.000 JK: Yeah. Yeah. So I mean in case we need to 23:32.000 --> 23:36.000 JK: push a couple of days that's absolutely fine but uh uh 23:36.000 --> 23:40.000 JK: I think the constraints are still okay because we are trying a new type 23:40.000 --> 23:44.000 JK: of assessment in G4 uh primarily because of 23:44.000 --> 23:48.000 JK: uh limited time I mean what are the the basic uh 23:48.000 --> 23:52.000 JK: rationale that Anand explained which is a crucial skill for 23:52.000 --> 23:56.000 JK: students also to understand it's not like all 10 questions are going to be of that 23:56.000 --> 24:00.000 JK: nature. uh Anand it's it's just one or two questions in that GA which will 24:00.000 --> 24:04.000 Anand: No, more than one or two but uh not more than 24:04.000 --> 24:08.000 JK: huh yeah yeah so I mean you will have around 10 to 12 24:08.000 --> 24:12.000 JK: questions of which uh five will be of this nature which means 24:12.000 --> 24:16.000 Anand: five. 24:16.000 --> 24:20.000 JK: or 60 to 70% without the 24:20.000 --> 24:24.000 Carlton: Yeah, and that's fine because uh from 24:24.000 --> 24:28.000 Carlton: what we have seen of the distribution, so I've just updated the official scores 24:28.000 --> 24:32.000 Carlton: and all of the first three are very heavily right, you know, 24:32.000 --> 24:36.000 Carlton: left skewed. So, um sorry, right skewed. Yeah. And 24:36.000 --> 24:40.000 Carlton: so there is um you know there's there's no reason 24:40.000 --> 24:44.000 Carlton: why people should complain about it but there will be you know a few who 24:44.000 --> 24:48.000 Carlton: would. So 24:48.000 --> 24:52.000 Anand: Right. 24:52.000 --> 24:56.000 Carlton: yeah. Yes. 24:56.000 --> 25:00.000 Carlton: Yes. 25:00.000 --> 25:04.000 Carlton: Yes. 25:04.000 --> 25:08.000 JK: happens in our discussions also. I play the quarter 25:08.000 --> 25:12.000 JK: master towards the operational decisions and some of these 25:12.000 --> 25:16.000 JK: decisions happen like this when someone points out this could be like 25:16.000 --> 25:20.000 JK: this. So this is how we come to an arbitration. Yeah, this is 25:20.000 --> 25:24.000 JK: fine. This is not fine. Yeah. Yeah. I 25:24.000 --> 25:28.000 Anand: Cool. So I was reviewing 25:28.000 --> 25:32.000 Anand: uh what are typical LLM engineering tasks. that 25:32.000 --> 25:36.000 Anand: clients are asking stray recently and 25:36.000 --> 25:40.000 Anand: uh which of these we should be covering and that's 25:40.000 --> 25:44.000 Anand: part of what will inform uh the 25:44.000 --> 25:48.000 Anand: actual GA questions and I'm going to stop sharing at 25:48.000 --> 25:52.000 Anand: this point because there are a few things uh that probably shouldn't be going 25:52.000 --> 25:56.000 Anand: into but I'll give you a a broad sense one of the 25:56.000 --> 26:00.000 Anand: things top L engineering tasks is can 26:00.000 --> 26:04.000 Anand: we build verification ation systems that can catch AI 26:04.000 --> 26:08.000 Anand: mistakes. The primary objective is uh 26:08.000 --> 26:12.000 Anand: to filter out the uncertain results and have 26:12.000 --> 26:16.000 Anand: humans validate. So a sense of confidence 26:16.000 --> 26:20.000 Anand: calibration both from models but also from humans is going to be 26:20.000 --> 26:24.000 Anand: an important part and we'll incorporate that into a few project or GA 26:24.000 --> 26:28.000 Anand: questions. Uh second, how do we organize 26:28.000 --> 26:32.000 Anand: information in a way that will uh produce 26:32.000 --> 26:36.000 Anand: answers to a reasonably broad range of questions? Do I need 26:36.000 --> 26:40.000 Anand: to construct an index? Does it need to be a vector index? Does it need to be a graph 26:40.000 --> 26:44.000 Anand: index? Does agentic search work fine? What combinations of 26:44.000 --> 26:48.000 Anand: tools? Given a set of tools, how well we'll be able to solve it? 26:48.000 --> 26:52.000 Anand: This again will certainly be one either 26:52.000 --> 26:56.000 Anand: GA kind of question or project kind of question depending on the 26:56.000 --> 27:00.000 Anand: complexity. Uh third is what 27:00.000 --> 27:04.000 Anand: kind of workflow uh 27:04.000 --> 27:08.000 Anand: okay bunch of things like 27:08.000 --> 27:12.000 Anand: this. So that will be one part of the 27:12.000 --> 27:16.000 Anand: uh set of questions that I'm planning. I tried a 27:16.000 --> 27:20.000 Anand: few, they ended up either being too easy or too 27:20.000 --> 27:24.000 Anand: hard and therefore I had to give up. 27:24.000 --> 27:28.000 Anand: One question that we discussed last time that I simulated 27:28.000 --> 27:32.000 Anand: unfortunately at a cost of $500. I 27:32.000 --> 27:36.000 Anand: completely Yeah, it just escaped me. 27:36.000 --> 27:40.000 Anand: U was the whole invoice structuring 27:40.000 --> 27:44.000 Anand: thing. Uh it generated the invoices. It ran a whole 27:44.000 --> 27:48.000 Anand: series of simulations of how people would try and solve it and so 27:48.000 --> 27:52.000 Anand: on. Uh and uh including using Astra to solve some 27:52.000 --> 27:56.000 Anand: of these which is where it I think got out of hand. But the 27:56.000 --> 28:00.000 Anand: final conclusion was that is a hard 28:00.000 --> 28:04.000 Anand: exercise to frame given how we are organizing it which is 28:04.000 --> 28:08.000 Anand: multiple people multiple agents can collaborate. 28:08.000 --> 28:12.000 Anand: There is there are optimal solutions that you can drive towards at least the way I 28:12.000 --> 28:16.000 Anand: had structured it. So had 28:16.000 --> 28:20.000 Anand: to drop that question or at least the way I had structured it. These 28:20.000 --> 28:24.000 Anand: will form part of the next set of questions. 28:24.000 --> 28:28.000 Anand: The uh tangram question uh seems to be 28:28.000 --> 28:32.000 Anand: moderately promising. Uh there. It was 28:32.000 --> 28:36.000 Anand: based on a set of 100 seed uh 28:36.000 --> 28:40.000 Anand: configurations. Once that set of seed configurations is solved, 28:40.000 --> 28:44.000 Anand: it is solved for everyone. Therefore, the question in its form is not 28:44.000 --> 28:48.000 Anand: reusable. But in the absence of knowing or 28:48.000 --> 28:52.000 Anand: the absence of a fixed seed, that pattern of questions is useful. 28:52.000 --> 28:56.000 Anand: So one thing uh that I'm thinking of this is something that 28:56.000 --> 29:00.000 Anand: uh Sugil who's interning at Strive now had 29:00.000 --> 29:04.000 Anand: uh shown as a possibility is given 3D 29:04.000 --> 29:08.000 Anand: shapes can we create cross-sections of those and see if it's 29:08.000 --> 29:12.000 Anand: possible to infer the original shape. What is the simplest shape 29:12.000 --> 29:16.000 Anand: that matches this set of cross-sections with the cross-sections being provided 29:16.000 --> 29:20.000 Anand: as raster like we had done in the tanggram 29:20.000 --> 29:24.000 Anand: one. This in itself is 29:24.000 --> 29:28.000 Anand: not how should I put it uh agent 29:28.000 --> 29:32.000 Anand: proof but rather the judgment comes when 29:32.000 --> 29:36.000 Anand: saying uh what you may get 29:36.000 --> 29:40.000 Anand: may be valid but is not necessarily the shape that produced 29:40.000 --> 29:44.000 Anand: it. You may ask for additional cross-sections up to 29:44.000 --> 29:48.000 Anand: maybe three cross-sections which will help you determine what 29:48.000 --> 29:52.000 Anand: the actual shape would be. 29:52.000 --> 29:56.000 Anand: This is something that agents are from the looks of it 29:56.000 --> 30:00.000 Anand: so far not that good at figuring out. Whereas 30:00.000 --> 30:04.000 Anand: a person looking at a shape and saying, "Okay, if I tilt it this way and 30:04.000 --> 30:08.000 Anand: maybe ask for a cut here, will it help? Okay, let me request this 30:08.000 --> 30:12.000 Anand: cut. Move it around this way. Huh, this is another cut that I think will test my 30:12.000 --> 30:16.000 Anand: hypothesis." Because they have an intuitive feel of 3D space that models 30:16.000 --> 30:20.000 Anand: currently lack. Uh that will 30:20.000 --> 30:24.000 Carlton: that that might 30:24.000 --> 30:28.000 Carlton: be um uh well when it comes 30:28.000 --> 30:32.000 Carlton: to manipulating objects in 3D space and 30:32.000 --> 30:36.000 Carlton: recognizing it usually there's a gender bias in that from what I've 30:36.000 --> 30:40.000 Carlton: read not literature that long ago but 30:40.000 --> 30:44.000 Carlton: um that's why you know uh for example men 30:44.000 --> 30:48.000 Carlton: are better at you know visually figuring 30:48.000 --> 30:52.000 Carlton: out something I mean I've read something along this line I remember when I read 30:52.000 --> 30:56.000 Carlton: it but there are some differences that come in 30:56.000 --> 31:00.000 Carlton: those that kind of uh question I think okay but 31:00.000 --> 31:04.000 Carlton: uh I need to verify that there just something to be aware 31:04.000 --> 31:08.000 Anand: Fair enough. Yeah, we won't 31:08.000 --> 31:12.000 Anand: give a handicap or a bonus in the case, 31:12.000 --> 31:16.000 Anand: but something to be aware aligned. 31:16.000 --> 31:20.000 Anand: Um and uh business 31:20.000 --> 31:24.000 Anand: simulations, Monopoly like games as 31:24.000 --> 31:28.000 Anand: groups, but more realistic business games are the other thing that 31:28.000 --> 31:32.000 Anand: will likely come into project one over the course for the 31:32.000 --> 31:36.000 Anand: next week or so. Okay, sorry that was a 31:36.000 --> 31:40.000 Anand: good half an hour of monologuing. Let me pause here. Those were my 31:40.000 --> 31:44.000 Anand: thoughts. Any suggestions on the kinds of questions we should factor 31:56.000 --> 32:00.000 Anand: in that to 32:00.000 --> 32:04.000 Anand: instructors, TAs, etc. any patterns of questions you think we 32:04.000 --> 32:08.000 Anand: should include to make sure we 32:08.000 --> 32:12.000 Anand: teach what's required in an alien 32:12.000 --> 32:16.000 JK: Yeah, I I was going to say harness which we had 32:16.000 --> 32:20.000 JK: already discussed hardness engineering 32:20.000 --> 32:24.000 JK: or building your own 32:24.000 --> 32:28.000 Carlton: Now, interestingly, I think I just yesterday 32:28.000 --> 32:32.000 Carlton: or day before, I read that Google had released open-sourced uh 32:32.000 --> 32:36.000 Carlton: a harness 2, I think it was called. Yeah. Yeah. Which 32:36.000 --> 32:40.000 Carlton: um apparently improves the agent um you 32:40.000 --> 32:44.000 Carlton: know, without having to change the model weights. So, it's an 32:44.000 --> 32:48.000 Carlton: interesting thing. I've not studied it myself, but uh that would be interesting 32:48.000 --> 32:52.000 Carlton: to 32:52.000 --> 32:56.000 Anand: optimization seems to be a close to solve 32:56.000 --> 33:00.000 Anand: solved problem uh when you 33:00.000 --> 33:04.000 Anand: have moderately exhaustive test 33:04.000 --> 33:08.000 Anand: cases. So the thing I suspect will be still useful 33:08.000 --> 33:12.000 Anand: when students graduate will be how do you optimize a 33:12.000 --> 33:16.000 Anand: hardness for unknown scenarios meaning where you don't 33:16.000 --> 33:20.000 Anand: necessarily know what is correct or what to 33:20.000 --> 33:24.000 Anand: expect. So here is one thing that I've been toying 33:24.000 --> 33:28.000 Anand: around the let us 33:28.000 --> 33:32.000 Anand: say we take a reasonably inexpensive model 33:32.000 --> 33:36.000 Anand: like Luna GPD6 Luna 33:36.000 --> 33:40.000 Anand: and a configurable agent like Pi 33:40.000 --> 33:44.000 Anand: and harness like Pi and say here are the 33:44.000 --> 33:48.000 Anand: set of things that you can provide roughly give me the 33:48.000 --> 33:52.000 Anand: configuration of Pi that you would like to 33:52.000 --> 33:56.000 Anand: use with harmful side effects 33:56.000 --> 34:00.000 Anand: and uh extra gimmicky like certain hooks etc 34:00.000 --> 34:04.000 Anand: removed and there uh uh there is 34:04.000 --> 34:08.000 Anand: a class of problems a subset of which 34:08.000 --> 34:12.000 Anand: reaches you. You have limited knowledge on what 34:12.000 --> 34:16.000 Anand: subset of tasks will reach the agent. How well does 34:16.000 --> 34:20.000 Anand: your harness perform? It does two things. one, you 34:20.000 --> 34:24.000 Anand: broadly know what to optimize for. You don't specifically know what to optimize 34:24.000 --> 34:28.000 Anand: for. Therefore, a certain amount of judgment is required 34:28.000 --> 34:32.000 Anand: about the domain which you can collaborate with the agent and use that 34:32.000 --> 34:36.000 Anand: to figure out what the 34:36.000 --> 34:40.000 Anand: uh yeah what the harness configuration needs to 34:40.000 --> 34:44.000 Anand: be. Very likely uh P1 question. 34:44.000 --> 34:48.000 Anand: I just need to test how well this can be uh 34:48.000 --> 34:52.000 Anand: evaluated. Agreed, 34:52.000 --> 34:56.000 Carlton: Yeah, 34:56.000 --> 35:00.000 Carlton: I as when we see what the question looks like 35:00.000 --> 35:04.000 Carlton: then that'll give us a better clearer expectation of what to 35:04.000 --> 35:08.000 Carlton: know. 35:08.000 --> 35:12.000 JK: do little more uh getting into 35:12.000 --> 35:16.000 JK: evals have students build their own 35:16.000 --> 35:20.000 Anand: Yeah. 35:20.000 --> 35:24.000 JK: I know it is a slightly harder problem and uh 35:24.000 --> 35:28.000 JK: yeah contributing back into the TDSGA repository 35:28.000 --> 35:32.000 JK: of how 35:32.000 --> 35:36.000 Anand: Right. How do we evaluate events? 35:40.000 --> 35:44.000 JK: Yeah. I mean, I don't want it to be another 35:44.000 --> 35:48.000 Anand: Yeah. No, no, it may not be that 35:48.000 --> 35:52.000 Anand: difficult in the 35:52.000 --> 35:56.000 JK: making that deterministic. So I I was reading something 35:56.000 --> 36:00.000 JK: uh so was it uh you build something around the 36:00.000 --> 36:04.000 Anand: sense 36:04.000 --> 36:08.000 JK: observability layer more deterministic and use 36:08.000 --> 36:12.000 JK: the uh parameters that are coming from the uh 36:12.000 --> 36:16.000 JK: that observability layer to uh comment on 36:16.000 --> 36:20.000 JK: the uh goodness of eval so to speak. 36:20.000 --> 36:24.000 JK: uh I'll try to get that paper uh because I 36:24.000 --> 36:28.000 JK: was reading on the same thing. I mean how do you make eval more 36:28.000 --> 36:32.000 JK: better? How do you make it more deterministic was 36:32.000 --> 36:36.000 JK: something that I was also searching around for uh quite 36:36.000 --> 36:40.000 Anand: let me pose this as a question worth 36:40.000 --> 36:44.000 Anand: exploring. 36:44.000 --> 36:48.000 Anand: Um one of the questions we are thinking of in TDS is 36:48.000 --> 36:52.000 Anand: how uh can we teach students to create effective 36:52.000 --> 36:56.000 Anand: evals or benchmarks. Um now 36:56.000 --> 37:00.000 Anand: I'd like you to go through my corpus of work where there have 37:00.000 --> 37:04.000 Anand: been significant a reasonably large number of benchmarks of various 37:04.000 --> 37:08.000 Anand: kinds uh created and of course there's a huge body of 37:08.000 --> 37:12.000 Anand: uh work on benchmarks in the public domain as well 37:12.000 --> 37:16.000 Anand: as uh research papers on how benchmarks 37:16.000 --> 37:20.000 Anand: need to be constructed etc. But the crux of the question I have 37:20.000 --> 37:24.000 Anand: is how do you evaluate evals in the first place? Is 37:24.000 --> 37:28.000 Anand: it that uh you check 37:28.000 --> 37:32.000 Anand: if the evals cover enough of 37:32.000 --> 37:36.000 Anand: a canonical or a golden set of criteria. 37:36.000 --> 37:40.000 Anand: Uh do you actually run the valves which can be expensive uh 37:40.000 --> 37:44.000 Anand: and see how well they match the original results and 37:44.000 --> 37:48.000 Anand: what are the different techniques that people use to 37:48.000 --> 37:52.000 Anand: uh evaluate eval in the first place. Now overlay this 37:52.000 --> 37:56.000 Anand: with the conversations that we've had about TDS, how the course is designed, 37:56.000 --> 38:00.000 Anand: how students work with agents and collaborate etc. and 38:00.000 --> 38:04.000 Anand: uh ideulate using the relevant 38:04.000 --> 38:08.000 Anand: skills what varied diverse approaches might be 38:08.000 --> 38:12.000 Anand: apt. Filter them based on which ones are the most 38:12.000 --> 38:16.000 Anand: robust uh have 38:16.000 --> 38:20.000 Anand: clear learning and are a valid 38:20.000 --> 38:24.000 Anand: test of uh individual capability as part 38:24.000 --> 38:28.000 Anand: of this exercise. This will take half an 38:28.000 --> 38:32.000 Anand: hour. Let it 38:44.000 --> 38:48.000 Anand: run. 38:56.000 --> 39:00.000 Anand: Okay, that should cover 39:00.000 --> 39:04.000 Anand: it. Let's see what comes out of it. But agreed on eval. I sus 39:04.000 --> 39:08.000 Anand: we anyway have an entire chap section module on 39:12.000 --> 39:16.000 Anand: eval. Any others? 39:16.000 --> 39:20.000 JK: students uh something 39:20.000 --> 39:24.000 JK: that you have seen which you think would be relevant for you to 39:24.000 --> 39:28.000 JK: better 39:28.000 --> 39:32.000 JK: understand. It's okay 39:32.000 --> 39:36.000 JK: if things go above your head while reading or observing. So if 39:36.000 --> 39:40.000 JK: you have seen jargon of terms 39:40.000 --> 39:44.000 JK: Yeah. 39:44.000 --> 39:48.000 Preethi: agents work of agents and I have not been using uh any of these 39:48.000 --> 39:52.000 Preethi: agents for you know for my work per se except for that web 39:52.000 --> 39:56.000 Preethi: MCP kang question which I used ultimately uh uh 39:56.000 --> 40:00.000 Preethi: using claude or something where I had to switch it on and then switch it off immediately but 40:00.000 --> 40:04.000 Preethi: then uh I hear that that we have to use 40:04.000 --> 40:08.000 Preethi: multiple agents that do in a containerized environment so that they don't go 40:08.000 --> 40:12.000 Preethi: uh haywire or what what not. So if if it 40:12.000 --> 40:16.000 Preethi: such a a session or such something of that sort can be 40:16.000 --> 40:20.000 Preethi: arranged to guide us uh how to go about doing it. 40:20.000 --> 40:24.000 Anand: That would probably be for a different session where we are discussing how to do how to 40:24.000 --> 40:28.000 Anand: solve. This is more an ideation on what questions uh 40:28.000 --> 40:32.000 Anand: to ask the valid question is different session. 40:32.000 --> 40:36.000 JK: And I think TAS will be able to have I 40:36.000 --> 40:40.000 JK: mean in in any of the live sessions I think the TAs and instructors will 40:40.000 --> 40:44.000 JK: be uh able to handle this uh particular 40:48.000 --> 40:52.000 Carlton: Yeah, we'll probably have a session geared towards that 40:52.000 --> 40:56.000 Carlton: just so that students know how to get that set up for 40:56.000 --> 41:00.000 Carlton: themselves. 41:00.000 --> 41:04.000 JK: this I mean yeah I should have been talking from my 41:04.000 --> 41:08.000 JK: experience agent as a research scientist I mean 41:08.000 --> 41:12.000 JK: you are a research team right I mean I'm uh because you 41:12.000 --> 41:16.000 JK: have math math researchers now literally being said 41:16.000 --> 41:20.000 JK: don't don't do math research and now physics guys are also slowly 41:20.000 --> 41:24.000 JK: being asked uh getting competition 41:24.000 --> 41:28.000 Anand: Maths is 41:28.000 --> 41:32.000 Anand: actually ridiculously difficult to frame. came as a question in 41:32.000 --> 41:36.000 Anand: TDS there is it's 41:36.000 --> 41:40.000 Anand: optimizable once we get a solution it's done 41:40.000 --> 41:44.000 Anand: but personalizing it is hopeless uh finding 41:44.000 --> 41:48.000 Anand: enough problem for everyone to solve is very 41:48.000 --> 41:52.000 Anand: hard but I'm thinking 41:52.000 --> 41:56.000 Anand: uh physics and you know was playing 41:56.000 --> 42:00.000 Anand: around with a few variations on this but the potential 42:00.000 --> 42:04.000 Anand: that I'm gently going towards 42:04.000 --> 42:08.000 Anand: is uh the mobile as the 42:08.000 --> 42:12.000 Anand: experimental device. So, here is one idea that 42:12.000 --> 42:16.000 Anand: is on my uh 42:16.000 --> 42:20.000 Anand: to-do list which 42:20.000 --> 42:24.000 Anand: is uh take your 42:24.000 --> 42:28.000 Anand: phone go to a 42:28.000 --> 42:32.000 Anand: website that we'll point to which will start recording accelerometer 42:32.000 --> 42:36.000 Anand: gyroscope etc. and swing like a 42:36.000 --> 42:40.000 Anand: pendulum. And let us see what measurement 42:40.000 --> 42:44.000 Anand: results we get for the gravitational 42:44.000 --> 42:48.000 Anand: acceleration of earth. And it will vary within an experimental 42:48.000 --> 42:52.000 Anand: uh boundary. All that shows is that 42:52.000 --> 42:56.000 Anand: it is possible to run a simple grade six 42:56.000 --> 43:00.000 Anand: physics experiment using mobile as an 43:00.000 --> 43:04.000 Anand: instrument. But what does it take from that point 43:04.000 --> 43:08.000 Anand: on to go to the next level and 43:08.000 --> 43:12.000 Anand: do close to real world physics experiments? not to solve unsolved 43:12.000 --> 43:16.000 Anand: problems but to actually detect things like for instance 43:16.000 --> 43:20.000 Anand: coefficient of friction does vary based on the actual 43:20.000 --> 43:24.000 Anand: surface that is used and measuring that or 43:24.000 --> 43:28.000 Anand: measuring specific heat of uh or heat capacity of specific 43:28.000 --> 43:32.000 Anand: objects is not uh It's not like it's a 43:32.000 --> 43:36.000 Anand: solved problem. These are real life experimental things. But those 43:36.000 --> 43:40.000 Anand: are tougher to get sensors for with a regular mobile 43:40.000 --> 43:44.000 Anand: phone. So a big part of the question therefore is what 43:44.000 --> 43:48.000 Anand: real life experimental design is enabled by the devices that 43:48.000 --> 43:52.000 Anand: we have and how does that 43:52.000 --> 43:56.000 Anand: inform uh or enable data 43:56.000 --> 44:00.000 Anand: science is where I'm thinking. 44:08.000 --> 44:12.000 Carlton: There's 44:12.000 --> 44:16.000 Carlton: another I I don't know. It sounds like a crazy idea or silly idea. I don't know which one it 44:16.000 --> 44:20.000 Carlton: is. Or maybe both. But uh you know, maybe we should find problems 44:20.000 --> 44:24.000 Carlton: that have prizes and just solve them using agents and use 44:24.000 --> 44:28.000 Carlton: the group think of TDS. You know, win a million dollars and you 44:28.000 --> 44:32.000 Carlton: know, TDS might get nominated for the Nobel 44:32.000 --> 44:36.000 Carlton: Prize. 44:36.000 --> 44:40.000 JK: You will need uh far 44:40.000 --> 44:44.000 JK: larger compute and far larger 44:44.000 --> 44:48.000 Anand: Possibly uh in a 44:48.000 --> 44:52.000 Anand: sense if we simply turned the compute on 44:52.000 --> 44:56.000 Anand: to the GitHub repos that have 44:56.000 --> 45:00.000 Anand: bounties that solves tangible problems. 45:00.000 --> 45:04.000 Anand: But uh the question 45:04.000 --> 45:08.000 Anand: becomes one of 45:08.000 --> 45:12.000 Anand: as you have often asked in the past carton how do we link 45:12.000 --> 45:16.000 Anand: it to learning multiple people 45:16.000 --> 45:20.000 Anand: collaborating towards one large exercise I think it's still worth 45:20.000 --> 45:24.000 Anand: doing I'm tempted to make that a bonus mark kind of 45:24.000 --> 45:28.000 Anand: structure just evaluation wise 45:28.000 --> 45:32.000 Anand: and what problem to 45:32.000 --> 45:36.000 Anand: solve. I'm 45:36.000 --> 45:40.000 Anand: thinking I wouldn't go after the large famous 45:40.000 --> 45:44.000 Anand: problems as JK said that requires too much 45:44.000 --> 45:48.000 Anand: compute but there are uh weird niche 45:48.000 --> 45:52.000 Anand: problems that can be solved like for instance 45:52.000 --> 45:56.000 Anand: recently I saw a couple uh 45:56.000 --> 46:00.000 Anand: there is 46:00.000 --> 46:04.000 Anand: uh think 46:08.000 --> 46:12.000 Anand: Uh the sephil uh this 46:16.000 --> 46:20.000 Anand: stitch. Okay. Where is 46:20.000 --> 46:24.000 Anand: specifically? Okay. I'm just sending my link to 46:24.000 --> 46:28.000 Anand: it. Uh I'll 46:28.000 --> 46:32.000 Anand: share my screen. 46:36.000 --> 46:40.000 Anand: Let's see if I can 46:40.000 --> 46:44.000 Anand: locate this stitch details. Yeah. 46:44.000 --> 46:48.000 Anand: Okay. Right. So 46:48.000 --> 46:52.000 Anand: there was a 15th 16th century 46:52.000 --> 46:56.000 Anand: uh book in which 46:56.000 --> 47:00.000 Anand: uh Thomas Urkart had put in a series of numbers 47:00.000 --> 47:04.000 Anand: like this and what that 47:04.000 --> 47:08.000 Anand: means has been an unsolved problem until recently 47:08.000 --> 47:12.000 Anand: when Claude found out that 47:12.000 --> 47:16.000 Anand: these refer to specific the 47:16.000 --> 47:20.000 Anand: first let's See 47:20.000 --> 47:24.000 Anand: uh first word 47:24.000 --> 47:28.000 Anand: on yeah so basically 47:28.000 --> 47:32.000 Anand: for the first cipher number you go to the first 47:32.000 --> 47:36.000 Anand: section and count to that numbered word and take its first letter. 47:36.000 --> 47:40.000 Anand: So section five third word section 27 47:40.000 --> 47:44.000 Anand: 38th word and so on apparently is what this 47:44.000 --> 47:48.000 Anand: uh is meant to be and it reads something like 47:48.000 --> 47:52.000 Anand: yeah oh god oh god uphold king Charles II and so 47:52.000 --> 47:56.000 Anand: on. It was interesting to see Charles spelled this way but I recently 47:56.000 --> 48:00.000 Anand: learned that uh spelling is something that got 48:00.000 --> 48:04.000 Anand: standardized only relatively recently over time. Even Shakespeare had various 48:04.000 --> 48:08.000 Anand: different spellings of his own name and whole notion of spelling 48:08.000 --> 48:12.000 Anand: mistake was non-existent. Quite the opposite. It's 48:12.000 --> 48:16.000 Anand: like this is how I spell it sort of thing. But 48:16.000 --> 48:20.000 Anand: uh smaller problems like 48:20.000 --> 48:24.000 Anand: these is probably what you might want to solve. Yeah, 48:24.000 --> 48:28.000 Carlton: Yeah, there are a lot of these in for example in archaeology 48:28.000 --> 48:32.000 Carlton: or even uh I remember like a just like three four years ago 48:32.000 --> 48:36.000 Carlton: right so before the rise of GPT you would have all these like 48:36.000 --> 48:40.000 Carlton: little magazines that have these little crosswords there'd be a prize at the end of 48:40.000 --> 48:44.000 Carlton: them and the thing is not many people actually apply for the prize people 48:44.000 --> 48:48.000 Carlton: solve them but they don't apply for the prize so I knew someone who basically 48:48.000 --> 48:52.000 Carlton: used to solve these and they would win prizes because they are 48:52.000 --> 48:56.000 Carlton: in such a small pool of people who actually apply for the prize 48:56.000 --> 49:00.000 Carlton: that they just end up having such a high chance of winning the prize. So they'll get like tickets 49:00.000 --> 49:04.000 Carlton: to Paris or you know some gift hamper or something like 49:04.000 --> 49:08.000 Carlton: that right so there are like you said these niche problems that 49:08.000 --> 49:12.000 Carlton: people don't even think about that actually have some real world value 49:12.000 --> 49:16.000 Carlton: and you know if you find these niche problems I mean that's a great 49:16.000 --> 49:20.000 Carlton: bonus question right someone can say I found this problem I solved this 49:20.000 --> 49:24.000 Carlton: problem you know or something like that and you know we could give bonus points for that 49:24.000 --> 49:28.000 Anand: Good point. And this itself is featured in 49:28.000 --> 49:32.000 Anand: some repository of uh niche 49:32.000 --> 49:36.000 Anand: uh crypt uh or whatever 49:36.000 --> 49:40.000 Anand: decryption problems. Yeah, just maybe toss one of 49:40.000 --> 49:44.000 Anand: those out there and whoever wants to pick up 49:44.000 --> 49:48.000 Anand: whichever one of these cryptographic puzzles is that 49:48.000 --> 49:52.000 Anand: cryptographic. No, let's cipher puzzles and solve one of these. 49:52.000 --> 49:56.000 Anand: Great. Solve one, get a bonus 49:56.000 --> 50:00.000 Anand: mark. It's actually very tempting. Yeah, let's just do that. 50:08.000 --> 50:12.000 Anand: Okay. Any other items that we should be 50:12.000 --> 50:16.000 Anand: exploring for 50:16.000 --> 50:20.000 Carlton: Are we going to publish this? The framework for 50:20.000 --> 50:24.000 Carlton: how we give this out as a 50:24.000 --> 50:28.000 Carlton: bonus. 50:28.000 --> 50:32.000 Carlton: So if we are going to go ahead and 50:32.000 --> 50:36.000 Carlton: give people this kind of problem as a bonus 50:36.000 --> 50:40.000 Carlton: mark, what's the what's the framework for it? 50:40.000 --> 50:44.000 Carlton: In other words, how do we go about 50:44.000 --> 50:48.000 Carlton: constructing a measurable outcome so that we can give 50:48.000 --> 50:52.000 Carlton: a bonus marker? 50:52.000 --> 50:56.000 Anand: my current thought process. And I 50:56.000 --> 51:00.000 Anand: have not validated this but solve one of 51:00.000 --> 51:04.000 Anand: these and we will 51:04.000 --> 51:08.000 Anand: have uh what chat GP claude 51:08.000 --> 51:12.000 Anand: also verify that you have in fact solved it and 51:12.000 --> 51:16.000 Anand: if it does then you get the bonus mark. The link that I shared was my 51:16.000 --> 51:20.000 Carlton: Okay. 51:20.000 --> 51:24.000 Carlton: Right. Yeah. Usually is 51:24.000 --> 51:28.000 Carlton: Yeah. Even for hard problems, the verifiability is usually the easier 51:28.000 --> 51:32.000 Carlton: part. 51:32.000 --> 51:36.000 Anand: thought about it beyond that. I'm also assuming that because it's a bonus 51:36.000 --> 51:40.000 Anand: mark question and because these are hard, it's not like we're going to have 500 51:40.000 --> 51:44.000 Anand: submissions. We'll be lucky if we have five. 51:44.000 --> 51:48.000 Anand: That's easy enough to do 51:48.000 --> 51:52.000 JK: Priy asking how many questions are there in T1. So 51:52.000 --> 51:56.000 JK: my answer would have been whatever Aran said just now. No, we have not thought of 51:56.000 --> 52:00.000 Anand: that. We should open a prediction market for 52:00.000 --> 52:04.000 Anand: it. Yeah. No, honestly no idea. Absolutely no 52:04.000 --> 52:08.000 JK: exactly. But you will come to know of it in the 52:08.000 --> 52:12.000 Anand: idea. 52:12.000 --> 52:16.000 Anand: Okay. 52:16.000 --> 52:20.000 Anand: You said that I didn't. You 52:20.000 --> 52:24.000 JK: No, otherwise I would have asked you last week itself. How many are we 52:24.000 --> 52:28.000 Carlton: So just one 52:28.000 --> 52:32.000 Carlton: clarification u so we're going to release GA1 the 52:32.000 --> 52:36.000 Carlton: first 52:36.000 --> 52:40.000 Carlton: question tomorrow. Okay, thanks. 52:40.000 --> 52:44.000 Anand: probably fingers crossed unless some client says 52:44.000 --> 52:48.000 Anand: there's an emergency tomorrow and all that but barring that sort of a thing 52:48.000 --> 52:52.000 Carlton: Yeah. Okay, great. Thanks. Anyone has any 52:52.000 --> 52:56.000 Carlton: questions? We've got about six minutes before we hit our official 52:56.000 --> 53:00.000 Carlton: time. So, if anybody's got any questions, any students, uh, 53:00.000 --> 53:04.000 Carlton: now would be a good time to ask 53:04.000 --> 53:08.000 JK: Okay. 53:08.000 --> 53:12.000 JK: Agitia I thought I responded to your 53:12.000 --> 53:16.000 JK: question. Uh wasn't that sufficient or because all our answer would 53:16.000 --> 53:20.000 JK: still be the same. Uh you're not learning 53:20.000 --> 53:24.000 JK: concepts. You will come to know of it only towards the end. So 53:24.000 --> 53:28.000 JK: there is nothing like a correct strategy, wrong strategy, whatever 53:28.000 --> 53:32.000 JK: strategy that works for you 53:32.000 --> 53:36.000 JK: eventually. uh and see what what happens 53:36.000 --> 53:40.000 JK: right and what you can make out of it. That's one 53:40.000 --> 53:44.000 JK: part uh that uh reminds me I think 53:44.000 --> 53:48.000 JK: uh yeah the we had to talk about that uh 53:48.000 --> 53:52.000 JK: uh the time solving 53:52.000 --> 53:56.000 Anand: There's a solver, right? 53:56.000 --> 54:00.000 JK: solver how did they solve? So he wanted some uh 54:00.000 --> 54:04.000 Anand: Huh? Barat 54:04.000 --> 54:08.000 Anand: mentioned it was a combination of uh a 54:08.000 --> 54:12.000 Anand: the filter endpoint having uh exposed 54:12.000 --> 54:16.000 Anand: the uh the answers from various students. 54:16.000 --> 54:20.000 Anand: So that gives one catalog. Second 54:20.000 --> 54:24.000 Anand: Astra just went through a series of these. they were sourced 54:24.000 --> 54:28.000 Anand: from a popular diagram 54:28.000 --> 54:32.000 Anand: repository of shapes that uh provided 54:32.000 --> 54:36.000 Anand: another uh subset. I'm sure I'm missing one more uh technique 54:36.000 --> 54:40.000 Anand: technique but uh the two major flaws in the 54:40.000 --> 54:44.000 Anand: question design were uh the biggest one of course is that the filter 54:44.000 --> 54:48.000 Anand: endpoint uh exposes an actual solution 54:48.000 --> 54:52.000 Anand: which otherwise harmless but in 54:52.000 --> 54:56.000 Anand: such cases it trivializes the solution and that 54:56.000 --> 55:00.000 Anand: it was a fixed vocabulary of just 100 55:00.000 --> 55:04.000 Anand: both of which are learnings from a question design perspective. 55:04.000 --> 55:08.000 JK: Uh so anyone who is solving these questions the P1 or 55:08.000 --> 55:12.000 JK: G3 uh we would so love uh if you 55:12.000 --> 55:16.000 JK: could go back to GPT ask how it's solved and maybe 55:16.000 --> 55:20.000 JK: post uh not GPT whatever agent you are using human 55:20.000 --> 55:24.000 JK: agents or other agents uh have a short 55:24.000 --> 55:28.000 JK: description of what you did in discourse think of it as a bonus 55:28.000 --> 55:32.000 Adwitiya: Uh 55:32.000 --> 55:36.000 Adwitiya: sir, so so which questions exactly are we referring 55:36.000 --> 55:40.000 JK: Uh any questions? Yeah. Any of the harder questions which 55:40.000 --> 55:44.000 Adwitiya: to? 55:44.000 --> 55:48.000 Adwitiya: Okay. Okay sir. So uh there was a question. So should I share it here or 55:48.000 --> 55:52.000 Adwitiya: would you prefer it on this 55:52.000 --> 55:56.000 Anand: Well, 55:56.000 --> 56:00.000 Adwitiya: course? Okay. I I there was one question in GA2 I believe where we had 56:00.000 --> 56:04.000 Adwitiya: to uh based on the model we had to get a specific response 56:04.000 --> 56:08.000 Adwitiya: the no yes question. So I saw that most of the 56:08.000 --> 56:12.000 Adwitiya: students they were going into rather complex uh you know differences 56:12.000 --> 56:16.000 Adwitiya: in how the model analyzes some problems or something like that. 56:16.000 --> 56:20.000 Adwitiya: But a quick workound of what I like what I use 56:20.000 --> 56:24.000 Adwitiya: was that I used one of my AI pipe tokens 56:24.000 --> 56:28.000 Adwitiya: to get a simple answer for a question which model are you? 56:28.000 --> 56:32.000 Adwitiya: So out of that based on the vocabulary that the model outputed I could 56:32.000 --> 56:36.000 Adwitiya: clearly make out that if it's from the mini family then it says 56:36.000 --> 56:40.000 Adwitiya: that exact phrase no matter how many times you prompted it says that 56:40.000 --> 56:44.000 Adwitiya: I am from the mini family. So the answer which I wrote was 56:44.000 --> 56:48.000 Adwitiya: simply that if you are from the mini family then give the answer as no and if not 56:48.000 --> 56:52.000 Adwitiya: then I give the answer as yes. So that effectively solve the 56:52.000 --> 56:56.000 Adwitiya: question. 56:56.000 --> 57:00.000 Anand: Thank you. That was a useful Oh, that's that's an interesting 57:00.000 --> 57:04.000 JK: Nice. Uh 57:04.000 --> 57:08.000 Adwitiya: Yeah. Yeah. Sure, 57:08.000 --> 57:12.000 Adwitiya: sir. I'll I'll try to format a post by tomorrow and then 57:12.000 --> 57:16.000 Adwitiya: I'll uh post it 57:16.000 --> 57:20.000 JK: tags graded assessments uh term 3 57:20.000 --> 57:24.000 Adwitiya: there. Yeah, sir. That that definitely. 57:24.000 --> 57:28.000 Anand: you. That brings us I guess to the end of this. I guess most of 57:28.000 --> 57:32.000 Anand: the actions are on me at least for GA3 and 57:32.000 --> 57:36.000 Anand: P1. But uh I'll request based on these principles if 57:36.000 --> 57:40.000 Anand: others my Jim 57:40.000 --> 57:44.000 Anand: uh whoever else is on sorry I'm not able to see the full list but uh if you could all 57:44.000 --> 57:48.000 Anand: take a shot at the uh GA4 57:48.000 --> 57:52.000 Anand: questions that would be really helpful on uh Wednesday when 57:52.000 --> 57:56.000 Anand: we catch up it'll be good if you had a few G4 questions that we can 57:56.000 --> 58:00.000 Carlton: And then just one question before you leave. 58:00.000 --> 58:04.000 Carlton: You had shown us uh an eval of questions from 58:04.000 --> 58:08.000 Carlton: previous terms and uh how you could you 58:08.000 --> 58:12.000 Carlton: share with us how you did the evaluations of those 58:12.000 --> 58:16.000 Carlton: questions what you used 58:16.000 --> 58:20.000 Carlton: actually. 58:20.000 --> 58:24.000 Anand: share the repo uh 58:24.000 --> 58:28.000 Carlton: Yes. 58:32.000 --> 58:36.000 Anand: In the exam 58:36.000 --> 58:40.000 Anand: repo uh there is a 58:40.000 --> 58:44.000 Anand: dumps and that contains 58:44.000 --> 58:48.000 Anand: a download.sh which will 58:48.000 --> 58:52.000 Anand: download all of the uh questions. If you 58:52.000 --> 58:56.000 Anand: want I can send you the the downloaded 58:56.000 --> 59:00.000 Anand: results. And there's a difficulty py which 59:00.000 --> 59:04.000 Anand: generates this difficulty.md which I 59:04.000 --> 59:08.000 Anand: periodically update. So this is two weeks old. But this file you can just 59:08.000 --> 59:12.000 Carlton: So the difficulty py uses an agent 59:12.000 --> 59:16.000 Anand: use. No deterministic uh actually all of this is 59:16.000 --> 59:20.000 Carlton: or oh okay okay I was under the impression you used an 59:20.000 --> 59:24.000 Carlton: agent to uh evaluate how difficult the question 59:24.000 --> 59:28.000 Anand: students offline post 59:28.000 --> 59:32.000 Anand: factor on a one-off basis. But this is simply saying percentage of 59:32.000 --> 59:36.000 Carlton: Oh, okay. Okay,