# Transcript

**Anand**: [00:40] While the others join, might as well start with a little survey with who's here. How many of you are planning to start a startup right after college? How many of you are planning to go for higher studies right after college? How many of you are planning to go to work right after college? How many of you didn't raise your hand so far? What are your plans?

**Student**: Working.

**Anand**: [01:09] Oh, you are already working. Fair, fair.

**Unsure**: Yes, faculty.

**Anand**: [01:12] Ah! You know, which is an option I should have mentioned.

**Unsure**: [01:16] We go to get a job after this course. [laughter]

**Anand**: [01:21] But no, that's interesting. So the vast majority are planning to work right after, and that is a difference from the last time I did this survey, which was 2010, I think. When- well, it was a larger batch, chemical engineering students, and one or no, sorry. Three people said they were going to run a startup right after. The majority were planning to study right after. I'm guessing that's changed. Interesting. And why work rather than study or startup?

**Student**: Finance.

**Anand**: [02:14] Money is a good reason. But would it- wouldn't your salary be higher if you had an extra degree? And over time, therefore, be a better ROI? I don't know, you've probably done the calculation. Or is it more need cash flow early? Okay, it's probably more a cash flow thing, by the few nods that I got. For those who joined in, we're just generally chatting if people are going to work, study, startup, whatever after college. We should get started, right? It's 2:00. Should we?

**Anand**: [04:13] Cool, let's dive in. Why do you bother attending classes? You can get this from a lecture online.

**Palani**: Well, it's good that you're asking the question towards the end of the course. [laughter]

**Anand**: [04:36] No, seriously, what is the point? You could get this from an online lecture from MIT, which would be better than what, certainly, what I can teach, and be better than what Bunny can teach.

**Student**: You have to come. Attendance.

**Student**: Credit.

**Anand**: [04:54] You have to- Good point.

**Student**: Credit is not an issue anymore, right? Like you can- there are people who don't attend the class, but they come and write the exam.

**Student**: For me, reading is boring.

**Anand**: [05:04] Good point. Videos?

**Student**: Like, this is like a dedicated slot so that we will come here and listen. But if I want to do that in my room, it's like it won't happen.

**Anand**: [05:16] This happened to me at SUTD. I was attending Brad's lecture, and it was interesting. I watched more or less his same class on video at home to teach it to my daughter. It is boring. Okay, fair. Once in a while, okay. Every day?

**Student**: Doubt clearance. Sometimes, we might have doubts.

**Anand**: [05:42] Which you can ask AI.

**Student**: We get to meet our friends here.

**Anand**: [05:50] You get to meet friends. In which case, you should be talking to them, not staring at me. And you don't get that opportunity, but okay, before class, after class. So basically, the one and a quarter hours is the tax for meeting friends among other things, which is fair. See, the education system is designed more for teachers, perhaps, than for students. Certainly, also designed for the convenience of teachers. It is far more convenient for us if everybody pays attention, and everybody gets the same kind of assessment and evaluation, and so on. But since you are here, and AI is anyway eating up our lives, why don't we do something slightly different today?

[06:33] Talk to your friends. Now, I have a specific question that I would like to invite all of you to talk to your friends about: pairs, triples, however you want to organize yourself. And let us take, maybe, five minutes to discuss the following question. I'm basically setting up the environment, the rules, for you to plan for the next short time.

[07:00] You've all submitted some homework. I don't yet have access to it, mostly because you've- not all of you have given me access to it. **So, if you want your grades for that, please find out my email ID and give me access.** I'm not going to give you my email ID. This much you should be able to do with Google. With AI, you should be able to do a whole lot more. Find out my email ID, give me access. I will be evaluating that definitely not by today, so if by some time by end of day today, or worst case, even sometime tomorrow, if you're able to give me access to your data visualization submissions and the phone assignment submissions, then that works. And if you have completely forgotten about this, ask your classmates, finish the assignment quickly, submit it. Not a big deal. How many of you have finished the assignments, just curious? And you don't have to have. Okay. Then, let's flip it around. How many of you have not yet finished the assignment? And it's perfectly fine, because you will get a little bit of time.

[08:03] There are 32 submissions for the data visualization and 31 submissions for the mobile application that I saw. There are more than that many in the class?

**Unsure**: Yes, 48.

**Anand**: [08:15] Okay, fine. Then the rest of you do not know that you have not yet submitted it. Please make sure that even though you think you have submitted, it is there where it is supposed to be submitted. And my AI is going to be doing the first pass. It doesn't care later on whether you've submitted it or not. It is just very happily going to give you a zero on that. It is probably going to give you a zero now itself, because you're going to have a few surprise quizzes.

[08:45] But let's dive in. These are going to be evaluated partly by my agent, partly by me, and Bunny will take a look at it. But the agent is going to have a strong point of view as well. How should we evaluate it? There is Varuna, the company. You've got some data, you've seen the data, you've presented a certain point of view. You've revised it in the last class, and now you have a revised replay visual. We are supposed to look at it and evaluate it. **How should we evaluate it is the point of discussion.**

[09:22] After you have discussed, what I'd like you to do is- I'll share this QR code again in a short while. You would be answering this question: created a visualization, how will you test? And when I say you've created a visualization, you're welcome to use the Varuna context, but the question is really aiming at a general visualization. Anybody has created a visualization: **how will you test if it achieves the intended purpose for the intended audience? What checks will you do? What experiments will you run? What evidence will you collect? What findings will make you revise or reject it?**

[10:04] But before you discuss as a group, just take a shot at filling this. Open your phones or laptops, and if you have neither, after your neighbor finishes it, borrow their phones or laptops. Scan, and finish. It's 2:07, let's say 2:08 right now. I will- there is a counter here that currently is at zero. I will wait for the counter to get to at least 25. The rest of you, answer when you answer. How will you test a visualization? After you've put in your answer, we will discuss, and you will discuss, and then we will collectively discuss.

[11:12] And in case anyone's worried that this is a little hard to type or you shouldn't speak in a class, please dictate. Feel free to use dictation. It works well.

[12:30] Give me 5 seconds, I'll put it back on.

[14:36] There's six responses in 4 minutes. You should probably just go a little faster and start sharing your thoughts.

[15:08] We are at 10 and it's 2:12. So yeah, just try wrapping up.

[15:35] Those who have finished responding, please feel free to browse the internet, play a game, whatever you want, including disturbing your neighbors.

[15:45] 17. We'll wait for six more, seven, eight more.

[16:27] 19. Okay, still need a few more.

[17:10] Come on, come on. Now, we really should wrap up. We're a couple of minutes beyond what I thought we would spend on the waiting.

[17:24] 23.

[17:44] Okay. This is one of the classic problems that educators have. Some guys get it; there's no point teaching them. Some don't; there's no point teaching them. What do we do with the middle? Or rather, the only thing that we can focus on is in the middle. But now we are just past the middle. Two people just- Who's pending?

**Unsure**: [18:08] Okay, maybe these two are visiting. So, they are not part of the class.

**Anand**: [18:13] Okay.

**Unsure**: Since you are lecturing, I asked them to come.

**Anand**: [18:15] Fair enough. No problem. Though you are very welcome to join in, that's also absolutely okay.

**Unsure**: [18:22] He seemed to have filled it.

**Anand**: [18:25] 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25. Actually, we are only 25 even if we count them.

**Unsure**: [18:39] Okay, no, no, but he seems to have submitted, did you? And you didn't submit.

**Anand**: [18:43] Okay, then fine. Then we are okay. Then I should just recalibrate and say I'm sure. Okay. Great. Now, why did I ask you to do this? Because around ninth standard, 10th standard, I you know used to get this feeling. I would read some book; it would make perfect sense. And I'd realize, "I haven't learned anything. I already knew this." Something makes perfect sense when you read it, and I started assuming therefore that I knew it beforehand. What I didn't realize was if somebody had asked me before I read that book, I probably would have given a different answer or would have said, "I have no clue." When explained, it makes perfect sense. At which point, it struck me that **one of the better ways of learning, therefore, is to put a hypothesis on the table, figure out whatever it is that I want to learn, see what is the difference between that and what I have learned. At least that difference now constitutes net learning.**

[19:51] So, you've done the first part. Now, let's open up the second question, which is actually the same one, and it will refresh on the same page. But now, I'd like you to talk to your neighbors- one, two, three, however you want to discuss. Same question. And tell me what you changed your mind on. Just what was the difference. And if you haven't changed your mind on anything, absolutely fine. For this, there are two time limits. It's 2:17, let's take it to 2:25. Stop discussion at 2:25 and start writing at that point.

[25:38] Okay, it's time for you to start wrapping up, start writing. Three minutes. Specific thing to write is: **have you changed your opinion on something? Have you changed your opinion on anything from the previous answer you gave?**

[27:24] Okay, nine responses so far.

[28:54] 15. Start wrapping up.

[29:27] Yeah, it's time you should just start submitting. Whatever it is that you have, just maybe one or two words, finish it off. 17, 18. We had 24 here, I'll wait till 20. Just a couple more and then I'm going to start, we'll start. Or okay, 20. Fine, the rest of you just quickly, four of you, wrap it up. Okay. So, I was just going through the earlier responses to the first question. I will-

---

**Anand**: [30:05] So, one thing that a number of you said was, "We will show this to people and see if they are able to understand it," which is a reasonably good design. And one interesting response was, "If people agree, the visualization works." Makes sense. Any problem with that, though?

**Student**: That person has to understand it. We are not sure whether they understand it.

**Anand**: [30:38] If everyone understands?

**Student**: [30:42] What if the- see, everyone understands, but what if something is- I mean, what if it doesn't clearly answer the question? I mean, what if the- like other factors also have to be considered, like the data is a fact and if the visualization is correct, have the facts from the data been taken properly, right?

**Anand**: [31:06] Fair point. Is it correct? I could- I'll just complete, and then you get your-

**Student**: There is a common mode, right? Like you might be asking all the people of same group. So, they might be blind to it.

**Anand**: [31:22] True. So, therefore, maybe you should ask enough people, the right people, which is part A. Part B is what you said, which is: are you showing them the right thing? If I show a graph which says men live longer, women live less, and everyone agrees on the same interpretation, except that, in truth, the reverse is the case, chart is just plain wrong.

[31:45] And, therefore, the other part of what you said is- and at least one of you had written a- quite an interesting one, which is: **recalculate the raw data, and see if it matches with the original.** That's one way of doing it. Someone else had written saying, **verify the aggregated values, but also verify whether the geometric proportions match the numbers.** Again, a useful and relevant way, because it's one thing to say, "I have written 24 and 25 here." It's another thing to show that difference as 3x, 4x, which absolutely is unwarranted.

[32:26] One of you said something interesting, which is: **deliberately look for ways to misinterpret the visualization.** That's powerful. It's not just saying, "Look, do people understand it?" But is it possible for me to find a way in which it could be misinterpreted? Which is a good protection against the common-mode problem. Somebody who we don't even know might be looking at it in a different way, and for sensitive data, that's a useful one.

[32:57] Some of you have said roughly that, "Use an AI audience." This is both good and bad. See, an AI audience is powerful in that you can get a counter perspective. What I mean by counter perspective: you can ask it to take the personality of some extremist who is looking at the data trying to misinterpret it, asking the perspective of someone who heavily holds a certain point of view, what are the ways in which they will use it. So, you can create new perspectives. Having said that, will the perspective that it take match the perspective of the audience? How well it can- how well can it see literally, and does it match the perception of a human? And it is crazy. Let me show you this.

[33:50] Join displays. No, I should have mirrored the displays. And files, AI vs Humans Online.

[34:03] I asked students to draw a bunch of things, asked humans and AI to label the object. Take this: just the letter C, when they were supposed to draw a caterpillar. Every human that was shown this got it right. They said caterpillar. AI is struggling. Doesn't realize, "Oh, this is a caterpillar."

[34:33] Take this: cave. Humans have no problem. All the models are saying it's a mountain. Take this: all the humans are aligned that this is a donkey. Cow, sheep, horse, that's what AI sees it as.

[34:51] Let's look at the reverse. Curious where the humans got it wrong, but AI got it right. This one: humans are completely confused about what this is. None of the models had even the slightest doubt that this is an owl. Yeah, I can't see the owl. This is live data as of this week really.

[35:14] So, if that is the case, one thing we can be sure of is **an agent has a very different kind of perception.** And if their perception is different, then the kinds of mistakes they will make will be very different. They have infinite patience or at least a lot of patience. They will check, double check, triple check, quadruple check. What human does that? And what human can't count the number of R's in "strawberry" if they know English?

[35:39] So, the kinds of errors are very different, which is not to say, "Don't use agents." **Don't use agents as a full substitute for humans.** That's something to keep in mind.

[35:51] Let me just put this back to join displays. Few other things that I wanted to flag, which I will do once I'm able to- Oh, okay, sorry. Wrong one. This.

[36:11] Now, let's talk a bit about what you changed your mind on. Anyone wants to share something that you felt, "Okay, I thought X before, now I think Y, and my point of view is different based on the discussion"? I'm very patient.

**Student**: [37:03] Like, initially, we had- like I had written that I will show it to someone so that they can, you know, kind of, someone from the domain or someone who is affected by the visualization, and they gave their inputs. After having discussed, I got to know that like, you know, it is better to show to the multiple, you know, such affected people and they will give their different opinion, and then maybe we can kind of decide upon which is that parameter which actually drives them to take the decision. Because eventually what I feel is the parameter may not really be the one which others feel.

**Anand**: [37:38] That's a powerful one. You are saying, "Earlier, I would have shown it to one or a few and would have asked them for their input. Now, I'll show it to many and ask them specifically on the parameter that will drive the decision." Which is a- which is, I think, a powerful lesson. Any others who saw something differently from the discussion? What?

**Student**: [37:59] I think just a visualization is not enough. I mean, the visualization could portray multiple perspectives to the same question. But probably another conclusion, or just a line or fact that I want to prove, would be a better-

**Anand**: [38:13] Sorry, a what?

**Student**: [38:14] Just a visualization is not enough. It should also portray another line probably that, you know, answers the question that I want to answer.

**Anand**: [38:22] Oh, you mean line by- you mean a sentence. Okay. That explains it, meaning a narrative is something that is required to support a visualization.

**Student**: [38:30] That was my suggestion.

**Anand**: [38:32] Which is an interesting one. What did you learn? What did you change your opinion on?

**Student**: [38:38] I think just the graph is not enough.

**Anand**: [38:41] Okay. Sorry, I thought your suggestion was- okay. So you- your point of view earlier was that a graph is enough, and based on the discussion, what you're taking away is a sentence would probably help. Got it. That is interesting. Any other thoughts? Any other- Yes, please.

**Student**: [39:01] Initially, I thought that we can validate the visualization using some labeled databases. But later, I realized that it is not only about the validation part. Visualization is a iterative one until the required information passed to the audience.

**Anand**: [39:17] **Interesting that visualization could be iterative.** That is also a new perspective, and I think it's an important one. It's not something I was referring to in this particular case. But yeah, we don't make one decision. We make a series of decisions. Sometimes, we keep changing our minds along the way. And, actually, **a good visualization is one that leads to more questions.**

[39:40] If somebody looks at the visualization, says, "Ah, okay, but," and moves on to another topic, you've done a fantastic job. It feels terrible, by the way, and I've had this feeling so often. Create something, a lot of sweat and blood, produce something, they say, "Ah, okay, fine, but-" and now they ask three more questions. So now, I have to go back and sit and create and all of that. But that's the thing. That is the best kind of visualization, which doesn't stand in the way of free flow of decision-making. "I got it, and now I have a set of follow-up questions." And, yes, that is precisely the value of possibly the same visual or perhaps a different one.

[40:26] You know what was interesting, though? I'm getting two interesting signals. One, it was far easier, I think, for you to answer the first question than the second question. First question was: how would you test a visualization? Second question was: what can you learn based on 5 minutes of discussion? And keep in mind that the second question is probably the more important question. The first question you can ask an AI and get the answer to. Second question is how you are changing yourself.

[41:02] If you haven't practiced deliberately learning how to learn, and effectively saying, "I have a point of view. I know my point of view. Somebody else has a point of view. I am able to listen to that point of view, and I am able to find out the difference between their point of view and my point of view, and adjust mine accordingly, or choose not to adjust mine accordingly," that's also perfectly fine. But being able to spot that, practice that.

[41:30] If you are doing an MBA, this is, in fact, one of the most useful things that you can do in a group discussion, where you're constantly looking at what you're agreeing with them, what you're disagreeing with them, understand and distill that very clearly, be able to synthesize, and be able to factor it in.

[41:50] The second interesting thing is none of the discussions have anything that would tell me what would make a visualization fail. What are the ways in which you would have- or, I mean, I don't yet have a perspective on the misinterpretation part, which somebody had mentioned in one of the first answers. That seems to be the adversarial testing part of it still seems to be something that you haven't collectively discussed.

[42:27] Now, what that means is, A, I am taking away that it is not something that most people think about. Journalists would. Journalists would think almost exactly about how would this chart be misinterpreted. A journalistic editor, in particular. But, in your case, that is a blind spot. What does that mean? It means that, all of you, when you produce visualizations, you will assume that your result is correct and will get tripped up maybe 5% of the time, but you are likely to get tripped up. Just be aware of that.

[43:05] Secondly, okay, this was- just taking a quick look at the results. Yeah. So, one thing that seems to have come up a lot is: when does AI help, and when does it not? So, one of you said, "So first, we should check the visualization against the data before AI agents are writing their opinions." Some of you have suggested using AI agents instead of people as part of what you're learning from this, but also realizing that numerical correctness does not guarantee understanding.

[43:53] I'm not projecting. I'm just reading my notes. So, it looks like what you're collectively saying is AI seems to have three roles. One: generate failure cases. That's useful. It can think differently. Second: independently recalculate from source data, which is a verification thing. But the third was particularly interesting. Some of you have said, "Propose alternate designs, using it as a design partner for visualizing the data," which anyway was one of the exercises that you were exploring. Which all three of these are useful.

[44:30] I suspect that you may not have initially thought of all three, but that is part of the power of what you may learn collectively from these discussions. But here's what I'm going to do. I'm going to take your collective thoughts, use this as the basis to evaluate your visualizations, both as humans as well as for AI, to see what you're getting right, what you're getting wrong, and take that feedback to see how you missed something.

[45:00] See, again, you have a prior. You've created a visualization. You've even improved on it, saying, "Now I understand, or I have a point of view on who the audience is, I have a point of view on what decision they need to make on that." The second visualization exists. Let's evaluate it against all of these criteria that you have laid out, and see what the gaps are between your collective knowledge versus that. Any questions?

**Student**: [45:26] The thing is, this data set is like in a- in multiple verticals. Not being from the domain, we will not be able to correlate how do they relate to one another. So, AI largely drives what we make out of the data. And then we go on to depend on it. But many a times also what I feel is that when we get the first, you know, clue of somewhere something is wrong, we generally tend to stop there.

[45:54] We do not really know like if at all there is any other, you know, problem in the pipeline. Something else is at fault. So, the very first fault we get, either maybe AI also might do similar, we give an undue weightage to only that first flaw or, say, anomaly that we detect, rather than trying to go from all the directions and then see which all are the, you know, faults that are there, how do they relate to each other, and then finally give a priority to all these faults and then visualize that which fault for our, you know, context is the most critical one.

**Anand**: [46:33] So?

**Student**: [46:34] So, I think, you know, how much of time are we supposed to spend on a particular visualization is an open-ended question. Like, I don't really get the hold of it.

**Anand**: [46:44] That's an excellent question. The theme is this. "Boss, I start solving a problem, I can solve it to death. When do I stop?" Currently, how do you decide?

**Student**: [46:56] Deadline. So like, if I have 2 days to do it and I can only spare a day out of it, so I sit a day, do it for half a day with half a day of rest, and whatever comes, then I just send it.

**Anand**: [47:10] Which is an excellent point. Now, supposing 10 things come up, you would probably only spend a tenth of a day.

**Student**: Yeah, but then I would obviously- if it is addressed, then I may not spend more time.

**Anand**: [47:20] Huh, which is the second part. So it's not like 10 things mean equal time. It's-

**Student**: Yeah, prioritize.

**Anand**: [47:26] Priority, exactly. You say, "This is the more important part. This is the available time." Which is great.

[47:31] Now, this is a useful first step. The question, by the way, incidentally, is not restricted to data visualization. It's just: how do I be more efficient, more productive, in general, on any task? So, what you've outlined is, firstly, be aware of how much time to spend, which is a prioritization part of it, which I think is an important one. Any other ways?

**Student**: [47:58] Second is that does the visualization at least do the bare minimum part? It has to catch one fault. If it is able to do that, then maybe you have done whatever with time constraints. Whatever I do extra, it's just a bonus.

**Anand**: [48:13] Good point. So, set a threshold. So it's not like I will spend even only 5 minutes. If within those 5 minutes, I'm not able to get to a certain minimum threshold, then I should spend 10 minutes, 15 minutes, half an hour, whatever, fair. But that does not actually allow you to be more effective, so to speak. What you're saying is, "I have to do this." What are the ways by which your visualization or, in general, any task can just be more effective?

**Student**: [48:44] Now, there is always a dilemma. One is that how much of time you spend to basically justify whatever minimum you have done. You can use the time for that. Or else, try to explore better ways of, you know, kind of coming up with a new, say-

**Anand**: [49:01] Yeah, and I think that's an important one. Supposing you have some time today and you say, "I'm going to use that time to learn how to create better data visualizations. The next time something comes up, I will be maybe 10% faster, 50% faster, 20% more effective." So, that's a good way to invest time. Now, whether it should be for data visualization or something else, that's up to you. What you want to be more efficient at is something that you can drive.

[49:27] And what you're doing here is effectively creating capacity. Ultimately, what we're doing in this class really is building- you're investing a certain amount of time to say, "At some point, something will go to the back of my head so that when I normally do some data visualization work, I will be more effective at it than somebody who has not had the benefit of their brains having gone through that kind of a thought process."

[49:51] Which partly also means the harder your brain works at that point, the deeper it sticks, the quicker it comes back, the payoff is high. But that means it's a lot of work at that point in time. "Do you want to spend that mental effort on this or something else?" is also a trade-off that you make in the present. But that generally works. What I mean is that the amount of effort that the brain puts in tends to be the effectiveness with which it sticks, whatever the topic is.

[50:19] Are there other ways of creating capacity? Meaning, are there other ways so where you can do even better and even more data visualizations more effectively?

**Student**: [50:29] But all of it actually depends on this feedback from people. Most of the times, I think we are working in isolation. So, without me interacting with any of the other- not only anyone, so like if I'm actually creating a visualization for someone like a manager-level who has to take a decision, how much of the bandwidth do I get with that manager to take any feedback? Because someone who is at a lower level may not be able to have that temperament.

**Anand**: [50:56] Correct. Very good point. So, if I was able to take feedback from a dozen people, that is great. Why won't people give me time? And, usually, they'll say two things. One: you take too much time. Two: it is- I don't see the point.

[51:14] So, as part of our next exercise, what we're going to do is see how one goes about running a small feedback kind of a process. Let's do that. What I'm going to do is expose the next question, and show you two visualizations.

[51:34] What you will have to do is imagine yourself as a student unfamiliar with either subject, which is easy. That's yourself. And you have to see each visualization for 10 seconds. I'm not going to give any explanation. The screen flashes, flashes, don't worry about it. Which visualization is more likely to help that student understand its main message? Some student being shown some random image, there is some purpose behind that image. Which image will help the student more likely understand the message? So, you'll have to understand the message and judge whether it's conveying the message well.

[52:20] So, you will have access to this on your mobile as well, but I'm going to open these. Just stare at it for 10 seconds. Sorry.

[52:37] Can you zoom in?

[52:46] Let me go to the second one.

[53:06] Okay. Now, I'm going to go back to this. The links are there. If you want to open it, take a look at it, that's fine. Spend no more than 1 minute, and just write: between these two, I think A will be easier for students to understand or B will be easier for students to understand. Begin with the letters A or B, and then explain what visual features led to your prediction or decision.

[55:10] Okay, start wrapping up. It's not that tough a- you know, it's- you don't have to be very elaborate. Just say A or B and whatever you felt. Seven responses. I'll wait till we get to about 15, and then open up the next question. So, you do need to look at the next question, so wrap up.

[55:42] Okay, we've gotten to 16. Fine. Let me now open the next one, and you can see a couple of more images. Exactly the same question. Two more images that would appear below, and you're welcome to check it on your mobiles. Here is one image.

[56:22] Second image.

[56:45] And same exercise.

[58:46] Okay, 15. Just wrap up then. You should just wrap up now, and okay, 18. Yeah, I'm going to go on to the last of this series. The third question. Okay, again, same question. Two visualizations. I'm going to close the earlier ones, and open them.

[59:30] And the next.

[59:55] Okay, do start filling out question five as well.

---

**Anand**: [59:59] ...question five as well, and then we will talk a little.

[61:16] Okay, 13 of you are just finishing it.

[61:32] Okay, I'm just going to let you finish and start sharing some of the things.

[61:38] Getting feedback is hard. No question about it. There are a few things we kind of know about getting feedback, but agents also know about getting feedback. But if I had to pick one mini, but very important lesson on gathering feedback from people who are busy and want to see clear impact, it is: **make sure that it is a binary comparison. Do you want this or this?**

[62:09] Let me give you an example of bad design. And this is bad design that I implemented in one of my exams just a short while ago intentionally.

[62:23] So, the students were asked to morph the faces from the left to the right. That is the task. And in this case, what they have to do is evaluate: is this morphing easier, or is this morphing easier, or is this morphing easier, this morphing easier? Across these 10, they have to rank order them.

[62:47] Now, let me tell you what would be—firstly, what is hard about it? 10 is hard. What would have been even harder is to ask them on a scale of 1 to 10, tell me what you would place each one of these as. What makes this slightly easier is between these, I say, "Okay, man to man, that's slightly easier. Black and white man to color woman, that's pretty hard. So, this is probably the tougher one. Chinese to African probably is easier." So, I will drag this, move this up somewhere if I can. Okay.

[63:23] Oh, quiz is over, is it? Oh, quiz ended. But I'd have been able to drag this up. So, some amount of simplification, but it's still going to take a lot of time.

[63:31] In contrast, if we just showed them—and let me show you another one. Yeah. So, this was a similar one: take the photos, create 3D models out of it. Which one do you think is better?

**Student**: [63:47] A looks closer.

**Anand**: [63:51] Let's take the second one. Which one looks better?

**Student**: [63:57] B.

**Anand**: [64:00] Now we're divided, right? How many people say A? How many people say B? So, I'm going to go for both are about equal.

[64:12] And of course, this is important, by the way. The fact that, just at a glance, you have some people say what matters is alignment to the facial features, which was A. What others say is, "Look, a 3D visualization has got to be a 3D visualization, not a face plastered on top of a sphere." Both are valid perspectives. But it's important to know.

[64:36] Okay. And between these, which is better? Anyone thinks otherwise? And finally, on this one? Any Bs? Okay, close enough to say.

[64:58] And so on. It'll go on like this for a while. But you saw that it was relatively easier. Look at this, I look at this, say one is better, one is worse. Faster. And there is a certain momentum to that speed. When I started playing this, I couldn't stop. I'm saying "playing," this is actually an exam question. Get to see, "Oh, okay, this is something people are building. Oh, this is interesting."

[65:21] Same with visualizations. When you start doing a comparison, "Okay, this, this," as long as you can make the task easy. Which is the other element: **what you want people to do is give them a clear, simple task, and they will do it for a long time**, because it becomes like playing a game. So, **if you can gamify the whole thing, nothing like it. Then you will get plenty of feedback.**

[65:48] Now, what exactly did we learn from what we saw on the results that you had shared themselves? I'm going to pass that to an agent and share. But we have only 12 minutes, which is slightly less than the time that I had planned. So, I'm going to go on to the last part of what we're going to cover. Keep in mind that while in the classroom, there is some amount of interaction that we can do. All the general _gyaan_ [knowledge], I can mail you. You can read at your leisure or give it to your agents to read. That's also fine.

[66:23] You had built a set of mobile applications. I hope you have them on your phones, because you're going to be swapping them with your neighbor. But before that, I will pose one question to all of you that I'd like you to answer first.

[66:47] Data visualization, yes, we've conventionally studied. Now, here you are happily creating a mobile application, some of you perhaps—how many of you created a mobile application for the first time ever? Okay, about half the class created a mobile application for the first time ever. Great.

[67:05] If that is the case, and I suspect that it's unlikely that you created a data visualization for the first—a chart for the first time in this class. You probably did that in school. Now, the thing is, the number of formats with which you can communicate data is exploding. The example that you saw of people creating faces from 3D images was a student exercise which had 0.5% weightage in a long series of questions. If anybody had spent more than 15 minutes doing that, they would have been wasting their time.

[67:42] And students who did this are very smart. All 600 of them created a WhatsApp group, just posted, "Has anybody solved this problem? Which tool does it right?" And they said, "Okay, Meshy.ai seems to be very good. The free tier is not bad. The paid one, okay, costs some a few dollars. You can use it if you want. It works much better, but up to you." So, okay.

[68:01] Now, the sad part is, even with this knowledge, even with this group, you saw the kinds of visualizations people were trying to create and submit. So, it is one thing that it's possible to create something well, the technology exists. It's another thing to know it. It's another thing to use it. So, there is a long tail. You, at least, hopefully, will know the stuff.

[68:20] But, stepping back, the forms in which we will be visualizing data is exploding. And it's worth exploring a few more of those forms. The mobile is one of those forms, and that's why you had an exercise to visualize mobiles—to create a sensor of some kind on mobile. But in your case, you weren't visualizing data in the mobile. You were actually using it as a sensor, as a data collection source, which I think, for obvious reasons, will be important because you probably by now have discovered the kinds of sensors that it would have created.

[68:55] Now, I want you to help me help you. I'm going to be evaluating this. How would I evaluate? You are going to be evaluating your neighbor's application. How should you evaluate? Have a think about this. You've built a mobile application. **How will you test whether it works correctly and is useful to someone else?** Exactly as we had looked at the intent, the audience, and the purpose for data visualizations.

[69:28] And on this, what I'd like you to do is propose specific test cases and criteria. What should you try? What should you observe? What should you measure? What constitutes failure? And specifically think about your phone sensor app, but also beyond that. Share your point of view.

[69:48] Hoping we will have—I suspect we may not have time for the discussion, so we'll jump straight to the last exercise. This, you get only 2 minutes. Very quickly finish this piece.

**Student (Female)**: [70:23] I had built this app based on what problems I faced earlier. So do I still have to...?

**Anand**: [70:29] Yeah, but after you finish this. We'll all be swapping phones.

[72:43] 10 responses. Just wrap up please for this one. Just how would you test, what would you try, what do you observe, what's failure?

[73:03] Okay, we have 16 responses. The rest of you will probably be wrapping it up. Great.

[73:08] So, I'm going to then—now, we have 5 minutes. Might spill over a little if anyone wants to head out, that's fine. But I really just need us to complete one thing. I'd like you to swap your phone, open the sensor app that you built, give it to one of your neighbors. Ask them to use the same criteria, review it. Record your observations.

[73:32] **I want you to submit a voice recording in which you read out saying, firstly, whose app you tested, mention their name; what it does—I want you to get an understanding of what app they built, talk to them, ask them; and mention what you tried, what works well, and one thing to improve.**

[73:55] Before you ask them what the app does, try it out once. See if you're able to figure it out, and if not, then just even the intent of the app is a useful one. Record and upload. Keep it to less than 5 MB. Don't make it a very long recording.

[74:10] And this is something you have to complete before I shut down my laptop, which will be in 10 minutes, because this is running on my laptop, you won't be able to access it. So, yeah. Swap phones, and or yeah, and try out your neighbor's mobile application.

**Student**: [75:31] My app requires a mic. Can we clap?

**Anand**: [75:33] Yeah, go ahead. Make all the noise you want.

[81:01] Anyone who's completed it, feel free to head out. Anyone who has not completed it, make sure you do it in the next 5 minutes. Make sure you share your homework—mobile app, innovative visualization, everything with me. I'll share a summary of the session. This part of the class is done.