What changed your mind?

Why are you even here?

While latecomers trickled in, Anand ran a show of hands. Startup right after college? Higher studies? A job? Most of the room wanted to work. The last time he had asked a batch this, in 2010, it was a larger class of chemical engineers, three of them planned a startup, and the majority were planning to study right after. 01:21 Why work? Finance, said one student. Wouldn’t an extra degree pay more over a career? The nods said it wasn’t about lifetime return. It was about cash flow now.

Then, at two o’clock, the real question, one that hangs over every class in the age of AI: Why do you bother attending classes? You can get this from a lecture online. 04:13

Palani got the first laugh: Well, it’s good that you’re asking the question towards the end of the course. The students gave real answers. Attendance. Credit, though a classmate pointed out that there are people who don’t attend the class, but they come and write the exam. Reading is boring. Doubt clearance, which Anand answered in four words: Which you can ask AI. 05:42 And the most honest one:

“This is like a dedicated slot so that we will come here and listen. But if I want to do that in my room, it’s like it won’t happen.”

— A student, on why class still matters 05:04

Someone said they come to meet friends. In which case, you should be talking to them, not staring at me. 05:50 Anand then admitted something teachers rarely say aloud:

“See, the education system is designed more for teachers, perhaps, than for students.”

— Anand 05:50

Having everyone pay attention to one speaker and take the same test is convenient for the teacher. Content is the one thing a classroom no longer has a monopoly on. A room full of people can do other things, though. They can commit to an answer without seeing anyone else’s. They can argue face to face. Most usefully for a course on visualization, they can be each other’s test audience. So: since you are here, and AI is anyway eating up our lives, why don’t we do something slightly different today? 05:50

That design didn’t come out of nowhere. That morning, Anand had asked ChatGPT to scan the possibilities:

The class kept that skeleton and dropped the props: predict privately, discuss, revise; then test charts on each other, then test apps on each other. Everything ran through a live form on Anand’s laptop. The form’s title was the agenda: Verify, compare, test.

Your first answer is a hypothesis

The students had already built a visualization for Varuna, the factory dataset from earlier homework, and revised it into a replay aimed at a manager’s decision. Now someone had to grade it: Anand, his agent, and Palani. How should we evaluate it is the point of discussion. 08:45 Before any discussion, though, each student answered one question alone, on their phone:

Then came the wait. Six responses in four minutes. Ten. Seventeen. Nineteen. Anand watched the counter crawl and said what every teacher thinks:

“Some guys get it; there’s no point teaching them. Some don’t; there’s no point teaching them. What do we do with the middle?”

— Anand, waiting for the counter to reach 25 17:44

Why write first and talk later? Because of something Anand noticed in ninth or tenth standard. He would read a book, it would make perfect sense, and he would conclude he had learned nothing: Something makes perfect sense when you read it, and I started assuming therefore that I knew it beforehand. 18:43 If someone had asked him before he read it, he would probably have given a different answer, or said he had no clue. Psychologists call this hindsight bias. Its classroom cousin is the illusion of explanatory depth.

Half a test, twice over

Twenty-four students answered. Read their answers side by side and they split into two families. One family asks “Is it true?” Do the numbers match the raw data, are the axes and labels honest? The other asks “Does it work?” Can a real person look at it and get the message? Each family was well represented. Few answers had both.

The common answer was a good start. Ask friends to skim through the visualization and tell insights they get from it, then compare it with our objective, wrote one student. Another: Find people who have no prior knowledge about thr data. Then show them the visualization and ask them to interpret it. Neither is wrong. The trouble is what neither one checks. Showing a chart to people tests whether the message gets across. It doesn’t test whether the message is true. Checking the data tests whether it is true, not whether anyone can read it.

The rare answers had the ideas that make a test strong. Four of them deserve a closer look:

The empty last row of the chart is the gap the rest of the class kept returning to. No one said what result would count as a failure before running the test. “If they understand it, it works” sounds like a test. Without a threshold it can’t fail, so it isn’t one. A real test sounds like this: “Shown the chart for 20 seconds, at least four of five plant managers pick the right station to fix.” (The four-of-five is an illustration, not a standard. The point is that you write it down before you look.)

“If people agree, the visualization works”

Anand read that line from one of the answers to the room. Makes sense. Any problem with that, though? 30:05 The class took it apart in three moves.

First, agreeing isn’t the same as understanding. Second, from a student who had clearly been thinking about it: everyone understands, but what if … it doesn’t clearly answer the question? … have the facts from the data been taken properly, right? 30:42 And third, the sharpest point in the hour:

“There is a common mode, right? Like you might be asking all the people of same group. So, they might be blind to it.”

— A student 31:06

Common mode is a reliability engineer’s idea, and it fits here exactly. Five classmates who share your background, your course and your assumptions are not five independent checks. They are one check repeated five times. Anand split the fix in two. Part A: ask enough people, and the right people. Part B: make sure you are showing them the right thing.

“If I show a graph which says men live longer, women live less, and everyone agrees on the same interpretation, except that, in truth, the reverse is the case, chart is just plain wrong.”

— Anand 31:22

He then praised the rare answers by name. Recalculating from the raw data. Checking geometric proportions: it’s one thing to say, “I have written 24 and 25 here.” It’s another thing to show that difference as 3x, 4x, which absolutely is unwarranted. 31:45 And deliberately looking for ways to misread the chart: That’s powerful. 32:26 Searching for misreadings also protects against the common-mode problem, because somebody who we don’t even know might be looking at it in a different way.

The owl nobody saw

A few answers suggested a shortcut: skip the people and test the chart on an AI audience. One proposed creating some ai agents as a audience lets say some roles like project manager, employee etc. Another laid out a six-step pipeline of independent agents checking each other, with a human deciding at the end. Anand’s verdict: This is both good and bad. 32:57

The good part: an AI can play roles you can’t easily recruit for. You can ask it to take the personality of some extremist who is looking at the data trying to misinterpret it. That is red-teaming on demand. The bad part is a question nobody had asked: does an AI even see the way your audience does? And it is crazy. Let me show you this.

He switched screens to a results page from his other course, Tools in Data Science. Students there had each drawn objects in a minute. Other students, and three AI model families, then tried to name each drawing from ten options.

The conclusion is plain: an agent has a very different kind of perception. 35:14 And if they perceive differently, they make different mistakes. They have far more patience: They will check, double check, triple check, quadruple check. What human does that? Yet they also stumble where no human would: And what human can’t count the number of R’s in “strawberry” if they know English?

“Don’t use agents as a full substitute for humans.”

— Anand 35:39

Put this next to the common-mode problem and a useful idea falls out. An AI reviewer is valuable precisely because it fails differently from you. It is a check whose blind spots don’t overlap with your friends’ blind spots. That makes it a good additional reviewer. It makes it a bad stand-in for your audience. If GPT reads your chart perfectly, you have learned something about GPT. You haven’t learned much about the plant manager. Anand’s write-up of the experiment, Drawing humans-only CAPTCHAs, goes further into what kinds of drawing fool which kinds of eye.

“I’m very patient”

The second question was the first one turned back on the student. Eight minutes of talking with neighbours, then a new box on the same form:

When he asked the room to share, there was a silence. Anyone wants to share something that you felt, “Okay, I thought X before, now I think Y” … ? I’m very patient. 36:11 Then three students spoke, and each named a different kind of change.

From one opinion to the decision that matters

The first student had planned to show the chart to someone from the domain. Discussion changed that: it is better to show to the multiple, you know, such affected people and they will give their different opinion, and then maybe we can kind of decide upon which is that parameter which actually drives them to take the decision. 37:03 Anand summed it up: show it to many, not a few, and ask specifically about the parameter that drives the decision. That’s a powerful one. This moves the test upstream. Before you check whether the chart delivers its message, check that the message answers the question the decision-maker actually has.

A chart may need a sentence

The second student had arrived thinking a graph is enough and left with this: I think just the graph is not enough. 38:38 In other words, a narrative line is part of the visualization. Another wrote on the form, I will add my final conclusion in addition to the visualization where the graph is an evidence. That is good practice, with one trap: a headline can prime readers to agree. So test the chart with and without the sentence. If people only “get it” when told the answer, the chart is not doing the work.

One visualization is a step, not a destination

The third student started with validating against labelled data and ended up somewhere bigger: Visualization is a iterative one until the required information passed to the audience. 39:01 Anand ran with it: a good visualization is one that leads to more questions.

“If somebody looks at the visualization, says, ‘Ah, okay, but,’ and moves on to another topic, you’ve done a fantastic job. It feels terrible, by the way.”

— Anand, on why the best charts create more work for you 39:40

It feels terrible because you put in a lot of sweat, they glance at it, and now they want three more charts. But that is what success looks like. The chart didn’t get in the way. The reader understood it, believed it, and moved on to the next question in their decision.

Then Anand pointed out what the answer counter had shown. The first question was far easier for the class than the second. Answers to “How would you test a visualization?” came quickly. Answers to “What did five minutes of discussion change?” came slowly, and they were short.

“The first question you can ask an AI and get the answer to. Second question is how you are changing yourself.”

— Anand 40:26

Here is the skill he meant: I have a point of view. I know my point of view. Somebody else has a point of view. I am able to listen to that point of view, and I am able to find out the difference … and adjust mine accordingly, or choose not to adjust mine accordingly. 41:02 It is also what wins group discussions, in MBA interviews and in meetings: keep track of where you agree and disagree, distill it, and fold it into your view.

The question nobody discussed

The second signal was an absence. None of the discussions have anything that would tell me what would make a visualization fail. 41:50 One student had written about deliberate misreading in their first answer. The groups never took the idea up together. Adversarial testing was the class’s collective blind spot.

Who does think this way by default? Journalists would. Journalists would think almost exactly about how would this chart be misinterpreted. A journalistic editor, in particular. 42:27 And the cost of not thinking that way:

“All of you, when you produce visualizations, you will assume that your result is correct and will get tripped up maybe 5% of the time, but you are likely to get tripped up. Just be aware of that.”

— Anand 42:27

He read on from his notes, an AI summary of the answers that had come in minutes earlier, and added with a straight face: I’m not projecting. I’m just reading my notes. 43:53 Taken together, the class had found three jobs for AI in testing a chart, and probably no single student had thought of all three at the start:

Note what is not on the list: standing in for the audience. The owl took care of that.

Then Anand announced how grading would work. The class’s collected answers would become the rubric, and he would apply it to their Varuna visualizations, as a human and with AI. Let’s evaluate it against all of these criteria that you have laid out, and see what the gaps are between your collective knowledge versus that. 45:00 The students had, without knowing it, written their own marking scheme.

“When do I stop?”

A student raised a real worry. On an unfamiliar, multi-domain dataset, AI largely drives what we make out of the data. And people fixate: when we get the first, you know, clue of somewhere something is wrong, we generally tend to stop there. 45:26 They give an undue weightage to only that first flaw instead of collecting every fault, relating them, and ranking them. Radiologists know this trap well. Once you find one fracture, you are less likely to spot the second. It is a cousin of anchoring. Under that worry was a bigger question: how much of time are we supposed to spend on a particular visualization is an open-ended question.

Anand restated it the way every working engineer has felt it:

“Boss, I start solving a problem, I can solve it to death. When do I stop?”

— Anand, restating a student’s question 46:44

The class answered it themselves, one rule at a time.

The fourth rule is why the course exists. What we’re doing in this class really is … investing a certain amount of time to say, “At some point, something will go to the back of my head.” 49:27 And the effort itself is the point: the harder your brain works at that point, the deeper it sticks, the quicker it comes back. 49:51 Learning scientists call this a desirable difficulty. It is also a fair warning about letting AI do the thinking for you.

A last student named the real constraint: feedback. If I’m actually creating a visualization for someone like a manager-level who has to take a decision, how much of the bandwidth do I get with that manager to take any feedback? 50:29 Anand agreed, and gave the two reasons busy people refuse: One: you take too much time. Two: it is- I don’t see the point. 50:56 The next exercise was built to answer the first one.

A or B, in ten seconds

Six charts came from Anand’s July workshop at VizChitra, The Curator’s Dilemma, where data-visualization practitioners voted on which they would sign their names to. Here they were paired off. Anand flashed each pair for about ten seconds, with no explanation: The screen flashes, flashes, don’t worry about it. 51:34

The wording of the question was chosen with care. When Anand asked ChatGPT to help draft it, it pushed back on “which is better?”:

Play along. For each pair, tap Show for 10 seconds, make your prediction, and then see how the class split and why. (Afterwards you can open any chart at full size.)

Same feature, opposite verdicts

The vote counts are the least interesting part of the result. The reasons are better. Students cited the very same feature as the reason a chart works and as the reason it fails.

Two students looked at the same GDPVal treemap and disagreed about whether it had “limited, precise text” or “a huge block of text.” Neither is lying. They were reading different things, or reading at different speeds, or bringing different backgrounds. That is the point of testing on people: a feature has no fixed value. Its effect depends on the reader and the task. Faces help you recognise someone fast. They make precise comparison harder. Which matters depends on the job the chart is supposed to do.

Three more patterns are worth learning from:

• Seeing a pattern isn’t decoding it. The colour differences are evident although I have no idea what the area denotes. That student predicted a chart would communicate well while admitting they couldn’t read one of its two encodings. Recognising that a chart has a signal is not the same as understanding what it says.
• Students judged as themselves, not as the imagined reader. The question asked about a student unfamiliar with the subject. Yet the reasons included it was a topic I have an idea on and most of the graphs we see are A type rather than B type. Predicting how someone else will read a chart is hard. That’s exactly why you test on real people instead of predicting.
• Taste may cluster, and it may not. Six of the seven students who preferred the Temporal Rosette also preferred Popular Actors, which suggests a group drawn to pictures and compact shapes. With 24 people, that could easily be chance. Treat it as a hypothesis to test, not a finding.

Finally, remember what this exercise measured: predictions of comprehension. Nobody checked whether a novice actually understood the GDPVal chart in ten seconds. Measuring that would be the next experiment, and it would be easy to run.

Make it binary. Make it a game.

So, how do you get feedback from people who are busy and want to see impact? If I had to pick one mini, but very important lesson on gathering feedback from people who are busy and want to see clear impact, it is: make sure that it is a binary comparison. Do you want this or this? 61:38

To show why, he pulled up bad design that I implemented in one of my exams just a short while ago intentionally. 62:09 TDS students had to morph one face into another. The same exam also asked them to rank ten face pairs by how easy each would be to morph. Ten things at once is hard. A 1-to-10 score for each would have been even worse. A ranking at least lets you reason locally: man to man, that’s slightly easier. Black and white man to color woman, that’s pretty hard. 62:47 But it still takes a long time.

Then he showed the alternative: the same work, two at a time.

He ran it live. On the first pair of 3D faces: A looks closer. On the second, the room split. Now we’re divided, right? How many people say A? How many people say B? 64:00 The split itself was informative:

“Some people say what matters is alignment to the facial features … What others say is, ‘Look, a 3D visualization has got to be a 3D visualization, not a face plastered on top of a sphere.’ Both are valid perspectives. But it’s important to know.”

— Anand 64:12

The other effect showed up at once: speed creates momentum. When I started playing this, I couldn’t stop. I’m saying “playing,” this is actually an exam question. 64:58

“What you want people to do is give them a clear, simple task, and they will do it for a long time … So, if you can gamify the whole thing, nothing like it. Then you will get plenty of feedback.”

— Anand 65:21

This answers the manager problem from earlier. Busy people won’t sit through your 20-minute walkthrough. They will tap A or B on their phone a dozen times. You also lose little information. Statisticians have known for a century that many simple pairwise judgments add up to a reliable ranking. That is Thurstone’s law of comparative judgment and the Bradley–Terry model, the same maths that sits behind chess ratings and LLM leaderboards. For your next chart, don’t ask for “feedback”. Show version A and version B, and ask which one helps them decide.

Six hundred students, one WhatsApp group

With twelve minutes left, Anand asked how many people had built a mobile app for the first time for this course’s homework. About half the class. Almost nobody had drawn a chart for the first time in this course; they did that in school. That gap is the point: the number of formats with which you can communicate data is exploding. 67:05

The 3D faces they had just judged came from a TDS exam question worth half a percent. If anybody had spent more than 15 minutes doing that, they would have been wasting their time. The students knew it, and they were organised:

“All 600 of them created a WhatsApp group, just posted, ‘Has anybody solved this problem? Which tool does it right?’”

— Anand, on TDS students and the 3D-face question 67:42

The group settled quickly on Meshy: a decent free tier, a better paid tier for a few dollars. And yet: the sad part is, even with this knowledge, even with this group, you saw the kinds of visualizations people were trying to create and submit. 68:01

“It is one thing that it’s possible to create something well, the technology exists. It’s another thing to know it. It’s another thing to use it. So, there is a long tail.”

— Anand 68:01

A phone is one of these new formats, with a twist. In their assignment the students weren’t displaying data on the phone. They were collecting it, using the accelerometer, gyroscope, microphone, camera and GPS (through browser APIs like DeviceOrientationEvent and the Web Audio API): You were actually using it as a sensor, as a data collection source. 68:20 A sensor app gets tested twice: is the reading true, and does the app work for the person holding the phone? It’s the same pair of tests as for a chart.

“Help me help you”

Anand would grade the apps, and the students were about to review each other’s. So he asked them to set the standard first: I want you to help me help you. 68:55

This time the answers were stronger. Weeks of building had given the students concrete failure modes to think about. Hand-coding the 24 answers:

Some of the answers were excellent checklists:

One student made the opposite case: the app should be reliable on at least single phone at a time. Not necessarily robust as time might be a constraint for a full scalable project. That is a fair position, if you say so up front. A one-phone prototype that is honest about being a prototype passes. The same app sold as “works for everyone” fails. The test depends on the promise.

The same gap showed up as with charts: phrases like close approximation, consistent results and work as intended sound like criteria, but they can’t fail. Here is how to make each vague check falsifiable:

Swap phones

The last ten minutes were the experiment itself. The instructions:

“Submit a voice recording in which you read out saying, firstly, whose app you tested … what it does … what you tried, what works well, and one thing to improve.”

— Anand 73:32

One rule mattered most: Before you ask them what the app does, try it out once. 73:55 That is the cold read from Act I, applied to apps. The room got noisy. My app requires a mic. Can we clap? asked one student. Yeah, go ahead. Make all the noise you want. 75:31 About twenty voice reviews came in, transcribed by Gemini after class. Here is a sample of what classmates had built and what their testers found:

Read twenty reviews in a row and the lessons of the whole class come back:

• The cold read caught real failures. At least three testers couldn’t tell what an app did until its maker explained. That is a usability failure, found in under a minute, for free. In real use, the maker won’t be around to explain.
• Where you test limits what you can learn. I couldn’t try much because I thought it would disturb if I shout in this classroom, and I couldn’t move outside. A driving app tested at a desk, or a noise app tested in a quiet lecture hall, can only pass the tests the room allows.
• The visualizations were the weakest part, in a visualization course. Most suggestions were about the charts: too few, too static, or, like the spectrogram, too technical for a newcomer.
• Good praise is specific, and so is good engineering. “Move the saddle back 1.5 mm” is the best instruction in the set. The light-meter tester’s noise floor of 2–5 was the best measurement. A reading that won’t go to zero is exactly the kind of thing ground-truth testing should catch.
• Almost nobody tried to break anything. Testers tilted, clapped and walked around, which is normal use. Hardly anyone denied a permission, switched off the network or rotated the screen, though those ideas were in their own written answers ten minutes earlier. The blind spot from Act III survived contact with a real app.

The form ran on Anand’s laptop, so it would go offline when he left. Anyone who’s completed it, feel free to head out. 81:01