# Transcript

<!-- Solo talk Visualizing Machine Learning, speaker S Anand, original presentation at DataHack Summit Bengaluru 9 November 2017; this chosen video was uploaded 28 March 2020: https://youtu.be/sORnCj52COw. Transcript made 17 Oct 2026 with ~/code/scripts/call, Gemini 3 Flash Preview, 10 minute chunks due to initial chunk-upload failure. Automated timestamped transcript, unverified by careful human comparison with video. Preserve generic speaker tags unless verified. -->

**Anand**: [00:19] Increasingly, our models are black-box models. Let me explain. We were working with a telecom company, and they said, "We want to predict the problem of churn." The issue is people leave the network. In fact, they said there's roughly 8% churn. 

**Anand**: [00:39] I'll be a little louder if you want to reduce the volume. That's fine. What that means is that 8% of the customer base is no longer part of the customer base every month, which means that roughly by the end of the year, the entire base has churned. They said, "Look, it costs us about 80 rupees to get a new customer—all the process of doing the KYC, get the marketing, getting the customer onboard—that's roughly how much it costs us. But it probably will only cost us 40 rupees to retain the customer because even if we give, let's say, a 10-rupee offer, and only one in four take it up—and the offer says that if you stay on for another three months, then you can avail this discount—then at least one in four people take it up. **As long as we're able to identify with a reasonable accuracy who the person is that is likely to leave the network and target the offer to those people, we will be able to easily double our loss due to churn—sorry, even halve our loss due to churn.** So, help us find out who the people are."

**Anand**: [01:15] So, we said fine, let's start building models, and one of the models that we built was a decision tree model. That said, "First, check when the person last made an outgoing call. If it was more than 15 days ago, then there's a good chance that you've lost them or are very likely to lose them. If they've, however, made a call in the last four days, you should be okay; they're not going to leave. But if the last call was made between five to 14 days, then check whether they've made in the last quarter more than one recharge." 

**Anand**: [01:45] If they've made more than one recharge, they probably won't leave. Check what their recharge amount was. And if it is larger than the cutoff—which I've not put the actual value on—then again, they probably won't leave the network; they're probably going to be with you. Now, this model gave them an improvement of 39%. That is, the original cost per customer that they were losing was about 6 rupees, 6.5 rupees, and this brought the cost down to about 4 rupees per customer. In other words, to re-acquire a customer, that's effectively what it was costing them, which was a decent improvement, and they were fairly happy.

**Anand**: [02:18] The next step, of course, was to improve the model. So, we said, "Let's apply the full suite of tools that we have at our disposal. Support Vector Machines, Neural Nets, Random Forest—let's apply them all." And it turned out that Support Vector Machines was, among the unoptimized models, the most effective. There was a tweaked version of Random Forest that we created that was slightly more effective, but the difference was marginal. We finally went with the standard model of Support Vector Machines, and that reduced the cost to 2.2 rupees. In other words, there was a 66% improvement, and the project manager on their side was totally thrilled. "Yes, this is it." 

**Anand**: [03:00] Four months later, the project was canned. They didn't implement any of it. **Why is that? The problem was actually the accuracy. The problem was that the model was a black-box model.** So, when our project manager took it to his management team, took it to the marketing team, and said, "Here is a model. It'll tell you who's going to leave." Their question was, "How?" and his answer was wavering between something extremely complicated that neither he nor they understood, to roughly, "I don't know, but there's something that works." 

**Anand**: [03:36] And they said, "Okay, but what if we go back to this model?" There was a bit of a back-and-forth, and it became very, very painful. **They got to the point where we felt we almost shouldn't have given them the black-box model. We should have said, "Look, this is all you can get. It doesn't get better than that," and they would have gone in for it. But the sheer confusion killed that project.**

**Anand**: [04:05] And this is not uncommon. We were—we are, in fact, in the middle of a project where we're forecasting agricultural prices for a commodity manufacturer, one of the largest in the globe. And when we looked at an evaluation of how accurately we can forecast the price of these commodities across various products—not going to the names of the products—and when we considered the various models, it turns out that between a moving average forecast versus an auto-regression versus an exponential smoothing and so on, the model that turns out to be consistently effective and more accurate is the neural network model. 

**Anand**: [04:41] The numbers here are showing the accuracy of the model, roughly as a percentage, and this model is—these neural net models are getting to anywhere between 70% to 90% accuracy across the board. Clearly winning. Now, when we're asked to explain how do these models work, we then get into, "See, think of it like a brain, and then there are these neurons, and they fire and they interact." 

**Anand**: [05:05] And there are two kinds of clients. Those that say, "Oh, this is such fantastic jargon. I don't understand it, so my boss also won't understand it; this is wonderful." And there are the people that say, "I don't care. Why is this idiot telling me all this stuff?" **The point remains that because these models are increasingly becoming effective and increasingly indistinguishable from witch magic, we need a way of trying to understand these.** And that's effectively what I'm going to be talking about. How do we understand these models?

**Anand**: [05:41] Well, let's take one model. I'm going to start with K-means clustering. Actually, I'm going to be focusing on K-means clustering; we don't have enough time to do multiple models and all that. So, let's take a small piece of code. This is—I'm really sorry about the aspect ratio; there's not much that I can do about it. But this is a clustering algorithm. Let's talk through it line by line. We import a couple of libraries, and I'm reading a data file. This is a census dataset. Let's open it—census.csv. 

**Anand**: [06:12] What I've done is collated a number of columns from various censuses. So, we have information on each district, and there are 640 districts according to the 2011 census in India, for various states and districts, in which we have the total population broken up by male and female, broken up by literate, which is further broken up by male literate and female literate, the proportion of scheduled caste, scheduled tribe, male, female, workers, marginal workers, by religion, by asset ownership, by education, lots of fields. We have all of this data. 

**Anand**: [06:48] Now, the reason we went about collecting and clustering this data was because a media company came to us and said, "See, we are treating the country as if it's one big market. Well, actually two markets. We say that there are the Hindi-speaking people and there is the rest of the people." Literally, how the market is divided, and that's how we create content for each of these. What we find, though, is Bilaspur and Hyderabad, though geographically disparate, are actually very similar markets in terms of behavior and audience needs. 

**Anand**: [07:12] **How can we capture that? How can we understand that these districts are actually very similar? So, we shouldn't be having such a monolithic strategy. We shouldn't even be having a state-based strategy, because intra-state differences can be quite huge and inter-state differences can be quite small.** Can we actually take all of this data and some other internal data that we have and create clusters? Similar districts, irrespective of where they are physically? We said, "Sure, let's do it; it's very simple." It takes five minutes—it took five weeks to collect the data, but the algorithm actually only took five minutes. 

**Anand**: [07:46] So, that's what we're doing now. We read the data using Pandas, and I'm ignoring everything except the first—I'm ignoring the first three columns, which is the district code, state name, and district name. I mean, who cares about those? And then we pick K-means. For those of you who may not be aware, K-means clustering basically says that I'm going to take all of these numbers and find out if two districts are very close to each other. If so, I'm going to treat them as part of the same cluster. If they're far away, I'm going to treat them as part of different clusters. That is the underlying basis. And then it does something. Let's not worry about what it does. 

**Anand**: [08:21] Now, once it's done this magical clustering, we take the labels. Now, these labels will be 0, 1, 2, 3, whatever, based on the number of clusters we have asked for. How do we determine the number of clusters? We can explicitly say, "Give me two clusters, three clusters," whatever. And fit it to the data. 

**Anand**: [08:38] In this case, I'm going to say, "Give me four clusters." Now, how do I know whether four clusters is right or not? Two ways. One, we can see how well these clusters look—I mean, is it like I have two clusters that are all over the place or very tightly decoupled from each other? If I get to three clusters, are they very tightly decoupled? I can do that by trial and error. Another way is to ask the client the question: "Suppose I gave you 20 clusters. Are you going to be able to create 20 different sets of channels?" "No." "Okay, if I give you three clusters, four clusters, how many can you manage?" They say, "Look, if you give me anything more than six, I'm going to have trouble. If you give me anything less than two, obviously it's useless." 

**Anand**: [09:23] **So, let's say between three, four, five, and six is the number, and in that range, you figure out what is the right number. So, it's a combination of practicality—which is how it's going to be used, which is pretty important and you better know that before you come up with your answers—as well as the mathematics behind it.** I mean, between three, four, five, and six, there's not that much of a difference. So, I'm going to pick four clusters for now. And get the labels, add the labels into a column called "cluster," and I'm going to just print or save four columns: the district code and the cluster ID, and for my own reference, state name and the district name. So, let's run this algorithm: cluster.py.



**Anand**: [10:00] And that's going to save the data into a file called census_cluster.csv. Census_cluster.csv. Let's open that and this has the same four columns that you would have expected. There's the district code, the cluster, which can have values of 0, 1, 2, or 3—I'm sorry if you can't see that from the back, but there are those four values—the state and the district.

**Anand**: [10:25] Great. I have my answer. Does this make any sense? **The thing is, the algorithm is the easy part. People have solved that problem for us. Our problem begins once the algorithm finishes.** Or before the algorithm begins. You've got to figure out what the problem is, you've got to figure out what the data is that needs to be given to the algorithm, and then the algorithm does its job. And then our job again begins where we have to figure out what it means, help people understand it, interpret it for them.

**Anand**: [10:53] So, let's do that. If I had this, first thing I'd do is say, "Okay, look, this is something geographical. Let's plot it on a map." So I'm going to plot it on a map. What I have is somewhere—yeah, an India district map in Excel. Now, in—which makes it relatively easy for me to plot. So, for example, what this means is that if I put in random numbers like 0, 1, 2, 3, 4, 5, 0, 1, 2, 3, 4, 5, you can see probably the colors changing on the right-hand side. So I'm going to apply a simple VLOOKUP. I know you can't see this, but don't worry about it. I'm not here to teach you Excel. I'm going to look up the district and figure out what the cluster number is and fill that in and put it on the map.

**Anand**: [11:39] And now I get something that is slightly more meaningful. I have a map in which it says, "Okay, there is a yellow cluster that occupies all of Northeast and parts of Jammu and Kashmir." Now I know that Jammu and Kashmir and Northeast have very similar characteristics. So, this makes sense. And then there's most of AP, Telangana, which seems to fall into one group. And then there's—hmm, this is interesting—the greens. So, this is Bombay, this is probably Pune, this is Bangalore, this is Calcutta. Okay, I get a pattern around there. **The greens are hovering around the metro areas mostly. Okay, that's one kind of a pattern.**

**Anand**: [12:13] Now, this is slightly more helpful, right? Now with this, I can start iterating. Okay, what if I were to divide India just into two clusters? Let's do that. And run the same algorithm again, but before that, I should close Excel, otherwise it'll crib, and rerun the algorithm. Now let's reopen the file. Do the same VLOOKUP again. VLOOKUP the district code against the clusters here.

**Anand**: [12:53] And if I divide India into two, that's kind of what it looks like. There's AP, Telangana, parts of Kerala, roughly the South and Maharashtra, and belts in UP, Bihar, West Bengal, that seem to have very similar characteristics. So, they can go back to their team and say, "**Look, if you still want to keep only two segments, that's fine, but what we're finding is that parts of UP and Bihar are actually closer in characteristics to the South than the programs that you have in the North.** You probably want to tweak your programs."

**Anand**: [13:16] That was useful. In fact, that was the first starting point we had for the engagement because you could just see stuff. Now, this seeing stuff needs to go beyond this, because when I had those four clusters, I had no way of interpreting them. What we need is a way of playing around with these. So let me show you a tool that we built just to do that. The point here is not the tool. The point here is the technique. So, let's focus on the technique.

**Anand**: [13:44] Same data. I have the district codes, the names, etc., the same characteristics that you saw earlier. I'm going to pick all of the districts and I'm going to choose a series of columns—let's just say I'm going to choose pretty much all of the column categories—and across these, use again K-means in which I'm going to partition it into, let's say, six clusters just to make it a little more complex, and say, "Go ahead and apply the clustering."

**Anand**: [14:08] Now, the same result that we saw earlier, which is on a map, is still visible. And based on the parameters that I chose—which actually have more data than the earlier examples—now give me a very interesting picture. It tells me that, for example, there is—let's take cluster number 3, the maroons. These are clearly in the urban areas. So, that's the Delhi belt, that's the Bombay-Pune belt, that's the Bangalore area, that's Chennai, that's most of Kerala. So you can see that 3 is the urban belt.

**Anand**: [14:39] And interestingly also see that Punjab seems to be a beast that's very different from the rest of the country. Completely different demographics from almost anywhere else. But which still begs the question: How are they different? **What helps is to know what the features of these are.** So, for example, I could take the average of every single characteristic that I have in this segment. So, what is the average population of cluster 0, 1, 2, 3, etc.? And I find that cluster 4 is distinguished by a very low average population. I'll make it bigger so you can see it.

**Anand**: [15:07] Cluster 4 has a very low average population. Cluster 0 has a very high population per household, and so does cluster 4. Cluster 3 is one of the very few, in fact, the only one, that is low on rural percentage. But this you already knew because we noted that cluster 3, the maroon cluster, was the one that was mostly in the metro areas, and by definition, the metro areas are the ones that are less rural. Every other area is more rural. So, this is just pointing us back to what we already knew, but also reinforcing that this sort of an interpretation is helpful.

**Anand**: [15:40] Now, I could go step-by-step. Now, if I wanted to find out what distinguishes a particular cluster, I could say, "Okay, cluster 0 seems to have an unusually low literacy rate, seems to have an unusually high Muslim population, seems to have an unusually high young population, seems to have an unusually low TV ownership," and so on. So I'm basically picking out the bright colors. Saying, "Okay, that's dark red. That's bright green." **I don't need a human to pick bright colors, right? The system can do that. In fact, that's what we did. We said, "Let's just have a slider that picks out the brightest colors."**

**Anand**: [16:20] And what this tells me is that the three most significant observations in this dataset are that cluster 3 has a low rural population—that's the 25.9% which we saw earlier. Cluster 4 has a high Christian population and also has a low total population. So what is this cluster 4? Let's get back to the map and look at cluster 4. Turns out that cluster 4 is almost entirely made up of the Northeast. So that's a very different segment from all of the others. And I also find that cluster 5 has a high Sikh population. You know that; that's the Punjab belt.

**Anand**: [17:00] What kind of programs should I write for some of these? Or what kind of advertisements should I go after for some of these? Well, let's look at this. Cluster 3 clearly has a high computer penetration, higher education index—and higher education index—which means that the kind of programs that I would take to them would be the kind of programs that I would take to an urban audience. And this segment itself, you would name it as the urban segment.

**Anand**: [17:27] Whereas for cluster 4, which is parts of the Northeast, you find that it's relatively low in terms of population density, which means that distances are going to be pretty huge between areas. High SC/ST population, and low Hindu population, high Christian population, we saw that. So, for example, if I were starting a religious channel, this at the very least tells me that this region I would have a very different religious channel—I'd have a Christian religious channel rather than a Hindu religious channel or a Muslim religious channel.

**Anand**: [17:53] So, what we have is not just the clusters; we have a rough understanding of what these clusters are and are able to derive the interpretation of what we serve these clusters with based on those characteristics. Helps us go a step further. And there's more that one can do with this. I could say, "Look, how are various characteristics related?" So, for example, do I find that the literacy rate versus the rural rate tends to consistently increase or decrease? And we find that there's a marginal decrease across these. These are other exploratory analyses that one can go into.

**Anand**: [18:29] But what happened as a result of this exercise was that we were able to come up with a map for our client saying, "**Look, this is what your India needs to look like. There are the—there are six clusters, and let's not worry about what they are, but roughly. So, there is the 'Poor' districts in which you need a very different segment from the 'Breakout' districts where yes, they're poor, but they're starting to get educated.** There are the 'Aspirants' where there's a highly skilled labor pool and they are at the cusp of economic development. Then there's the 'Rich' who are both educated, economically forward, as well as living in metro areas partly. People with relatively low skills but high income. Combinations of several areas. And ultimately, they get a feel for what these segments mean through these words and what the strategy for these segments needs to be."

**Anand**: [19:11] It doesn't matter what we cluster. We took the same thing for a—we took a very different project for the same organization, which was to look at advertiser clustering. People buy advertisements. Sometimes they buy lots of ads, sometimes they buy less ads, sometimes they buy ads in only a few channels like regional channels, sometimes they buy ads across the national channels, sometimes they buy ads only in festive seasons, sometimes they buy ads across the entire year, sometimes they buy only in the really expensive slots, sometimes they buy only in movie slots.

**Anand**: [19:46] Based on all of these characteristics, can we club all of them and tell me what are the clusters that I can go after? We found that, for example, P&G, HUL, Cadbury, Reckitt—these belong to one group. They have very similar characteristics. They're big buyers, they buy across categories, but they buy at very low price points. Okay, it's very difficult to make money on...



**Anand**: [19:59] ...on an individual transaction; you make money only on scale. But very similar are Godrej, L’Oreal, ITC, Amazon, Coke, who again are big buyers across categories but at a significantly higher price point. Dabur, Nestle, Maruti—these are companies that have relatively low regional advertising—sorry, I'll make it bigger so some of you can see it at the back—they only advertise on national channels, very little on the regional channels. 

**Anand**: [20:25] Whereas companies like Heinz, Apple, Future Group, these tend to advertise a little less and they advertise at average price points and they get average viewership in return. Now, how does this help? **What I can do is go to an OLX and say, "Look, you belong to a peer of companies that have very similar advertising characteristics.** Now Samsung has come up with a strategy of advertising in which they're focusing heavily on these kinds of slots. Maybe that will be helpful for you." 

**Anand**: [20:53] What you do is put an account manager for this group of companies because they have similar buying characteristics and that person can learn from one and teach the other. And the clients welcome this; the organization obviously gets a lot of benefits. Creation of products also gets done through these groups. **Now, the—what helps here is not just the cluster identification. It is the cluster interpretation that comes along with it**, which in this case is simplified to the point of just a few words below that explain what the cluster is. But what needs to go behind the scenes is an understanding of how one goes about creating these clusters. 

**Anand**: [21:31] How does one do that, though? One way that I've seen, apart from visualizing these that I showed you, is a very powerful technique called "moving up and down the ladder of abstraction." Some of you may have seen Bret Victor's talk or blog post titled "Up and Down the Ladder of Abstraction." I wish I had loaded this... [typing] up and down the ladder of abstraction... okay, yeah, here you go. Hopefully my internet won't be too bad. Actually, it is terrible. 

**Anand**: [22:11] All right. I've most certainly made a mistake in not pre-loading this. I assumed that the data card that worked there would also work here. Okay, it’s coming up. We can do a mini data project by just moving this around to see where exactly the network is going to be better. But this is a very heavy page. Let's try my luck and hold it closer to the tower. Maybe a few inches will make a difference. 

**Anand**: [22:40] Now, what I'm going to do is, while that loads up, I'm going to talk about what Bret Victor talks about. **Bret Victor, incidentally, is—if you haven't seen his videos, watch them. They are mind-blowing.** There is just no other word for it. What he talks about is the way in which we understand models. We don't understand models through the mathematics of it; that's not the language that we are used to. **We understand models through kinesthetics, by touching and feeling these models**, by exploring what exactly the model is doing when we feed it a certain kind of input. So, I'm going to give up on this and perhaps just talk you through... [someone hands him a connection] yeah, actually if you've got a better connection, I'd like that please. 

**Anand**: [23:25] Okay, that is much faster. So, let's try and understand how a car would swerve on a road. Okay, so we’re building a model for a self-driving car and obviously that requires a fair bit of intelligence. Let's take a very simple problem. Supposing the car encounters a situation where it's on the wrong side of the road and it has to get back to the right side of the road and there's a bend happening. The logic we can put—that we can put against this—can be fairly straightforward. 

**Anand**: [23:55] So what you do is at every point, move a certain step forward; if you're on the wrong side of the road, turn towards the right side of the road by 2%. I’m sorry, 2 degrees. Very simple algorithm. Now, this has only two variables: one which is how fast to move forward and the other is how much to swing to the left. Let's normalize the problem and bring it down to a one-variable problem. So assume that I only move forward by one pixel—in other words, at a constant speed—and I will turn by 2 degrees. Now what does that do? Well, turns out that if you were to simulate it, this is how the car goes. 

**Anand**: [24:30] Now that is kind of useful. But wait, let's play that again. Now, at this point it went out, it went out again, oh no—now it's back on—wait... okay. So it's not going too fast. But it is going too fast for my ability to interpret. What I want is: some parts I don't care, skip this; some parts pause here; and wait, go back. **What I really need is effectively a slider that says, "Okay, wait—ha—wait, that's one area where it went out.** That's an area where there's a huge outward swing and that was certainly more than out here where it just went past the edge. This is definitely bad. And now, okay, this is slightly bad but probably only slightly worse than what I saw here and then beyond that, yeah, this is okay. Yeah, a little bit outside the road but on a straight road that's probably less of a problem." 

**Anand**: [25:11] Going back and forth—that helps. So now I'm interacting with the model. But even this—yeah, yes, this is kind of useful. Um, but look, could I just see the whole thing at one shot and see what happens? Now, what I want to do is, as I move this, I get to see that this is clearly wrong. But the question is: What is the correct model, right? 

**Anand**: [25:31] **If it's not 2 degrees, maybe it should be 3 degrees?** So let's change that 2 degrees to maybe 3 degrees and at—close enough, 2.9, 3. Now let's see if the model's doing better. Okay, that's still a bad turn, but not too bad. This is slightly better and this is slightly better. So yeah, looks like 3 degrees is better. Is 3 degrees optimal? That comes—that becomes my next question. So now okay, maybe 4 degrees is better. 4 degrees... slightly better, slightly better, slightly better, slightly—yeah, so okay out here... hmm. Yeah, so looks like 4 degrees is definitely better. 

**Anand**: [26:03] Maybe just increasing the angle will be great. Maybe let's get to something close to 8 degrees. It’s started to change; it's taking very sharp curves and it's almost—okay, but for this edge, it was almost completely inside. Now, I'm now not quite able to—I kind of know that increasing degrees make it better, but I'm trying to—I've now even forgotten what were the numbers that I used in the past. 

**Anand**: [26:29] **What we want is to be able to put a curve like this that summarizes all of what I saw.** [Shows multiple trials as transparent trails]. Is it like for 2 degrees, this was what the curve was like. Okay, so there's a little bit of white here, there's a little bit of white here, a little bit of white here. So that's roughly how bad the model was. This is terrible obviously at very low values. You know, it gets a bit—it gets a bit better. There's still a little bit of white here. It's only after we get to maybe around 6 and a half, 7 degrees that it almost fully stays in the black—or the gray. See yeah, I have to get to almost 9 degrees for it to be perfectly within the road. 

**Anand**: [27:01] So this is going to help me a lot more because it seems to be almost monotonically decreasing—the quality of the model—as I become—as I increase the pace of my turn, it definitely becomes better. But we can go one step further and say, "Look, what I want to see is when I take a specific curvature degree, what the curve is, but I also want to see how the car is moving at that particular point." Moving the—moving one level down the abstraction. 

**Anand**: [27:28] See, we started off with movement. Then we said, "Let's abstract that by putting that onto a line that summarizes it at a higher level." Now we are back down by looking at the movement. Let's take all of this and abstract it out two levels now. I want to see what the curves are for 1 degree, 2 degrees, 3 degrees, 4 degrees, 5 degrees—across all of those. And that gives me a picture of what the curvature is across and that there are clearly a bundle of models that are going to keep me on the road and a few of them that are taking me very wide outside of those. 

**Anand**: [28:03] But I still don't know which one is which. So I'll have to come down one level of abstraction now. I'll have to... okay, let's skip this. This incidentally was a summary of the accuracy or the quality of the model. So there's this range where it's not doing very well clearly and an interesting range here between 0 and a few degrees where it's doing surprisingly well, but we'll come to that later, maybe. 

**Anand**: [28:30] So what I can do is take the same curve and say, "Okay, wait. What exactly do we mean by picking 7 degrees?" Okay, so that's where 7 degrees is. Now we found that the curves at the very low levels were fairly accurate. Why is that? Not because they were staying on the road, but because they were kind of like so far off the curve that the numbers are failing. I mean, look at this one: at 0 degrees, I'm not moving away from the road. In other words, I'm just completely ignoring the bend and just plowing straight through. Yeah, so technically that's a better model than something like this which is going way off. But we kind of see why the model was accurate at the left edges. 

**Anand**: [29:06] Now yet another parameter that I could add is: What is the curvature of the road? We had one parameter that we were changing: how much does the car bend by. The other is: how much does the road bend by. And I could take these both as parameters and—let's—I'm going to skip in the interest of time some of these abstractions and jump straight to the third level of abstraction where this shows me for various bend angles and for various turning rates, what are good models. 

**Anand**: [29:35] So if your bend angle and turning rate are in the white region, you have a good model. So for—for bend angles that are relatively large, 80-90 degrees, you need very high turning rates. But if your bend angles are medium, then a wider the range—a wider variety of turning ranges will work. What we've done here is abstracted this model out to so many levels that we can go up and down these, understand the model in a variety of... [audio cuts].



**Anand**: [29:59] ...understand the model in a variety of different ways. And this is a trivial model. The model literally says go forward by one pixel, turn left or right by X degrees. That's it. **And this is true for almost any kind of model, no matter how simple or complex.**

**Anand**: [30:14] I'm going to take a data set which is currency data and apply this principle. So I have, for various dates, the currency rate against the US dollar of various currencies. So this says that on the 6th of November 2013, the Australian dollar was trading at 0.9497 USD. Now, this is the raw data level. Let's move this up one level of abstraction.

**Anand**: [30:40] What I'm looking at now is, for each of these currencies, what the values are across time, which can be quite easily summarized by this simple curve, which shows me the Australian dollar over time, the Brazilian real over time. Now, while the curve itself is one level of abstraction, summary metrics like the mean, the growth, the variance, the range—these are other metrics that take me one level of abstraction higher.

**Anand**: [31:04] Now, this is useful. So I can see that the Australian dollar is gently increasing; the Canadian dollar dipped and then increased; the Indian rupee dipped and then increased; the Philippine peso dipped and then increased. And then I say, "Wait a second, I said that three times." So maybe there's something in common between the Philippine peso and the Indian rupee and the Canadian dollar. They seem to be moving together.

**Anand**: [31:31] So now, could I take this up one level of abstraction and say, "Are these currencies related?" Could I do a scatter plot, for example: when one is high, is the other also high? So let's take it up to level 2, where I have here a plot between, for example, the Australian dollar and the Canadian dollar. And you find that yes, there is a mild positive correlation.

**Anand**: [31:53] Australian dollar versus Swiss franc—there seems to be an even stronger correlation between these two. Australian dollar versus Chinese yuan—even stronger, about 80% for those of you who can't read from the back, but you can see from the curve that—and Australian dollar versus the euro is even more strongly correlated. So we find that many of these currencies are positively correlated, but we don't necessarily know if that is true for every single currency.

**Anand**: [32:17] I mean, maybe there is something—perhaps the Icelandic krona? Okay, here you go. The Icelandic krona is marginally negatively correlated with the Australian dollar. So with this, what we're doing is moving from the realm of looking at one metric to pairs of metrics, specifically in comparison to the Australian dollar. But why stick to the Australian dollar? I should be able to take it up one more level further and say, "Let's take every pair and simply compute the correlation coefficient across these."

**Anand**: [32:43] Which is what we have here. For every pair of currencies, using the Excel Data Analysis Toolpak, we just created the correlation matrix. Which shows me that, for example, the FTSE is 100% correlated with the FTSE—of course—and 87% correlated with the S&P, 75% correlated with, sorry, I forget what CAC is, but only 2% correlated with the Canadian dollar, negatively correlated with the Japanese yen, strongly negatively correlated with the Pakistani rupee, and mildly negatively correlated with gold, platinum, and silver.

**Anand**: [33:14] So I find also that if I look at it at this level, Pakistani rupee is interestingly negatively correlated with a large number of currencies. So are gold, silver, and platinum. And this led us to a publication a few years ago where we were looking at the pairwise correlations across a wide set of securities.

**Anand**: [33:34] So this 68 is the pairwise correlation between the Australian dollar and the euro, shown also as a scatter plot on the bottom left. What we then did was group together the securities that were most similar. So for example, there's a 98% correlation between the FTSE and the S&P; so let's bring them close together. There's a 92% correlation between the Swiss franc and Chinese yuan; so let's bring them close together. **In other words, we were doing clustering on top of correlation.**

**Anand**: [33:56] What that does is create blocks of correlated securities. So the Singapore dollar, Japanese yen, gold, Swiss franc, Chinese yuan—these five currencies, they tend to move very similarly. They're one block. Similarly, here's another block that move together; the green indicates that they are positively correlated.

**Anand**: [34:14] Now what happens, however, is that when anything in this block goes up, everything in this block falls. That's the red area in between, telling us that you have distinct chunks—clusters—of securities that tend to move together. And for a portfolio manager, it becomes very easy. **For a portfolio manager, there are two kinds of decisions: When should I consolidate? When should I hedge?**

**Anand**: [34:37] Consolidate means if I'm, let's say, holding stock in two oil companies, you tell the person, "Don't. It's practically the same stock. Consolidate and move all your wealth into just one of those." Second, if you hold oil, then you find a good hedge to it because if oil tanks, you need something that will hold your wealth up. So what is a good hedge? Typically gold, because when oil goes down, gold goes up.

**Anand**: [35:00] Now that is something that they can infer very trivially from this by saying, "Okay, so I, let's say, trade with Singapore and Japan." I walk up and down—I find firstly that they're in the same block. So there's no point holding both. I will consolidate and just put all my money into, let's say, the Japanese yen. Now if I hold money in Japanese yen, if it tanks what should I do? I'll go up and down, find the most negatively correlated one—turns out to be the Sensex at the moment, the BSE Sensex.

**Anand**: [35:24] So park maybe a third or half of your portfolio on the Sensex, and that makes your portfolio both de-risked as well as consolidated. This incidentally, when we shared this with one of our clients, they made their largest trade in the portfolio segment using this visual. Not because they weren't using correlations before. **The problem was really about communication.**

**Anand**: [35:45] **Models, even the simple ones, can be quite complex.** And what I want you to take away from this is this: **Increasingly, the models that are accurate are black-box models.** They were complex enough before; now they are beyond hope in terms of understanding. Don't even try. **And therefore, they need interpretation even more so than ever before.**

**Anand**: [36:09] Methods are emerging to understand these, and a few powerful methods are simply visualizing these in terms of output and trying to interpret them. **Build a visual summary that explains the model.** Don't expect people to say, "Okay, because it's Random Forest, it has to be right." That's the same as saying because it's witch magic, it will work.

**Anand**: [36:28] **Build a model that will help them understand from their domain and help them interpret this in a way that they can relate to. Help them move up and down the ladder of abstraction.** Start at a low level, build it up, summarize it. Then go one level lower and show an example. Build it up two levels. Allow them to change the parameters. Allow them to interact. **Walk up and down the ladder of abstraction. And that's what really gives them the power to be able to do this.**

**Anand**: [36:54] Every time I mention this, though, the immediate next question that comes up is: What about tools? What tools do you use to do this? So I just preclude this question because if we get into Q&A, I don't want to get into a long discussion on tools. So this is the stack that we use at Gramener; we recruit across this entire stack.

**Anand**: [37:14] We stick to Excel for simple analysis. See, the thing is, there's the need to analyze; there's the need to communicate. And that happens both to our clients as well as internally. What we do when we want to analyze is stick to either Excel or R. On occasion, we go to Python, but we find that Python's really a productionization tool. We find it more useful for communication.

**Anand**: [37:40] When we need to communicate something, it needs to be productized. And internally, on the server side particularly, it's all on Python, with Pandas and Tornado being part of our stack. And when it comes to pushing it out to clients, JavaScript on the browser—effectively there's really no choice on the browser other than JavaScript and a templating library, the likes of Underscore or Lo-dash, and a visualization library, the likes of D3. These are what constitute our stack.

**Anand**: [38:05] But the thing that we find is that even on simple tools—or tools that have been around for a long time such as Excel and tools that are potentially a lot more complex such as D3—**the skill gap is roughly the same.** In other words, it's not as if people find the traditional tools much easier to understand; there's just more of them. The skill gap is something that is consistent.

**Anand**: [38:29] Every time I'm asked, "So should we pick tool X or tool Y?", my answer is simply this: **The variation across tools is not that much.** What you can do in R, I can do in Python. I would argue that R's an easier tool to use, but I'm better at Python. But I'm even better at Excel. I've been doing Excel for 25 years and Python for less than 10 years. **Anything that I can do in Python, I can do in Excel. Without exception.**

**Anand**: [38:57] Including HTTP requests, including—I mean, every single thing that I showed you and more, I can do in Excel because it's a full-fledged programming language. **The point is that tools are versatile enough and are rarely the constraint. People are the constraint. Techniques are the constraint.**

**Anand**: [39:13] The variation in people's ability is several orders of magnitude higher than the variation across tools' abilities. **Pick a person, let them work with whatever tool they're familiar with.** That is the single biggest—single most useful advice that I've been following at least for the last few years.

**Anand**: [39:27] So I've shown you some examples. You can find more examples on gramener.com/demo—there's the whole—including the ones that I showed you. I do hope that you will be playing around with machine learning models, but please do try explaining them as well, and all the best with that. Thank you. [Applause]



**Unsure**: [39:59] [check?] Ladies and gentlemen, if you have any questions, please... okay, we have one there. Can we pass the mic there?

**Partho**: [40:22] Hi, my name is Partho and I'm from General Motors. My question is about the way you had that four blocker for different tools. Do you have any kind of recommendation for which visualization to pick, like the dendrogram and heat map combination you showed?

**Anand**: [40:40] Tool for which purpose? Okay, the four blocker. There is a **simplified grammar of graphics** that is making its way around these days. **I'd suggest you read a book by Noah Iliinsky.** I'm forgetting the name of the book—it'll come back to me—just search for Noah Iliinsky. There is a chart in one of his books on what are the different kinds of encodings that are most appropriate for different kinds of data.

**Anand**: [41:16] If I have categorical data, it makes sense to use color. If I have quantitative data, slightly less so. If I have a huge variation in quantitative data, significantly less so. If I have, let's say, a few categorical data points, it makes sense for me to use angles as the encoding and therefore, potentially, a pie chart. But if I have a large number of categories, then it's almost impossible. If I have ordered categorical data, so like for example primary, secondary, tertiary education—that's orderable but still categorical; I can't say that secondary is twice as high as primary, I can't quantify it—in that case, you would have a very different kind of visualization.

**Anand**: [41:53] But the thing is, these are also increasingly making their way into tools. So you could simply do a search for "chart picker wizard" or some such thing. Tableau, for instance, would suggest charts; PowerBI suggests charts; Google Sheets suggests charts. And all of these use the same principles. They analyze the data, they say, "Here are the kinds of characteristics of the data; let's pick the output." **The trouble with this approach is that you still end up with a huge number of potential charts. In other words, by following these techniques, you will only learn new ways of going wrong.**

**Anand**: [42:31] **You're better off finding out what the real objective is and what the audience is familiar with. Go for something simple. The idea is not to necessarily create something new; the idea is to communicate clearly.** Show something, ask the user to make an inference. If the first inference that they make is not what you want, try something else. **The way in which, or the process by which, you come up with new visuals is less important. The fact that you have tried it out, experimented, and validated that what you're communicating is in line with what you want to communicate—that process is more important.**

**Unsure**: [43:06] Gentlemen, now can we please take this question offline? We're short of time. Really sorry.

**Anand**: [43:11] Sorry about that.
