# Transcript

<!-- JSFoo Bangalore 2012, 20 October 2012. Original video: https://youtu.be/JAlZ0uJnqkA. Transcribed from original audio with ~/code/scripts/call (Gemini 3 Flash Preview) on 17 Oct 2026; timestamps refer to the video. Automated draft, not yet manually verified against all speech and slides. Unclear words or speaker attributions should be checked against the video before quotation. -->

**Anand**: [01:17] Am I audible from the back? Is it clear towards the back? Okay, I'll start off there. **We're going to be talking about visualizations using D3, the library**. This was originally planned to be a workshop where I talk about how you could create visualizations from D3. In the one hour that we have, I don't think we're going to be able to do that. So instead, I'm going to make a compromise. I'm going to show you stuff that can be done with D3, just excite you on the possibilities of what this library can do, but also not just about the library; **it's also about how one can conceive of visualizations and can create some fairly interesting things out of what might be potentially mundane data**. And after that, I'll try and show you some simple things, the basic model of how one creates stuff using D3.

**Anand**: [02:22] To begin with, I'll start with a very, very simple picture. Okay. I'm going to start with a very simple picture. Some of you might know of this thing called the National Rural Employment Guarantee Act, or NREGA as it's called. There's a fair bit of political overtones around it. The full name is the Mahatma Gandhi National Rural Employment Guarantee Act. And what that Act says is it's sort of like a social security act, and it's center-funded primarily, but also to a certain extent state-funded, and there are various rules around that. And each state gets a certain amount of money, which they use to promote—well, which they use to give to unemployed people directly or indirectly. There are two ways in which they can spend it. They can either spend it on labor, or they can spend it on raw materials. It's curious to see how spending on raw materials would actually help unemployed people, but it is part of the Act. And the funding can either come from the government or the funding can come from the state. 

**Anand**: [03:32] What we have here is a very simple picture that shows on the vertical axis what percentage is spent on labor versus right at the bottom what percentage is spent on raw materials. So **each bubble is a state**, and we have here Tamil Nadu, for instance, spending a fair bit on labor as opposed to, for example, Madhya Pradesh, which is spending a lot lower on labor and primarily on raw materials. I haven't put it on the scale, but the bottom is roughly 50%. So Madhya Pradesh is spending a little over 50% on labor, whereas Tamil Nadu's spending close to 100% on labor. Now, you'll notice that most of the states are on this side. The size of the state is the total amount that they're spending on the National Rural Employment Guarantee Act. So Madhya Pradesh is clearly spending a lot, whereas this tiny bubble, which is Jammu and Kashmir, is not spending that much. Andhra Pradesh is spending a lot; Maharashtra's spending a whole lot less. But you'll also see on the horizontal axis, there are two states on the left extreme. These are primarily state-funded, meaning the bulk of the money that's going into this Act is coming in from the state itself. One state is Karnataka. The other state is Orissa. Now, this is a fairly marked divide between these two states, and I'm not making any comments on, you know, the Center being Congress and these two governments being BJP or anything like that, but—and it maybe—it may have nothing to do with this at all. 

**Anand**: [04:54] Now, **this is a very simple visualization, a scatter plot that's done in D3**, and towards the second half, I'll be showing you how exactly this visualization is created and walking you through it. But in the meantime, let's walk through some more interesting visualizations. Let's take, since we are in a JavaScript conference, let's take JSLint. What we did was crawl GitHub, about 10,000 users or so, and took all of their repositories. And these are JavaScript users and their repositories and said, **what kind of JSLint errors do they make often?** How many of you use JSLint? Practically everybody. For those who do not, JSLint is a sort of error checker for JavaScript. If you get a JSLint error, you might want to correct it. It checks for things like whitespace, you know, good style practices; it also checks for potential security errors and so on. **Each bubble is one JSLint error**. And the size represents—now, there are two ways in which we can see the size. For now, the size represents the total number of files that contain an error. 

**Anand**: [06:05] Let's look at this large bubble. This one is the—it says "expected something at column B not column C." To give you an example of what this sort of error looks like or what kind of errors generate this, let me open a sample set of errors. I ran JSLint on jQuery. Now, the error that you saw there, the most frequent one, was "expected something at column B not column C." So let's search for "at column," and you've got all of these. So, "expected `var` at column 5 not column 1," "expected `window` at column 13 not column 5," "expected" and I mean, "`undefined` at column 9 not column 5." Notice that all of these are almost always four columns away from where they are. **These are classic cases of indentation**. Let's, in fact, look at these. I think I have a version of jQuery 1.3.2 out here. Yeah. So line—let's start with line 13, which is here. Now, it's saying that this `var`, which is starting at column 1, should in fact be starting at column 5. In other words, make sure that your JavaScript is indented. And **that is the most violated rule of JSLint across**. Which leads one to question, should we even have this rule? I mean, if so many people are violating it, there's probably a good argument to say it's okay not to use it. 

**Anand**: [07:37] The second most violated rule is missing spaces between something and something. What kinds of missing spaces? Let's see. Missing space between close bracket and open curly brackets. That's typically when you're defining an anonymous function. You just say `function()` and immediately put your curly brackets. That's not good practice according to JSLint. Then there's "exactly one space between something and something." When do we want exactly one space? Between `function` and open bracket. So if you look at jQuery here, between this `function` and this open bracket, there should be exactly one space, but no space has been provided. I didn't even know that was a rule. But these are among the most violated rules of JSLint across. And then there are lots of smaller ones. Let's take something really small. "Weird assignment." Okay, that is weird. "Strange loop." I'm not picking these intentionally, by the way; I'm just, you know, picking. "Variable A was not declared correctly." What's the least? "Unnecessary use strict." So there are people that are using strict when you don't need to. Good for them. 

**Anand**: [08:58] Now, let's see how this—now, remember I said this is by the number of errors in a file—sorry, by the total number of errors. Now, there are two ways in which we can count errors. You can say there are 10,000 files, and if an error occurs in a file, I'll count it once. Or I can say if an error occurs 10 times in a file, I'll count it 10 times. You can choose to count the presence of an error in a file, or you can choose to count the number of occurrences of the error in the file. And this is counting the number of occurrences of the error in the file. **What would happen if we just counted the files?** Now that changes a bit. What we had earlier as the most prominent one, which was "expected something at—well, the indentation error really—when you move it, it's no longer significantly more prominent than the others. In fact, this one, which is "A was used before it was defined"—so some variable's being used before it was defined—is the error that occurs in most files. It doesn't occur the most number of times, but that's the error that occurs in the most number of files. This error, "missing `use strict` statement," again occurs in the most number of files. This is one of those errors that can at best occur only once in a file. If you've got `use strict`, you've used it. And if not, it'll be missing. So when you move this back to the total number of errors, it becomes relatively small. 

**Anand**: [10:04] So there are a number of errors like this that can occur only once in a file and tend to misrepresent the total size. Now, **this was again created using D3**. Among other things, it tells us what kinds of errors we are most likely to make, and the single most error is indentation followed by missing whitespace followed by wrong number of whitespaces followed by unexpected spaces followed by expected one thing, saw another thing. Now, to give you a sense of what this might mean, "expected"—let's search for some other word—"and instead saw." Okay. So in line 17, we expected an identifier and instead saw `undefined`. Now, that's because jQuery is doing something weird. In this line, it's just defining a blank variable called `undefined`. Why? Because that means that `undefined` now becomes a variable instead of a keyword, and you can minify it if somebody has in the JavaScript environment redefined `undefined`, which you can do, and that can lead to some security issues among other things. This will still take care of it because if you don't define a variable, by default, it always gets the value `undefined`. So it's doing a rather clever trick, but JSLint is obviously saying, look, you shouldn't be doing stuff like this. So that's on JavaScript errors.

**Anand**: [11:11] Let's pick something else. Let's see what people have been tweeting or when people have been tweeting about JSFoo. **Every dot here is a tweet that has the hashtag #jsfoo**. And what we're doing is seeing this over time starting from Thursday afternoon where a few people have tweeted, and time's flying along. **The black dots are original tweets, the red dots are retweets, and blue dots are replies**. And if the reply is connected to some tweet that exists in this list, then it's connected to this particular tweet. So early in the morning, there aren't too many tweets, thankfully. 3:00 AM, people are still asleep. 4:00 AM, 5:00 AM, people have not yet woken up. And let's see when they wake up. 7:00 AM. So we woke up at 7:00 AM yesterday. 8:00 AM, a lot of tweets, people are coming over to JSFoo, and then JSFoo starts. So people are tweeting through the day. The event becomes richer and richer. Around lunchtime things die down; we're obviously, you know. In fact, towards the afternoon, things get a little slower. Sleepy after the food, or what? I don't know. But towards the evening, things get somewhat quiet. 8:00 PM, but people are still tweeting or in many cases retweeting.

**Unsure**: [12:35] `weird.js`. 

**Anand**: [12:36] Ah, okay. `weird.js`. Okay. But seems to be going on till almost 1:00. Did it go on till 1:00 yesterday? 

**Unsure**: [12:43] I reached home at 1:00. 

**Anand**: [12:44] Wow. Okay. And tweeted. Okay. So you were probably the last dot. Hey, but there was something at 5:00! Somebody's waking up really early. 7:50 this morning, 8:00, people are slowly starting to tweet. 10:00, hmm, now's when the tweets should start. Actually, hmm, not that much. Huh, it's around 12:00 that they really picked up. And that's 1:57 just before this talk is where I stopped. So today there haven't been as many tweets as there were yesterday. Let's see what the nature of these tweets are. So, like I said, the red dots are the retweets, the black dots are the original tweets, and the blue dots are replies. Not too much conversation happening here; I don't see that many blue dots. Let's pick a cluster that's got a reasonable number of replies. 

**Anand**: [13:30] So we've got this one. Okay, that's `@shameerc` saying, "I'm kind of interested to know how or where all of you use Backbone.js? Links much appreciated." And that's gotten 1, 2, 3, 4, 5 replies. So somebody's said, "We use it for building single-page or highly interactive app." Somebody's said, "It's a great addition for single-page apps." "If you'd like a walkthrough of some Backbone.js code and some examples, feel free to drop by at our table, blah, blah, blah." There's also a chain here. So somebody's replied to somebody who's replied to somebody who's replied to somebody. So what's the original thread here? That's `shari_p`: "Just came back from JSFoo, didn't feel that it was that interesting." Okay. "Hoping for better tomorrow as `@karthik_` is giving a talk." Did Karthik give a talk?

**Unsure**: [14:15] No, it was a lightning talk.

**Anand**: [14:18] Oh, okay. "Hmm, looks like we too catching up is going to have to wait, won't be able to make it. Oh dear." Okay. And `shari`'s replied: "What? No, was relying on your talk to improve the conference, but it's an emergency. Okay. Not coming to Bangalore and I'm sure tomorrow's talks will be awesome. Let's hope." So that was a long chain. 

**Anand**: [14:42] Here's something with two replies. Let's see what that is. So that's `zainab`—oh, is that—do the quotes mean retweeting something? Potentially. Oh, okay. And what were the replies to that? Oh, okay. The reply was actually a retweet of the same thing. And, you know, the sheer number of reds here leads me to think that JSFoo isn't the kind of crowd that tends to talk original as much as tends to pass on stuff. **Let's compare it with PyCon**, which happened, well, September 29th. So in the morning, a bunch of original tweets coming in, new tweets, probably people getting into PyCon. It started a little later. And now's where the event starts. A few retweets coming in, a few replies coming in, but still mostly original tweets happening out there. Afternoon, a bit more, not as many as there were in JSFoo. So clearly it's not a tweeting crowd, but the few that do are putting in their own messages. There still aren't that many retweets. Goes on till the evening around 8:00 things start dying down. Hmm, and then pick up—picks up after dinner maybe? Okay, this is a slightly late-night crowd. They're tweeting all the way till midnight, 1:00 AM, and phew, finally, okay, they do get some sleep. Nothing in the early hours—no, 4:00, 5:00. Between 4:00 and 5:00, there are some people. In fact, somebody's actually replying to something. There is a chance that this might be someone in a different geography; we'll have to check. And then 9:00, that's when things begin on Sunday. Lot more tweets pouring in that time around afternoon, bit more of activity, and then slowly it starts dying down. Not too much activity, which seems to be a common pattern; there doesn't seem to be too much activity post-lunch on Twitter. And around 4:00 is when I stopped getting the tweets. But **look at the pattern there. Far more black dots as a percentage than red dots**. 

**Anand**: [16:34] Let's take Paris Web, which has been happening over the last couple of days at Paris. It'll still be a bit slow. But yeah, you know, Paris Web is a much bigger event, at least in the Twitter universe. It's quite slow here; I wish I could show it to you on my laptop. It comes on like an explosion. This is over a very short span, less than a few hours. It was from 4:00 AM to 8:37 AM or something. And clearly you see it struggling. Now, if you look at the percentage of reds versus blacks, it's about the same, meaning roughly as many reds as there are blacks, perhaps. So that seems to be a uniform mix. But the number of blues is certainly higher. Not very high, so clearly Twitter's not being used as a conversation medium; by and large, it's used to say stuff that you're doing, to pass on stuff that other people are doing, not much to reply. But at least in Paris Web, there seems to be a lot more activity. 

**Anand**: [17:35] I was looking at the Twitter trends this morning, and there was #SOTY, which seemed to be a big trend. Anyone knows what SOTY is? *Student of the Year*? Karan Johar, is it? Okay. So I just pulled the tweets; I haven't seen what it looks like. So 3:00 in the morning, people are—oh wow, okay. They're clearly very excited about it at 3:00 in the morning. And goes on, 6:00 AM. I stopped—well, a fair bit of activity there this morning. Combination of retweets, replies, and so on. I stopped at around half-past eight or that's when I pulled the tweets. But yeah, no, that's certainly a trending topic. And it started at 3:00 in the morning, so people must have been either waiting for it to start and, you know, push the messages, or maybe this is some marketing campaign that's going on. 

**Anand**: [18:25] Anyway, let's look at something else. Let's look at shopping. So if you walk into a store, and this happens to be a store in the UK, you'd normally come in from the outside and make your way to one of the aisles. Each of these bars is an aisle, and the thickness of the line tells you how many people make their way in that direction. So people start outside. A reasonable number of people go to the general merchandising section, but the vast majority of them come to the fresh produce section. By the way, how do we know this? Because in this shop, we had a bunch of carts which had an internal GPS kind of a system. So as people pushed the carts along the aisles, this thing would track where exactly the cart went. So we have a rough sense of which aisle they hit or—so for a given cart, we know that it hit aisle A first, then aisle C, then aisle D, then aisle X, and so on. We don't know how fast they moved and stuff like that; that wasn't—or I don't have that data. But at least we know the direction in which people went. So they start from the outside, they go on mainly to fresh produce. 

**Anand**: [19:35] Okay, so after going to fresh produce, where do they go? They go to buy meat, fish, and poultry primarily. But a few of them also go outside—zoom out—a few of them go outside. Now, remember, this does not mean that they came in and went out immediately. All it means is that of the people that came to fresh produce, and they may have come after finishing their full shopping round, they're exiting from here. So it's a relatively small fraction that are exiting from this location. They primarily go to meat or fish or poultry; a few go to ready meals; a few go to dairy; very few go to general merchandising. Almost no one goes to any of these directly. So **there certainly seems to be a strong proximity effect**. There aren't that many people that go straight to a section and buy stuff from there and head out. 

**Anand**: [20:17] But what about those people that went to general merchandise? They primarily come to fresh produce or entertainment. A lot of people also go—general merchandise—before going out. They also go to the apparel. So there are two sections on the left, apparel and cafe, which are actually on a different floor. So let's not count them. People come outside and then climb up the or go near the gate, climb up the stairs, and then go to the sections. Okay. So from fresh produce, they primarily go to meat, fish, or poultry. From there, they go to the next section, which is ready meals. From there, they go to the next section, dairy. From there, they go to the next section, yogurts and desserts. From there, now there's a breakup. See, up to here, so **it seems almost a linear flow**. And there's a reason why these sections are arranged that way. From fresh produce to meat, fish, poultry to ready meals to dairy. **These are the most common purchases that people make, and it's so easy for them to just go to one, get the next, get the next, get the next**. But from then on, when you get to yogurts and desserts, now that's not something people buy as often. World foods, certainly not as often. So from dairy—or sorry, yogurts and desserts—I got that wrong, yogurts and desserts is essential, and a lot of people buy that. But from there, a few go to world foods, but a few go to canned foods, rice, pasta, recipes, bread, cakes, rolls. And across all of these, there's also a little bit of migration towards the opposite aisle. So people cross the aisle. But **notice how few people are crossing the aisles. Most people are going along one track**; they just say, I'm staying on this side, finishing all my shopping, going right to the end and coming back. Which almost means that you can have lanes in this store, one lane for going all this way and another lane for going all that way. 

**Anand**: [21:55] Oh, it's a half-hour session, is it? I thought it was a one-hour session. If it is, then I'll wrap up. Thank you. It is a one-hour session. We'll wait for a reconfirmation. It is, right? Okay. Okay, thanks. So, yogurts and desserts, blah, blah, blah. So notice that they stay on this side. From canned foods, they go to the next section. From rice, pasta, they go to the next section. Understandable; there's not that much jumping. But from all of these sections, there's always a small probability of them going outside. Let's go to the last section. From there, there's clearly a transition to the opposite aisle, and then they start moving backwards. 

**Anand**: [22:31] Now, it could have happened the other way as well. You know, it's just as possible that people start from fresh produce and then go to baby, health, pharmacy, but no, they don't do that. They go to the opposite aisle and then come back here. A lot of people seem to be exiting from this section, which I would worry about. You know, what's this section got that—I mean, either it's the last spot people shop for. I don't know what BWS is, by the way. But whatever it is, either that's the last spot people are shopping for and therefore are not interested in buying anything more, or it's got such bad—sorry? 

**Audience**: [23:05] Beer, wine, and spirits. 

**Anand**: [23:06] How many people go there straight? No one. Okay. Where do people come to beer, wine, and spirits? Okay, some from nuts and snacks. You know, this never hit me. This is, you know—okay. Not many from sugar and tea and coffee, nor eggs, breads. You know, **what was this thing about beer bottles being stacked next to diapers? Let's test that**. Baby foods, baby healthcare, and pharmacy. No. At least in this store, that doesn't seem to be the case. But from beer, yeah, okay. From beer, they—okay. They've got their priorities right. **Buy the beer first, then we'll worry about the diapers**. Which sort of makes sense. 

**Anand**: [23:56] So that's one way of exploring this sort of data. Let's take some other data. Let's take weather data. Now, some of you may have seen this visualization of mine before, and it's a somewhat slow one. This is a map of India in which every district is highlighted. I've got a few gaps there because early this morning I didn't have the patience to cleanse the data; there were a few district names that were misspelled, and that's what's causing those gaps. But **the color here that you see is the average temperature of the district across a century**, and every flash is one month's worth of temperature data starting from 1900 all the way down to 2000. So we are probably somewhere in the 1900s—early 1900s right now. Now, the darker the red, the more hot it is; the lighter the green, the cooler it is. And you can see that J&K stays cool throughout the year. The Northeast is reasonably cool most of the time. The West Coast is surprisingly cool almost right through. And so is Porbandar; that's Porbandar. So, you know, if you want to retire, barring the rain, you now know where you ought to be. This flashing spot, if you notice, has a slightly offsetted weather—



**Anand**: [24:59] ...offsetted weather patterns. Relatively hot when the surrounding areas are cool and cool when the surrounding areas are hot. That's Bilaspur in Chhattisgarh. It's a valley. You also will notice a somewhat similar pattern in Shimoga. It's not coming through very clearly here and it becomes starker over the second half of the century. But Shimoga also has a counter-cyclic weather pattern, which means hot when the surrounding areas are cool and cool when the surrounding areas are hot. They're both valleys, but there are many other valleys; we don't quite know why this is happening. But, you know, it's one of those things that seems to be happening.

**Anand**: [25:35] You've seen word clouds, I'm sure. But **what if the word cloud were in Sanskrit?** Why do we have all of these word clouds just in English? So, we took the *Ramayana* and said, let's plot all the words there. For those of you who know Sanskrit, I guess most of it will make sense, but for those who don't, the epic is mostly about *Raja* [King], *Ramaha*... Now here's where you start getting into a big problem in Sanskrit, or many Indic languages, that you wouldn't get into if you were dealing with, say, English. 

**Anand**: [26:13] The fact is in English, by and large, most words are discrete. If you have a word like "has," it's usually spelled "has" and there aren't too many variants. "Hasn't," for instance, is a variant. And you can detect many such variants; it's not trivial, but you can get the vast majority of those variants easily enough. And if you simply split based on spaces—just take any corpus, split based on the spaces in between and say those are my words—you wouldn't be wrong too often. You wouldn't get too many overlaps and you'd be right, say, 80–90% of the time. 

**Anand**: [26:45] In Sanskrit, you have problems like: okay, so there is *Ramaha* [Rama] here, which is one variant of Rama. Then you have *Ramam* [Rama as the object]. Then you have *Ramo* [Rama], which is exactly the same as *Ramaha*, just that if you have an "aha" at the end, you can also use "o" if there's a vowel succeeding it. Where else do we have variants of Rama?

**Audience**: [27:07] *Rama*.

**Anand**: [27:07] *Rama*. Okay, that's just plain Rama. 

**Audience**: [27:13] *Ramam*. There is *Ramam*.

**Anand**: [27:14] *Ramam*. See, yeah, we saw *Ramam* somewhere, right? Where did...? Okay, you're probably able to spot it a lot better than I am this close. So, you know, the thing is, so there is all of these variants: *Ramaha* [Rama], *Ramam* [Rama], *Ramena* [By Rama], *Ramaya* [To Rama], *Ramasya* [Of Rama], *Ramat* [From Rama]. All of these variations, which are effectively "from Rama," "by Rama," "to Rama," which get combined with this word. And segregating that out is a nightmare. All I've done is removed the stop words, which is things like "and" (*ca*), "also" (*api*), stuff like that. And it's not complete; there are a huge number of stop words still left out there. 

**Anand**: [27:48] But one gets a sense of what the epic is about. And incidentally, the *Ramayana* is a lot simpler an epic. So there's a lot about kings, *vakya* [people saying stuff], *dadarsha* [showing stuff], *vacanam* [saying stuff]. Now as a main character, you see Rama in a number of places. You'll probably see Ravana in a few places. You'll see Sita in a few places with the variants. Do you see Lakshmana? 

**Audience**: [28:16] Above *Raja*.

**Anand**: [28:18] Above *Raja*. Ah, okay, Lakshmana. And you'll probably see—okay, I see Sugriva there. But the thing is, you don't see as many characters as you would in the *Mahabharata*. And that's also the nature of the epic. **This is a somewhat simpler, more linear epic. Few characters. It's more like a, you know, Telugu masala movie if you will**—relatively simpler plot—as compared to the *Mahabharata*, which is more like a soap opera.

**Anand**: [28:48] Incidentally, let's do this for the JSFoo tweets. Okay. So it's primarily about HasGeek and Kiran and Zainab. There's a lot of talk about live streaming, talking, watching, live, tomorrow. Why are people talking so much about tomorrow and not today? A lot of talk about hacknight, Raspberry—the Raspberry Pi talk I'm sure has been quite interesting. Awesome, multi-node, Bangalore JS, JavaScript, blah, blah, blah. That's roughly the pattern. 

**Anand**: [29:23] Now, let me show you one last—or maybe two last—visualizations. Let me show you this salary bit. This isn't something that I've created; this is on `wealthfront.com`. And what it shows you is a visualization of, on the horizontal axis, what percentage of equity are different kinds of professions given, and what is the average cash on hand that they're given. 

**Anand**: [29:57] So, for example, if you were into business development, then as a sales engineer or as product management, you'd be getting a fair bit of equity and a fair bit of salary as well. Whereas if you're in administration, different story altogether. And one can start breaking that up. So let's take software engineering. That's scientific research; that's not quite what we wanted. Let's get to software engineer. **As a software engineer, you might earn as much as $182,000 to $200,000 if you were a director, and you'd get a lot of equity**, 30%. So anywhere between the 25th and 75th percentile are between 8.5% to 30%, which is quite good. As an architect, you might even get more equity, though slightly lower pay. And then there are the lower levels; managers, reasonably fine. 

**Audience**: [30:52] How do you define what director means? It's not what it means in India. The word "director" doesn't mean a director of the company but rather a product director.

**Anand**: [31:00] Oh, yeah, of course. Yeah, sorry. By director, you would be a product director, for example, or the director of a business unit, etc. Not necessarily the owner of the company. And obviously, if you were the owner of the company, your stake is going to be much higher, which actually leads one to wonder: is there potentially some confusion even there between director as the owner and director as a role? Because that equity...

**Audience**: [31:22] In US, it is board member, not director. Director is a post.

**Anand**: [31:25] Okay, fair enough. Let's look at some other profession. Let's look at what you would make if you were in design or UX. Design or UX. Not as much. So in smaller companies, you'd make—you'd get a lot more equity, but there's more variability around the pay. Larger companies, there's less variability about the pay, less equity, and you'd make around $100k or so. 

**Anand**: [31:57] Well, those are all examples of stuff that can be done using D3. And I'm going to run you past a showreel that was prepared by Mike Bostock that gives you an idea of the kind of animated transitions D3 can make. So this takes four companies' stocks—Apple, Amazon, IBM, and Microsoft—and shows the same data in various forms and gives you a sense of what kinds of charts are possible and how these can be transformed between each other. So that's D3, what D3 can do. 

**Anand**: [32:33] And in the next 15 to 20 minutes, what I'm hoping to do is explain how D3—how you can use D3 to create some visualizations. Before I jump in into that, any questions? Yes.

**Audience**: [32:50] Your first chart was about the bubble chart, about NREGA.

**Anand**: [32:54] NREGA.

**Audience**: [32:56] Yeah, the NREGA one. So how does it make sense to have it as bubbles? Okay, I think the second chart where you were showing the JavaScript errors, JSLint. **How does it make sense to show it as bubbles instead of having in a graph form where it is more easy to make sense out of which error is made more? Because in circles you don't really know which radius is bigger.**

**Anand**: [33:20] Very true. Now, **the reason why I put it in bubbles is not because it would be a better data visualization; it is simply because I wanted to show circles and it looked cool**. If I were doing this for a client, I would have certainly put it in a different way. We tend to avoid circles. For those of you who are unaware of this point: it's a lot tougher to compare areas of circles and a lot tougher to compare the radiuses of circles than it is to compare horizontally across bars. **Areas are slightly tougher to compare than distances, especially when they're laid out next to each other, and angles are among the worst things that you can use to compare**. So in general, circular visualizations tend to be more wasteful of space, less effective from an Edward Tufte perspective of data-to-ink ratio. But in a lot of cases, if done well, they look more visually appealing. So that's your answer: it was for jing-bang effect, not for accuracy. 

**Audience**: [34:11] So as a follow-up question: we have—I have used D3—and we have a lot of options for doing various visualizations. But to understand which visualization makes sense for which type of data is the most difficult task, not the programming part. So how do you tend to get around that?

**Anand**: [34:29] I will give you a very, very brief answer to that, but first begin by saying that that is by far a tougher problem by several orders of magnitude: knowing what visualization to use when. The rough—okay, let me give you **two rules of thumb that would probably be the most effective. First: if you've seen a similar visualization that you liked, copy it**. That is probably the best way of figuring out what is the right representation, which therefore means you need to be on the lookout for various kinds of visualizations. 

**Anand**: [34:59] **The second thing is, all other things being equal, follow this hierarchy. If you want to compare something, make sure that you're comparing it via distances: X-axis, Y-axis, specifically position**. We are excellent at comparing position. We are slightly worse at comparing sizes. We're slightly worse than that when comparing colors, especially when they're close to each other. In fact, colors you should just keep for discrete categories and never have more than 20 colors ever. And lastly, angles. Avoid angles at all costs. 

**Anand**: [35:34] Using this hierarchy, you can choose a visualization where you say, these are my most important metrics and therefore my most important metrics should lie along the position axis. Then if I have a secondary axis—if I have a secondary dimension that I need to compare—and it is possible to do it using size, do it using size. If it's possible to do it using color, do it using color. If not, just put that into a separate visualization. 

**Anand**: [35:58] The last part actually I just want to add on: you may benefit from two things that visualizations on a computer can do, which is you can start putting it in a drill-down manner. You can say I'll start with a top-level visualization and then click to get to the next level. And using that to break up a visualization. Or you can use **the principle of small multiples**, which is: I don't have to have one graph. So let's say I want to see India's weather and I want to see it on paper, or that animation was too complex for me. What I could do is take a grid: January, February, March, April, all the way to December; 1900, 1910, 1920, all the decades. And I have a grid. In each grid put a map of India. And that's the principle of small multiples. So that's one way you can break up visualizations.

**Audience**: [36:34] What is the browser support for D3?

**Anand**: [36:36] **Browser support for D3 is pretty much everything except IE**. I should clarify: IE 9 works okay, IE 10 works well.

**Audience**: [36:45] Yeah, you've shown quite a lot of interesting data out there. So is there a public data available, like the temperature thing that you've shown? So what is the data source that you primarily use and which of those—I know Twitter is something we can access—but other things which are publicly accessible?

**Anand**: [37:04] So data sources... let me tell you that the easiest way you, this audience, will probably find data is by just asking people. Okay, I'm sure most of you are experts at scraping, and as an audience, you know, **a programmer crowd is much better at scraping than asking for help. Best way is ask for help**. And it took me many years to get to a point where I, A, have a large collection of datasets and, B, have a large number of people I know whom I can reach out to for datasets. It does take time as well. Anything more detailed than that is not for this session, I would say.

**Audience**: [37:42] Just a follow-up on the guidelines—the rule of thumb you gave on deciding. I'm wondering whether you have it documented somewhere because your words were very quick and...

**Anand**: [37:53] Okay, so I'll give you the hierarchy again: it's position...

**Audience**: [37:56] No, it's okay, I might forget it still. So I'm just asking...

**Anand**: [37:58] I might forget it as well! I just am remembering it from—well, mostly experience—but I have seen this somewhere. I honestly can't think... 

**Audience**: [38:08] Okay, sure, I'll find it.

**Anand**: [38:09] I'll try and tweet it. 

**Audience**: [38:12] So the other question was generally about D3. The examples and other things are really overwhelming. If you go to the site, the possibilities are really overwhelming and you actually forget what you want to visualize. So do you have anything about what to look for when you look at the charts and which could be of helpful for general usage?

**Anand**: [38:33] I'd suggest do it in a two-phase fashion. **Spend some time looking at all the charts so that it goes in the back of your head and sleep over it**. And then use a different time slot to think about what you want to create. When you say, "I'm going to create something, this is the problem I have, let me see what I can find," you will easily get lost in the visualizations. Just split it out. Turn the internet off; might help.

**Audience**: [38:58] Just like Google Data is a pretty good source of public data. And there's a book called *Visualize This* by Nathan Yau; he covers like everything you need to know about visualization. 

**Anand**: [39:09] And Nathan Yau's also published a small compilation of what are the various sources of data. So has a guy called Pete Warden, W-A-R-D-E-N or W-E-R-D-E-N, I can't quite remember. Both of them have a collection of a number of good data sources that you can pull out and play with.

**Anand**: [39:28] Let's do this. What I'm going to do is show you the code for the NREGA visualization, if I can find it. Let's start: `nrega.html`. Let's do a "View Source" and I'm going to talk you through this. I'll make it a bit bigger, hope that it's visible from the back. Okay, yeah, that size is probably all I can risk. 

**Anand**: [39:54] So this is where the interesting stuff is happening, but I'll just for formality's sake cover the rest of it. We start with a regular HTML5 doctype, we put in a title. I've got a stylesheet `common.css` which has very little. I'm using Bootstrap simply out of habit. I'll show you what `common.css` has. I'm using Bootstrap, like I said, out of habit and I'm saying, you know, define the body's width and for SVG blocks just make sure that it looks a little pretty. So all it does is draws the shaded border around it, nothing more. So that's all `common.css` has. Let's skip all of this. 

**Anand**: [40:32] Then we get into the heading and a paragraph and we've defined an SVG tag. Inside the SVG tag, we've got a bunch of labels. So you see this "Spend on labor" top left, "Spend on raw materials," "State funded," blah, blah, blah. A very crude way of labeling it, and all we've done is put text there. **One of the prerequisites of getting the most out of D3 is that you know some amount of SVG**. You don't have to—D3 can work with divs as well—but you'd get a little more out of D3, a lot more out of D3, if you knew some SVG. 

**Anand**: [41:07] Then we include D3. It's a mid-sized—well, maybe even large library; unzipped, un-minified, it's about 200 KB. But for the kind of visualizations that you use, the alternative is to use an image which will invariably be larger anyway, so most people don't worry about the size of the payload. 

**Anand**: [41:23] Now, what we do then is `d3.csv`, which is an AJAX call to pull out a file, and then we get the data, and then we do a bunch of stuff. Now before I get into this "bunch of stuff," I've got to explain some basic concepts of D3, which is what I'll do over the next few minutes. What I'm going to do here is reduce the size of this, get into the... might be a bit of a challenge, let's see. 

**Anand**: [41:50] First thing I'm going to do is show you how D3 is like jQuery and then I'll show you how D3 is not like jQuery. To begin with, **D3 has selectors similar to jQuery**. So you say `d3.select` and what that does is selects one element. Unlike jQuery's dollar, which selects all the elements in a selection, this selects one and only one element and can trip you up the first few times you use it. So if I say `select("svg")`, that returns one element. 

**Anand**: [42:23] If I want to remove everything under that, what I've got to do is say `select` from `svg *`, but I've got to make sure that I say `selectAll` because otherwise it would only select one element. Now `selectAll("svg *")` would pick everything under the SVG element; that's 60 elements, mostly the states plus four labels. And say `.remove()`, which clears up the canvas. Now I can start drawing stuff. I can, on this SVG element—I get the SVG element first—`d3.select("svg")`. Now that I've got this SVG element, I can append to it, that's the next thing we have, another element. Let's say I'm going to append a rectangle to this. And I can set various attributes to it. 

**Anand**: [43:08] Now, in order for a rectangle to be visible, at the very least I need an x, y, width, height, and a color. Color is optional, but I'll still put it in. And you can specify those using the attributes: `x`, I'm going to say 100; `y` is 100; `width` is 100; and `height` is 100. **Notice that it has jQuery-like chaining**, but also notice that I did not use the jQuery equivalent of a dictionary here, and that's because it isn't yet supported. And I'm going to add another attribute, which is `fill`, which I'm going to give color red. And I have a red rectangle. 

**Anand**: [43:56] Straightforward so far. So **this is how D3 is like jQuery: you can do stuff that jQuery can do, not as easily as jQuery can do in some cases, but the facility is there for most cases**. The power of D3 comes when you start taking data and tying that into the dataset. Now, **D3 stands for Data-Driven Documents**. Now this is a very powerful concept. What it says is: you've got data and you map that data onto elements of the document. For example, let's take NREGA again. I've got a list of states. I want each state to map to a circle. That is the concept, which means that every attribute of the state will map to the attribute of the circle as well, or at least I'll have the ability to do that. If an attribute of the state changes—let's say the percentage of center funding changes—I can make it move as an object. I can map the percentage of center funding, which is a variable of that object, to a position, another aspect of the DOM element. Or you can map it to any other thing for that matter. 

**Anand**: [45:04] So how does one do that? What we do is—firstly, I'm going to remove this rectangle—or, you know, I'm going to let it stay, it's fine. `d3.select("svg")`. Now at this point, I have selected an element that exists, which is the SVG element. And then I'm going to do something weird, which you wouldn't do in SVG, which is `selectAll("circle")`. 

**Anand**: [45:28] Now, here is the first place where D3 differs from jQuery. **`selectAll` does not necessarily only select stuff that exists; it also selects stuff that could exist in the future**, at least from a certain perspective. So when you say `selectAll`, what you're really saying is: take all of the stuff that is and could be inside this as circles. 

**Anand**: [45:48] Now the second thing that D3 lets you do is bind data to a selection. So at this point, you can say `.data` and you can put in an array. I'm going to say just `[1, 2, 3, 4, 5]`. Now at this point, what we're saying is: inside SVG, there may or may not be circles, but there ought to be five circles corresponding to each element here. I've got `1, 2, 3, 4, 5`; it doesn't have to be `1, 2, 3, 4, 5`, in fact it typically is an array of objects, and the attributes of the object then start becoming the attributes of the element. 

**Anand**: [46:28] Now with this, I have defined what is called—I'm just going to call it `nodes`, let's not get into the technical terminology. Now I've just defined this. I've got to tell it to draw it and how to draw it. Now there is a very—**the third important concept with D3 is knowing the update cycle**. There are three possibilities: you have a set of elements that exist and there's more data that does not exist; or you have as many elements that exist as there are data; or you have more elements that exist than you have data. All three are possible. Total number of elements versus total size of the data could be less than, equal to, or greater than. And what D3 lets you do is say: if there's more data, how should we add it? If there's less data, then how should we remove the stuff? And if there's the same amount, then how should I change the existing stuff to map it? So which is effectively the **Create, Update, and Delete operations**. 

**Anand**: [47:30] And in order to specify the create operations—right now, we know that there are exactly five elements in the data and we know that there are no circles—so what we want to do is create it. So we say `nodes.enter()`. When you say `nodes.enter()`, it tells D3 we've got to start doing some creation work. The creation begins with appending an object, so we say even though we've done a `selectAll` of circles, we say anytime you find more data than you've got elements, start putting in a circle for each element and then start setting the attributes of these. 

**Anand**: [48:06] So what I'm going to do is set the attributes as: `x` to be—now here's another interesting feature of D3, which is that you can set this to be a function. Oops. And in that function, you're by default passed the data element. So I'm going to return—we've got 1, 2, 3, 4, 5—so I'm going to return that element times 100. And I'm also going to put in a `y`, which is `function(d) { return d * 100; }`. So it's going to draw x and y as the same, so it'll start with 100, 200, 300, 400, 500. And I'm going to put in a radius of, let's say, `function(d) { return d * 20; }`. And I'm going to set a color—let's just put green for now. Let's hope this works. 

**Anand**: [49:07] Okay, not quite. Looks like they're all sitting on the top left. Now let me show you the command again and let's see if you can spot the error for those of you who are familiar with SVG. 

**Audience**: [49:25] The coordinates are negative. 

**Audience**: [49:32] D is always zero; descending order. 

**Audience**: [49:36] The last circle's sitting on top; y should be negative. 

**Anand**: [49:42] No, not the radius, no. The position of a circle is determined not by x... 

**Audience**: [49:52] It's centered on top of each other. 

**Anand**: [49:53] Yes, but it shouldn't be, right? We said x is 100, 200, 300, and so on. 

**Audience**: [49:56] Top left corner is the default. 

**Anand**: [49:58] The name of the attribute...



**Anand**: [49:59] The name of the attribute is wrong; it's actually `cx` and `cy`, not `x` and `y`. So what I'm going to do is delete all of these: `d3.selectAll("svg *").remove()`, and create the nodes again. And now, instead of `x` and `y`, let's try putting in `cx` and `cy`. And we've got the circles where we want them to be.

**Anand**: [50:30] What we've done here is mapped a dataset onto objects. And **this is the exact same principle that has been used in every one of the visualizations that you've seen so far**. Let's take an example; let's take our tweets. So, what we're doing here is periodically adding an item to the data and then saying `data.enter()` and then putting it at a certain position. Now, incidentally, D3 has a layout that automatically moves the position around. Let's not get into it; I probably took the wrong example to show you how this works. Let me pick something simpler. Actually, let me just show you the NREGA thing; that's the simplest of the lot.

**Anand**: [51:15] So, let's take a look at the code. What we're doing here is `d3.csv` gets this file `nrega.csv`. And using that data—what `nrega.csv` has is: A, the state (each row is a state); B, the percentage of center funding; and C, the percentage of spend on labor. We say `d3.select("svg")`—that takes the top-level SVG element. We `selectAll` the non-existent circles, and then we bind the data. We say the data that I want to use for all of these circles is in this data. `.enter()`, which means now we want to start telling D3 what to do when it finds more data than it has circles.

**Anand**: [51:57] First, append a circle. Put in attribute `cx`. Now here, we're putting in a function which returns `d.center`, which is the percentage of center funding, multiplied by 900 and add 40. That's just to position it on the scale because the width is 940, so just 0 to 100 maps to 40 to 940. `cy`, we again do some transformations so that the bottom is zero or something else, and get a function. Lastly, we take the radius `r`, and that is the square root of the total NREGA funding. **Why do we use square root? Because if we use the actual value, then the radius would be proportional to the value. Usually, we perceive size as how important something is, and size is proportional to the square of the radius**. So we take the square root of the size, which is the funding, and use that as the radius.

**Anand**: [52:50] And then we put in a color. The color is—we return an HSLA-based color which is just taking various hues on the edge of the circle and putting in what is close enough to a random color. And then inside it, we append a title and put in the text which is the name of the state. So **with those 30 lines, you get a scatter plot. Every single one of these visualizations, without exception, has been created in less than 100 lines. That includes all of the HTML and so on**.

**Anand**: [53:21] So, that is where I want to end before we take questions. What you can do with D3—this was a sample of—it's a fairly powerful visualization library. **The crux of it is you can map data to visual elements**. Thank you for your time. Happy to take questions.

**Audience**: [53:48] In the tweets example that you had shown us, does D3 handle the animation aspect of figuring out that the objects need not get overlaid over each other?

**Anand**: [53:58] Yeah, the question was: does D3 handle the animation in the tweets example? The answer is absolutely yes.

**Audience**: [54:03] Can you just kind of show me the API for that? Just a snippet?

**Anand**: [54:07] Sure. So, let me show you what's not relevant. There's this entire section which I've very crudely coded, the sole purpose of which is to update this timer. The real code is here. And the important pieces are this call `force.drag` and `force.start`. So we create a **force-directed layout**, which is the visualization that's being used here, and we give it some basic parameters. And then we say every time there is a "tick" (which means there is a movement), this is how I want you to update the lines and the circles. And `force.start` keeps calling those ticks every now and then. `force.drag` lets me move each of these to wherever I want. All of this is built-in. And like I said, **this code is still under 100 lines**.

**Audience**: [55:07] Hello. So, could you shed some light on the different extensions and plugins that are used along with the D3 to make some useful visualizations, like something literal like some databases or some other storage engines that are used with D3?

**Anand**: [55:26] The question is: what other libraries or tools are normally used along with D3 to, like, get the data from databases or let's say elastic search or some other similar data storage engines?

**Anand**: [55:39] Those are usually server-side tools. I would rather not take that now; maybe we can talk about it offline. But I guess there were many presentations focusing on how to get data from a database. Remember, though, that this is an entirely different problem. **D3 can natively consume XML, CSV, and JSON**, and you've got to pass it data in this format, or you can pass it in any other text or binary format that JavaScript can parse, but then you've got to build your own parser. What you're asking is the problem of: what libraries can I use to create that data from a database and pass it to this?

**Audience**: [56:15] Extract that data, yeah.

**Anand**: [56:17] Extract the data from the database—which would be a server-side problem. And I think that is completely independent of D3, meaning it doesn't matter whether you're using D3 or FusionCharts or Raphael or any of those visualization libraries; that would be common. And I would, if you don't mind, go into that offline.

**Audience**: [56:39] Hi Anand, thank you for the wonderful presentation. I have a small doubt in the spheres visualization. Did you use any sort of force-directed layout for placing the spheres?

**Anand**: [56:54] No, not quite force-directed layout. It has what is called a **pack layout**. And let me show you what that looks like. So, if you look at the code here, there is a different layout called `d3.layout.pack` instead of `d3.layout.force`, which what it takes is a bunch of shapes and puts it in the most compact representation possible. Not shapes—circles, clearly.

**Audience**: [57:22] Yeah. So, I mean, I can see a spiral there. So that the layout takes care of the spiral or like you need to manually...?

**Anand**: [57:28] It isn't actually creating a spiral intentionally. What it's doing is taking the objects one by one—so it's taking the first object and, since I've sorted it in descending order, the first object is the biggest and it puts it somewhere. Then it takes the next largest and puts it as close as possible so that it's all packed. Takes the next, packs it as close as possible, and so on. And **it isn't doing it very efficiently if you notice, because this circle would have fitted here. So it doesn't necessarily do the most optimal packing; what it is doing is taking a good trade-off between speed versus size**. So you just lay it out and hope for the best. If you were to randomly sort it, you might get—not might, quite often will get—a better packing than if you were to sort it. But if you sort it, it tends to look nicer sometimes.

**Audience**: [58:13] Cool, thank you.

**Anand**: [58:14] **If you sort it ascending, it looks almost exactly like a spiral**. In fact, let's see if I... no, I won't have the time to show it to you, but all of this code is on GitHub. You can just pull this in and sort in descending order; it'll look like a near-perfect spiral.

**Audience**: [58:40] I missed the first five minutes of the presentation, so maybe you mentioned this, but what other visualization libraries do you use at Gramener and how do you choose between, like, when you start a project, which one to pick?

**Anand**: [58:53] Okay, the question was what do we use at Gramener. For those—I don't think I introduced myself at all. My name is Anand. I'm in this company called Gramener and we do data visualization. Sorry about the lack of introduction. As for what we use at Gramener, we do not use JavaScript at all. If we do, we use D3 and that is sparingly. If for any reason we could not—and I'll tell you why we cannot use D3, and that has happened only once so far—we use Raphael.js.

**Anand**: [59:27] **The only problem that we have with D3 is that it does not work on lower versions of Internet Explorer**. Other than that, it is fit for purpose for almost every other visualization. **The secondary problem that D3 has is the slightly higher learning curve**. It isn't as easy to learn. I'm not expecting—for those of you who don't already know D3, the 15 minutes of D3 overview that I gave you, I assume will make no sense whatsoever. It's just a starting point. **The learning curve is steep and that is the other problem**. So that is the one of the few occasions where we use another library, but that is always when we don't have a choice. So short answer: always D3, perhaps Raphael.

**Audience**: [60:12] So, I have a question regarding the weather map, the India map that you had. So was that even created in the 100 lines of JavaScript or HTML?

**Anand**: [60:21] Yes. See, the good part about it is, **while it's 100 lines of JavaScript, the SVG file behind it is more than a megabyte**. I don't know if that's cheating or not, but that's data.

**Audience**: [60:30] So you like imported an SVG file and directly did it? Okay.

**Anand**: [60:34] Yeah, yeah, absolutely. I mean, I wouldn't draw 100 lines of... you know, I wouldn't use 100 lines of JavaScript for a map, or at least not this detailed a map.

**Unsure**: [60:43] Anand, unfortunately, we're out of time. So I guess all questions will have to be offline from now on. Where's the next speaker? Is he set up? Thanks Anand, that was great.

**Anand**: [60:55] Thanks, everyone.
