# Transcript - Social Code Analysis I'm going to be talking about social code analysis. And what that means will become obvious as we go along. At IIT Madras, I teach a course called Tools in Data Science. And one of the projects that I assigned the students was, you've got to write a Python program that automates data analysis. I'm going to give it a data set. It has to use LLMs to analyze that data set and create a story all by itself. The evaluation and the feedback were also LLM-based. We had a set of 27 checks. For instance, item number 17 was, the code should only send carefully chosen analysis to the LLMs, not the whole thing. And the evaluation was done by an LLM, which in some cases told the student, "Yeah, look, you're only sending selective analysis, that's good," or "No, you're sending all of it, that's not particularly good," etc. But this wasn't the fun part for me. What was more interesting was another experiment that I was trying, which is how exactly does copying work amongst students? So, in this course, copying is not only allowed but encouraged. The project specifically says, you get eight bonus marks for code diversity. For your base marks, you're welcome to copy. But over and above that, if you produce something that's unique, we'll give you bonus marks. So effectively saying, yeah, it's good to create original stuff, but please feel free to work together as a group. If you want to freeload, that's perfectly fine. That's all right. And what I did was evaluated the code similarity between students. Specifically, the way this was done was, take all the submissions, remove all the fluff—the comments, the docstrings—and tokenize it. Take chunks of five tokens and see how much they overlap. And based on this, what we found was, here's what the similarity looks like. So each of these is one student's submission. And at a 100% threshold, here are a bunch of students whose submissions were identical. Zero difference. And here's another bunch whose submissions were identical. Here's another bunch whose submissions were identical, and so on. The color indicates something that I will explain later, but the size represents the marks that they finally scored. So some clusters are bigger because the students in that cluster scored higher. Some clusters are smaller. Now, even though students have copied responses identically, they don't necessarily get the same marks because the LLM is evaluating and there's a certain amount of randomness in that evaluation. How we manage that is a different story. But the thing is, across this... by the way, what I also got to see was at different levels of similarity. So that's at 100% similarity. But if I look at it, let's say at around maybe 80% similarity or 70% similarity, a larger cluster forms. So here are a group of students who've kind of copied from each other, but it's not identical copies. It's almost like there are three clusters of students who've probably gotten the same idea. I'm not sure if it's independent or if it's emergent, but there certainly seems some commonality across those solutions, and we get to see what the nature of that commonality is. Of course, at some level, there are huge differences, but we'll go to that in a bit. Now, one of the things that I saw was the way in which the timing of copying worked. So for instance, there was one student who submitted the original, and I know this is the original because it was the earliest submission, on December 11th, and they scored seven marks. The first copy that was made of this submission was a day later. The next day, in the evening, on 12th December evening. The second copy was made two days later in the night. And then there was a spate of copies that happened right before the deadline. Massive. On another cluster, somebody put in an original late night on 12th December, and then on 14th December afternoon, the first copy arrived. And then there were several copies just minutes or hours before the deadline. Now, the interesting thing is that the students who copied from the first one, which was submitted on 11th December, on average they scored about seven marks. The ones that submitted from the second cluster on average scored about 10 marks. Now, the second cluster was significantly better than the second original was significantly better than the first original, partly because it was a loose copy of the first original. By looking at the code, it was obvious that the second original student had... well, I call them original, but it's semi-original, but close enough... had taken enough good elements from the first, incorporated it into theirs, and then put it in. Which means that all of the students who copied from this late submission benefited from the wisdom of the crowds. So, a good rule for students is, if you're planning to copy, copy late. It's a good strategy. Yeah, exactly. Exactly. And the amount of trouble that I have teaching people who enter the workforce. Look, it's all right to copy. I just want you to get the job done. I don't want you to produce original stuff. Just get it done. Takes some time and effort. But now, some students make sensible changes. So here was one on the left, there's the original. This student added Python .env and a few other related minor changes to the original code so that the API key is taken from the environment. Makes perfect sense. Another student made a similar change. They took this line and replaced the actual API key from their environment, effectively leaking their own API key. Which... I mean, why would you do that? Among other things, that lets every other student use and exhaust your API key limit. So, the second rule is, if you're changing code when you're copying, make sure you know what you're doing when you're changing it. Try not to mess it up. Otherwise, just submit the same thing. It's fine to get points for originality, but you lose points for something like this. What I also realized was that roughly half the students put in original submissions. That is, I have a measure of similarity, which is let's say about 50% Jaccard similarity. That is, if I see pieces of code that look like what you see on the left versus the right. Now, the left says, "You are a data analyst." The right also begins with, "You are a data analyst." The left says, "Given the following data set information, provide an analysis plan." The second says, "Provide a detailed narrative based on the following data." Yeah, kind of similar. But this is the threshold of what I consider distinct submissions. Anything closer, it becomes a copy in my crude definition. At this level of a cutoff, what we find is that half the students neither copied nor allowed anyone else to copy from them, despite them being clearly told that copying is allowed, you're welcome to work in groups, collaborate, do whatever you want. And yet, half of them choose to work in isolation. Now, what this implies, however, is that they have a disadvantage. I had students broken into four groups, and that's what the colors represented. The stand-alones. They didn't copy. They didn't allow anyone else to copy. And on average, their score was about 6.23. The lowest performing. Those who did slightly better were the ones who copied early. The first ones to copy as soon as someone publishes a result. And this, we know, is not a great strategy because if you wait till late, you get a better pool of submissions. And therefore, those who copied late did a little better. But the people that scored the highest were the originals, meaning people who put in something by themselves, but more importantly, others had copied from them. That's where they had gotten, I'm guessing, the maximum benefit because somebody or the other would have said, "Oh, this didn't work for me," or "That, maybe you could have done it this way." And that feedback really helps them improve. So from my perspective, I see this as a win in two ways. At the very least, if you're really tired, if you've got a hundred other things to do and my course is just one among a dozen, it's all right. Copy. It's better than nothing. On the other hand, if you are putting in some good work, then you can get a bigger boost by letting others copy from you and learning from each other. So, those were the lessons for students. But as an educator, there were a few other things that I learned. First is that there's a lot of stuff that I can learn by analyzing similarity of code. I can figure out who are the students who are isolated, and therefore I probably need to connect them together. Who are the students who are future teaching assistants? And I do need a bunch of teaching assistants, and this is a pool that I can pick from. When do they learn? Or how close to the assignment deadline do different people learn? The curve helps. And as you said, scale is a nightmare, right? I love scale. Because what that allows me to do is run large-scale experiments like this, and we get a beautiful curve of the distribution. We get to see what are the concepts that they're tweaking and messing up. What are the concepts that they are copying and doing it right? So a slightly better understanding of where the learning was effective and where the learning was not so effective. **Question**: My question is why choose that particular method? Why choose that particular method? Similarity? I tried three techniques. In fact, the first, which I announced in the project, was I would use embedding similarity. It did not work well because chunks of code that were... so if somebody had taken roughly the same kinds of words but put them around in very different ways, the sequencing information was lost, which Jaccard similarity preserved. Jaccard similarity is also more explainable. I'm just saying I'm taking phrases of five words and overlapping. Explainability becomes important with difficult students who come back and say, "No, no, no, you should have given me more marks, I didn't really copy from this student." That sort of thing. So, in short: sequencing preservation and explainability. **Question**: Could you explain Jaccard similarity a bit more? Sure, it's looking at... so you will be able to see the code, but this is what I did. I took the code without any of the comments or docstrings, and then tokenized it, meaning converted the Python code into words. If it had an open bracket, open quotes, if you have a function call, all of that gets converted to words. So now it's roughly English. And then I took all phrases which are five words. Take the first five words, the next overlapping five words, next five words, and so on. And these are chunks. Then I take all of the chunks in Program A, all of the chunks in Program B, and see roughly what percentage of overlap there is. That's the Jaccard similarity. **Question**: Are students allowed to see each other's code and modify it? Yes, so students can do whatever they want until the deadline. Actually they can do whatever they want even after the deadline. I just take a frozen copy of it. **Question**: So they can also use Generative AI tools? Yeah. So in this course it's allowed. They can also pay sombeody to take the course on their behalf. If they're able to cheat well here, then maybe they're able to do that when they join the workforce. So, that I don't mind. I don't mind at all. **Question**: The ones who allowed others to copy from them, are they allowed to make changes? Yes, so they are allowed to make changes right up to the point where I'm taking a copy, which is the assignment deadline. So if any of them had submitted, so to speak, three or four days before the deadline, and had possibly even shared amongst each other well before that and yes, there were commits after they had put in the initial submission. They just put it into their Git repository. All I want is the link to the Git repository, **Question**: Could you share the link to the repository? It's the Tools in Data Science cource. The link is .