Slashdot Mirror
More Than Half of GitHub Is Duplicate Code, Researchers Find (theregister.co.uk)
Richard Chirgwin, writing for The Register: Given that code sharing is a big part of the GitHub mission, it should come at no surprise that the platform stores a lot of duplicated code: 70 per cent, a study has found. An international team of eight researchers didn't set out to measure GitHub duplication. Their original aim was to try and define the "granularity" of copying -- that is, how much files changed between different clones -- but along the way, they turned up a "staggering rate of file-level duplication" that made them change direction. Presented at this year's OOPSLA (part of the late-October Association of Computing Machinery) SPLASH conference in Vancouver, the University of California at Irvine-led research found that out of 428 million files on GitHub, only 85 million are unique. Before readers say "so what?", the reason for this study was to improve other researchers' work. Anybody studying software using GitHub probably seeks random samples, and the authors of this study argued duplication needs to be taken into account.
You you don't don't say say.
Table-ized A.I.
It's good to have implemented a report of de-duplication of the files and directories in the cloud.
Other research has discovered that over half of Github comments are the thumbs-up emoji, and of the non-duplicate code 73% of it is horrible JavaScript.
Yeah, it can be rough to learn how to use Git submodules...
Honestly though, the few times I've directly integrated with someone else's code, it hasn't exactly been library-ready. There was a lot of massaging that had to be done the last time I did this, so a straight up duplication of their stuff was actually not a bad idea (AFTER I submitted them a PR to try and help manage this.) Their application wasn't designed as a library though, so I'm not sure what the right thing to do when you library-ify someone's code actually should be.
I don't understand how you can come to that conclusion. Forking under your own account is the most natural way of interacting with the code base.
Richard Chirgwin, writing for The Register:
I can't even imagine the number of times jquery is directly copied into a project
Richard Chirgwin, writing for The Register :
70% is a lot more than half. In this case the difference between half and 70% is a casual 129,000,000 duplicated files.
Kudos for not going in mega-clickbait mode, but still, "nearly 3/4 or more than 2/3" would be a better title.
lucm, indeed.
For the most part. There are many more crappy "me too" projects that were obviously knocked off from other well known projects for the sole purpose of cheating to make resumes look better. Really, this should surprise nobody given the level of cheating already going on in CS degree programs. If coding is what's needed to get a job in the future, you will always have people willing to fake it until they make it, at least until they finally get a real project and fail.
If half of the code is duplicate does that mean it is just a duplicate of the other half? If so then how would you know what the duplicate is and what the original is? Unless you count the duplicate code in with the original code in which case only one quarter of the code is a duplicate of the other quarter. Or maybe in my post thanksgiving carb haze I am over thinking this?
put all the code in there and link it to the associated github accounts, providing the code is 100% identical it should work, but they must consider forks and even one line of code in one file will make a lot of difference in the compiled software
Politics is Treachery, Religion is Brainwashing
I need library XYZ. I include the source in my commit. Alice needs library XYZ. She includes it in her commit. Bob needs library XYZ. He includes it... and so on.
Even if our hello world apps are different, the libraries to build them are identical.
Sure, lots of people will rely on separate installation of dependencies, but a lot of people won't.
70% doesn't surprise me.
Do they mean (obv. I didn't read TFA) code is duplicated in non-forked code, or are they just observing that lots of projects will be forked by other users in order that they can play with it and post their pull requests to them?
'cos if it's the latter, then that's kind of obvious isn't it?
I wonder how much is just people trying to avoid dependency hell?
Because let's face it, when I just want "that one bit" of some gargantuan framework / solve-all / codeball-from-hell then I'd rather spend five minutes of disentangling and integrating than a lifetime playing in "follow the library".
Wow.
No surprise here, this is how this stupid thing works: in order to submit a one-line bugfix, one have to fork the repository, patch, commit, pull request.
That's the hilarious part; duplicating code is also most of the purpose of github!!
Wetness detected in local river!
reused/recycled code. One would be stupid to event/develop everything from the very beginning yet again...
- haven't looked at the study though, no time..
And the only way to push a change back to a repository you don't control! You fork, push your change to your fork, then create a pull request. This is by design - I have no idea why this is in any way a surprise.
W..w..W - Willy Waterloo washes Warren Wiggins who is washing Waldo Woo.
I wonder if autotools was factored into this number as this gets pretty much copied to everything.
Makes sense... it's called a fork. Several of my projects are forked more times than they contain files..
Geeze, can't you guys stop copying each other?
Great minds think alike so perhaps helping speed up the journey might help. Coffee seems to work for me in both situations. :)
People don't care about analysing code properly, learning from it or even adequately adapting it to whatever other situation. In fact, I think that a big proportion of programming-related people aren't even able to analyse/understand random pieces of slightly complex code. There is an (ignorant) tendency towards ridiculously-specific specialisations and a systematic promotion of copy-pasting, absolute-truth-repetition and arbitrary, group-based assessments; and this is precisely why you see so many problems in software everywhere: many people with lots to say in the industry not doing it properly, not knowing how to do it properly and not even able to recognise who does (not) do it properly. Personally and after having been releasing my biggest open-source code so far during the last months, I will be notably reducing my activity on this front. It is very discouraging seeing how a so lost system misuses and misinterprets my work.
DISCLAIMER: I am the sole author of all my public code (including associated resources like comments, documentation, etc.), in the sense that I have developed it completely from scratch. Additionally, note that I release all of it as public domain and that's why I am not precisely concerned about random people using it or referring me. I am exclusively interested in knowledgeable programmers analysing it to get a good idea about my skills.
Custom Solvers 2.0 = Alvaro Carballo Garcia = varocarbas.
No matter what Perl looks to you (even if it is valid code written by your cat walking across the keyboard), not every random jumble of noise is valid code.
Yes, it is entirely possible that two files of size > sizeof(SHA1) (= 128 bits) will have the same hash.
But on the other hand, it's very likely that none of them is valid code, but gibberish.
Once you intersect both requirements (must share a hash and must be legit code) suddenly the probability drops a lot (because "must be code" is a very stringent criteria that drastically reduce the search space of possible files to a infinitesimal fraction).
At that point you're in "Shakespear-typing monkeys" territory. Yes, the probability is non-null. But at that point you're better off playing lottery until the collapse of the civilisation, you'd have better odds of winning.
As a matter of facts, "Shattered" the current known computed collision of SHA-1 is a pair random nonsensical blocks of gibberish. It can only be exploited in systems that can embed arbitrary blobs (attachments) and feature a turing-complete language (post-script) that can react upon the blob - PDF files.
"Sufficiently advanced satire is indistinguishable from reality." - [Tips: 1DrYakQDKCQ6y52z6QbnkxHXAocMZJE61o ]
It is a surprise because they explicitly excluded forks from their dataset...
So we have 1000's of round wheels, instead of a few squared and octagon ones.
Business logic can be all over the place, but basic functions and patterns tend to be closely tied to the natural designs of computing and/or logic
That's funny. In nearly every legacy system or codebase I've inherited I've found that at least 75% of code is clearly useless at minimum. I call it the 75% rule. It's always possible to rewrite it as something 25% the size while at the same time making it faster, less resource intense, more portable, with less dependencies, more secure, more flexible, bug free, more defensive, more maintainable, more auditable, more readable, etc.
It gets worse than that though. There's a lot of duplication that isn't easily detectable by automation. For example unrolling a loop wont show up on a duplicate line measurement. Often people put what would be variables as language constructs such as procedure_a, procedure_b where you should really have procedure(letter).
This study has the potential to be flawed though because of the way git works (deduplication). A lot of things can throw it off, there are several ways to look at a problem.
That's the hilarious part; duplicating code is also most of the purpose of github!!
Wetness detected in local river!
How about reading the point made in TFS?
The researchers did this study because Github is used as a source of data for identifying trends in computing. As they say, this duplication of code skews the results, and anyone wanting to draw serious conclusions from this data needs to account for this.
The important data isn't the headline, it's... well... the data. I'm hoping there will be less (virtual) printing of sensationalist "JavaScript is the best language in the world" headlines due to this prompting people to question the methodology.
Got them moderator blues I blieve I walk out the do', With these mod-points I been gettin', I 'most never post no mo'
I'm doing my bit to keep the stats up though, There are no 'duplicates' of any of my code ;-)
In an open-source project aimed at in-house usage - I don't want my "customer" to suffer denial of service just because a 3rd party neither of us controls (or the internet provider) went down.
I wonder what the proper procedure could be? Put it under /3rd-party? Add as a build-time dependency? Something else?..
WYSIWIG, but what you see might not be what you need
Where does it say that? I just re-read it and I'm pretty sure you made that up. There is a mention of copy and paste also contributing at the bottom, but that's it.
W..w..W - Willy Waterloo washes Warren Wiggins who is washing Waldo Woo.
I helped review the presentation and personally know the people that put together the paper, I can confirm they excluded forks
> You can clone/download all what you wish and enjoy it on your own machine, but why having publicly accessible codes which have been basically developed by other people
There are a couple major reasons to make your version of the project accessible on the internet. Maybe the most important is so that other people can see your pull requests. As an example, I used to do a lot of work on some software called Moodle, which is used by many schools. Moodle has a mature development process, so any changes to Moodle code are reviewed, commented on, and approved by at least two people other than the author. Typically three to five people comment on a pull request. It would be pretty hard for my peers to make suggestions about my proposed changes, or approve them, if they couldn't see them. Making the changes I propose available allows us all to work together - very much the spirit of open source.
Additionally, "enjoy on your own machine" brings up the question which of my machines? Primary desktop at work, where I type most the code? The development server where I test it? My laptop I use when I work at home? Having the source available on the internet is useful for the same reasons it's useful to be able to access your Gmail from anywhere, not just from one "local computer".
At my current job, our *company* has forks that our *team* works on before submitting a pull request upstream. Which local computer would you save our copy on that our whole team could see it and work with it?
Also, a few dozen schools use changes and additional modules I wrote which never made it into the official distribution. They aren't currently cross-platform enough for the project to include them because the main project runs on Mysql, MariaDB, Ms-SQL, Postgres, Oracle, and some others. It's still useful for my stuff to be accessible for those who want to use it. They'll just have to use either MySQL/Maria or MS SQL, or make their own adjustments to my code if they use Oracle.
To me the main reason is the first reason, though - it allows other people to see and comment on my change, review it, before the change is integrated into the official package everyone uses.
All the online (or most anyway) MOOC courses use github. A lot of assignments involve forking a repo and modifying one or two files.
Maybe the most important is so that other people can see your pull requests.
But this makes lots of sense. This is precisely the whole point of forking: actively and publicly contributing in others' code. It doesn't matter if the PR is accepted or not, you have already modified the original version. What doesn't make sense is forking something which you don't touch at all; perhaps temporarily and under very specific conditions, but not that being the general rule.
Additionally, "enjoy on your own machine" brings up the question which of my machines?
I meant this in case you weren't interested in modifying the original code (what the forks/PRs are for), but just in using it.
At my current job, our *company* has forks that our *team* works on before submitting a pull request upstream. Which local computer would you save our copy on that our whole team could see it and work with it?
Exactly the same than the previous scenarios: in the moment you perform whatever modification and save it in GitHub, the corresponding file stops being identical to the original one. It doesn't matter if you do a PR or not, that scenario shouldn't count as a duplicate anyway.
Custom Solvers 2.0 = Alvaro Carballo Garcia = varocarbas.
While the article about the paper may neglect to mention this fact, the paper itself (linked from the article) is quite explicit: "We skipped forked projects as forks contain a large amount of code from the original projects, retaining those would skew our findings."
With "I will be notably reducing my activity on this front", I logically meant my public-source activity. I have already lots of public codes which can help anyone interested in (and capable of) understanding my programming skills and working attitude. I will most likely continue having a quite relevant programming-related online activity, but will not be wasting my time in over-commenting and making codes everyone-friendly to be ignored or cluelessly misassessed by those only knowing how to count stars/lines of code and to run ready-to-be-used programs. In any case, I do look forward to the current ridiculously-bad-for-everyone situation to gradually change and to modify my behaviour accordingly. Under equivalent conditions, I will always prefer to share/show/give/contribute/help than to keep everything to myself.
Custom Solvers 2.0 = Alvaro Carballo Garcia = varocarbas.
> corresponding file stops being identical
Yep, the two or three or four files I change are no longer identical. The other 4,997 files in the project haven't changed, they are identical in both versions (forks). GitHub, presents my version of the *project*. It doesn't only show the differences and force users to download from someone else's fork, then apply my changes. They can just download my version of the project. (GitHub can also show the differences, if that's what someone wants to see.)
That does NOT mean GitHub physically stores all those different copies on disk. It just presents my version of the project, including files that are the same as someone else's version.
And as far as I am clarifying issues which should be evident to virtually anyone, also note that "my public-source activity" represents a net loss for me (potentially beneficial in the long term, logically). It is a mere self-promotion where I don't earn a penny; in fact, I lose a lot via time/effort investment and having to worry about addressing the most clueless concerns of random idiots (the sensible, knowledgeable people truly interested in properly understanding, learning, contributing, etc., on technical aspects or, eventually, to properly use my activity to determine my suitability for whatever project seem to be a minority!). The idea of me only earning money via being hired as a (remotely-working) programmer to work on whatever development (although being quite picky with clients/projects) seems particularly difficult to understand for some people, despite my multiple repetitions in many places (+ the evidence that a remotely-working programmer is usually mostly interested in working as a programmer). This is another aspect which puzzles/bothers/tires me a lot about a big part of the online programming (or whatever you want to call it) community: how are these people exactly working/getting money if I have to systematically explain them so evident issues?! Anyway, I guess that this is more than enough on the evident over-clarification front for today.
Custom Solvers 2.0 = Alvaro Carballo Garcia = varocarbas.
You did NOT read the paper, which states in many places that forks were excluded. Here is one such passage, "We skipped forked projects as forks
contain a large amount of code from the original projects, retaining those would skew our findings."
Yep, the two or three or four files I change are no longer identical. The other 4,997 files
This is explained in other comments in the thread: GitHub doesn't seem to internally care about the non-modified forked files (it only shows everything to the user) and, in any case, the counting methodology has to forcibly care about this issue, otherwise the proportion of duplicate files would be extremely high (easily over 99%) and not descriptive of the real usage of the platform. For example, I have a couple of forks to the public .NET repositories, each of them might contain millions of files and I only modified 2 or 3 (reasonably relevant modifications though); if you count all these files in my dupe counter, I would have over 99.99% duplicates just because of this what doesn't make sense.
Custom Solvers 2.0 = Alvaro Carballo Garcia = varocarbas.
That's like calling identical twins "duplicate twins" and saying we should drop half of them in any study of population genetics.
If two code files are the same, that's not just noise - a person made that happen for some purpose. It makes no difference whether you find that "bad" or "sloppy" - it's a legitimate part of the in-use population.
Now, that doesn't mean some studies shouldn't still drop them - for example, if I'm studying the *writing* of code, I might want a sample of unique stretches of code that were directly written, not just copied or forked. It just means we shouldn't presume we're improving the work of "other researchers" by casting all these files as useless filler (and I'm guessing most people who are smart enough to research code have already thought of this and are either accounting for it in some way).
Also this could affect the surveys of what programming languages are most used.
At worst the current surveys only shows in which language programmers do most copy-paste code.
I wonder how much of that duplicate code is copied from stackoverflow?
My own investigation shows that the majority of github code is made of only distinct 95 characters!
All the rest are dupes!!!
a lot of bots and stupid people use the fork button to bookmark or make themselves look legit and most forks go nowhere. so yeah.
Thanks for pointing that out, I had no idea that the word "wet" fails to describe the local river with the maximum known precision! Golly.
Thanks for pointing that out - I followed the link to the abstract and then downloaded the paper, and you are correct. The Register article is misleading... lesson learned.
W..w..W - Willy Waterloo washes Warren Wiggins who is washing Waldo Woo.
By design to make a one line change you fork a project, change that line, commit (for me usualy to a new branch) and send a pull request.
Now what surprises me is that 30% of files actually differ. Or is that just that a lot of forks didn't update to the latest version so what actually differs is the main project since it has moved on?
Óbh-óbh.
Look, this is more like pointing out that you're measuring the total length of the world's rivers wrong when you measure the source of the Rio Negro and the Rio Amazon from source to sea, because for a fair portion of that length, the Rio Negro is the Amazon. If hydrological researchers were making such a fundamental error, someone would have to point it out.
But code researchers were making a completely analogous error, and it needed quantified. And now it is.
Got them moderator blues I blieve I walk out the do', With these mod-points I been gettin', I 'most never post no mo'
It is kind of like that, except in your example there is one mistake that goes away when you apply the fix, and in the story, it is still really fuzzy and the remaining code might even still be mostly copied.
So it is like if you didn't have maps of the rivers, and didn't know which ones overlap, and so the data is complete crap, and then you find a fragmented map and now you know where some parts of a few of the rivers are. It is progress towards a good goal, but the data is still crap so far.
If you're asking WHY do folks fork and NOT modify, it's to "lock" a version, and to be able to build in an automated way. Granted, git supports this via checking out a specific commit, but for some reason a LOT of folks find it better to fork it, and then clone off that fork. The only advantage I can think of is it protects you from the original deleting the project altogether.
So imagine if you're developing a commercial software that uses LibraryA. You write it to how LibraryA looked when you pulled it and developed it. You want to automate your build so that you pull the dependencies from a URL. So you fork it, and then you just always pull from that fork. If at any point in time you're allocated resources to do updates, then can merge into that branch.
So that's why someone would fork and not change it. Another part to which maybe you're referring is to why "so much cold would remain unchanged." Well, a pull request could be a single line, it could touch 20 files out of a project of 5000. Most code is likely going to remain a copy in any fork scenario.
You want to automate your build so that you pull the dependencies from a URL
Curious! Inefficient and uncontrollable but the kind of thing which a big number people might do. I would never have thought about doing something like that myself; so, very helpful information, thanks! In fact, it kind of explains a weird issue which I have been seeing while streaming from the site of a major TV network in my country for some months (I think that it isn't there anymore). There were always problems/delays while connecting for the first time and, after that, regular pauses and reduction of quality. I started noticing that when that happened the given application was connecting to GitHub (you know in the lower part of the browser where you can see regular connections to ad providers and similar)! And I found it extremely crappy! Why not having your own (ideally local) copy! Or, at least, connecting to a site precisely meant to perform these actions, which isn't the case with GitHub! I even visited that repository and it was a library meant to simplify the implementation of streaming services, but it was quite small file!
Custom Solvers 2.0 = Alvaro Carballo Garcia = varocarbas.
Lots of projects use "generic build tools" and unfortunately, this may be the easiest and safest way to integrate AND get the project that's under-budgeted by months out the door.
I personally don't have too much experience on this specific front, but I guess that, under very specific conditions, there is no harm in building from a source code in GitHub. On the other hand, doing something like what I described in my previous comment and letting an in-production streaming application systematically communicating with GitHub to get a small file seems gross incompetence; to not mention the fact that streaming is precisely closely related to the core business of that company. It is incompetence of the person who takes a ready-to-be-used code without properly understanding/debugging/adapting it; also of those who originally developed that code, for not having setup a better/more efficient alternative (and/or clear warnings/instructions); it is even incompetence of the managers mishiring/mismotivating/mispaying and pushing beyond what is logical to meet ridiculous milestones; even the tolerance of the viewers, accepting problems and errors as normal, might be partially blamed. All this seems wrong for many reasons, easily improvable and very difficult to be justified. At least, for me, for my expectations and the kind work I do/conditions I accept.
Custom Solvers 2.0 = Alvaro Carballo Garcia = varocarbas.