Slashdot Mirror
Linux Is the Largest Software Development Project On the Planet: Greg K-H (cio.com)
sfcrazy writes: Greg Kroah-Hartmant, the Linux superstar, delivered a keynote at CoreOS Fest where he gave some impressive details on how massive is the Linux project. Kroah-Hartman said the latest release (4.5) made two months ago contains over 21 million lines of code. More impressive than the amount of code, and what truly makes Linux the world's largest software project is the fact that last year around 4,000 developers and at least 440 different companies that contributed to the kernel. Kroah-Hartman said, "It's the largest software development project ever, in the history of computing -- by the number of people using it, developing it, and now using it, and the number of companies involved. It's a huge number of people."
"Greg Kroah-Hartmant, the Linux superstar,"
Uh, what? There are only two superstars in Linux: Linus and the guy who came up with systemd.
"by the number of people using it, developing it, and now using it" ...I was using it before BUT now, after developing it, I'm *really using it now... :)
The VMS operating system was estimated to contain over 25 million lines of code, and that was measured over 10 years ago - I'm sure it's quite a bit more by now.
There is no God, and Dirac is his prophet.
That is because unit tests are useless and so is most QA/QC.
In your first sentence you demonstrate your total and complete ignorance of the subject matter at hand.
Let me guess, you think Agile development improves quality too.
No, it improves the productivity of shitty prima donna developers who think they can spend 8 years writing 4 lines of code to get it just right. It allows middle managers to reel in asshole developers who think their shit doesn't stink while actually producing nothing of use to the business.
Its a really shitty development pattern for a good developer, but 99.99999% of developers are shitty and Agile is far better than free roam for them. Your average run of the mill developer in most software shops is actually less useful and efficient than most janitors.
For every Linus there are 7 billion people who aren't.
By the time you have defined all your "unit tests" we have already built the entire system and have it running in the real world.
Of course mine will work and do what its supposed to do, yours will be buggy, not meet the design requirements, not function as expected in random ways.
You are the definition of ignorant on this matter. You really should just keep your opinion to yourself because the only people who are going to agree with you are going to be the same people in the unemployment line with you.
Software development is an engineering practice when done right, which you clearly have no fucking clue how real engineers work. Fortunately people like you will never be allowed to do anything that actually matters like build bridges or buildings, and your ignorance and arrogance will prevent you from even understanding why you'll never be able to do it.
So congratulations there snarky ... you've shown us you're nothing more than a completely ignorant, smug asshole who has no fucking clue when to shut his stupid mouth :)
Persistent Volume manager for Kubernetes - https://github.com/dwimsey/openshift-pvmanager
That is because unit tests are useless and so is most QA/QC. Let me guess, you think Agile development improves quality too. By the time you have defined all your "unit tests" we have already built the entire system and have it running in the real world.
Only if you think nine 5's is acceptable reliability.
Some of us have higher standards than that.
Because when you build systems of systems, it doesn't take many constituent systems built with the attitude that "unit tests are useless and so is most QA/QC" to turn the entire assemblage into a steaming pile of unreliable shit.
But hey, if you like producing steaming piles of unreliable shit, go right ahead.
Just get of your damn high horse - because it's stinking up everything with horse shit.
That bit about shitty programmers...that's gold. But only a corporation would have to resort to such methods to rein in developers. In Linux land, if you fu is weak then your code goes nowhere (hopefully, at least).
PS: I don't reply to ACs.
Oh look: another butthurt QA person who is jealous because he doesn't make as much money as a prima donna developer! I've been writing software for decades that you are currently using every day. Guess how many units tests Linus has written? Zero.
The VMS operating system was estimated to contain over 25 million lines of code, and that was measured over 10 years ago - I'm sure it's quite a bit more by now.
This is just the kernel. But most of it is arguably "not" kernel code... it's drivers. This is directly addressed in TFA:
All versions of VMS and OpenVMS together come nowhere near to running on as many different hardware platforms as Linux, so it would be shocking if Linux's drivers weren't massive in comparison.
"You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"
Shitty programmers are shitty programmers. No amount of unit tests and Agile development is going to fix that.
I'd say having a university programming project become one of the biggest operating systems in history, and with the vast number of contributors and collaborators, all for a project that you can freely download, yeah, that's a pretty impressive badge of honor.
If Linus died tomorrow, he'd be viewed as one of the pantheon of great computer innovators, not so much because he produced anything in and of itself unique, but rather because he transformed the *nix ecosystem, and lead to greater penetration than I suspect Unix's original creators could ever have imagined.
The world's burning. Moped Jesus spotted on I50. Details at 11.
This is not entirely true when the software in question is being directly converted into hardware. This happens for VHDL, Verilog, and SystemVerilog. People call these hardware design languages, but the reality is that they are pure software. Other tools (Synopsis Design Compiler for example) turn this high-level software into low level netlists which in turn are used to produce silicon. A mistake in the high level software can cost millions of dollars to fix and spin a new piece of silicon.
The definition of 'unit', however, is not necessarily always agreed upon. What is the 'smallest possible' unit that can be tested? I would argue that it is the smallest possible unit that makes sense to test. For example, a PCI-Express root-port has at least one very well defined interface (the actual PCIe bus), but often also has very well defined internal-interfaces for how the root-port 'unit' plugs into the rest of the chip. Since all interfaces are very well defined, I can write unit level tests to fully exercise and prove that the root-port works correctly, and there is very little need to write full-chip tests which would end up taking days to run vs the hour or two it takes to run against just the unit.
The full-system tests are much harder to develop as they require writing assembler to get the CPUs to send traffic down to PCIe. But instead I can directly send write/read commands which is much simpler test case to develop. IT also gives me finer grain control of my test and the timings, as I don't have nonsense like cache-hits, misses, etc getting in the way and perturbing my stimulus timings.
The logic portion of developing hardware is almost entirely a software problem these days, with the exception that this software has to meet some real world constraints of physical setup and hold timings. But software for real-time embedded systems have similar timing requirements that must be met, so it's not a completely foreign concept to software development.
Unit testing is critical to certain sectors of software development, and large corporations spend hundreds of millions of dollars a year doing unit testing because in the long run it saves a ton of money and time.
What you think is the "true" reality is really a Docker container running in a VM in the Cloud.
Unit testing of software is a great way to prove that a given software routine performs the wrong function with perfection.
Google is not a software development project.
I nearly fell out of my chair when I read your LKML post claiming that the 2.2.x/2.4.x kernels were more stable than they are now. I'm guessing you either weren't involved with Linux back then or have a short memory.
Here is a short history lesson:
The development ended up dragging out to the point where the base kernel features needed for new devices not existing.. so you could use the stable kernel which didn't support much, or you could try the development kernel which tended to be wildly unstable or you could use the disto kernels (RedHat etc) which attempted to patch drivers from the unstable branch into the stable branch resulting in something almost as unstable as the development branch.
I still have nightmares of the day someone presented me with a brand new IBM server in which the Old kernel couldn't see the disks, the dev kernel crashed on boot, and the distro kernel would crash several minutes after starting. (I ended up going with a hand built custom kernel + patches).
It doesn't matter how many times you whine on slashdot or troll the kernel list. No one who remembers the old stable/unstable branches wants to go back to that nightmare.
Also FYI: There are several projects that QA the Linux kernel testing branches and report the results back to the kernel devs.
You have a way over optimistic view of Agile as generally implemented.
I contribute to a project that has CI going to run unit tests on every commit. Those unit tests take hours to complete. I have not seen those unit tests catch a single bug (other than style formatting issues, which are non-functional). The human testers have. The problem being that people equate hours of test as the sole metric of quality. The quantity of test does not by itself assure quality of test. The other facet of this is the phrase '100% coverage', which I've seen *many* project managers jump up and down all excited. I have seen QA teams *fired* at that milestone, because it's a misleading phrase. When management sees '100% automated test coverage' they think 'Human QA is no longer needed'. They don't understand those unit tests are just as likely to have problems, and almost certainly will not be all-encompassing (in fact unit tests tend to hit the corner cases that the developer *recognized* would be tricky, and had those edge cases in mind as they wrote the code anyway)..
My problem is less that unit tests are a waste of time or that they can't be good, it's that too many advocates of the process oversell the benefit and lead to poor decisions with respect to quality based on overconfidence in the process. The conundrum is that in practice, either you oversell it and cause these bad consequences, or you try to describe it accurately and project management will express frustration on 'wasted effort' in writing unit tests since they can't really replace human testing.
Incidentally, the sentiment that the project burdened by unit tests will be beaten to market is a likely consequence. It's not necessarily a *good* thing, but if 'good enough' hits the market before 'truly good', most will ignore 'truly good'.
XML is like violence. If it doesn't solve the problem, use more.
Taking the numbers at face value you get the following stats:
- with 4000 developers
- 2.7 lines of code added per day per developer
- 1.3 lines of code removed per day per developer
- 0.47 lines of code changed per day per developer
In electrical engineering circles we call these two aspects of testing 'verification' and 'validation'. Verification is verifying that the code matches the specification. Validation is making sure that the specification is really what we want in the first place.
Fortunately, us real engineers demand complete specifications, and when we find mistakes in those specifications we make sure they get fixed. Doing things in your haphazard land of writing code by the seat of your pants must be stressful at times.
...because they still don't support Linux with their official logins, tax, banking etc. services.
In fact - BankID which is the most used login method just came out and said they'll drop Linux (which they did) support as the userbase was too small.
I also use Telia ID card login - with a card reader, even this is "hackishly" supported (officially unsupported, but I've gotten it to sorta kinda work), so people with Linux can't officially even register at the unemployment offices unless they have windows or a smartphone.
But hey, hopefully we're heading in the right direction - I've been a registered Linux user since 1998, and now exclusively use Linux for my Desktop. Every time I go to work and use Windows - I'm constantly reminded of an inferior system with endless updates, endless disk-trashing and endless limitation as a user.
What this world is coming to - is for you and me to decide.
I have never, and will never, work for the government. Real engineers use specifications and test against them.
I'm just joking. I'm sure you are a better engineer than I am.
The problem being is that many managers *think* that some magical process can allow shitty programmers to do great work, and then lay off the great programmers to get cheap-o developers because Agile means quality, no matter how bad your workforce is.
Of course this isn't how most of the well-liked software is done, but it's done *all the damn time* in enterprise software land. Leading to a paradox of free and cheap 'consumer' software generally being a *ton* better than equivalent enterprise software packages which generally cost a whole lot more (when equivalents exist).
XML is like violence. If it doesn't solve the problem, use more.
Software development is an engineering practice when done right, which you clearly have no fucking clue how real engineers work. Fortunately people like you will never be allowed to do anything that actually matters like build bridges or buildings
I just had to laugh at this one. You praise Agile but it is everything but the way you'd build a bridge or building. You absolutely don't create a building one room at a time, you have architects and construction engineers and blueprints all ready to go before you start implementing. If you did it the Agile way you'd rewire the house ten times over before you're done. And you don't in the middle of construction find out you want to add another floor and an upstairs bathroom, not unless you want a huge replanning. Agile is more like Star Trek, full power to shields/weapons/engines/life support and things magically rewire themselves to serve the most pressing business need.
To be honest, I feel like waterfall overplan and overthinks things, Agile underplans and underthinks things. I like waterfall projects that act "agilish", small initial scope and iterative releases, clear deliverables, priorities and rapid prototyping. Or Agile projects that act "waterfallish", someone with a clear long term vision of what the system eventually will look like and how the components we build will scale and fit the big picture. Bad waterfall is just mental masturbation over plans that'll never work in reality. Bad Agile is just timeboxed cowboy coding, making up shit as you go along. The problem is some project managers think you can make uncertainty go away by sitting around a table and discussing it further. Some things you just won't know until you've tried and seen what kind of progress you actually make.
Live today, because you never know what tomorrow brings
I notice you didn't include things like servers, routers, automobiles, industrial machinery, televisions (esp. the "smart" TVs), all the IoT shit (e.g. thermostats), avionics systems, and etc.
The embedded stuff outnumbers smartphones and 'PCs' by at least a factor or two... and nearly 50-60% of those run some embedded variant of Linux... embedded Windows ekes out maybe a sliver of that market.
Quo usque tandem abutere, Nimbus, patientia nostra?
You're making one big mistake, and that's equating software development with construction. It's not. Software has achieved automation of construction of final artifacts based upon detailed specs: it's called compilation. What programmers do is much closer to creating specs than to construction according to specs. People writing recipes are not cooking.
I think you confuse "the art of programming" with "the craft of programming." If it were the former, you'd be allowed to put the noses on sideways, like Picasso, and call it "art". But as a craft, you need to produce the best work possible, even though it certainly contains an element of art.
I work on safety-critical software. We are required to do rigorous unit testing on every line of code by The Authorities. And I regularly find bugs in my code in the process. In fact, since I will be developing unit tests anyway, I (try to) make it a practice to do develop a module and its test simultaneously, which is more efficient than doing the unit test at the end, as is the common instinct by those who regard unit tests as "useless."
I also think you confuse "useless" with "efficient." Depending on the use of the software, unit tests may be an inefficient use of development time - as the Linux folks evidently believe. So, I don't do unit tests for every category of software. For example, there's no need to do that for a simple utility script. But I wouldn't want to fly in an airplane whose autopilot code was never unit-tested, and whose developers instead simply assumed that all the code they write is bug-free.
Of course mine will work and do what its supposed to do, yours will be buggy, not meet the design requirements, not function as expected in random ways.
Exactly true. I've found over many years of engineering that there is a vast difference between something that nominally works and something that works perfectly under all conditions. Once you've achieved the former, you've done perhaps 1/3 or less of the work required to get to the latter.
You're making one big mistake, and that's equating software development with construction. It's not. Software has achieved automation of construction of final artifacts based upon detailed specs: it's called compilation. What programmers do is much closer to creating specs than to construction according to specs. People writing recipes are not cooking.
No. In the waterfall model, first you write the spec then you implement the spec. The developer breaks down the spec to code, the same way the compiler breaks down code to machine instructions. Implementing the spec is not supposed to be a creative process, in fact figuring it out everything you're going to do first so you know how to best implement it and don't go down dead ends and waste time rewriting, refactoring or switching tools/libraries/architecture is pretty much the cornerstone of waterfall methodology. Then you write it once and test that everything was done according to the spec, at least that's the theory.
In fact cooking is a pretty good analogy, say the customer wants to have a pizza. The requirements are that it be tasty, crunchy and spicy. So you make a recipe for a pizza. Then you create a big and expensive industrial robot to make a pizza exactly like that. And you test it extensively, but not in terms of the requirement but that you added exactly the right ingredients, in the right shape and amount placed in perfect order and arrangement, that it's baked for exactly the right duration at the right temperature. And after ensuring everything is exactly to spec, you hand it to the customer and he goes "Yuck! That's not what I wanted"
Live today, because you never know what tomorrow brings
Wow. It took you many years to figure that out? I'm pretty sure everyone in the industry knows that the last 5% of the project takes 90% of the time.
Maybe I'm not as smart as you if it took me longer years to figure it out. Even so, based on our figures, it sounds like it takes you about three times as long to get something working well after the point that it nominally works. That might explained why you noticed this effect almost immediately. ;-)
I've been writing software for decades
I'm the new RPGLE programmer in an AS400 shop and I'm surrounded by developers like you. Long story short, the 20+ year old code base is crap, none of the neck breads know what the hell is going on in the system and then blame it on bad data. Everything in the system from a user's perspective is confusing. And after a few years of using the system these folks wrote, our clients scream for SAP or RedPrairie. And these folks keep telling themselves, "They don't know what they're missing! We can roll out changes in days not months! They'll be sorry!" All the while I sit there and see our company bleeding money out because people like you think you can do no wrong and are artist and the code you write is somehow magical.
I'm not saying unit test and agile development fix bad programmers, but what I am saying is the free roam model of programming leads to a lot of crap and loss of money. And the self righteous nature means they'll never understand why our clients go elsewhere and never return. It just is so frustrating to work with people like you because you all believe that you can do no wrong. People want software that works. Sometimes that means unit testing, sometimes it doesn't, but if just go around yelling, code is art and the users exist only to ruin it! You're just blowing your company's money out the window. Just bloody hell, and you wonder why they keep outsourcing? Well just look in the mirror one day, there you go. Please never work within a 500 mile radius of Atlanta.
I'll take my troll mod now. For this person it was worth it. And trust me bud, I know that everything I've said has fallen on deaf ears.