Report: Aging Java Components To Blame For Massively Buggy Open-Source Software

itwbennett writes: The problem isn't new, but a report released Tuesday by Sonatype, the company that manages one of the largest repositories of open-source Java components, sheds some light on poor inventory practices that are all-too-common in software development. To wit: 'Sonatype has determined that over 6 percent of the download requests from the Central Repository in 2014 were for component versions that included known vulnerabilities and the company's review of over 1,500 applications showed that by the time they were developed and released each of them had an average of 24 severe or critical flaws inherited from their components.'

Comments

The root cause : poor unit testing

[Dr_Barnowl]

Why?

Because if you don't test your code, you don't know if changes to it break it.

Changing the components your code is composed of is a big change.

Therefore : people get nervous about changing the components they have used (even changing the version).

What should be happening : when you're planning a new release, raise the component versions to the latest and run your test suite. If it passes, good job, release it.

What is actually happening : the version numbers never get edited, because that version worked, and if you change it, OMG, it might stop working.

Re:The root cause : poor unit testing

[david.emery]

Testing is no cure for bad design or bad coding, which are the -root cause-. The specific design and code techniques to prevent vulnerabilities need to be better communicated and enforced (by open source code reviewers, as well as commercial developers).

That's not to argue testing is unimportant. But it's not the root cause of vulnerabilities, and it's not clear to me that we know how to test for a lot of vulnerabilities.

Re:The root cause : poor unit testing

[Dr_Barnowl]

Oh, yes, testing doesn't fix bad design. But it helps to avoid the problem mentioned - which is that projects use versions of components with known problems that are known - and thus have been fixed in newer versions.

Re:The root cause : poor unit testing

[DickBreath]

The root cause: poor management (in most cases)

The root cause is not poor unit testing. Not bad developers. It is managers who won't allow the change to be made. It ultimately will always come down to money. They are unwilling to spend on having a reasonable staging environment that closely mimics the production system such that making these changes could be done safely and receive proper testing. And people to do that work also cost money.

In short: management doesn't care, due to money. So the product can just self-destruct. (like SourceForge)

Re:The root cause : poor unit testing

[Yaztromo]

What should be happening : when you're planning a new release, raise the component versions to the latest and run your test suite. If it passes, good job, release it.

What is actually happening : the version numbers never get edited, because that version worked, and if you change it, OMG, it might stop working.

Part of the problem I run into with this is that sometimes projects stick with old dependencies because at some point, some major version came along that significantly changed the organization of the API in such a way that the latest component version an't just be dropped in, but requires significant resources refactoring your code to use it. Getting management buy-in for that when there aren't any big customers breathing down their neck to get a flaw fixed can be neigh on impossible.

I ran into this recently myself. During internal testing, I discovered a flaw in our product when accessing any of our web resources using an IPv6 destination IP in the URL (i.e.: http://18080./ A quick bit of debugging showed that an external library we had been using for several years was doing some brain-dead parsing of the URL to pull out the port number; it was just doing a string split after the first colon it found, and presumed the rest was the port number.

Modifying the Maven POM to use a newer version of the API in question was initially difficult because the project had since reorganized their own library structure, breaking things into multiple smaller JARs. Except that some of the functionality was actually _removed_, and isn't available at the latest API revision (functionality we had been using, naturally). Classes had moved around to different packages than where they were previously, and various interfaces appear to have been completely rewritten.

Upgrading to a version of the library that actually fixed the flaw was going to be akin to opening Pandora's Box. Unfortunately, our former architect (from whom I inherited this code) was the type of guy who just liked to throw external libraries at every problem. In the end we had to document the fault for all current versions of the product, and now I'm trying to get management buy-in to do the work necessary to upgrade the library in question for the next version of our product. And this is for just one library out of over 100 that need similar attention.

Suffice to say, I'm not happy about this state of affairs. Unlike the previous architect, I push against using third-party libraries as our solution to everything. If I were allowed to rewrite everything from scratch, we could avoid these problems. Things are unfortunately messy out here in the real world, and when libraries decide to significantly change their interfaces your program uses to access their functionality, no amount of unit tests is going to make upgrading those libraries any easier.

Yaz

Re:The root cause : poor unit testing

[lgw]

These are flawed Java components, not complete systems. What kind of component-level testing is generally useful for avoiding security issues? Most of the issues I've seen have either been from each component assuming the other was checking for something, or from anti-patterns like depending on "string cleaning" to avoid injection attacks - implementation choices that are bad practice, but have no flaw you can point to at the time the component is written.

Plus in general insecure code tends to be the result of implementation choices, where the component would be fine with any normal unit test of its interface, but is vulnerable to very specifically crafted malformed input.

For more complete systems, fuzz-testing and a variety of common injection attacks are great, but that makes little sense to me at the component level.

Re:The root cause : poor unit testing

[RabidReindeer]

This is somewhat deceptive. Sonatype supports Maven component archives.

One of Maven's chief claims to fame is that when you build a project, it doesn't grab "the latest" versions of dependencies, it grabs the selected versions of dependencies. On the grounds that "If it ain't broke, don't fix it".

This ensures a predictable product because everyone who does a build, no matter when, no matter where, will be pulling in the same resources to build with.

The problem arises when one (or more) of those selected component versions turns out to have issues. The build ensures that the product will be consistent, and thus will pass its own tests, but as the old observation goes, testing cannot prove the absence of bugs, only their presence. So if there was a vulnerability, an old project's tests wouldn't see it. And because you're asking for a specific library release version, later fixes don't get automatically included (of course, neither do later breakages, but they ignored that aspect).

In theory, then, this is simple to fix. Just update the project (POM) to pull in newer, better dependencies.

And the NEXT version of Windows will fix all your problems, and I've got a very nice bridge in NYC for sale cheap.

If you're working on a project, you generally have all you can do to keep up with issues in your own code, let alone some supposedly trustworthy third-party libraries. You cannot afford to be constantly updating the dependency versions and even if you could, there's the issue of "dependency Hell", where changing the version of Hibernate can conflict with the version of slf4j which can conflict with junit, which can conflict with... I usually like to budget 2 or 3 DAYS when I'm ready to start upgrading dependencies.

Sonatype doesn't get a pass here, though. If they/Maven supported a mechanism that could flag builds that have known weak dependencies, it would help a lot. Management, of course, would promptly command it to be turned off to ensure "productivity", but at least we'd have some help short of periodically manually auditing every library in a complex project (like that's ever going to happen).

Re:The root cause : poor unit testing

[UnknownSoldier]

Hit the nail on the head with the problems of external libs.

> Unlike the previous architect, I push against using third-party libraries as our solution to everything.

You're not alone, my boss and I are of the same mind set, and I've noticed the same pattern amongst coders:

* The more better programmers minimize the amount of external libs.

* The inexperienced / junior programmers are so gung-ho to include every library under the sun that it is almost dizzying. And then they complain why their project takes 5 minutes to start up or so slow to navigate into their IDE (because code completion is usually to blame.)

If a problem _requires_ a library then, sure, we'll use that, but we would rather write almost everything ourselves so we fully understand the context of the problem.

Re:The root cause : poor unit testing

[Yaztromo]

Wait, what?

If you write code, part of the documentation before you start should be a "risks" statement, where you state that a dependency on X external, third party library, exists, and that any vulnerability could cause issues in your application. Also, that substantial upgrades to the library interface will affect maintainability if any interfaces are changed, or are deprecated.

When someone throws a pile of libraries at a problem, that risk statement gets lengthy.

Which is all well and good if you're doing greenfield development. It's not so good when you're inherited a codebase where none of this was done in the first place, and you're tasked to keep it going. As I said, the real world can be a messy place. In my case, the previous lead architect just threw immature libraries at every problem willy-nilly, at a time before I worked for the company. I get to inherit the problems this lack of foresight caused, and don't have the benefit of going backwards to fix it.

Your problem isn't with using external libraries, it's using ones without service contracts, or immature ones. And you're reacting by throwing out the baby, bathwater, bathtub, house, plumbing infrastructure, and electrical grid.

I don't disagree -- I'm hardly anti-library (you won't get too far in development without them, unless you're doing low-level embedded work). I like a good, stable, well-designed library that has been around for a long time, and where breaking changes are rare. Unfortunately, when the previous architect would throw a library at every problem that may have required eight lines of code, without care for anything other than ensuring the library license permits us to ship it.

And FWIW, I have brought these issues up with management. Their stance is they don't want to see any backend changes of any sort (where the bulk of our library woes lie) -- their focus is on the front end only. They're more than happy to defer the risks to some unspecified future.

As I said, the real world of development can be messy. I'm stuck with a codebase that is full of immature libraries, libraries that are no longer maintained, and libraries with bugs where getting new libraries introduces major changes requiring major refactors. If I were permitted to start fresh, I'd be doing things the way you described, and we wouldn't rely so much on "randomLibrary.jar" that was someones pet project seven years ago with 'LGPL' attached to it, but which has since been rewritten or abandoned. It's a willy-nilly application of such libraries to every problem that crops up without any analysis that I'm against, and not the use of libraries in any and all situations.

Yaz

Is this unique to Java?

[gstoddart]

I'm betting if you have a large enough pool of open source things, which depend on other open source things, then the bugs in the dependencies will trickle up to the projects which rely on them.

Though, admittedly, Java has also made this more annoying -- a decade or so ago when I was actively working on a Java project, it always amazed me how a new version of Java could completely break everything and then you'd have to re-test and re-certify everything.

It got to the point we put in very large bold characters in our release notes ... we work on this version of Java, if you get clever and introduce your own version of Java, we won't talk to you until you confirm the bug in the version we support.

A surprising number of clients were willing to blaze trail with whatever version of Java came along, and then kept expecting we'd be supporting custom versions from vendors or features which didn't exist when our version was built.

Eventually we learned to dread a new release of Java. Because invariably things went to hell and stopped working.

Re:Is this unique to Java?

[Anonymous Coward]

The last major release that "broke" things for me was the 1.4.2 -> 5 transition in 2004. Since then (5 -> 6, 6 -> 7, and 7 -> 8) have been relatively painless, If you were relying on an undocumented feature, or compiling against com.sun.* or sun.* classes you did so at your own risk. If you stuck to the documented JDK, you were usually ok.

Re:Is this unique to Java?

[Dr_Barnowl]

I've only ever seen instances where new versions of Java broke things by removing deprecated components like JINI, but I generally tend to stick to OpenJDK for everything - as the "official" Java (the benchmark for certification) maybe it has less "clever" in it than the others.

Clients may be keen to move onto newer versions of Java because of the immense litany of security defects that get listed by Oracle when they release a new version, and because of their apparent enthusiasm for end-of-lining support (alas, Java 7, we knew you well, but no more public updates after April 2015).

Re:Is this unique to Java?

[TheCarp]

And others. I had a hell of a time back when my job involved compiling all third party software we used that wasn't part of the OS distribution. Several really fun PHP updates. My particular favorite was when a minor point release actually moved entire modules out of the core and into a seperate download..... talk about incompatable versions!

Re:Is this unique to Java?

[Daniel Hoffmann]

Yeah this is a common problem in pretty much all platforms, what makes Java stand out is too many Java things are actually specifications, not implementations. It kinda mixes all the headaches of conventional development (dll hell, outdated libraries, testing against multiple hardware/OSs) with the headaches that developing for browsers have (multiple implementations of the same specifications). One of the things that make people like the Spring is that, unlike J2EE, Spring is an implementation, not a specification so it usually works is you change your application server for example. Well some parts of Spring relies on Servlets, which is a specification, but Servlets implementations are ok (although Java6 does not support Servlets 3.0 which is a pain in the ass if your client is on Java6 and refuses to update).

Re:Is this unique to Java?

[TheCarp]

> It got to the point we put in very large bold characters in our release notes ... we work on this version of Java, if
> you get clever and introduce your own version of Java, we won't talk to you until you confirm the bug in the
> version we support.

It gets really fun when open source folks do this, I actually had this conversation recently:

"Have you tried the latest version? That module has been updated since the version you are using"
"No, but I am looking at your code on github now, line X would be where there should be logic to handle this case, but it isn't there."

I then did verify it with the latest version, but it failed as expected. :)

Re:Is this unique to Java?

[houstonbofh]

It got to the point we put in very large bold characters in our release notes ... we work on this version of Java, if you get clever and introduce your own version of Java, we won't talk to you until you confirm the bug in the version we support.

Which is how we ended up with the management nightmare of different hardware requiring different and incompatible versions of JAVA for the "Web Client" to manage it. So, one workstation to manage Cisco. One workstation to manage EMC. One for HP. One for the phone system and a different one for the voicemail... And hope to God no one clicks "Update" on the popup before reading it!

Re:Is this unique to Java?

[gstoddart]

I haven't directly touched Java in years ... but one of the things which struck me was that it just seemed too damned brittle.

What should have been core APIs for published interfaces would suddenly change in the number of parameters between versions, or not be there at all, or return something new.

They'd "fix" something by simply deprecating it/removing it.

It felt very much like a young language which was constantly shifting under your feet, constantly calling for a do-over, and often breaking backwards compatibility.

Flash forward a decade, and I see the same kinds of things you mention -- software which ships with exactly one version it supports, and woe to anybody who ever updates it or has security people who say "you can't run that version".

It feels kind of half-assed and not very maintainable if you need to spend that much effort not breaking it. All the while the platform itself constantly nags you to update, but if you do that you're probably screwed.

And then Oracle started putting the Ask.com bullshit in, and made it essentially a platform you can't really trust because it's constantly trying to subvert your machine.

Probably not only Java components affected

[QuietLagoon]

It wouldn't surprise me if similar audits found the same level of vulns in component libraries for other development environments.

.
Java developers are no different than other developers.

Just block them

[sunderland56]

One basic problem seems to be that repositories are providing downloads of known vulnerable components.

Once a bit of software has a known vulnerability, it should *immediately* be deleted from all repositories. Responsible developers will post a fix in a timely manner; hacks will wait weeks/months/years to update, Eventually people will move away from the badly written bits of software - because they aren't available. Problem solved.

Re:Just block them

[houstonbofh]

Delete stuff from the Internet... Hmmm... Sounds like a wonderful idea. How?

Actually it is a terrible idea, even if it could work, because looking at how the code progressed is how you learn. Not to mention that I can patch and old version to fix the vulnerability, but not have to move to the new and incompatible version.

Re:Just block them

[ERJ]

You don't know how the software is being used. Maybe it is Apache's commons-net which has a vulnerability in the FTP client while my software only uses the SMB client. Maybe the next revision up has API changes that break compatibility.
In the ideal world everything would be kept up to date but time is a finite resource and if there is not a compelling reason to update it seems silly to waste time on it.

This is not surprising

[ErichTheRed]

This basically defines some of the problems of "enterprisey" software:
- It's composed of a million glued-together libraries.
- It's written by chronically understaffed/overworked IT department employees.
- Rigorous testing either (a) doesn't exist, (b) is so onerous that most developers try to avoid it, or (c) is outsourced/offshored to the lowest bidder, and therefore isn't completed without the staff basically doing the tests for the outsourcer.
- Anything that breaks it is avoided at all costs because of all of the above.

By extension, this is why some companies are stuck running IE 6 for key applications, or Office 97 because rewriting the scary mess of macros that runs a process isn't something anyone wants to do. I do systems integration work, and new versions of Java, web browsers, etc. are miserable. They introduce bugs small enough to be annoyances (rendering problems, etc.) and big enough to break the entire system.

The key to fixing this is for the software architects to require that developers move up to at least a semi-modern release of their key libraries, test everything against them, and remove the old outdated ones once all the bugs are fixed. The problem is that this is never done.

Re:This is not surprising

[JustNiz]

>> The problem is that this is never done.

The reason is that many Software Director positions are now filled with technically clueless people that are basically salesmen rather than engineers.
They have no comprehension of the concept of technical debt, or the need to spend time on activities that don't directly translate into new features.
The net result is that you're always just piling more crap onto the top of a steaming turd pile so making it worse, instead of working to replace the shit.

Accept the fact

[cloud.pt]

It's about time everyone stops whining. There are things in life you're better safe than sorry, but then there are things in life you just can't change: not every single entity can keep maintaining what they create. Human beings are limited, and so are human organizations - they lack money, workforce or simply the patience to put up with some "critical flaws" that are just too rooted in bad design to be solved without a restructuring.

THAT IS THE REAL FLAW.

There are good ways and bad ways to create reusable components. Black boxing (containing) everything for starters (sans the closed-sourced part) is something people tend to limit the scope to testing and/or to services outside a fully-fledged system's component border. Technologies like SOA are just one of many ways to plug&play every new piece of technology that performs a very specific task in a different way of a previously flawed one. Think project Ara. It's not only fun to develop like this (although some have problem conceptualizing it), but it's also more robust in the long run. Using such paradigms is what we, as the "clients" of such "aging and flawed" components can do push better development of individual components.

Now, each and every component developer has to find ways to keep their work atomic, so as to not conflict with the principles of technologies they are developed to work for. This might all seem like an utopian way of what to expect of the coding community, but then again we are also still looking for the best ways to apply near-perfect political views designed hundreds of years ago, which are yet to achieve full potential. I keep my hopes up for both issues, but my expectations low.

Dependency Vulnerability != Product Vulnerability

[dark.nebulae]

It is important to note the following:

Sonatype has determined that over 6 percent of the download requests from the Central Repository in 2014 were for component versions that included known vulnerabilities.

That means that when building a project the devs are using an older version of a dependency than a newer, fixed version. You don't pull your own artifacts from Sonatype, just dependencies.

Yes this can mean there's a bug that might be exposed to end users, but frequently a dependency is just a dependency used by the developer's code. Sure there could be a transient vulnerability, but I don't think vulnerabilities will be that transparent but that depends upon the nature of the vulnerability and the use of the component in the dev's code.

This is more of a "DLL Hell" situation. Vulnerabilities can be solved in one component, but due to interdependencies it may or may not be possible to use the fixed version. If you're choosing to use an older version of a dependency, well then that's a bad choice. But sometimes those older versions you can just be stuck with due to interdependencies, business direction (only version X has been approved by corp even if Y has critical fixes you may still be stuck with X), etc.

Re:What about JavaScript code?

[Dr_Barnowl]

He's saying that Java, because of it's nature, and the type of programmer that uses it, could reasonably be expected to be more rigorously programmed than Javascript, so if there are horrific problems with Java, then the problems with Javascript are like the Elder Gods descending upon the web.

Possibly not so bad at the client side, but all this Node.js stuff that's popular....

Re:Java is the Devil

[oh_my_080980980]

Because .NET and C# are better....

Re:What about JavaScript code?

[oh_my_080980980]

Which still has nothing to do with the original topic.

No mention of Sonatype's business?

[Captain Damnit]

It should be noted that the company releasing this report, Sonatype, markets a product called Insight Application Health Check that scans your binaries for libraries with known vulnerabilities.

I have never used their service, and can offer no comments on its utility or value. However, it is a bit unseemly that TFA doesn't mention that the source of their information about this very real problem also sells a service that solves it. This is a knock on IT World, not Sonatype.

K.I.S.S. - Ugly Lasts

[Tablizer]

If you want software to last more than one season, then rely on old-school "plain jane" HTML for most of your UI with a little JavaScript only where absolutely needed.

Plane-jain HTML is not glamorous or fancy, but it has been supported and will be supported for a good while. If your org wants fancy, it has to pay the Fancy Tax.

If you have to repaint the entire screen for an activity, so be it. If it's a relatively small-user-base app, the overall bandwidth overhead is not really bigger than downloading fat JS libraries. If you code your HTML to mostly rely on CSS, then HTML redraws will be small packets.

Re:What about JavaScript code?

[DickBreath]

> JavaScript lacks most of the language features needed for writing robust, reliable software at a large scale.

There is no reason that it should not be possible to write huge enterprise applications in JavaScript. (or assembly language).

It's just that it becomes a huge unmaintainable mess.

The reason for higher and higher level languages, including type checking, is to make the compiler to more and more of the bookkeeping of writing software. You could do it in assembly language if you were willing to do enough bookkeeping.

What JavaScript (and similar dynamic languages) bring is that you can write smaller projects very quickly. See Ruby on Rails for example. But then Twitter started with Ruby, and in 2012 switched to Java for scalability. They said that Ruby was the right language when they started.

Blame Maven

[_xeno_]

This is a problem that Maven has created, mostly.

What the summary doesn't mention is that "large repository of open source software" is a Maven repository. Maven allows you to specify dependencies for your Java project.

The problem is that you have to specify a specific version of whatever you use. So let's say you use OpenFoo 1.1 and that at the time you write your code, the latest version of OpenFoo is 1.1.3.

Now assume a horrible vulnerability is discovered in OpenFoo 1.1.3, so they release OpenFoo 1.1.4 to fix it. Well, your Maven POM says you require OpenFoo 1.1.3, so until you go in and manually change that, you will only ever use 1.1.3. There is - by design - no way to say "I want the latest 1.1 version." You can only describe a single, specific version.

So it's no surprise that Sonatype will see a ton of old Maven projects continuing to download outdated Maven artifacts. There's no way to say "I want the latest version of a specific branch" you can only specify a single version. Which means that a project that hasn't changed in years will still pull in the old versions of the libraries, even if it would work with the later versions.

Re:Blame Maven

[Anonymous Coward]

There's no way to say "I want the latest version of a specific branch" you can only specify a single version. Which means that a project that hasn't changed in years will still pull in the old versions of the libraries, even if it would work with the later versions.

No, Maven does support version ranges: You can say stuff like this: <version>[1.0.0,2.0.0)</version>. Here's a pretty good thread on the subject from StackOverflow.

To be fair, I don't think that most projects do this, but at least it's supported. Also, I'd guess that an analysis of a lot of the other artifact repositories like PyPi, Bower, or npm would produce similar results.

Re:Blame Maven

[davidleelambert]

And the comments on that answer say that the "LATEST/RELEASE features" are deprecated or no longer supported, although the links they give to back that statement up are broken.

Re:FUD

[DickBreath]

It's not FUD. It's clickbait. It will make up for the revenue shortfall from SourceForge.

Sonatype FUDs Open Source ..

[nickweller]

April 2013: "Sonatype's annual survey of 3,500 software developers and shows struggle in setting corporate policy on open source and enforcing it" ref

April 2013: "Control and security of corporate open source projects proves difficult | New Sonatype survey finds 80 percent of most Java applications comes from open source" ref

Nov 2014: "Software developers use a large number of open-source components, often oblivious to the security risks they introduce or the vulnerabilities that are later discovered in them." ref

April 2015: "open-source also represents a vast, unpatched quagmire of cyber-risk that’s putting public safety at grave risk. That’s the assessment of Joshua Corman, CTO at Sonatype" ref