Does Open Source Need a Red Team?

garyebickford writes "IMHO the Open Source community (whatever that is) needs a Red Team project. This would be an open source project, but its output would be a process rather than a piece of software. If such a group exists, I'm not aware of it. This document and this page [from the Google cache] are from a commercial company (picked at random from a Google search) that provides similar services. The OS Red Team would provide 3rd party security testing, code review and evaluation for open source projects prior to release, providing a 'report card' stating what has been reviewed and tested, and recommending fixes. When a package is released, the Team's 'weather report' stating the probabilities that a package would survive different kinds of attack would be a valuable piece of information for prospective users." Do you think the Open Source Community would benefit from such an effort?

"The Team could also provide a set of recommended processes and tools for O.S. projects to follow prior to submission to the Red Team test queue. This by itself would be a valuable tool.

Such teams are sometimes used by companies to test the security of their networks and software. The O.S. community have done an excellent job so far, but as open source is used more and more by the mainstream computer users, vetting by a 3rd party would help make many organizations more likely to accept a piece of O.S. software.

The Team would, like any open source project, be comprised of both experts and newbies. The newbies would have the opportunity of doing real testing under the guidance of folks who know more, thereby becoming more expert themselves. The experts would provide a centralized open-source-oriented set of recommendations and specialized review as needed.

Either the Red Team or its members could also provide paid services for commercial software, and could participate with university CS departments in training students, providing the opportunity for valuable cross-training between schools. It might even be possible to arrange course credit for work on the Team.

Many Open Source projects could benefit from such a 3rd party group to recommend development procedures, code styles, and actual testing to teach and motivate better security practices in code design. The plain fact is that many (most?) of us developers are not completely 'up' on the issue of security - it's a very dynamic area of specialization. This initiative could be another resource that will be useful in establishing OS in the mainstream."

Comments

Funding? Needed at All?

[hbo]

OK, so you are going to hire highly experienced and expensive talent to do security audits for open source projects that don't have a revenue model? Where's your revenue model?

And of course, the benefit of open source is that all sorts of motivated, talented people from all over the world pitch in to do a similar analysis for free, and without a formal "red team." This breaks down quite a bit with the volume of Free Software being produced nowadays, however. But the important pieces of infrastructure (Apache, e.g.) DO get the scrutiny their importance demands. Not to mention pounding by black hats.

Someone mentioned OpenBSD. But even they don't audit everything. They confine their attention to the core of the OS. That's quite a lot of software, but the ports tree is quite a bit more. The ports get somewhat more attention than they would simply because you've got a large set of security conscious users.

Re:Funding? Needed at All?

[garyebickford]

A couple of points on funding were made either in the original or by others - I noted that either the project as a whole or members individually might sell the service for commercial software or business clients who want the security. I also noted (this is a bit more difficult) that some universities might offer credit for participation - many open source projects are now essentially university projects, such as PHPWebSite.
The paper by _iris (92554) suggested previously that this function might be part of an insurance package. Business insurers often require IT emergency plans, risk analysis and 3rd party network security audits, and might reasonably require a company moving to open source software (read, "unknown vendor") to have the software reviewed. This is exactly where the idea of a group specializing in OS software would fit and quite possibly make money.

As for "all sorts of motivated, talented people", I think you're making my point. The "Red Team" project could provide a focus for those folks to achieve some synergy - I'm not involved in that area much but I suspect that a lot of those folks are feeling 'behind the curve' lately. It's amazing how well the community has done. A project that focusses that effort could greatly improve the availability of the 'right' information and fast access to folks who have the experience.

I do disagree with your thought that the "important pieces" get scrutinized, if you mean to say that's sufficient. The problem is that, using Apache for example, Apache itself can be bulletproof but many scripting projects using modperl, PHP, JSP, etc. are not. The result is a server that is nearly as vulnerable as if Apache itself had a vulnerability.

Apache scripting is what triggered my thought on this topic. A large number of scripting vulnerabilities have been reported lately. I can't guarantee that the scripts I've written have been bulletproof. I'd like to have a resource that, perhaps with some configuration by me, could run through a set of tests of my scripts. This couldn't guarantee anything but it could help developers.

Yes.

[nadador]

The first phase of the open model was the developer stage - individual people contributed their talents to produce interesting software. Their products were raw and unrefined, but very powerful. All of the best practices that (supposedly) happen at commercial software houses - all of that process - was chucked out the window in favor of devoting time to the very real creative experience of molding and bending and shaping new code.

The second phase of the open model was the documentation phase. When they collected code from the net to make their products, the commerical vendors of open software took the raw, unrefined code, and harnessed its power into a form that PHBs could recognize. Now we have Ximian - a refined product that PHBs recognize, built on the creativity of GNOME developers. Now we have MontaVista and Timesys Linux kernels - products that PHBs recognize, refined to their needs, but built on the creativity of the kernel developers.

I suppose that the third stage of the open model might be to do this - to help open projects apply best practices for software creation, test, and maintenance. I just don't know who you're going to get to do it. Individual developers, I would imagine, will be more concerned with the raw creativity of hacking at code in vi. Commercial companies will more be more likely to apply these practices to the code that they ship their customers, not the code that lives in the repository at SourceForge, although maybe they coincide.

I suppose my point is that you have to find people who want to do it, or money to make people want to, and I'm not sure where you're going to find either.

Maybe you should try asking the OSDL

[Alpha27]

http://www.osdl.org/

I recall they are an organization sponsored by big names in the IT industry, that could possibly emplore such an idea. Their idea is to proviude enterprise class testing to help advance the linux community. I don't see why this couldn't be an extension of it.

I'm sure a nicely worded, thought out paper explaining the benefits would at least get a response, and possibly spike some interest.

Red Team Go! Red Team Go!

[TitaniumFox]

I'd rather have an OSI Red Team that was more like Delta Force.

They could wear MIT wearables, have an internet uplink, and code-fu your ass into submission.

The Two-Edged Sword of Open Source Software

[_iris]

A while back I wrote a paper titled The Two-Edged Sword of Open Source Software, which might be of interest.

Re:Already have an ecosystem.

[GigsVT]

Which security companies pay for notification of flaws ?

http://www.idefense.com/vcp_faq.html

I've seen a couple others, I can't recall offhand.

Process will make it better?

[madmaxx]

You think a process will make OSS better? The act of defining a process renders it useless, as the principles become static. Software will only get better when we do away with processes. Software is complex, it requires us to think -- not follow procedures.

I've been a part of many small and large processes, and none of them were effective. The best any of them were able to do was to soften what was produced by the morons. In lessening the effect of retarded developers, the processes become a hindering block to those who know wtf they are doing. Process is so fun.

Software development needs to be organic. OSS needs more mentors, gurus of the deep, dark, unknown to become one with the new blood. It is about community, and about collaboration - the real sort of kinship where people build things together. Process is about as un-personal as it gets.

Re:Process will make it better?

[garyebickford]

I don't mean to suggest that some Cathedral be built. My first engineering job was as a Software QA Engineer - the most hated title in the company. Projects were almost always late and over budget before we got them, and when we rejected them yet again with another 200 'blue sheets', managers' blood ran cold.

I see this as just another project that is itself a resource to projects that want to make use of it, just as GTKlib is available as a resource.

Lest you go off the deep end waxing poetical in the dark (sorry... :O) What I'm proposing is a focal point for the mentoring you're talking about. It's an Open Source version of the Red Team concept.

Re:Process will make it better?

[ComputerSlicer23]

That's a really nice thought, but Engineering is all about process. There should be a process, and guidelines, and a list of things that happen for code to be "good to ship". OSS does this informally, by the users using it, essentially out of appriciate for the scratching of the itch.

One of the more complicated things we ever did, involved sending a guy to the moon. That involved lots of Engineering. That's the closest thing I can think of to writing new software. First, they didn't have the tools to make the stuff they needed. They had to make tools, to make the final products they needed. Then they did testing, documentation, procedures, failure cases, procedures for handling all of the failure cases. Then they did training. Then they did more training.

They had to create materials, and process, and solve invent whole new areas of Engineering to do that. They had all kinds of problems they overcame, by letting the Engineer's imagination run rampant, and then having it, checked, re-checked, and double checked. They did it, by having redundant checks, and redundant design teams. They had fail-over systems in triplicate. They tested literally everything they could. When something failed, they investigated way, and change the procedure to mitgate the probability of that happening again.

It'd be wonderful to see some process be implemented. Right now, to get code into say the stock kernel, there is a process. You convince Linus it's good enough. You can do that a myriad of ways. If it's small you just send it to him. If it's larger, you send it to a large list of people who peer review it. You send it to a maintainer of a popular tree. They let is sit in their tree for a while. Nobody reports a bug, some people report success. Whoopie, it'll probably end up in a source tree. It's very, much a process. It's just not a process a single company can execute. They can't have the internal talent lying around to do it.

There are a lot of projects that have little to no internal review. They have only a handful of developers examining the code. Some of whom might not be the greatest coders in the world.

The reason Apache, Linux, FreeBSD, and other large projects are successful, is because they have an over abundance of people who care, who are extremely talented. They also tell people who are crappy who try and contribute to go fly a kite, until they can get up to snuff. It'd be nice to see a group of people, who care, precisely because they are paid to care. Right now, a lot of those people are paid by Linux distributors, or other distributers of OSS.

Personally, I'd rather see an open source static source checker developed so auditing could be streamlined and automated. You run the checker against your code, it analyses all of the possible cases, and emits warnings. You have a way to notated the code (preferrably, out of band of the source code) to say, I've checked this one, don't warn me any more. Have all that integrated with a SCM tool, that will identify when you have changed a section that has previously been human approved, and nullify the approval. So you reduce a lot of the grunt work of doing an audit.

Something even more sophisticated for the Linux kernel, that says, function call foo, can only be called if lock bar is held. Then having a compiler check to see if there might be a problem. Have rules about function bar can't call anything that can sleep. Then have a the tool check it. So for each project you can establish a ruleset, and enforce the rules in a relatively automated way. This function can't be called from interrupt context, and build the call tree to see if it ever can be called. I believe smatch is the beginnings of such a project.

I guess in the end, the gurus, execute their own personal process, and if you have really great guru's the project works. If you have guru's whose process sucks, it's a failed OSS project. The reason you don't see big catastrophic failures in OSS, is they die while they are

Insurance, yes

[garyebickford]

Yes, one of the possible sources of revenue could be insurance-related. I would think that the way such a thing might work would be that an established carrier would want to hire the group to vette a piece of code before offering clients hacker or data loss insurance. Such insurance already exists, so this is a potential market for someone wanting to do the project. If I were an insurer I might well require 3rd party evaluation of any new software in a mission-critical or financial or other high-cost-of-risk environment.

"Community"?

[fuzzybunny]

Nice idea. However, it fails on one problem: there is no "Open Source Community". Or rather, there is, but it's not the sort of homogenous, integrated entity/organization that gives managers and powerpoint jockeys warm fuzzy feelings.

Rather, it's a bunch of dudes knocking out code. And for the same reason you're not going to get most of them to provide adequate documentation, which is thoroughly understandable given that (a) they're doing something for fun, and (b) they're not getting paid for it, you're not going to get these people to submit to procedures and processes, on the whole. Hobbyists will continue to build stuff on a lark, doing it the way they feel like doing it.

Now if you want to provide something like this as a service for companies hoping to use OSS, great. However, someone would have to pay for it, which takes away one of the big pluses of OSS. In fact, that's one of the reasons your average commercial entity goes for proprietary software--it's the management perception that there is an organized set of procedures and such behind its development (usually true to some degree) as well as an organization they can sue if things go pear-shaped.

Nice idea, but needs practical development.

other ways...

[kevin lyda]

red team sounds like something a closed package would need. linux and other free software offer additional options for testing. openbsd does a continuous code audit. linux has the kernel janitors. in addition there are numerous citations for fuzz - here's one.

i get the idea you want a company to do all this work and then place a certification on distros or packages. you confuse the issue with the buzzword scented "red team" references, but it really sounds like you want to use the services of such a company - or create one and create buzz for such a company.

This already exists

[Tuross]

Folks... it's called "bugtraq" and it's been around for decades.

Anyone else amused by the irony that someone is advocating open source software should start practising the things closed source development is now getting buzzword compliant with, which is made popular in that arena because its already such a success with open source software? ;)

"Best Practices" are context-dependent

[korpiq]

I suppose that the third stage of the open model might be to do this - to help open projects apply best practices for software creation, test, and maintenance.

Are you implying that open development (with its world-readable version trees, communication through archived, public message systems, bypassing monetary systems as the controlling aspect of software development, etc.) has somehow proved itself so inefficient that it should be given up in favor of whatever the closed development sector has to offer?

It's the closed commercial sector that is supposed to bend toward open methods, not vice versa. That is happening through grass-roots efforts like "stealth" installments of Linux-servers in the end of 1990's followed by "stealth" installments of Linux-workstations right now, as well as governmental and communal bodies around the world already embracing the open model as a cost- and result-effective method unbound by the insecurities of commercial offerings.

I'm sorry to sound this flamy, but your comment (as well as this whole subject, actually) reminds me of quite a few people who claim they have a grasp of the open development model, while they still look at it through a 1980's commerce school's window.

As for the security of Open offerings, mature projects' insecurity (the cumulative time window of exploits open against product's lifetime) should be compared to that of closed-development (=non-patch-accepting) offerings. From what I gather, on that basis insurance prices against IT disasters should be considerably cheaper with mature Open products.

Re:"Best Practices" are context-dependent

[nadador]

Maybe I mispoke. I didn't mean to imply that open model will bend to the way that the closed software world works. I do agree that the commerical companies are bending to do things the open source way, and not the other way around.

But lets be realistic about whats going on.

Where I work, half of our product is the documentation that we supply to our customers detailing design, implementation, and testing. A big chunk of our time is spent going over records of integration testing and through the logs of the version control system. This totally stiffles people like me who want to go and fix the ugly, nasty code that lives in our version control system, but I know every line of code change is going to be another couple of minutes in a code review with our customer. It has been my experience, and correct me if I'm wrong, but most open source projects pride themselves on the design behind the technical aspects of their projects, not by their quality assurance.

If I understand the point of the story, the question is whether or not open projects need a devoted test and process organization, which is what we used to call our Software Engineering Process Group before the last reorg. They do black box testing, make tool suggestions, etc.

I can see how that service would be a tremendous service to an open project, but I don't know if you're going to find people who want to do QA in their free time.

Sardonix?

[packetknife]

Not exactly what you are discussing but there was a lot of hoopla around Sardonix many months back and it doesn't appear WireX has done anything real with it yet. I'm on the mailing list and it sounds of crickets.

Another thing to remember is that there are decent references out there, some quite well known, that people could follow and use but simply don't (Viega's book, and number of HOWTOs, etc.).

In anycase, you might want to approach WireX and see what, if anything, can be done to resurrect Sardonix. Cheers, -Pk

Red Shirts

[Jack Comics]

... Sure, so long as they don't join Starfleet.

Captain Kirk, Mr. Spock, and the Red Team beam down to an alien planet -

Kirk - "Rodriguez, check to see what's causing that buzzing sound coming from the rock nearby."

Rodriguez (Red Team) - "Bleep you! Go check it out yourself! We've lost three Red Team members this past week that beamed down to strange worlds with you!"