Does Open Source Need a Red Team?

garyebickford writes "IMHO the Open Source community (whatever that is) needs a Red Team project. This would be an open source project, but its output would be a process rather than a piece of software. If such a group exists, I'm not aware of it. This document and this page [from the Google cache] are from a commercial company (picked at random from a Google search) that provides similar services. The OS Red Team would provide 3rd party security testing, code review and evaluation for open source projects prior to release, providing a 'report card' stating what has been reviewed and tested, and recommending fixes. When a package is released, the Team's 'weather report' stating the probabilities that a package would survive different kinds of attack would be a valuable piece of information for prospective users." Do you think the Open Source Community would benefit from such an effort?

"The Team could also provide a set of recommended processes and tools for O.S. projects to follow prior to submission to the Red Team test queue. This by itself would be a valuable tool.

Such teams are sometimes used by companies to test the security of their networks and software. The O.S. community have done an excellent job so far, but as open source is used more and more by the mainstream computer users, vetting by a 3rd party would help make many organizations more likely to accept a piece of O.S. software.

The Team would, like any open source project, be comprised of both experts and newbies. The newbies would have the opportunity of doing real testing under the guidance of folks who know more, thereby becoming more expert themselves. The experts would provide a centralized open-source-oriented set of recommendations and specialized review as needed.

Either the Red Team or its members could also provide paid services for commercial software, and could participate with university CS departments in training students, providing the opportunity for valuable cross-training between schools. It might even be possible to arrange course credit for work on the Team.

Many Open Source projects could benefit from such a 3rd party group to recommend development procedures, code styles, and actual testing to teach and motivate better security practices in code design. The plain fact is that many (most?) of us developers are not completely 'up' on the issue of security - it's a very dynamic area of specialization. This initiative could be another resource that will be useful in establishing OS in the mainstream."

Comments

Yes.

[nadador]

The first phase of the open model was the developer stage - individual people contributed their talents to produce interesting software. Their products were raw and unrefined, but very powerful. All of the best practices that (supposedly) happen at commercial software houses - all of that process - was chucked out the window in favor of devoting time to the very real creative experience of molding and bending and shaping new code.

The second phase of the open model was the documentation phase. When they collected code from the net to make their products, the commerical vendors of open software took the raw, unrefined code, and harnessed its power into a form that PHBs could recognize. Now we have Ximian - a refined product that PHBs recognize, built on the creativity of GNOME developers. Now we have MontaVista and Timesys Linux kernels - products that PHBs recognize, refined to their needs, but built on the creativity of the kernel developers.

I suppose that the third stage of the open model might be to do this - to help open projects apply best practices for software creation, test, and maintenance. I just don't know who you're going to get to do it. Individual developers, I would imagine, will be more concerned with the raw creativity of hacking at code in vi. Commercial companies will more be more likely to apply these practices to the code that they ship their customers, not the code that lives in the repository at SourceForge, although maybe they coincide.

I suppose my point is that you have to find people who want to do it, or money to make people want to, and I'm not sure where you're going to find either.

The Two-Edged Sword of Open Source Software

[_iris]

A while back I wrote a paper titled The Two-Edged Sword of Open Source Software, which might be of interest.

Re:Already have an ecosystem.

[GigsVT]

Which security companies pay for notification of flaws ?

http://www.idefense.com/vcp_faq.html

I've seen a couple others, I can't recall offhand.

Process will make it better?

[madmaxx]

You think a process will make OSS better? The act of defining a process renders it useless, as the principles become static. Software will only get better when we do away with processes. Software is complex, it requires us to think -- not follow procedures.

I've been a part of many small and large processes, and none of them were effective. The best any of them were able to do was to soften what was produced by the morons. In lessening the effect of retarded developers, the processes become a hindering block to those who know wtf they are doing. Process is so fun.

Software development needs to be organic. OSS needs more mentors, gurus of the deep, dark, unknown to become one with the new blood. It is about community, and about collaboration - the real sort of kinship where people build things together. Process is about as un-personal as it gets.

Re:Process will make it better?

[garyebickford]

I don't mean to suggest that some Cathedral be built. My first engineering job was as a Software QA Engineer - the most hated title in the company. Projects were almost always late and over budget before we got them, and when we rejected them yet again with another 200 'blue sheets', managers' blood ran cold.

I see this as just another project that is itself a resource to projects that want to make use of it, just as GTKlib is available as a resource.

Lest you go off the deep end waxing poetical in the dark (sorry... :O) What I'm proposing is a focal point for the mentoring you're talking about. It's an Open Source version of the Red Team concept.

Re:Process will make it better?

[ComputerSlicer23]

That's a really nice thought, but Engineering is all about process. There should be a process, and guidelines, and a list of things that happen for code to be "good to ship". OSS does this informally, by the users using it, essentially out of appriciate for the scratching of the itch.

One of the more complicated things we ever did, involved sending a guy to the moon. That involved lots of Engineering. That's the closest thing I can think of to writing new software. First, they didn't have the tools to make the stuff they needed. They had to make tools, to make the final products they needed. Then they did testing, documentation, procedures, failure cases, procedures for handling all of the failure cases. Then they did training. Then they did more training.

They had to create materials, and process, and solve invent whole new areas of Engineering to do that. They had all kinds of problems they overcame, by letting the Engineer's imagination run rampant, and then having it, checked, re-checked, and double checked. They did it, by having redundant checks, and redundant design teams. They had fail-over systems in triplicate. They tested literally everything they could. When something failed, they investigated way, and change the procedure to mitgate the probability of that happening again.

It'd be wonderful to see some process be implemented. Right now, to get code into say the stock kernel, there is a process. You convince Linus it's good enough. You can do that a myriad of ways. If it's small you just send it to him. If it's larger, you send it to a large list of people who peer review it. You send it to a maintainer of a popular tree. They let is sit in their tree for a while. Nobody reports a bug, some people report success. Whoopie, it'll probably end up in a source tree. It's very, much a process. It's just not a process a single company can execute. They can't have the internal talent lying around to do it.

There are a lot of projects that have little to no internal review. They have only a handful of developers examining the code. Some of whom might not be the greatest coders in the world.

The reason Apache, Linux, FreeBSD, and other large projects are successful, is because they have an over abundance of people who care, who are extremely talented. They also tell people who are crappy who try and contribute to go fly a kite, until they can get up to snuff. It'd be nice to see a group of people, who care, precisely because they are paid to care. Right now, a lot of those people are paid by Linux distributors, or other distributers of OSS.

Personally, I'd rather see an open source static source checker developed so auditing could be streamlined and automated. You run the checker against your code, it analyses all of the possible cases, and emits warnings. You have a way to notated the code (preferrably, out of band of the source code) to say, I've checked this one, don't warn me any more. Have all that integrated with a SCM tool, that will identify when you have changed a section that has previously been human approved, and nullify the approval. So you reduce a lot of the grunt work of doing an audit.

Something even more sophisticated for the Linux kernel, that says, function call foo, can only be called if lock bar is held. Then having a compiler check to see if there might be a problem. Have rules about function bar can't call anything that can sleep. Then have a the tool check it. So for each project you can establish a ruleset, and enforce the rules in a relatively automated way. This function can't be called from interrupt context, and build the call tree to see if it ever can be called. I believe smatch is the beginnings of such a project.

I guess in the end, the gurus, execute their own personal process, and if you have really great guru's the project works. If you have guru's whose process sucks, it's a failed OSS project. The reason you don't see big catastrophic failures in OSS, is they die while they are

Insurance, yes

[garyebickford]

Yes, one of the possible sources of revenue could be insurance-related. I would think that the way such a thing might work would be that an established carrier would want to hire the group to vette a piece of code before offering clients hacker or data loss insurance. Such insurance already exists, so this is a potential market for someone wanting to do the project. If I were an insurer I might well require 3rd party evaluation of any new software in a mission-critical or financial or other high-cost-of-risk environment.

"Community"?

[fuzzybunny]

Nice idea. However, it fails on one problem: there is no "Open Source Community". Or rather, there is, but it's not the sort of homogenous, integrated entity/organization that gives managers and powerpoint jockeys warm fuzzy feelings.

Rather, it's a bunch of dudes knocking out code. And for the same reason you're not going to get most of them to provide adequate documentation, which is thoroughly understandable given that (a) they're doing something for fun, and (b) they're not getting paid for it, you're not going to get these people to submit to procedures and processes, on the whole. Hobbyists will continue to build stuff on a lark, doing it the way they feel like doing it.

Now if you want to provide something like this as a service for companies hoping to use OSS, great. However, someone would have to pay for it, which takes away one of the big pluses of OSS. In fact, that's one of the reasons your average commercial entity goes for proprietary software--it's the management perception that there is an organized set of procedures and such behind its development (usually true to some degree) as well as an organization they can sue if things go pear-shaped.

Nice idea, but needs practical development.

other ways...

[kevin lyda]

red team sounds like something a closed package would need. linux and other free software offer additional options for testing. openbsd does a continuous code audit. linux has the kernel janitors. in addition there are numerous citations for fuzz - here's one.

i get the idea you want a company to do all this work and then place a certification on distros or packages. you confuse the issue with the buzzword scented "red team" references, but it really sounds like you want to use the services of such a company - or create one and create buzz for such a company.

"Best Practices" are context-dependent

[korpiq]

I suppose that the third stage of the open model might be to do this - to help open projects apply best practices for software creation, test, and maintenance.

Are you implying that open development (with its world-readable version trees, communication through archived, public message systems, bypassing monetary systems as the controlling aspect of software development, etc.) has somehow proved itself so inefficient that it should be given up in favor of whatever the closed development sector has to offer?

It's the closed commercial sector that is supposed to bend toward open methods, not vice versa. That is happening through grass-roots efforts like "stealth" installments of Linux-servers in the end of 1990's followed by "stealth" installments of Linux-workstations right now, as well as governmental and communal bodies around the world already embracing the open model as a cost- and result-effective method unbound by the insecurities of commercial offerings.

I'm sorry to sound this flamy, but your comment (as well as this whole subject, actually) reminds me of quite a few people who claim they have a grasp of the open development model, while they still look at it through a 1980's commerce school's window.

As for the security of Open offerings, mature projects' insecurity (the cumulative time window of exploits open against product's lifetime) should be compared to that of closed-development (=non-patch-accepting) offerings. From what I gather, on that basis insurance prices against IT disasters should be considerably cheaper with mature Open products.

Sardonix?

[packetknife]

Not exactly what you are discussing but there was a lot of hoopla around Sardonix many months back and it doesn't appear WireX has done anything real with it yet. I'm on the mailing list and it sounds of crickets.

Another thing to remember is that there are decent references out there, some quite well known, that people could follow and use but simply don't (Viega's book, and number of HOWTOs, etc.).

In anycase, you might want to approach WireX and see what, if anything, can be done to resurrect Sardonix. Cheers, -Pk