Slashdot Asks: Are You Ashamed of Your Code? (businessinsider.com)

Programmer and teacher Bill Sourour wrote a post last week called "Code I'm Still Ashamed Of," where he recounts a story in which he was hired to write code for a pharmaceutical company. Little did he know at the time, he was being "duped into helping the company skirt drug advertising laws in order to persuade young women to take a particular drug," recaps Business Insider. "He later found out the drug was known to worsen depression and at least one young woman committed suicide while taking it." Sourour was inspired to write the post after viewing a talk by Robert Martin, called "The Future of Programming," who argues that software developers need to figure out how to self-regulate themselves quickly as software becomes increasingly prevalent in many people's lives. Business Insider reports: "Let's decide what it means to be a programmer," Martin says in the video. "Civilization depends on us. Civilization doesn't understand this yet." His point is that in today's world, everything we do like buying things, making a phone call, driving cars, flying in planes, involves software. And dozens of people have already been killed by faulty software in cars, while hundreds of people have been killed from faulty software during air travel. "We are killing people," Martin says. "We did not get into this business to kill people. And this is only getting worse." Martin finished with a fire-and-brimstone call to action in which he warned that one day, some software developer will do something that will cause a disaster that kills tens of thousands of people. But Sourour points out that it's not just about accidentally killing people or deliberately polluting the air. Software has already been used by Wall Street firms to manipulate stock quotes. "This could not happen without some shady code that creates fake orders," Sourour says. We'd like to ask what your thoughts are on Sourour's post and whether or not you've ever had a similar experience. Have you ever felt ashamed of your code?

Comments

Maybe we should mimic civil engineering

[Parker Lewis]

In civil engineering, when any project is bigger then a certain amount, it's required to have a civil engineer signing the project, responsible for all the stuff. Sometimes I wonder if we need a similar regulation in software. By example: if it's covering something sensible to life, like medical, airport, etc, law should require a software engineer signing the project and responsible for it.

Like on civil engineering, probably this will force software people to really invest in QA (until current days, QA is really, really bad in software).

Re:Maybe we should mimic civil engineering

[Anonymous Coward]

What does this mean in terms of software? Software crashes all the time.

Not in safety critical applications. Writing software for them is a different beast.

How could an engineer sign off on a system like this?

With the proper documentation.

I don't think we have proper methods of describing and solving modern safety issues in embedded systems.

Google for machine safety standards. IEC 60601-1 seems to be a good starting point for medical devices.
I've only written code for industrial machinery so I can't say for sure if it contains the necessary information. You typically have to go through quite a lot of standards to figure out the full requirements.

You have to document not only how the software will handle all plausible input cases but also how the device won't endanger anyone in the case of common hardware failures.
Some electromechanical devices can be assumed to not fail if you never approach half the marked current.
Some components are designed to have a defined failure state. You can use capacitors that always break, never short circuits.
For transistors you have to document how the device will operate in the different possible ways the transistor can break.
For complex circuits like a CPU you are not allowed to assume that it will remain functional and because of this you need at least two CPUs and have software or hardware that detects if one of them doesn't act as it should.
Depending on what safety class you are aiming for you might have to use CPUs of different architectures and have different programmers writing the software to minimize the risk of them failing in the same way.

As you might have figured out you can't just throw in a Raspberry Pi or anything running Windows CE and hope to write life critical applications.
If you need an OS it will be something like SafeRTOS but most of the time you will skip it.
You typically have to use window watchdogs to make sure that the code executes within the right time and you need to add checkpoints to make sure that the code executes in the right order.
You should try to avoid using pointers and dynamic allocation. Yep, that rules out high level languages no matter how safe some people seem to believe they are.
Exceptions is a big no. You avoid code that doesn't have a determined path trough it.
If you actually use pointers you will have to document every usage to make sure that it can never be used uninitialized or trash other parts of the memory.
If you allocate things dynamically you will have to show that allocation failure doesn't lead to safety issues.

TL;DR;
We have the methods to write safe software. It's not easy and it is very time consuming.
If you are interested in doing it I recommend going for an EE degree rather than CS. Reading the standards will be hard otherwise and understanding the possible failure modes even more so.

Re:Maybe we should mimic civil engineering

[swillden]

One difference between the civil and software engineering examples that strikes me is that the civil engineer only has to ensure that the building never falls down due to natural and expected forces. If someone sets off a large truck bomb in the basement, the building *will* fall down, and everyone understands that's not the civil engineer's fault because that attack was outside of the normal and accepted design parameters.

That's not to say that we don't defend buildings against truck bombs. We do, but we do it with other mechanisms. We have regulations that attempt to restrict the availability of explosives. We have law enforcement and court systems that attempt to deter people from blowing up buildings by threatening them with punishment if they do. In some cases, for buildings that seem to be at particularly high risk, we apply various other security measures to control what vehicles can be driven into the basement, and by whom. We also have infrastructure in place that attempts to monitor whether or not some individuals or groups might be interested in trying to blow up a specific building, and devise and implement countermeasures dynamically as needed.

In the case of software, the responsibility of software engineers is not nearly as clear as it is for civil engineers. Largely this is because software engineering is still a very young profession as compared to civil engineering, and it's still evolving rapidly. In some cases, the tools and techniques used by attackers didn't even exist when the software was written. In most cases, the tools and techniques did exist and were well-known to attackers and security engineers, but not to the people who wrote the software. This indicates a failure of the profession to educate its members... but given the pace at which attack techniques develop and the pace at which the software industry is and has been expanding, it's a failure without obvious solution. Simply applying the same sort of regulation and procedures applied to civil engineering would be massive overkill that would dramatically decrease the ability of the industry to produce software and probably wouldn't solve the problem.

Clearly, we need to create more secure software. The status quo is generally terrible. There are exceptions; there are organizations that do excellent security engineering and we have a good collection of tools and practices for making software that is much better than the norm. On the other hand, no matter what we do during development there will always exist the potential for a truck bomb, an attack which was simply outside the parameters that it made sense to defend against. That means we'll always need additional, "active" defenses.

In the case of the hospital equipment, that means that processes developed for medical equipment not based on software simply don't work. FDA approvals hinder security because they make patching far more expensive and difficult than it should be. We can attempt to build security perimeters around all of the equipment, but experience proves that that's a fool's errand. There's always some way in and once inside the perimeter attackers can run amok.

Our current (but rapidly evolving!) best understanding of how to make software reliable in the face of active attack is a multi-layered strategy. It starts with good software engineering practices that attempt to minimize well-understood risks (buffer overflows, SQL injection, XSS, etc.). Then we try to add firebreaks wherever possible and reasonable, so that compromise of one component doesn't compromise the system as a whole. Such firebreaks mostly consist in locking down any communication channels between components that aren't actually necessary, within processes, between processes, and between devices on networks. We also try to authenticate users and keep them restricted to the functions they can legitimately perform. Then at every level we do regular penetration testing and work to identify and patch vulnerabilities before they can be exploited -- because there will

Re:Maybe we should mimic civil engineering

[ranton]

Google for machine safety standards. IEC 60601-1 seems to be a good starting point for medical devices.

We have the methods to write safe software. It's not easy and it is very time consuming.

There is still a fairly large difference between quality control in civil engineering and software development, even for safety critical devices. In college I worked with a professor whose area of research was requirements engineering, specifically requirements trace-ability. I did some work on a research project involving Siemens and the FDA where the goal was to improve specifications given to the FDA so they could monitor safety critical devices better. It was very eye opening just how difficult it is for the FDA to perform approval on devices which include a software component.

Right now their approach is basically to look for what they called "bad smells". It is impossible to thoroughly go over every software system with the same rigor they would over electronics or mechanic systems without an astronomically higher cost. So the best they can do is use their experience on where problems are most likely to be and to focus on areas where documentation is light. Just like an experienced software QA engineer would. My professor's research focused on AI and information retrieval to build tools which assist in investigation because a thorough review would be impossibly costly.

People often pontificate about whether civil engineering, electronics, or software engineering products are inherently more or less complex than each other. They look at number of bolts in a bridge and the lines of code in a software program and pretend they can compare the two. But coming from the actual mechatronics engineers and FDA officials whose job it is to oversee the safety of these products, software systems have simply too many external and internal inputs for software quality control to reach the rigor of other engineering disciplines. Human beings are simply not capable of handling the level of complexity it would take for software engineers to have the same confidence in their products as a civil engineer has in his.

Engineer experience and good QA practices really do make a big difference, but no software engineer will ever be capable of taking on the same level of responsibility for his products as a civil engineer does when he signs off on a bridge. This doesn't mean the industry cannot improve (it certainly can), it just means comparing the results of software QA with the results of civil engineering QA will always be faulty. Comparing each other's procedures to find ways to improve is still a good exercise though.