Court Orders Breathalyzer Code Opened, Reveals Mess

Death Metal writes with an excerpt from the website of defense attorney Evan Levow: "After two years of attempting to get the computer based source code for the Alcotest 7110 MKIII-C, defense counsel in State v. Chun were successful in obtaining the code, and had it analyzed by Base One Technologies, Inc. By making itself a party to the litigation after the oral arguments in April, Draeger subjected itself to the Supreme Court's directive that Draeger ultimately provide the source code to the defendants' software analysis house, Base One. ... Draeger reviewed the code, as well, through its software house, SysTest Labs, which agreed with Base One, that the patchwork code that makes up the 7110 is not written well, nor is it written to any defined coding standard. SysTest said, 'The Alcotest NJ3.11 source code appears to have evolved over numerous transitions and versioning, which is responsible for cyclomatic complexity.'" Bruce Schneier comments on the same report and neatly summarizes the take-away lesson: "'You can't look at our code because we don't want you to' simply isn't good enough."

Comments

But does it work?

[will this name work]

Poorly written code is one thing, but does it ultimately work?

Re:But does it work?

[Jason1729]

Does it matter? The real question is "Can a prosecutor convince a computer illiterate judge beyond reasonable doubt that it does ultimately work?".

Re:But does it work?

[Yold]

I read the report earlier, and there are some very valid issues with the source. The first is that in incorrectly averages readings taken, assigning more weight to the first reading than the subsequent ones. It also has a buffer overflow issue, where an array is being written past its end, and even if this results in an error, it goes unreported.

You would have to be a fricken moron not to have a problem with mis-averaging, however in my experiences with law-people, they can be even worse than PHBs.

Re:But does it work?

[gd2shoe]

Good question, but it needs to be reworded. Does it always work for all inputs?

Also important, if it's a poorly written mess, why is the company claiming that it works? I see no indication that they've done due diligence for a device used to convict people. Just because they've never observed it to fail, doesn't mean a thing.

Re:But does it work?

[mea37]

My first thought as well.

Of course, with poorly written code, it's hard to show whether or not the code ultimately works by examination of the code.

Then again, proving that the code works (which should be the standard when the code is analyzed in court) by code examination is very difficult even for well-written code.

Perhaps a better approach would be documented, repeatable testing of the device. When I challenge a radar gun, I get to ask about its calibration documents, but I don't think I get to debate the blueprints from which it was built.

My personal opinion - and before getting on an "innocent until proven guilty" kick bear in mind that I'm not a part of the court system in this case - is that the defense realizes that almost all software systems look awful and are trying to game their way out of a conviction they've probably earned.

That said, if for no other reason then to eliminate such gaming, there should be standards for testing and documenting the proper function of these devices. Any device that can't be calibrated and tested with sufficient certainty should be banned from use as evidence in court. If the device passes the test, then exactly how it does it shouldn't really matter.

Re:But does it work?

[MozeeToby]

I'd be more interested in their test plan and test results than their source code if I were trying to convince a computer illiterate judge of something. Find a missing test case or an uncovered corner condition and you might have a decent case, code that doesn't pass static analysis and is ugly... well that pretty much defines 99% of the code out there.

Re:But does it work?

[vertinox]

Of course, with poorly written code, it's hard to show whether or not the code ultimately works by examination of the code.

Of course it works because it gives an end result instead of an error message.

The question every should ask is "Does it work accurately?" or "Does poorly written code skew the results?"

Can the defense prove that the code was written so haphazardly that it ignores some data or does it round incorrectly like Excel does? These things do and can happen with sloppy code.

That said, if the code is just poorly commended and indented correctly (*wink*) but does the math right and makes sure there isn't a sampling or rounding problem, then it isn't a problem.

Re:But does it work?

[plague3106]

code that doesn't pass static analysis and is ugly... well that pretty much defines 99% of the code out there.

It's more than ugly, it's difficult to maintain. Also, this point is largely irrelevent; 99% of the code out there isn't spitting out a number that says you're guilty of a serious offense.

Re:But does it work?

[wiredlogic]

Regardless of the state of the code no breathalyzer truly "works". None of them can directly detect blood alcohol content. All they do is use a proxy to estimate using the reaction products from your breath. These devices are wholly unscientific. There is no possible way they can derive a credible estimate with a precision of 0.001% or even 0.01%. There is no accounting for body size, type, or metabolic rate. Furthermore these devices can be triggered by more than just ethanol. Chocolate is reported to cause a false reading as well as other common foods. This is why one should always refuse a breathalyzer test even if you haven't been drinking. There are too many chances for a false positive to risk it.

Re:But does it work?

[internerdj]

Also it looks like their out of range error scheme was to set it to the closest legal value and report it if it was recurring and continuous. Assume for a moment you took a test right after the last good reading, you took 32 samples. It would only report an error if all 32 samples failed. Otherwise 31 of the 32 will report the maximum legal extreme closest to that reading. Couple that with the fact that the averages were taken incorrectly, this isn't just reasonable doubt it is worse than using a RNG to find if they are drunk.

Re:But does it work?

[Volante3192]

If I read the report right, they coded the thing to never actually fail in the first place. It'll always return what can be passed off as a legitimate answer.

Re:But does it work?

[digitalunity]

Looks like the answer is no. It's a black box that doesn't report internal errors except when it can't ultimately decide on an answer.

The source code is useful only for showing the machines can be unreliable in certain circumstances, but unless he has substantiating evidence to show it gave an incorrect result he is unlikely to prevail.

Example: Guy blows .09 after drinking 2 beers. He might have a case that the machine was wrong. Example 2: Guy drinks 8 beers and blows .18. Machine might be wrong, but even if it was off by a bit due to rounding averages, he's still guilty as sin.

Sucks, but that's just the way the law looks at it.

Someone mentioned earlier that the weighting of samples under repeat tests give weight to the first blow, which is a big red flag. The initial blow is probably the sample most likely to be contaminated by liquid from the mouth which will skew the reading dramatically, leading to higher BAC's than actuality. If someone blew a .12 and then a .07 on the same machine, he could be found guilty but it's possible the second sample is more accurate.

Re:But does it work?

[Carnildo]

Perhaps a better approach would be documented, repeatable testing of the device. When I challenge a radar gun, I get to ask about its calibration documents, but I don't think I get to debate the blueprints from which it was built.

Calibration and testing won't reveal all the edge cases that might cause errors. Consider a radar gun designed to take the average of five samples. You've got a car moving away from you at 70 MPH, and a duck flies into the beam for one sample, moving towards you at 5 MPH. This gives the following five samples:

70 70 70 -5 70

I can see a way that badly-written code would turn that into an average speed of 106 MPH (storing a signed char as an unsigned char, which would turn the -5 into a 251), and yet it would pass calibration and every test someone's likely to perform.

Re:But does it work?

[DnemoniX]

Well the problem with calculating the averages should honestly be enough to get this tossed. The defense can put up an exhibit with a set of numbers using the flawed methodology which shows a person to be over the limit. Then call an expert witness with a math degree, or an accountant for that matter. Show that the average when calculated normally is below the the legal limit. Even better is if you can show that the machine has calculated an average that falls below the legal limit but should have been above. That would show the device to be a public safety hazard for clearing persons who are actually impaired.

Re:But does it work?

[Hognoxious]

There is no accounting for body size, type,

There's no need to. It measures a concentration (per unit volume), not how many beers you drank.

or metabolic rate.

Irrelevant. Just because you'll sober up quicker doesn't mean you aren't drunk now.

Try not to hit anybody on your way home.

Re:But does it work?

[Yold]

In Minnesota (and other states), it is a crime to refuse a roadside breathalyzer test due to "implied consent" laws.

Re:But does it work?

[fracai]

Presuming it's the same summary that I read, it contained a mistake.

Readings are Not Averaged Correctly: When the software takes a series of readings, it first averages the first two readings. Then, it averages the third reading with the average just computed. Then the fourth reading is averaged with the new average, and so on. There is no comment or note detailing a reason for this calculation, which would cause the first reading to have more weight than successive readings.

This actually places more weight on the final reading, not the first.

Re:But does it work?

[Man On Pink Corner]

The code in question, among other things, calculates an arithmetic mean of a sequence of values by successively averaging each value with the mean of all the previous ones, and reduces 12 bits of precision coming from the hardware sensor to 4 for some unspecified but undoubtedly stupid reason.

Well, it's not hard to imagine why they throw away all those bits. Prospective LEO customer: "Wow, this thing never gives the same reading twice. How am I supposed to secure convictions with numbers this flaky?"

The averaging thing, on the other hand... that's just a dumbass bug. One that's probably wrecked a few peoples' lives.

Re:But does it work?

[erroneus]

Whether or not it "works" isn't quite enough in my opinion. It needs to be clearly written in such a way that the purpose and methods used in sampling input from hardware and the making of calculations are verifiably accurate and true in all cases. This is an instrument that measures whether or not someone is within a prescribed legal limit and needs to be as provably clear and accurate as possible. We are talking about taking away freedoms from people as a result of this test machine and there should be as little room for error as possible.

If I were to prescribe a system for analyzing breath for alcohol content, I would require that a single test unit be comprised of two machines from two different manufacturers and that any single sample be split equally between the two machines for measurement such that when both machines return results and are both in agreement within a prescribed "reasonable" difference from one another, then we might begin to say we have a reasonably accurate measure from which judgements can be reasonably made.

In the mean time, software architecture needs to be held to the same legal standards as ACTUAL architecture and engineering. I recall being involved in a cabling project where all terminations were reading perfectly, but when I inspected the raceways, the bed radius of the cabling was way too tight and much of the cable was tied to various pipes and conduits and not fixed to the hardware intended for the handling of the cable. All of the cabling was not installed according to clear and complete specification and I was furious at what I found. The first answer offered to me was "but it all works right?"

If you took your car in for repair and they charged you the full price of the repair with parts and you found that it was repaired with duct tape and bailing wire, would you accept "but it works!" as a reasonable answer to your complaint? I think not!

Back to this situation: "Does it work?" The real answer? If you cannot read the code and make clear sense of it, you cannot prove that it works, only that it works under the practical conditions of testing. That is simply NOT good enough for any scientific measurement and especially not good enough for measurements that may be used to determine whether or not a person is sent to prison.

Re:But does it work?

[JCSoRocks]

I'm not generally someone that insists everything needs to be open source. However, in a situation like this, where this device makes the difference between a life changing conviction and exoneration, it's pretty obvious that people should have the right to examine it. The court was able to order it opened here, but it makes you wonder how many people have been screwed by this.

Sadly in the majority of cases where evidence based on something like this (DNA, hair analysis, etc) is shown to be based on someone or something that's not good - nothing comes of it. I saw a blurb about a "forensic expert" that would give the prosecution any testimony they wanted. The state he was based in refused to reexamine the cases he was involved in even after he was shown to be a liar.

It's depressing but it's one reason I steer clear of the law as much as I can. As much as we Americans like to think of our legal system as dispensing justice, the sad fact is that it frequently doesn't.

Re:But does it work?

[mea37]

Yes, but to GP's point - if the code had been subjected to proper tests, then it wouldn't matter how hard it was to maintain. Either the maintainers overcame that difficulty and it passed the test, or they didn't and it failed.

Re:But does it work?

[sexconker]

Show me a programmer creating "perfectly engineered code", and I'll show you a programmer building up its resume.

Re:But does it work?

[mea37]

"Calibration and testing won't reveal all the edge cases that might cause errors"

Then you aren't testing correctly.

"I can see a way that badly-written code would turn that into an average speed of 106 MPH "

You may need to revisit the legal definition of "reasonable doubt". Being able to contemplate a scenario where the evidence could be wrong is not sufficient to overturn the evidence.

Regardless, if the testers don't know that "an input suffered overflow or underflow" is an edge case they need to test, then they aren't testers.

Re:But does it work?

[Anonymous Coward]

You would have to be a fricken moron not to have a problem with mis-averaging, however in my experiences with law-people, they can be even worse than PHBs

Writing lousy code seems to be a prerequisite for being considered a great and invaluable corporate developer in some organizations. As long as it appears to work correctly who cares whether it actually does? .... well until you get sued for incompetence but the common consensus on that issue seems to be that this is a risk that comes with the territory. If you haven't moved on and covered your tracks properly by the time the results of your scramble to meet those deadlines is flushed out into the open it's your own fault anyway. The older I get the less it surprises me what kind of, utter, utter crap people actually have the gall to charge you money for!

Re:But does it work?

[HisOmniscience]

...programmer building up its resume.

I always knew programmers weren't human.

Re:But does it work?

[Darkness404]

The problem isn't that it can break, the problem is it can return bad readings. For example, a dentist's X Ray machine isn't suddenly going to show cavities everywhere because there is no code in the X-ray. Worst thing with a credit card machine is that it doesn't work, most of the time it doesn't overcharge you or something like that, or if it does a few phone calls will sort it out. Again, the worst thing that happens with fuel injectors is they break, your car doesn't run, you pay a few hundred and get it fixed. Worst thing with stoplights is they break, there is always a human driver who can figure out if all the lights are on red or green and call the police to manage traffic.

Breathalyzers are basically black boxes, there is no human to really check them out. With the code more apt to return false readings then simply break, it is dangerous code, when those readings can be the difference between a crime and a non-crime.

Re:But does it work?

[DeadCatX2]

You are correct. In the biz, we refer to this as an exponentially-weighted moving-average-filter. Recent samples are weighted more heavily than older samples.

y(n) = alpha*x(n) + (1 - alpha)*y(n-1)

The alpha value controls how much of the current input makes it to the output and how much of the old output stays. i.e. with an alpha value of 0.5, half of the new value is added to half of the old value. With an alpha of 0.1, 10% of the new value gets added to 90% of the old value.

This filter is nice because it doesn't require you to remember all the values that you want to average together, but it's a horrible way to get over the inherent noisiness in sensors.

Re:But does it work?

[Tanktalus]

No, writing lousy code is not a prerequisite for being considered great and invaluable. As /.ers are so fond of saying, correlation is not causation.

It's just that if you are a lousy coder, you probably have time for proper amounts of sucking up, while if you're a great coder, you probably are too busy getting things to work properly to concern yourself with interpersonal relationships. And it's easy to see why the former get more promotions than the latter.

Re:But does it work?

[pjt33]

This filter is nice because it doesn't require you to remember all the values that you want to average together

Why would you need to remember all the values? As long as you remember the number of values and their total you're fine.

Re:But does it work?

[bcrowell]

Like most first time offenders, I took a plea deal to avoid significant jail time and paid the ridiculous fine and took alcohol awareness classes. The whole thing was a farce, intended to make money.

When you use words like "farce" and "ridiculous," it makes it sound like you don't want to take responsibility for your own actions. I don't think DUI laws are "a farce, intended to make money." I think they're intended to protect people like me from getting killed by people like you.

Re:But does it work?

[Grishnakh]

Our legal system and government are about as non-corrupt as Mexico's. Ours just isn't quite as blatant about it.

Re:But does it work?

[Timmmm]

You know you don't actually need to do that.

average = (new_value-average)/++n + average;

I think that should work.

Re:But does it work?

[Grishnakh]

You need to get your head out of your ass and learn to read. My point is that any car company putting out engines that run like crap and get poor economy (relative to their competition for the same vehicle type of course) will not last long. I did not say that GM and Chrysler are going down because of crappy engines. In fact, I'd say that every engine out there by any major manufacturer runs just fine these days; I don't think any of them have highly shoddy FI code.

What I said there was that a car company has to do better than selling some [cars] to survive. I.e., you might be able to get a few morons to buy a car that runs like crap, but it's not going to be enough customers to keep the company afloat. Especially in today's economy, a company needs to do well in most aspects of its business in order to survive.

But that doesn't apply to this breathalyzer company, because their customer is the government, and you don't have to even make a decent product when the government is your customer; you just have to secure a contact, maybe with a few bribes. It also helps a lot when the customer (law enforcement) doesn't care about how well your product works, as long as they can use it to make lots of money, even if it's fraudulent. This is the same situation we have with red-light and speed photo enforcement. If you get snapped by a speed camera, and you're sure you were under the speed limit, what are you going to do? Threaten to take your business away from Redflex? Your local police department or municipal government certainly doesn't care if the ticket was in error, because they're making money from you because of it. What are you going to do, vote them out? Good luck with that.

Re:But does it work?

[Thinboy00]

Strange as it looks, "his" is the correct singular, gender-neutral, third person possessive pronoun.

Speak English, please (also removed a wrong comma -- it's not a serial comma because there was no "and", "or", or similar).

Re:But does it work?

[decoy256]

No, our system doesn't dispense "justice", which is why I'm a defense attorney.

Re:But does it work?

[0100010001010011]

The majority of crashes and deaths aren't caused by people that blow a 0.086. They're habitual offenders who blow .25s.

Re:But does it work?

[Toonol]

That may or may not be a flaw. It certainly isn't a true average, but an average isn't the only legitimate way to combine successive data values.

That said, I'm not sure why a rolling average like this would be used for this purpose. Is there a medically sound, scientifically reviewed 'proper' procedure to measure alcohol in breath? I have a fear that this was just thrown together by some programmers.

Re:But does it work?

[drsmithy]

If I were to get pulled over, and knew I was over the now ridiculously LOW limit of .08...

That's not a "ridiculously LOW" limit. Indeed, for most of the world it's _over_ the limit.

Re:But does it work?

[bcrowell]

a first offense can cost the driver as much as $10,000 in fines and penalties alone

A drunk driver could kill me. I don't think $10,000 is excessive as a deterrent. My life is worth a lot more to me than $10,000.

Re:But does it work?

[JackieBrown]

It really should be more subjective somehow...I mean, you can have someone that is severely impared on 2 drinks, and another big fella that can have 4-6 beers in a short period and be perfectly fine.

As much as .08 sucks, I'd rather have that than a cop "subjectively" deciding if I am drunk.

Re:But does it work?

[cayenne8]

"You're an idiot. Blood alcohol is blood alcohol. The big guy can drink a little more than the little woman. Just not as much as you'd like."

Sure, BAC is BAC....but, different people can handle and function to MUCH different levels at given BAC. At .04, someone might be really out of it...but, others at .08 are just barely starting to show any signs of intoxication at all. Some people just have more tolerance than others.

Re:But does it work?

[ClosedSource]

But laws are usually based on the measured value rather then unmeasurable and subjective determination of the driver's ability to function.

Re:But does it work?

[bh_doc]

"Its" referring to a person is almost certainly not correct. "His" is arguably correct, as it is both the male-specific and, historically, gender neutral pronoun. In recent time it has been assumed almost exclusively to be male-specific and its gender-neutral use has fallen out of favour. "Their" as a singular gender-neutral pronoun has since become popular. Welcome to evolving English. http://en.wikipedia.org/wiki/Gender-neutral_pronoun#Historical_solutions

Re:But does it work?

[ppanon]

You can think that you're doing fine because you've gotten good at compensating. For instance, dancers and figure skaters can learn to compensate for inner ear/balance issues from spinning at speeds and duration that would have most people nauseous or throwing up, but the spinning doesn't affect their reflexes. However you don't have to have your cochlear sense of balance feel impaired for intoxication to be affecting your ability to drive. It doesn't take much alcohol for your reflexes and cognitive response to be impaired enough to cause an accident, even if it's not obviously apparent. While there is some variation, the acceptable BAC levels were based on correlation with average results from testing for significant reflex and attention deficits.

You might be one of the outliers, but the odds are much better that you might are one of the myriad of people who delude themselves into thinking that they are outliers because their judgement is impaired. Unless you've actually personally undergone reflex/response testing by a third party in conjunction with BAC testing to judge your personal susceptibility to alcohol, your judgement on the subject after alcohol consumption is unreliable. However your ability to compensate for impairment in normal driving conditions wouldn't save you from an accident in an unexpected situation the way unimpaired reflexes would.

The small restriction on the few outliers is not a high price to pay for the safety of innocents. Nobody says you can't drink or drive, just that you have to exercise some level of personal responsibility and not do both (or for that matter, drive and consume any other drug that impairs your ability to drive safely)

Re:But does it work?

[centuren]

"You're an idiot. Blood alcohol is blood alcohol. The big guy can drink a little more than the little woman. Just not as much as you'd like."

Sure, BAC is BAC....but, different people can handle and function to MUCH different levels at given BAC. At .04, someone might be really out of it...but, others at .08 are just barely starting to show any signs of intoxication at all. Some people just have more tolerance than others.

Police officers, at least in this state, still use their judgement when interviewing and observing someone suspected of driving drunk. For example, you don't actually have to have .08 to be arrested. If you're below .08 but still relatively high, you can be arrested for reckless driving. Obviously you can be arrested for reckless driving without a breathalyser if you are driving recklessly, but it's my understanding they can use their judgement applying that charge to drivers who are intoxicated but within the legal BAC.

Of course, that doesn't mean they'll judge you to be fit to drive if you're over .08, but the line has to be drawn somewhere. Since it is so low for some people when comparing BAC to motor skill impairment, the best advice is always to not drink at all if you know you'll be driving, or not drive at all if you find you've had more than one drink. All the same, I wouldn't be surprised if I found out the average DUI arrest recorded a BAC of well over .1.

Lint is crap

[xenocide2]

Lint, as a static code analyzer, is bound to have false positives. More so in embedded systems, where you're dealing with registers and occasionally "violating" type safety where no type adequately exists. It's really not super surprising that 60 percent of the code is reported by Lint.

Code

[Quiet_Desperation]

not written well, nor is it written to any defined coding standard

Ah, so it's like most of the code in the world.

Re:Code

[Anonymous Coward]

except that this code is used to convict people.

Re:Coding Standard

[kailoran]

Because the output is used as evidence in court?

Good!

[SanityInAnarchy]

Ok, I'm not happy that some people almost certainly were measured inaccurately by these things. I'm not happy that this company was allowed to pull this kind of shit -- when you do government contracting, the government should own what you do.

However, I am very glad that the precedent has been set.

And I am especially glad that not only is there precedent, but there's a real live example of why we need this stuff to be open.

Re:Good!

[Red Flayer]

when you do government contracting, the government should own what you do

But they weren't doing government contracting. The produced a good that was purchased by the government. There's a very big difference.

The key here is not that the government, or anyone, should own what they produced -- it's that when what they produced is used to convict someone, that person has the right to examine the methods used.

It's not about openness, at all. It's about the right to a fair trial; openness is just a side effect.

Re:Good!

[Midnight Thunder]

The key here is not that the government, or anyone, should own what they produced -- it's that when what they produced is used to convict someone, that person has the right to examine the methods used.

I will call out the company for doing shoddy work. The question is whether the device was ever certified for the purpose, and if it was who did it and what was the process used. If you are going to use something to prosecute, then there needs to be evidence that the device was tested and certified using a publicly documented process. This is black box testing and if the government never did it, then why is it allowed in court?

No surprise

[infinite9]

80% of the code in business fits this description. With 20 year old legacy code written by 50 consultants, then upgraded in India, then ported from one platform to another to another, and a database engine switch or two. Code gets senile. What do they expect? Good thing we're all just commodities... human lego bricks easily replaced with cheaper plastic.

Re:No surprise

[legirons]

80% of the code in business fits this description

how much of that code is given police powers to arrest someone?

not written to a coding standard?

[AliasMarlowe]

Just because code is not written to some official standard does not mean it is guaranteed to be buggy. Undisciplined coding is as bad as undisciplined specifications - results can indeed be ugly. It is preferable if the coders follow good practices, and there ideally would be a clear system for specifying program behaviour in testable ways. It is easier to produce good code with robust behaviour if good practices are followed from design through coding to testing and documentation, but it is not impossible to achieve good results in other ways also.
Did they find any coding bugs, or did they just criticize the approach to coding?

Re:not written to a coding standard?

[SanityInAnarchy]

Did they find any coding bugs,

Yes. RTFA.

2. Readings are Not Averaged Correctly: When the software takes a series of readings, it first averages the first two readings. Then, it averages the third reading with the average just computed.

There you go. It's also inaccurate:

The A/D converters measuring the IR readings and the fuel cell readings can produce values between 0 and 4095. However, the software divides the final average(s) by 256... Further, because of an attribute in the IR calculations, the result value is further divided in half. This means that only 8 values are possible for the IR detection...

And, if there were a catastrophic bug, you wouldn't know it, you'd just keep getting readings:

An interrupt that detects that the microprocessor is trying to execute an illegal instruction is disabled, meaning that the Alcotest software could appear to run correctly while executing wild branches or invalid code for a period of time. Other interrupts ignored are the Computer Operating Property (a watchdog timer), and the Software Interrupt.

This belongs on The Daily WTF.

Re:Coding Standard

[SirGeek]

Er, why would it need or be expected to be? It's a commercial product. I don't think most bank websites are "coded" to any specific standard either.

From the article:

1. The Alcotest Software Would Not Pass U.S. Industry Standards for Software Development and Testing: The program presented shows ample evidence of incomplete design, incomplete verification of design, and incomplete "white box" and "black box" testing. Therefore the software has to be considered unreliable and untested, and in several cases it does not meet stated requirements. The planning and documentation of the design is haphazard. Sections of the original code and modified code show evidence of using an experimental approach to coding, or use what is best described as the "trial and error" method. Several sections are marked as "temporary, for now". Other sections were added to existing modules or inserted in a code stream, leading to a patchwork design and coding style.

Ok. Would you want to have something that can cause you to get convicted because it wasn't documented or even tested fully - ("Oh, Crap. That constant should have been 0.001, not 0.01. Ooops. Blood Alcohol level was 0.008, not .08. Sorry !")

Common sense (if it WERE common) should indicate that there should be full tests for a wide range of values performed with the written tests and expected values verified and available to prove that the device/software actually does detect the proper levels of alcohol.

Just remember

[captnbmoore]

This will not stop the state from using this to make a felon of you.

Don't forget the false positives

[Chmcginn]

In general, I was under the impression that the standard for criminal cases were weighted heavily to reject any technique, evidence, or device that had any appreciable chance of a false positive.

Re:No.

[Ohio Calvinist]

The problem in a lot of states is that .01 can make a huge difference between a DUI, a DUI with a "high BAC kicker", a wet-reckless, or nothing at all. It has to be accurate to at least a few 9's or for those "on the bubble" cases do have a severe level of doubt. Because driving with a .07 is not illegal (for the most part), but .08 is. The question in court is not "were you drinking tonight", but "how much did you drink" which is a very specific very objective, very deturminable piece of information.

As states lower their legal limits to the point where they intersect with non-impaired drinking drivers, especially with a .01 or more margin of error, you're going to get a lot of overzealous cops in cities with revenue shortfalls taking innocent people in for DUIs and hopefully more and more of these "border cases" will bring these devices into question more than the over-the-top blacking out, pissing his pants multiple-offender does in court.

A pyhrric victory for open source and code review.

[Darth_brooks]

The good: This particular breathalyzer has been proven to be the unreliable POS that it apparently is. This unit, and others like it, will finally start being held to a stronger coding standard.

The bad: every sleezeball, ambulance chasing, "call lee free", douchebag of a lawyer will use this case to attack the credibility of any and all breathalyzers made in the past, present, or future, spreading enough FUD to juries everywhere that an unacceptable number of drunken idiots get the God given right to keep their license until they finally end up killing someone.

As a person, I think groups like MADD spend most of their time trying to scare monger politicians into pushing us as close to prohibition as possible. I believe that alcohol can be used responsibly. But I also know that this case is going to result in DUI's getting overturned for people that damn sure don't deserve it. Borderline cases will get knocked down, cases will get thrown out, and the people that broke the law, that did something wrong, will walk out of a court room 'vindicated.' They didn't do anything wrong when they had six beers and drove home, it was that confounded *machine* that *said* they broke the law. The *machine* was busted, ergo they didn't break the law. In short, this case is going to make a lot of O.J. Simpson's. The jury said they didn't commit a crime, so they didn't. No harm no foul. Technicality? Bah! They're as innocent as the sweet baby Jesus.

I'd like to think things will wash out in the end. This case will probably end up making it harder to get off on this particular technicality in the long term. In the short term? Here come the appeals. Maybe the state is partially at fault for buying shoddy equipment. (Or maybe not. Did they do a code review? Do they have the resources to one? Probably not. Did you do a code review of the 3com switch in your server room? Their selection criteria can certainly be questioned, but it probably doesn't change the fact that someone drank enough to blow a .22 then decided to drive home.)

But in the end, the drunks are still going to be drunks. And tomorrow some of them will probably get to file appeals, and some of the ones that shouldn't be on the road, or even in public, will get to slip out of this brand new loophole. I'm not sure that that deserves a cork-popping celebration.

(and yes: We all handle our booze differently. Arbitrary limits that determine "drunk" may or may not be the answer. Hardcore drunks will keep driving even after losing their license. DUI's are as much moneymakers for the States as speeding tickets. Yadda yadda yadda.)

Re:No.

[FatAlb3rt]

Do you really think that you should receive the same consideration as a guy that's 3x over the limit? Blowing .08 and .18 are quite different in terms of state of mind.

Definitely possible that there's no false negatives, but for a device that can have such this level of an impact on someone's life, you'd think it would be held to a higher standard.

Re:No.

[Lord Ender]

In embedded systems programming, it is common practice to disable interrupts if they are not used. It is certainly possible that this app simply does not need to handle these interrupts, whether they are enabled or not.

It is also possible that the other flaws mentioned, which clearly reduce accuracy, do not do so sufficiently to change the outcome in a meaningful way.

The problem with drunk driving law is not primarily one of testing. It is that it presumes someone is incapable of driving with even trace amounts of alcohol, while treating other forms of more dangerous driving (such as driving while texting or on the phone) as being OK or far far less severe.

The way the laws themselves are written is a horrible miscarriage of justice. This is the result of the perverse and hypocritical views of MADD and its ilk, the bastard children of the prohibition movement.

Re:And does it work?

[legirons]

Perhaps they're coded inelegantly or poorly, but do they actually spit out inaccurate numbers?

Irrelevant - the test is: do they always spit-out provably correct numbers?

Re:"Reveals Mess"

[cdrguru]

The code is protected in the US by copyright. It is not protected anywhere else, especially in countries where it is cheap to reproduce the hardware. US Customs has proven over and over they will not block the import of infringing devices.

This means that once the software gets out - and it is - look for cheap copies that will put the original manufacturer out of business. Because law enforcement and just about everyone else in the market for such devices is going to jump on the price difference. Same functionality for 1/10th the price.

Do not believe for a second that there are any safeguards left for this sort of thing. There are not.

Re:Coding Standard

[0100010001010011]

An example of a DUI conviction using 5 numbers (Assuming 0.10 for simplicity):

Say the breathalyzer gets 5 numbers: [0.0625,0.0625,0.1,0.12,0.15].

If you average the numbers you get 0.099. However if you do a 'rolling' average, you get 0.13. Quite a bit of difference (not to mention, legal vs not legal).

Now imagine that last number was a burped packet of gas that had a ton of alcohol in it and was a 0.25.

Instead of the average being 0.119 (yes still illegal) it's now .18, at which point you hear on the news "he was nearly twice the legal limit!".

(Although playing with the numbers, it looks like the site is wrong, it incorrectly weights the later numbers because one wild number can instantly pull it up very high, or low.)

Re:DUH....

[shutdown -p now]

The fault is the State using output of a device which is an undocumented, unverified black box in legal proceedings.

Yes, of course, most of code out there is a similar mess. But if it fails, the worst that can happen is that your desktop crashes, or your iPod hangs... which is bad, of course, but not as bad as getting a criminal conviction for drunk driving.

These things should be held to the same standards as code in military equipment or nuclear reactors - mistakes are inexcusable.

Re:DUH....

[Tablizer]

the fault is the Executive staff that refuse to listen to their experts (programmers) and do what they recommend. Instead we get morons that know nothing about programming making unrealistic deadlines and forcing death march coding marathons to give up the mess we have today.

The US's comparative advantage is speed-to-market, for good or bad. Any service or product that becomes a stable commodity flows to lower-wage countries. Thus, churn-and-burn is the order of the day in the 'merica's.

Now if you could show through some kind of statistical analysis that companies that spend more time planning are more profitable, you may get more listen. I've worked with some crappy code for big, well-known companies that otherwise are financially successful. Thus, bad code and practices are not dooming them (although it would give a sense of satisfaction if it did).

One required 12 programmers because it was a combinatorial mess of factors being reported. They said they tried to use a meta-programming approach once before, but the programmer, who was otherwise well-regarded, got confused. They just instead switched to an army to produce the copy-and-paste combinations. They should have perhaps recruited somebody better at meta-techniques, but that was outside of their familiarity zone. Rather than experiment in their unfamiliar meta-land, they decided to byte the bullet and hire the army. I almost felt like building a demo at home and then showing them. Sometimes that's what it takes to break thru the status-quo.
     

Re:No.

[johannesg]

Readings are Not Averaged Correctly: When the software takes a series of readings, it first averages the first two readings. Then, it averages the third reading with the average just computed... There is no comment or note detailing a reason for this calculation, which would cause the first reading to have more weight than successive readings.

Maybe that is the intention? Just because Schneier *thinks* it is an average, doesn't make it so. Maybe the device becomes more accurate as more samples are taken, and therefore gives more weight to the last (not the first!) sample.

The A/D converters measuring the IR readings and the fuel cell readings can produce values between 0 and 4095. However, the software divides the final average(s) by 256, meaning the final result can only have 16 values to represent the five-volt range (or less), or, represent the range of alcohol readings possible. This is a loss of precision in the data; of a possible twelve bits of information, only four bits are used. Further, because of an attribute in the IR calculations, the result value is further divided in half. This means that only 8 values are possible for the IR detection, and this is compared against the 16 values of the fuel cell.

Who cares if there is loss of precision? In the end, it spits out a value that is essentially binary: drunk or not. You are not less drunk if it says so with 9 extra bits of precision, and in fact the extra accuracy could itself be considered an error as it shows a certainty about the result that may not be warranted by the design of the machine.

Catastrophic Error Detection Is Disabled: An interrupt that detects that the microprocessor is trying to execute an illegal instruction is disabled, meaning that the Alcotest software could appear to run correctly while executing wild branches or invalid code for a period of time. Other interrupts ignored are the Computer Operating Property (a watchdog timer), and the Software Interrupt.

If the code is tested properly, there is no need to keep the first two interrupts going. As for the "software interrupt", it may sound ominous that it is disabled, but there is absolutely no way to tell if and why it should be enabled, and in fact disabling it is probably correct. There is absolutely nothing in software engineering that says you should always enable all interrupts because otherwise your code would be less reliable.

Besides, a typical result of executing illegal instructions is that the device hangs or reboots. Since alcohol testing devices don't do that (to the best of my knowledge), it appears that disabling those interrupts does not cause any harm.

So, basically, it's designed to always return some value, even if it's wildly inaccurate, and even if the software is executing garbage at the time.

In other words: It appears to be a very low-level equivalent of Visual Basic's "on error resume next".

That conclusion is entirely unwarranted. It appears to be designed to provide a weighted average, not show undue accuracy, and is sufficiently well-tested that it does not need emergency measures like the illegal instruction interrupt. In other words, even though the software may look messy it is working fine.

Re:DUH....

[D Ninja]

the fault is the Executive staff that refuse to listen to their experts (programmers) and do what they recommend. Instead we get morons that know nothing about programming making unrealistic deadlines and forcing death march coding marathons to give up the mess we have today.

To some extent, you are correct. However, I also blame the developers. There are many "software engineers" and "computer scientists" I have worked with who didn't understand the basics of algorithms, design, testing, and other topics that are necessary to our field.

Re:Is this 1968?

[DeadCatX2]

With an attitude like that, it's obvious that you have little experience with embedded systems...

Re:No.

[MadShark]

> In embedded systems programming, it is common practice to disable interrupts if they are not used. It is certainly possible that this app simply does not need to handle these interrupts, whether they are enabled or not.

There is rarely a good reason to shut off the interrupt for an illegal instruction if it exists on your micro. It is entirely possible for a stray bit of electromagnetic radiation(cosmic ray, electric motors turning on or off, etc) to flip a bit in the micro, causing an illegal instruction. The illegal instruction interrupt exists for situations like this. It should be caught and handled in an appropriate manner, ESPECIALLY in safety critical or (as in this case) legal evidence gathering applications. I've seen it happen at work when we do our electromagnetic interference testing.

Re:No.

[osu-neko]

Maybe that is the intention? Just because Schneier *thinks* it is an average, doesn't make it so. Maybe the device becomes more accurate as more samples are taken, and therefore gives more weight to the last (not the first!) sample.

It damn well better be an average -- having worked with cheap, 12-bit ADC chips before, I know you're getting trash for data if you aren't taking multiple readings and averaging. You must average the readings because the readings are noisy, particularly "in the field". The point of the averaging is to get rid of the noise. The noise doesn't go away as more samples are taken. The average needs to be done properly across the range because if you're giving greater weight to the last reading, your failing entirely to eliminate the noise.

I'm certain it's throwing away that least-significant bits from the 12-bit ADC precisely because they're effectively RNG output. The problem is, that's the wrong way to do it. Keeping the entire value and then averaging properly gives a reasonably accurate value, even from a noisy ADC. You can discard the lower bits after the averaging (it's false precision anyhow), but not before, and you do it by rounding, not truncation. Doing what is described in the article gives you trash.

Re:Coding Standard

[colinrichardday]

Testimony is subject to cross-examination (at least in the US). Opposing counsel has the opportunity to exploit weaknesses in the witness's testimony. Also, the witness is subject to prosecution for perjury for lying. What penalty does a faulty (if it be faulty) device face?

Re:Is this 1968?

[SnowZero]

Assuming the microcontroller has a 10-bit A/D converter to get the reading, I'm pretty sure such a chip could add 32 numbers together. With the speed of 8-bit microcontrollers these days exceeding 1MHz even at ~$1 price points, emulating 16 bit numbers to get your sum is not a problem. Take a power-of-two number of readings and your average can be a simple bit shift. It will take more horsepower to convert to base-10 on the display than to take the average.

This is not a cheap child's toy or a toaster, it's a law-enforcment grade breathalyzer going for above $100; there is no excuse for being so lazy. Code that runs on small systems should be *clean* because bugs are harder to find without easy I/O, and the efficiency of it needs to be obvious. Also, code that can put someone in jail should not be spaghetti, regardless of the scale of the system running it.