Slashdot Mirror
OpenBSD's De Raadt Pans 'Incredibly Bad' Disclsoure of Intel CPU Bug (itwire.com)
troublemaker_23 quotes ITWire:
Disclosure of the Meltdown and Spectre vulnerabilities, which affect mainly Intel CPUs, was handled "in an incredibly bad way" by both Intel and Google, the leader of the OpenBSD project Theo de Raadt claims. "Only Tier-1 companies received advance information, and that is not responsible disclosure -- it is selective disclosure," De Raadt told iTWire in response to queries. "Everyone below Tier-1 has just gotten screwed."
In the interview de Raadt also faults intel for moving too fast in an attempt to beat their competition. "There are papers about the risky side-effects of speculative loads -- people knew... Intel engineers attended the same conferences as other company engineers, and read the same papers about performance enhancing strategies -- so it is hard to believe they ignored the risky aspects. I bet they were instructed to ignore the risk."
He points out this will make it more difficult to develop kernel software, since "Suddenly the trickiest parts of a kernel need to do backflips to cope with problems deep in the micro-architecture." And he also complains that Intel "has been exceedingly clever to mix Meltdown (speculative loads) with a separate issue (Spectre). This is pulling the wool over the public's eyes..."
"It is a scandal, and I want repaired processors for free."
In the interview de Raadt also faults intel for moving too fast in an attempt to beat their competition. "There are papers about the risky side-effects of speculative loads -- people knew... Intel engineers attended the same conferences as other company engineers, and read the same papers about performance enhancing strategies -- so it is hard to believe they ignored the risky aspects. I bet they were instructed to ignore the risk."
He points out this will make it more difficult to develop kernel software, since "Suddenly the trickiest parts of a kernel need to do backflips to cope with problems deep in the micro-architecture." And he also complains that Intel "has been exceedingly clever to mix Meltdown (speculative loads) with a separate issue (Spectre). This is pulling the wool over the public's eyes..."
"It is a scandal, and I want repaired processors for free."
You know, he's not wrong. This is, in impact, way bigger than Intel's FDIV fiasco and that ended up in recalls.
Also it might be possible that he was intentionally left out due to trust issues after patching the Krack attack and thus disclosing info about it prematurely.
That doesn't explain why FreeBSD wasn't notified until 5-6 months after Intel and ARM knew about the issue and until after Apple had shipped a patch. It also wasn't helped that there was no real coordination in releases. Apple shipped a binary update and there were patches in the Linux tree containing mitigation before the official end of the embargo period.
I am TheRaven on Soylent News
This is a question of quality, not idealism and perverse incentives.
We aren't talking about IME here. You seem to be blindly assuming that Open hardware is always free of faults.
Not sure about others but some are available for purchase.
"SiFive has declared that 2018 will be the year of RISC V Linux processors" - Linux Now Has its First Open Source RISC-V Processor, Slashdot.
To answer AC's question a few moths later: "What's the big advantage with RISC over ARM or x86?"
Meltdown, Spetre.
Funny, both me and my friend worked at companies where we were told to ignore risk. Why would intel be different?
Avantgarde Hebrew science fiction
This has been extremely worrying. What's more worrying are the number of 'security researchers' regurgitating Intel's bullshit verbatim. We have yet to fully see the fallout from this.
He's also dead right in that Intel has been mixing up the two issues, Meltdown and Spectre, deliberately, so they could tell everyone that it wasn't just Intel that was affected, and they also gave the impression that Spectre had been fixed when it was Meltdown that had been mitigated - with a patch that creates unacceptable performance problems, to a lesser or a greater extent.
Yes, all processor manufacturers are affected by Spectre, but it is Intel that is mostly affected because they implemented speculative loads badly without much attempt at segregation. They've also attempted to pass this off as 'historical architectural decisions we can do nothing about, but it is working as designed'.
>is hard to believe they ignored the risky aspects. I bet they were instructed to ignore the risk
The specific issue that Pentium line CPUs: a) do privilege check asynchronously; b) do it only for the "winning" execution branch was very well known among CPU design community.
Intel architects even bragged about that as their "innovation" in industry journals and filled a number of patents for that (this is the reason amd privilege checker runs on all branches)
Intel makes a monumental decision to benefit the short-term interest of their corporation at the long-term expense of their customers, then tries to weasel out of a equitable fix for their customers? It's not only their product that can't be trusted, it's their judgement at all levels. Heads need to roll at Intel for this....
Everything in the Universe sucks: It's the law!
"It is a scandal, and I want repaired processors for free."
And I want a pet unicorn. Come to think of it, unicorns are about as real a thing as a "repaired processor" since they physically cannot be repaired. He wants a replacement processor which almost certainly is never going to happen. Basically he's asking for every processor produced in the last 20 years to be replaced for free. If you think that's realistic I've got a bridge to sell you.
There will be plenty of legal action over this and the results of that will be the full extend of any compensation. Furthermore to get compensation he will have to show actual harm incurred. Simple fact is that at least so far there has been little to no tangible harm from this problem to date so standing will be an issue for anyone who sues. This might change in the coming months/years but until it does the chip makers aren't going to pay a dime to replace anyone's chip - flawed or otherwise.
Open Hardware doesn't fix problems in silicon that has already been manufactured. It might help with the next generation but it won't prevent bugs from appearing in the first place.
Bear in mind that the reason Open Source software works so well is that the marginal cost of (re)production is close to zero and that there are (comparatively) minimal capital costs. Really you just need a PC and a lot of time. Open Hardware is a worthy goal but it's going to be a LOT trickier to pull off in the real world for mostly economic reasons. Furthermore hardware isn't protected by copyright for the most part. It's protected by patents and those are expensive. Worse once someone has one on a piece of kit they can basically shut down any open hardware that uses that idea for the next 20 years.
Sure it does. If you want to keep something quiet until you are ready to announce it, then you DO NOT tell any of the people who have a track record of spilling the beans. Regardless of where you personally stand on the idea of embargos and standing up for principles, Theo refused to go along with an embargo previously and it was quite likely that he wouldn't do so this time either. Google's Project Zero team presumably had discussions with Intel and select others they felt they could trust about what was required to address the problem and how long it would take, and that group collectively agreed on the original release date of January 9th, plus who else to notify and when. Clearly that larger group did not include anyone in the BSD camp.
Standing up for your principles can have a cost attached, and I suspect we've just seen what that was for Theo and the BSD developers.
UNIX? They're not even circumcised! Savages!
"... I want repaired processors for free."
So do I. I want backdoor-free processors without payment. I will send Intel the faulty processors.
Intel CPU Backdoor Report (Jan. 1, 2018)
My opinion: Intel is a world-class company, with poor top-level management. Brian Krzanich is not the kind of person who is necessary. He is not a person with enthusiasm for technology combined with the social ability to lead a large company. One story about Krzanich: Intel CEO sold all the stock he could after Intel learned of security bug.
Paul Otellini, the previous CEO, was worse, in my opinion. Otellini "joined the finance department in 1974" I complained about Otellini 11 1/2 years ago in a Slashdot comment: More Intel employees should say in public what they have told me in private: Intel CEO Paul Otellini is not a competent leader. He lacks social ability. (June 09, 2006)"
Intel's health and strength is important to everyone on the planet, it seems to me. The technological part of the company can be excellent, but recent top management has not been able to handle the challenges.
The underlying issue, it seems to me, is that the process of choosing new CEOs tends to be defective. Perhaps all employees should have 50% of a vote, with the board of directors having 50%.
It also wasn't helped that there was no real coordination in releases.
There was, the problem is exactly what they were trying to avoid. It was under non-disclosure back in November. The actual full disclosure date wasn't supposed to be until tomorrow (9th Jan) so coordinated releases could take place. Unfortunately someone jumped the gun and we're left with the clusterfuck that this has become (Microsoft patches released, Apple patches released, UEFI patches not ready yet, Linux patches scheduled for original date).
This one of the very few downsides of Opensource. It's difficult to release bug information under NDA as efforts to patch the bugs result in the bug being revealed. That and Theo shat the bed with his past irresponsible patching of a bug early that was covered under a coordinated release schedule.
Secrets are only kept when few people know about them.
ARM is a RISC architecture, and plenty of RISC architectures suffer from Spectre. Meltdown is an Intel-only bug -- AMD doesn't have it because they implemented an obvious security rule, and presumably Cyrix and other x86 implementations didn't either.
I dont care how much better their next CPUs might be, im jumping ship for AMD on my next upgrade. I did the same after NVidia fucked me over.
Why did OpenBSD silently release a patch before the embargo?
OpenBSD announced an errata on 30 August 2017 that silently prevented our key reinstallation attacks. More specifically, patches were released for both OpenBSD 6.0 and OpenBSD 6.1.
We notified OpenBSD of the vulnerability on 15 July 2017, before CERT/CC was involved in the coordination. Quite quickly, Theo de Raadt replied and critiqued the tentative disclosure deadline: “In the open source world, if a person writes a diff and has to sit on it for a month, that is very discouraging”. Note that I wrote and included a suggested diff for OpenBSD already, and that at the time the tentative disclosure deadline was around the end of August. As a compromise, I allowed them to silently patch the vulnerability. In hindsight this was a bad decision, since others might rediscover the vulnerability by inspecting their silent patch. To avoid this problem in the future, OpenBSD will now receive vulnerability notifications closer to the end of an embargo.
09F91102 no, 455FE104 nope, F190A1E8 uh-uh, 7A5F8A09 that's not it, C87294CE no. Ah! 452F6E403CDF10714E41DFAA257D313F.
You do realise that OpenBSD and FreeBSD are two different entities, right?
This is a question of quality, not idealism and perverse incentives.
We aren't talking about IME here. You seem to be blindly assuming that Open hardware is always free of faults.
This is a question of quality. You seem to be blindly assuming that starts and ends with hardware faults. It does not, and it was the main point Theo was making here. Quality also has to do with how you handle faults when they happen.
And I'd sure as shit trust an open community a lot more than a proprietary closed one hell-bent on protecting profits at all costs. How many more bugs does Intel know about right now that they refuse to disclose because it might affect stock price? I rest my case.
Sure it does. If you want to keep something quiet until you are ready to announce it, then you DO NOT tell any of the people who have a track record of spilling the beans.
When has FreeBSD ever disclosed a security vulnerability under embargo? FreeBSD has a security officer and a secteam group that are the only ones that have access to any embargoed security information and have separate infrastructure from the rest of the project for preparing fixes. Only people who have signed the relevant NDAs are allowed access to anything shared with this group and they are normally given information about embargoed security issues as a result.
Regardless of where you personally stand on the idea of embargos and standing up for principles, Theo refused to go along with an embargo previously and it was quite likely that he wouldn't do so this time either
You do realise that FreeBSD and OpenBSD are entirely different projects, run by different people, with different infrastructure and different codebases and that Theo De Raadt has no connection to the FreeBSD project?
I am TheRaven on Soylent News
To quote Linus "A *competent* CPU engineer would fix this by making sure speculation doesn't happen across protection domains."
That's bullshit. When Intel introduced speculation across protection domains everyone including Linux was happy because it reduced system call costs. Without this, as soon as you get to a syscall / sysenter instruction, you stall the pipeline until all pending instructions have been committed. On a modern Intel CPU with close to 200 instructions in flight at a time, that's a measurable performance overhead.
We've known for a long time that side channels of this kind were possible, but not that they were performant. The new attacks are not interesting because they're side channels that allow data to be disclosed, they're interesting because they're side channels that allow disclosure far faster than previously believed. CPU designers believed that this kind of attack could only be exploited to get a bit every few seconds, at which rate it's not really worth trying as an attack and is pretty easy for software to spot (hmm, why is this thread at 100% and triggering insane numbers of cache misses? Looks malicious...). Now we know that you can use these attacks to get data at about 0.5MB/s, so you can scan the whole of memory in a few minutes.
I am TheRaven on Soylent News
I was one of those who called "no way" at first, but just yesterday I found this quote from an Intel engineer. It was originally posted in a reddit thread but has since been deleted - but not before being confirmed by other former engineers at Intel.
"We mustn't be caught by surprise by our own advancing technology" -- Aldous Huxley
Wider disclosure of a vulnerability means a greater chance that it's leaked to those who would exploit it for malicious purposes.
This has a very poor underlying assumption: White hats always know about exploits prior to Black hats. Considering the latter are far better funded, have better incentives, and don't disclose; I think the betting is that it is already used for malicious purposes.
Additionally, this is the information age, not the dead tree, secret squirrel, stamped "for your eyes only" days. In this age, does it really matter if 100 or 10000 people know a confidential piece of information? You only need ONE individual in that set to provide the information to malicious actors. Additionally, there are a multitude of incentives and multitude of avenues for sharing with little chance of getting caught.
Given the above, full disclosure is far better than limited disclosure.
Also, due to the nature of the BSD license, it's code is in a ton of software around the world. The assumption that it isn't widely used is wishful thinking of the unknown.
Risk isn't just a matter of how likely the discovery is, but how serious it is and how widespread the negative impact is. Given that Meltdown affects a huge population, even a tiny chance of discovery represents a huge risk, Intel doesn't want to do a recall since even chipzilla would be sunk by the cost. There's a big risk for you. Next time you consider Intel, just remember, you bought intel before and right now they're pointing at you and saying "HA-HA".
Care to spin the wheel again?
When you look at the actual affected processors vs. the number of people who will actually get off their ass and make a claim to have a processor replaced for free, it becomes very clear how affordable this really is.
A good point but not as good as you make it seem. Quite a lot of affected chips are not user serviceable for reasons such as being soldered to the board. This means that it isn't a simple matter of sending a customer a new chip. In many cases the boards in question won't be available for any price so the most a chipmaker could do would be to pay for a replacement. A repair simply isn't going to be an option in many cases.
But you are right, 99.99999% of people will probably have forgotten about this in a week and the only people concerned about it are the 5 or so of us who still read slashdot. Most people won't have even heard about the problem and most of those who do wouldn't know how to deal with it anyway. Basically it will go away when the current generation of PCs is retired in a few years.
If I may, I'd have to call this an anecdote rather than a quote. The description is from years after the Intel meeting, and doesn't have direct quotes of speech or writing of the personnel involved in the policy change.
With that understood about the anecdote's provenance, it is _completely_ believable. It is precisely the sort of mandate that can save a company in the short term, preserving the jobs and careers and technological development the company is doing, at the risk of a deadly failure down the road. It's the sort of business risk assessment that occurs on an annual basis when testing standards and guidelines are set. It also occurs on a daily basis when security practices are created: do we accept the risk of a breach today, while this is unpatched, versus the risk of service failure or loss of business during system updates?
No, it's not tricky to pull off.
If it wasn't tricky to pull off then it would have already been done on a wide scale. I'm not saying it's impossible but it is going to be a much tougher nut to crack than open software. Mostly for economic reasons rather than technical ones.
- Research and make use of expired patents extensively, file new ones defensively.
Who is going to do this? Who has the funding and more importantly the incentive to do this? IBM received 8000 patents in 2016 and numerous other tech companies received thousands more each. Exactly how do you plan to match that sort of pace? How do you plan to produce anything really useful without infringing on a pile of those patents? Not to mention fending off the flesh eating lawyers that give those patents teeth...
It's more capital intensive than software, but it's also not that expensive either.
I'm a certified accountant and an industrial engineer. I do cost accounting for a living. It is a LOT more expensive than software no matter how clever you are. There is a reason gross margins in manufacturing hardware are far thinner than in software. You don't escape these costs by just doing design either. Someone eventually has to make the product and that will require substantial capital. Then you have the cost of distributing the product. Unlike software which can be sent across the net for nearly free, hardware has to be shipped, stored and turned into products, all of which cost non trivial amounts of cash. If you think it isn't substantially more expensive than making and distributing software you haven't done the math.
Of course, when Linux was new the argument was that an OS was just too big for a bunch of Free Software fans to manage.
You are making a false equivalency here. Making and distributing software is COMPLETELY different than making and distributing hardware. The economics could not be more dissimilar. The legal protections (patents vs copyright) are different. The amount of up front capital required is different. You can modify software after it has been release but you cannot do that with (most) hardware. Basically just because it worked out well for software is does not mean it will work out well for hardware. Hope for the best of course but it's likely to be a difficult nut to crack.
Only a big corporate structure could support development of anything as complex as an OS.
Ultimately that turned out to be true. Basically all the developers of linux and most other major OSS projects are employed at large tech firms (and a few large foundations) and are paid to maintain them. It isn't a bunch of hobbyists in their garages.
Open hardware is harder, but probably not impossible.
Not impossible but for non-trivial applications it appears pretty close to it. The obstacles are predominately economic ones and some legal ones and they aren't minor obstacles. I'm not about to hold my breath for patent reform anytime soon and the patent swamp is a real problem. And the economics of making and distributing hardware are immutable. I think Open Hardware is a very worthy goal but it's going to be quite the challenge.
Yes, I do realise that - but it doesn't change the situation that *all* the BSD devs were kept out of the loop regardless of which family they work on. If you consider that there is still a lot of overlap between the two teams in terms of the code and contributors, then it would be kind of hard for FreeBSD to be making changes and testing them - especially if they wanted to roll them out publically like Linux did - without at least some of the OpenBSD devs either being in the loop or figuring out that something was up - and that then becomes a risk of premature and uncontrolled disclosure. Yes, it's possible that the FreeBSD security team could have kept everything under wraps, but from Google's point of view that's not the question they would have been considering; they'd have been considering how likely it was each potential insider was to result in a breach of the embargo. We now know FreeBSD wasn't in the loop, and we have to think they might consider that Theo might be a problem for maintaining the embargo, so regardless of whether they just treated all of BSD as a single entity or that the "Chinese walls" between the two teams wouldn't be strong enough, the end result is the same - a lot of BSD users, in all families, in a race to secure systems before the exploits hit.
UNIX? They're not even circumcised! Savages!
Funny, both me and my friend worked at companies where we were told to ignore risk. Why would intel be different?
Because all companies are not identical? Do you think everything you and your (imaginary?) friend have in common is also common to every company?
Also, what do you mean by "risk"? Do you think it's reasonable (or even possible) to eliminate all risk of a security breach? When did anyone accomplish that?
Was this attack known, and deemed an acceptable risk because of the incredible low rate at which data could supposedly be extracted?
Not this specific attack, but it was known that any source of nondeterminism in a processor was a source of side channels. These were largely ignored because these attacks get lots of papers at top security conferences but are really hard and slow to carry out in practice. Most of the existing attacks used the cache, but there are others involving things like the fact that computation on denormals is much slower than on normal floating point values (a fun one of these lets you scrape browser contents via WebGL and I don't believe has been mitigated yet in spite of being published a couple of years ago).
Speculative execution was known to be a potential source of these side channels for a while. Now that it's shown to be feasible, expect a lot more similar attacks.
I am TheRaven on Soylent News
From a big picture perspective, the making of the hardware has already been detached from the design of the system.
Doesn't matter. You still have to make it and that still will cost money. Doesn't matter if you make it in house or if you hire someone else to do it. If doesn't matter if you have the secret formula for Coke, you still have to put sugar water in bottles and ship it somewhere which is expensive. It's FAR harder to bootstrap the funding for an open source hardware design than open source software.
Would a manufacturer take the risk of making a huge investment that relies on Open Source designs? They already do. Most mobile phones are entirely worthless without Android, an Open Source software.
You're conflating issues. You can already send an open source chip design to a chip fab or a hardware design to a contract manufacturer. My day job is general manager of a contract manufacturer (wire harnesses) so I'm more than passingly familiar. But just because you have outsourced production doesn't mean that the costs for it go away. Your analogy to Android is a meaningless one here.
Just because someone else makes it doesn't make the patent swamp go away. Open source software works precisely because how copyright law is written. The GPL and every other license basically only works because of copyright law. That doesn't apply to hardware. To protect hardware designs you have to get patents on the design and that costs serious money. Not only that you have to avoid infringing other companies patents which is not a trivial exercise when companies like IBM, Google, Apple, etc are getting thousands of new ones every year.
Companies that rely heavily on open source software can do so because they have an alternative revenue source. Typically service or engineering - sometimes ads. What is the alternative revenue source for open hardware? Service? Maybe but the revenue streams aren't quite as clear for open hardware. And even if they become clear it still doesn't solve the capital costs and patent issues.
I'm not saying it's impossible but it definitely will be difficult for open hardware to achieve the sort of success we've seen with open software.
Re "Open hardware is always free of faults."
We have seen what the best names in some sectors of the computing community did for security for years.
PRISM (surveillance program) https://en.wikipedia.org/wiki/...
Open software and open hardware is a good start at having a few people have a look at computer security.
The big brands keep failing.
Generations of failed hardware, junk encryption, CPU's, OS and networking. Backdoors, trapdoors.
Domestic spying is now "Benign Information Gathering"
With a vulnerability disclosure of that scale, I can forgive a researcher for making a poor decision. Maybe you spend months to years bug hunting and make one discovery.
But someone who deals with a near constant stream of high severity bugs, should take a more considered position.
09F91102 no, 455FE104 nope, F190A1E8 uh-uh, 7A5F8A09 that's not it, C87294CE no. Ah! 452F6E403CDF10714E41DFAA257D313F.
(Posting anon to protect myself.)
I was involved with Intel and their Curie module.
If you know how bad that was and how many silicon level bugs there were, well, at this point I think you would believe it.
They made a TV series from that chip (as a PR move). Just a year later, they EOLd the chip and now Mouser/etc have thousands of unsellable Curie chips that no one wants. Intel even removed all traces of their TV show event, as if it was an embarassment to them. Everyone that was part of that event left Intel, as far as I know.
The Intel CEO grabbed a lot of the glory of dealing with celebrities and just wanted to be on TV. I can tell you that people who worked that project were treated like crap, but the execs got TV time and glory.
The chip still sucked and its unsellable, now.
Intel gave up on IoT. They had to. They would not listen to anyone and of course they failed since this is not their core competancy.
Intel is now a joke. H1B's walk the hallways. And contractors, tons of contractors. What does that tell you, when they hire temps more than fulltime people?
I would not integrate with Intel chips, given what I've seen.
But, wasn't Meltdown an issue with Intel CPU's far before this change in validation happened?
How is this different (aside from magnitude/number of units sold) from Takata's airbag recall?
Nobody is going to die from this mistake. Pretty big and important difference there. Product defects that result in actual provable fatalities tend to get a lot more scrutiny.
I suspect a recall this large would bankrupt Intel, much like the airbag recall is bankrupting Takata.
Won't happen and while this is a serious issue, it isn't THAT serious. I expect Intel will pay some cash to settle some class action lawsuits (and so will some other chipmakers) but that's probably about as far as it will go unless there are revelations we haven't heard yet.
Bullshit you say, and yet it's only Intel and a few, comparatively insignificant ARM chips which are affected by meltdown, which btw, was what Linus was referring to.
Ye, because Intel patented the technique and didn't license it to anyone else.
I can only presume AMD is an imaginary entity in your little world, because they apparently managed to solve all these impossible problems without handing out the keys to the kingdom to everyone who asked for them.
Nope, AMD pays a higher penalty on system calls, though they mitigate this to some extent by having shorter and narrower pipelines.
I am TheRaven on Soylent News