Slashdot Mirror
The Fedora-Red Hat Crisis
jammag writes "When Linux journalist Bruce Byfield tried to dig for details about the security breach in Fedora's servers, a Red Hat publicist told him the official statement — written in non-informative corporate-speak — was all he would get. In the wake of Red Hat's tight-lipped handling of the breach, even Fedora's board was unhappy, as Byfield details. He concludes: 'If Red Hat, one of the epitomes of a successful FOSS-based business, can ignore FOSS when to do so is corporately convenient, then what chance do we have that other companies — especially publicly-traded ones — will act any better?'"
The problem with a lot of corporate Open Source is that they ignore the ethical foundation of Open Source. And eventually we find out that Open Source isn't quite as good without the ethics.
Bruce
Bruce Perens.
They have to buy people's trust again now with their actions, and it's going to take years, if they even do it.
Bruce Perens.
It's happened numerous times. Consider the Bruce's comment regarding Debian above.
Frankly "a real business situation" sounds a lot like a metaphor for covering your ass at other people's expense.
Platform advocacy is like choosing a favorite severely developmentally disabled child.
There are still ways to handle this which cover both the need to minimize chances of a recurrence and the desire of users to know what happened and whether they are also at risk. This could include specifying whether this was due to a software bug still under investigation, a configuration error which has been fixed, or possibly an internal sabotage. Exact details could remain forthcoming until such time as complete mitigating solutions are in place, especially if a patch needs to be released to handle it, which should take no more than a few weeks.
You can never go home again... but I guess you can shop there.
"Frankly" when business is more important than the customer, often the business isn't worth a damn.
Does this justify the word "crisis?" I doubt it does. In my opinion "conundrum" would be a better word.
At first read, the heading made me think that Red Hat and Fedora communities were bickering big time, threatening timely releases of software we have [all] come to rely on. Of course this is not the case.
So why the sensational heading?
This goes back to the whole "trusting trust" concept. You have no way of knowing if the source you've been given reflects the binary you're using, unless you yourself compiled it (and hand-crafted the compiler you're using in assembly, and made the assembly language for your CPU, and made your CPU, but those are a different discussion.)
The point is, Red Hat signs their packages. If their signing mechanism has been compromised, it is quite conceivable that every single Red Hat package is untrustworthy. The dates on the packages are only as trustworthy as the key, so there is no beginning or end time for this: you must throw out all Red Hat packages on your system, because any could be compromised.
Source really gives you very little assurance unless you compile it.
If we want to look at this in contrast to Windows, there's not really any comparison, since we barely even begin to have a grasp of their Byzantine updating system, and couldn't even speculate as to the effects of a similar problem on their side.
I used to be 100% redhat and fedora... Now I've moved almost all my systems to ubuntu, but I still run centos on a few servers.
Every reputable tech company I deal with (ISP, Software, Hosting, Colo) has very clear, very open policies about outages, breaches, and security in general. If they don't I don't do business with them.
I know the ins and outs of my ISP, Hosting, and Colo companies processes because I get emailed whenever I have an outage that says "we experienced an outage from x-y on day z, the outage was caused by our dumb admin who tripped on the power cable, we rewired our entire data center to move all of the power cables to the ceiling to prevent a similar outage in the future".
Obviously that is a made up report, but it is extremely standard practice to let all your customers know a) when the problem happened, b) what caused the problem, c) concrete steps taken or procedures implemented to prevent similar problems in the future
That RedHat has fallen so miserably short of this basic tenet of IT procedures is extremely scary.
Not exactly.
Fixed it for you.
(And I even did it without shredding the evidence, NDA/gag orders, DMCA take down letters or all of the other CYA tactics peculiar to the legal profession.)
You can't send a takedown notice to an already printed newspaper.
This seems to be, from reading the Fedora and Red Hat statements, an ongoing investigation. The same way the police don't comment about investigations in progress, Red Hat is keeping mum. Keep in mind, the breach may be very complex and not something that they can confidently say "we understand" without a very detailed analysis.
They announced the issue immediately and took steps. For now, give them the benefit of the doubt that further details will be forthcoming once a proper investigation has been completed.
Learning HOW to think is more important than learning WHAT to think.
Disregard that. OP has it right. I suck cock and need grammar lessons.
But, it doesn't matter - it's all open source, you can look at the lines of code and verify for yourself that they're safe, right?
Wrong. I know this is common wisdom in the open source community, but it really isn't that simple when compilers are involved.
The reason is that the hackers COULD potentially have modified the binary of the compiler used to bootstrap the whole RedHat distribution. You can modify the compiler such that it takes harmless code and compiles backdoors into it. In particular you could modify it so that it always propagates the change when it compiles a version of itself. Since every system bootstraps from an already compiled version of the compiler, a well hidden backdoor could propagate forever, unless people actually analyze the machine code.
Read Ken Thompson's 1984(!) Turing Award lecture for the full nitty gritty details. This should be required reading for everybody in security (and all open source advocates, for that matter):
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.91.5728&rep=rep1&type=pdf
(PDF)
There are a number of possible scenarios that would recommend against being 100% candid on how far you were breached. If I was violated, I think I'd like to take a moment to do a "self-check" on all of my important bits before I started telling everyone all of the nitty-gritty details. As the article pointed out, people were told that there was a breach, and that they should not update for a few days. How is this "anti-FOSS"?
Perhaps they were on the trail of who did this? Perhaps they were comparing notes with the Ubuntu breach cited in the article, with the goal of finding the M.O? Perhaps, like any police detective, they were keeping certain clues to themselves while they investigated further? If the crimes were found to have similar approaches, keeping quiet might improve the odds of capture?
I use Fedora, and had been using Red Hat before Fedora came along. I don't think this kind of hysterical "anti-FOSS" reaction is really fits the facts as I just read them. Perhaps they have not handled this in the best possible way, but that's far from "anti-FOSS." Just because you didn't get your precious packages today, doesn't mean they've gone all corporate spin-zone on the FOSS community. Again, I'm not saying that they've handled it as well as they could have, I'm just making the point that there might be reasons for not detailing publicly the many many disgusting ways that each and every one of their private bits have been violated and penetrated numerous times, over and over again....
Give-em a break guys, I'd be more concerned if they didn't tell anyone about the break-in at all. That would really be "corporate" behavior. Simply deny the breach and lawyer-up. As it is, they're trying to fix it, and if you're so antsy to get your packages immediately, the source and diff's are there for you to check yourself. If they start getting in the habit of this, folks will start contributing to, and using other distro's.. isn't that how FOSS is supposed to work?
Yeah, but that is the techie paranoia.
Just because something can be done doesn't mean it actually happens. If I go to holidays and leave the door of my house open, it does not mean that something actually happens.
The point is, Red Hat signs their packages. If their signing mechanism has been compromised, it is quite conceivable that every single Red Hat package is untrustworthy.... you must throw out all Red Hat packages on your system, because any could be compromised.
Nonsense. Why should you "trust" RedHat Packages signed by employees?
The whole signing shit is a troll for the privacy church. What they forget are the proportions and what is really important. We know exactly that the problem didn't affect us in the past and it won't affect us in the future now we found out. No need to panic.
IT managers now know that RH is going to go unresponsive when there's a problem.
The issue isn't even fully known, so you're jumping to conclusions.
For some reason Fedora has to re-key all their repos and, while I think that's done, it's still being mirrored. One would assume a signing key has been lost.
Redhat isn't doing that. They apparently have a signing server, and a user's credentials were apparently lost, and some packages got signed, but not put in the repos. If you run a RedHat machine and get an unsolicited contact to install some new OpenSSH packages - don't.
I think Fedora has the bigger problem at the moment. Let them work through the problem, they know how to do this. When the users are safe (still an ongoing topic of discussion on how to best ensure this) my guess is they'll be releasing more information. I further suspect we'll learn that prior disclosure would have put users at more risk. We'll see.
How can they trust Red Hat again?
Historically the Fedora guys have been trustworthy to the extreme. That's why not everybody is jumping on them right now, despite the distro-partisans who smell blood in the water. Again, we'll re-evaluate our position on that once the dust settles.
My God, it's Full of Source!
OUTSIDE_IP=$(dig +short my.ip @outsideip.net)
"If you're running GPL software, and someone hacks your system, you must make all details of the hack known"?
Sure you must, under the GPL even a hack would count as a derivative work, so the hackers have to make the source available, wouldn't they? ;-P
There are two rules for success:
1. Never tell everything you know.
I would have phrased it differently: The issue isn't fully known, thus there's a problem.
There's been quite a lot of time.
Bruce Perens.
OK, some servers got hacked, the attackers didn't inject rogue packages into the repository servers so no customers/users were affected. Red Hat/Fedora responded by auditing everything and releasing a statement, along with tools to detect packages with the attackers' signature. Big deal.
Seriously, what else is there to be known about it?
Yeah, say whatever you want, but it's not as if Debian never had its servers compromised in a similar fashion, and never had to perform some PR damage control.
Unlike Debian, Red Hat is a publicly traded company with a whole bunch of customers with signed SLAs. Handling such matters without press trolls all rolling over it spreading FUD and causing unnecessary panic is _not_ an easy task, as can be beautifully shown by TFA.
I respectfully disagree with Bruce Perens. The Debian OpenSSL fiasco was so much more serious, damaging and dangerous to users all over the world, it's not even fair to compare. We're talking about millions of known networks and sessions compromised in Debian over a year and a half period, versus none in Red Hat over a week.
I appreciate how Debian acted _after_ the fact, but was there any other way to handle such a terrible mishap?
This is not about flawed Open Source policies, this is about seriously flawed journalism, where conspiracy theories are used to make a story where there is none.
- Otaku no naka no otaku, otaking da!!!
TFA says:
However, as of September 8, the crisis continues, with Fedora users still unable to get security updates or bug-fixes.
Not true. Go here: https://fedoraproject.org/wiki/Enabling_new_signing_key, follow the instructions and voila... updates available.
Anybody want a peanut?
"Affect" DOES mean "to have an effect upon". That's not the disputed definition.
Perhaps when you switch them in your defense of your chronic switching of them, that's evidence that you're wronger than the other wrong people. :)
[Meant lightheartedly, I don't honestly care what you type.]
The United States of America: We do what we must because we can.
Exactly. It's not a breach of any FOSS licence. It's possibly a breach of FOSS project best practice, but that isn't clear either, because we don't know how the problem happened or what code had to be modified to fix it.
Even if some FOSS code was modified, there is no licence obligation to distribute the changes unless you are distributing the binaries.
As I understand it, the security breach was that someone gained remote access to their servers. It doesn't necessarily follow that any of the code served by the servers was faulty. Last time I checked, not all the code running Redhat sites was open-source.
And the breach could well have been down to a sys admin error, rather than a problem with the codebase itself. It would obviously be acutely embarassing if Redhat's in-house team turned out to have made the kind of mistake that causes people to fail their RHCE exam, but it wouldn't have anything to do with FOSS.
Also, there may not be a simple answer to the 'what does this mean for me?' question. In the Debian case, the answer was quite simple, and so was the solution. The Redhat announcements sounded to me like "We know there was a breach, we don't know exactly what happened as a result, we don't think anything serious happened, but, to be on the safe side, we are changing all the locks."
Redhat's PR department obviously misjudged the best way to handle this incident, but the expectations of the FOSS community also seem unrealistic. When a company open-sources some code, it doesn't mean that anyone in the world gets unfettered access to all the information in the company. Reading TFA, I can't help but think that it is at least partly motivated by the blogger's outrage that Redhat didn't roll out the red carpet all the way to the server room for his terribly important blog.
Virtually serving coffee
You can't really say they are keeping things quiet while things are still in progress. This isn't being swept under the rug, this seems to be pursued in all areas currently. If after everything, there is still no more information, then that is a story.
"Thanks for all the money you paid to us. We've used it to buy off ISO among other things" -Microsoft
I see very often this quoted without any substantiation.
I thought that the responsibility of a company was to stick to whatever they say they will do in their chapters of incorporation, then shareholders sharing that vision would finance the venture.
If the companies' own rules mandate that openness and accountability are part of how the company functions, and shareholders used their judgement and accepted that, profit may take a second seat in the view that in the long term, the business strategy of transparency is deemed to be necessary in turn to make the enterprise profitable.
The problem with many investors is their short-sighted, quarterly short termism and companies that do not ensure ways to handle that in a way that makes sense in a longer term.
IANAL but write like a drunk one.
For some reason Fedora has to re-key all their repos and, while I think that's done, it's still being mirrored. One would assume a signing key has been lost.
Have you already read the Fedora report? Fedora did release a report about the incident. Within it they say that while an attacker was able to reach a Fedora signing system they do not believe that the key's passphrase was compromised. However it states that as precaution they have decided to create a new key.
The Red Hat side of things is different and far... trickier. I point you towards this LWN article about the intrusion as I think it's hard to say such simple statements about it.
Well, gee. Thanks for explaining the meaning of "bootstrapping" to me.
The problem is: when can you consider a compiler "clean"? The only way to be sure is to develop it yourself in machine language (no, you can't even use an assembler, because it could generate a backdoor, too), or to fully scrutinize the machine language of an existing compiler binary.
In practice, if you are using gcc, you have a compiler that has been recompiled by itself over and over again for at least a decade. Can you be absolutely sure that there wasn't somebody somewhere in that chain who added some malicious code that has propagated on? Not unless you audit the machine code of a specific gcc binary. The most likely party to have performed such an audit would be the NSA, but I am not sure I would trust them if they report there is no backdoor (in fact, they are pretty high on the list of who might want to plant a backdoor to begin with).
Nice try. The problem with Techies is that they don't get the larger picture. They focus on the blinking red herrings they are so used to and where they believe in.
We are talking about a serious flaw of a security model. True. But consider that most people run operating systems where executables are not signed at all.
There is no indication here at all that anyone externally found out about the problem before. It is basically that you found out that what you did over the last two years was vulnerable to potential attacks. How will it affect the future? Not at all, as the issue gets fixed.
Ah, and right now no one unauthorised actually has the key yet. It is only technically possible to crack it much easier...
If Red Hat, one of the epitomes of a successful FOSS-based business, can ignore FOSS when to do so is corporately convenient
Sorry, but I must have missed the clause in the GPL that requires full and immediate public disclosure of any security breach on your servers, or a duty to maintain 100% availability.
OTOH I do remember loads of stuff in the GPL about how there was no warranty.
There also seems to be a presumption that this "breach" represents some sort of systemic vulnerability in the Fedora/Red Hat product - TFA and several comments here reference the Debian SSL problem. What about the good old standbys of "inside job", "social engineering", "weak password" or "bugger, I knew I should have password-protected my SSH key"?
What if they're planning to fire someones ass, or even press criminal charges over the incident? That would place serious restrictions on what they could publicly announce.
In a survey of 100 programmers, 111111 thought that duck-typing was a good idea.
This is why the GCC build process builds the compiler three times. First it builds it with the existing compiler. Next it builds it with the new version. Finally, it builds it with the version built with itself and compares the binaries. If the last two are different, then the old compiler is likely to have been trojaned.
I am TheRaven on Soylent News
I initially read the article title as The Fedora Hat Crisis. I was wondering if there was a hat shortage brought about by FOSS people wanting to Cosplay. I now realise my error but have this mental imagine of people walking into a Linux conference all bedecked in red Fedoras.
You may think me a tired, old, cynic. I'd have to disagree about the tired bit.
Sorry but
Company has a problem with a server breach - no publicity, no comment - note even a hint
FOSS project - We're busy go away ...
Fedora - We have a problem we're sorting it out, we'll let you know when we know, the Server is Red Hat's and they have the same problem so they are dealing with it ....
Looks fair enough to me ....
Puteulanus fenestra mortis
I hope they continue to ding you, because you still have it wrong. Here's a rule that works 99% of the time:
So if you find yourself writing "effected" again, all you have to do is recognize that you want a verb and then use "affected" instead.
The most rabid believers in American Exceptionalism are the exact same people whose policies are destroying it.
https://www.redhat.com/archives/fedora-devel-list/2008-September/msg00842.html
In due time you can probably expect a more complete picture of what happened. I think the "Fedora/RedHat keeps us in the dark" view is overly alarmist.
I usually like Red Hat, but every once in awhile they do a really abusive something. This is another.
I was a Red Hat customer for years. Then they dropped the professional edition without ANY warning. Fedora didn't show up for over a year (or so it seemed). Well, now I use Debian, and occasionally investigate one of the other distributions. (Ubuntu, Mandrake*, one of the small ones...NOT Novell's offering. I don't trust them.)
I still want to trust Red Hat. I feel that their corporate intentions are honorable...most of the time. OTOH, I'm not about the rely on them again. They aren't trustworthy, merely well intentioned. So I want to trust them, but I know it's a bad idea.
OTOH, CentOS *seems* to have come through this without scars. Their comments indicate that they got cooperation from Red Hat in containing the problem. Perhaps companies can trust Red Hat more than individuals can...perhaps. Or maybe they were just lucky this time.
*I know they're officially Mandriva, but that's for garbage legal reasons. To me they're still Mandrake. (This isn't totally good. They've pulled some boners too.)
I think we've pushed this "anyone can grow up to be president" thing too far.
Debian's a non-profit. Comparing the two isn't useful, for a couple of simple reasons. If a Debian build server is owned, what's the financial damage, and to who? How about Redhat?
It's a lot easier for RH to show direct and indirect financial damage due to a breach, which brings in the FBI. Once the FBI is involved, your whole reply is a "No comment." It's an ongoing federal investigation. If somebody found the trojaned openssh on a DoD server, you can bet that the NSA is probably involved as well.
Once the feds are involved, their hands are tied. If I'm right, it took a lot of work and negotiation by the lawyers to release as much info as they did.
-30-
one. It was pretty evident there was something being done because none of the update servers were available. By knowing this it was just a matter of minutes to realize that something fundamental was wrong. Me knee jerk reaction was to conclude that they had been compromised to some extent and that spawned an special hands-on audit on all of my systems running with any derivative of Red Hat. Less than a day after we get a post telling us that they are working on it... meaning it is deep and they haven't found the rabbit in the hole yet. For me this is enough information to go in "extra alert mode" on all my machines that would be in the realm of the same problems.
remote logins log checking being a mere fraction of the full audit.
What is the diff against Microsoft and similar? Big difference... I had the choice to compile my version of sshd (and other remote offerings) and prep it on the servers that I had that could potentially be effected by a bad transient build. I could do the diff between the updated packages if there was any, on source level. Maybe this is going too far BUT at least it gave me the option to do my stuff pre-emptively while waiting for the final dictum from Red Hat and their investigation. I call this Pro-active guarding.
Most likely... once all major customers had done something similar they were able to disclose a bit more of the problem.
Anyone who mentions Ken Thompson's Reflections on Trusting Trustshould also mention David A. Wheeler's "Countering Trusting Trust". Those who don't should be punished by having to argue both sides of the debate.
I occasionally post the counter argument in a reply but no one sees it... Next time you see someone else with this behaviour tr, here's ammo for countering it.
(I believe the gcc rebuilds aren't so much to remove this type of intentional bugging but rather ensure the final binary is free from things like first compilation optimisation issues... Comparing the compiler binaries would probably indicate differences due to things like dates being present BUT hopefully what they would output on a given source would be the same)
You could just build yourself a tiny ELF binary with it and take a look..
And if the original compiler was gcc, and trojaned in the way the paper describes, then the triple compilation wouldn't catch it. Why? Step 1: the existing compiler builds binary 1 and inserts the backdoor. Step 2: Binary 1 builds binary 2 and inserts the backdoor. Step 3: Binary 2 builds binary 3 and inserts the backdoor. Step 4: binary 2 and binary 3 are compared, and if they are different, then there is an error. However, since all versions have the backdoor, there is no difference, and no error will be flagged. Try reading the linked article again.
The triple compilation is not for detecting trojans, but "because the compiler will be tested more completely and could also have better performance."
And if the original compiler was gcc, and trojaned in the way the paper describes, then the triple compilation wouldn't catch it.
But given the significant, massive changes that have been made to gcc over the years (not to mention all the other compilers that have been used to build it), the hack that the paper describes would need to involve hard-AI beyond what we have been able to achieve, and would probably take weeks to complete a single pass of compilation on the typical sort of machine used to compile gcc.
Systems were smaller, simpler, and hadn't been evolving for as many years (or had as many major components rewritten from scratch) back in the days Thompson wrote that paper.
(I built gcc with a C interpreter once, and then used the interpreted gcc to compile gcc. I did it mostly as a stress test of the interpreter, but it also served as a quick check for Thompson's trojan, which I didn't find--not that I was expecting to for the reasons cited above.)
You are correct about the purpose of the triple compilation, though. Trying to catch Thompson's hack that way would be pointless both because it wouldn't detect the hack and because the hack is no longer practical.
Ah I see.
Still if 5.2 is trojaned then it will take the clean 5.3 source and emit gcc53at. 53at will then emit 53bt. 53bt will then emit 53ct.
53b and 53c should be equal, and the trojan will also be equal, so 53bt == 53ct. You could comsider the trojan as part of an "optimisation pass", which will be performed by all builds of gcc.
A trojan compiler that always compiled printf with an extra check for if("crashme") exit; and always compiled the preprocessor with the printf mangler would always emit:
* A compiler that repeatably compiles printf with the trojan.
* A compiler that repeatably compiles the preprocessor with the trojanned preprocessor.
3laws: No freebies, no backsies, GTFO.
When can you consider a compiler "clean"?
Countering "Trusting Trust"
If you have any concerns with that, they should be answered in: David A. Wheeler's Page on Countering Trusting Trust through Diverse Double-Compiling (Trojan Horse attacks on Compilers)
If you find any holes in the theory that were not discussed, then consider writing up your findings for publication.
Happy moony