Slashdot Mirror
Heartbleed OpenSSL Vulnerability: A Technical Remediation
An anonymous reader writes "Since the announcement malicious actors have been leaking software library data and using one of the several provided PoC codes to attack the massive amount of services available on the internet. One of the more complicated issues is that the OpenSSL patches were not in-line with the upstream of large Linux flavors. We have had a opportunity to review the behavior of the exploit and have come up with the following IDS signatures to be deployed for detection."
Was this badly translated from another language, or have I been out of system administration too long?
We have to thank the security researchers that chose to break the embargo on the news before OpenSSL coordinated with downstream project.
Thank you for the mess, guys!
Someone knows if OpenVPN is affected? The tests do not work on it since it uses TLS in an unusual way.
An IDS provides the means to detect malicious patterns in traffic. It is by no mean a remedy.
If it's using one of the affected versions, how did it get past the famed OpenBSD audits?
General Relativity: Space-time tells matter where to go; Matter tells space-time what shape to be.
NSA, heartbleed, whatever. you'll tell your grandchildren about "back in the day" internet.
I'm running Linux Mint Olivia -- the next to current version -- an no openssl patch is yet available as of this afternoon. I image there are quite a few similar distros. Since I have actual work to do, and can't risk wasting two hours on a potentially borked upgrade, I'm stuck to trying not to use programs affected by the exploit for the duration.
While something tells me this exploit is somewhat overblown, what really ticks me off is that this is all the result of delegating memory management to C pointers and basically mmap. As far as I'm concerned, in this day and age, that amounts to spaghetti code and I can't say it endears me to the reliability of openssl.
Please, we need SSL to be secure, not fast. Just use a less efficient method to make things more secure.
May the Maths Be with you!
Except now pretty much every affected machine needs to have its SSL certificates and private keys revoked and trashed, and new keys/certificates issued.
In the meantime, thousands (if not millions) of sites leaked sensitive data to anyone who wanted to snoop on it.
Yeah, no big deal, none at all...no repercussions will come of this.
PFsense VRT rules (paid version) already updated, I'm sure every maintained IDS has been updated for this.
"If any question why we died, Tell them because our fathers lied."
There is well written C, and there is poorly written C. I've been through the bowels of OpenSSL, and there are parts of it that frighten me. Ninety percent of the issues in OpenSSL could be solved by adopting a modern coding style and using better static analysis. While static analysis tools can't find vulnerabilities, they can root out code smell that hides vulnerabilities. If, for instance, I followed the advice of two of the quality commercial static analyzers that I ran against the OpenSSL code base, I would have been forced to refactor the code in such a way that this bug would have either been obvious to anyone casually reviewing it, if the refactor did not eliminate the bug all together.
C and C++ are not necessarily the problem. It's true that higher level languages solve this particular kind of vulnerability, but they are not safe from other vulnerabilities. To solve problems like these, we need better coding style in critical open source projects.
And don't forget the GnuTLS failure similar to Apple's
Now we're just waiting to hear that Microsoft's IIS was also borked in some unexpected way, and it'll be a royal flush eh?
That is why part of the remediation process is new certs. I didn't say it wasn't a pain in the ass, but it's trivial with regards to the amount of work involved.
I didn't say it wasn't a pain in the ass. It's just easy to fix.
Let me guess. You must be in management.
I've now read that: "No versions of OS X or OS X Server are affected by the OpenSSL Heartbleed bug, because the last version of OpenSSL shipped by Apple in an OS was 0.9.8y, which is a branch not affected by this bug. So unless you've installed OpenSSL via MacPorts or Homebrew, your public-facing OS X servers/services should be immune to this bug." What say the wise ones here?
DaveyJJ
Nope. I am a senior engineer for an IT security firm. I fix this shit for a living, thank you.
Like some site that is (like "what that site is running?" (Apache, IIS etc)) where we can see who gets what fixed when. No point in changing my passwords on a still-affected site.
Slashdot's rate-of-post filter: Preventing you from posting too many great ideas at once.
It's not easy to fix leaked data.
You can revoke keys, change passwords, and patch the software, but you can't revoke the data that was already sent with them (and can now be decoded) no more than you can you revoke the bits of data that could have been stolen.
Then that is truly sad that you have no understanding just how disastrous from a security standpoint this is. once data is leaked it can't be unleaked. it is a fucking mountain not a molehill.
trivial? excellent then you can show us how to trivially identify what data has been leaked/exposed and what needs to be reported to the various authorities that require reports on exposed privacy data.
I think you completely missed my point. The hand wringing is useless. Fix it, mitigate it, and try to move on. Any damage that has been done is one. All that cane be done now is to patch and mitigate. All the wrangling going on on the 'net is amusing. The past can't be changed. We can learn from it and move on. There are plenty of ways to stop the bleeding. People are acting like the sky is falling. It's truly sad that you're one of them.
WOW, bad spelling and typos. I chalk that up to the beers I am drinking :)
Seriously though, it's not as big a deal as it's being made out to be. Yes it caused a security scramble, and rightly so. No, the sky is not falling.
Well, Microsoft's CAPI (CryptoAPI) actually, not IIS. IIS uses CAPI, but IIS is no more a crypto toolkit than Apache or lighttpd are. A vuln in CAPI (they've happened before) could also affect clients (IE, Outlook, anything else using the platform APIs...).
Besides, we're still waiting on a NSS issue. NSS isn't so much *broadly* used - I know of only a few product families that use it - as it is *heavily* used. The product families in question are Mozilla anything (Firefox, mostly; the N stands for "Netscape") and Chrome (for PCs). Very few browsers (though not zero; Chrome on Android 4.1 uses a vulnerable version of OpenSSL) are/were vulnerable to Heartbleed, but they'll get their turn eventually!
There's no place I could be, since I've found Serenity...
There have been a number of sites.
SSLLabs scanner has been updated to check for Heartbleed, and also will report when the cert validity starts (handy if you want to see whether they're using a new cert). https://www.ssllabs.com/ssltes...
LastPass has a pretty decent scanner that just focuses on Heartbleed (without all the other info that you get from SSLLabs): https://lastpass.com/heartblee...
There are some others out there as well, of course.
There's even one for client-side testing (almost as critical):
Pacemaker is an awesome little POC script (python 2.x) for testing whether a *client* is vulnerable (many that use OpenSSL are...). https://github.com/Lekensteyn/...
There's no place I could be, since I've found Serenity...
There are many organizations that not only can't patch, do not know how to patch, or simply haven't completed patching, but also don't _have_ an IPS or IDS in place. In fact, even if a company is in a position (and has the know-how) to install one, using either one of these options may come with what is perceived as an unacceptable performance impact.
I managed to write an exploit for this issue within about 30 minutes. The bug is almost trivial to exploit. In my meager tests, i gathered usernames, passwords, session cookies, and oauth2 client tokens from an unrelated location on the internet. So, yes, i'd say the issue leans a bit closer to mountain than molehill.
That said, the fix is also trivial, and the fact that several distributions still don't have updated packages is downright embarrassing.
It's worse than that. Most browsers don't check certificate revocation lists, and the certificate authority might not have a CRL infrastructure in place that can support the number of revoked certs generated by this incident.
Granted, they could take the money they receive from all the reissued certificates and use it to build such an infrastructure, but they probably won't.
Web-SSL was already a broken system. Now that it's been cracked open even wider, maybe we can throw it out and implement something better.
Oh, who am i kidding? We'll just pretend everything's okay again after most people have patched the hole.
The Canadian Revenue Agency has shut down online reporting of taxes. 80% of Canadians use this service and most corporations file electronically anyway. Get ready for a wave of paper heading for Sudbury!
http://www.cra-arc.gc.ca/menu-eng.html
your popular PHP5 platforms will be so safe on that
Is bypassing/wrapping/whateverthey'redoing OS calls changing how memory is managed not a big enough change to warrant an entry in the 1.0.0h -> 1.0.1 log?
One of my current roles is to provide technical support/advice for a group of project managers and business analysts. This morning a few of them had watched the Crash News Network over breakfast and came in convinced that privacy, as we know it, had come to an end. My job is to talk them off the ledge (and I actually enjoy it, they're smart people and as long as I explain it correctly, they get it... I've found that's pretty rare).
1. The issue only exposes 64k at a time. Let's assume that the average enterprise application has at least a 1G footprint (and that's actually on the low end of most applications I work with). That's 1,048,576K. At best, this means that this exploit can access 0.006% of memory of an applications memory at one time.
Ahh you say, I will simple make 16,667 requests and I will retrieve all the memory used by the application.
2. The entire basis of this issue is that programs reuse memory blocks. The function loadAllSecrects may allocate a 64k block, free it and then that same block is used by the heartbeat code in question. However, this code will also release this same block which means that the block is free for use again. Chances are very good (with well optimized code), that the heartbeat will be issued the same 64k block of memory on the next call. Multi-threaded/multi-client apps perturb this but the upshot is that it's NOT possible to directly walk all of the memory used by an application with this exploit. You can make a bazillion calls and you will never get the entire memory space back. (You're thinking of arguments to contrary, your wrong... you wont.)
Congratulations, much success... you have 64k internet.
3. Can you please tell me where the passwords are in this memory dump:
k/IsZAEZFgZueWNuZXQxFzAVBgNVBAMTDk5ZQ05FVC1ST09ULUNBMB4XDTEwMDMw
MzIyNTUyOFoXDTIwMDMwMzIyMTAwNVowMDEWMBQGCgmSJomT8ixkARkWBm55Y25l
There will be contextual clues (obvious email addresses, usernames, etc) but unless you know the structure of the data, a lot of time will be spent with brute force deciphering. Even if you knew for a fact that they were using Java 7 build 51 and Bouncy Castle 1.50, you still don't know if the data you pulled down is using a BC data structure or a custom defined one and you aren't sure where the boundaries start and end. The fact that data structures may or may not be contiguous complicates matters. A Java List does not have to store all members consecutively or on set boundaries (by design, this is what distinguishes it from a Vector).
Long story short. Yes, there is a weakness here. However, it's very hard to _practically_ exploit... especially on a large scale (no one is going to use this to walk away with the passwords for every gmail account... they'd be very, very lucky to pull a few dozen).
This doesn't excuse developers from proper programming practices. It's just putting "Heartbleed" in perspective.
I work for a large financial organization. While fixing the hole itself was easy- having to tell a bunch (I can't even legally give you a ballpark, but its a lot) of customers to change their passwords (or forcing them to change) is very bad PR. Plus we don't know if any financial data was accessed. The data could literally bankrupt very large companies or my own company. This is no small problem!
What if you work for an organization that has hundreds or thousands of users who connect to a SSL VPN? Re-issuing a single certificate isn't so bad, but re-issuing many certs (and working with end users to roll them out) sounds like a nightmare. Many businesses are also responsible for more than one website, and / or are heavily regulated. Just getting lots of users to change their passwords is bad enough, but if you have to tell them that their credit card number or medical information may have been compromised, possibly provide credit monitoring services for awhile, etc., is ABSOLUTELY a lot of work for a department or an organization.
Facts have a liberal bias.
While the proof of concept exploit used an unencrypted attack, the vulnerability can still be exploited AFTER the session is encrypted.
Since the IDS probably cannot decrypt the SSL connection... it is unlikely to detect an attack that occured after encryption was negotiated, and the extension message is invisible to the IDS
You were better off using non-SSL, unless you were on wireless or something easily snooped. I'm not aware http:80 servers have a little query that gets you memory dumps. Do I misunderstand?
But you know the vulnerable host is running OpenSSL 1.0.1 -> 1.0.1f, so you can look at the source code to figure out what the memory around the private key is supposed to look like.
What is the point of IDS? If you detect an attack, your private keys are compromised and the game is over.
And then you try to recover, you make new keys, renew certificates, revoke the old one... but since certificate revocation is quite broken, you never recover. An attacker that stole your old private key will still be able to masquerade as the legitimate server.
Follow the proposed specification at http://heartbleedheader.com to tell your users when you've patched your servers. This eliminates the guessing: "is it OK to update my password now? Do I even need to? Can I trust that I'm not being MITMed with their old SSL key that an attacker stole?" It's bad enough using the tools at hand to detect that information from a single site, let alone the hundreds you might have in your password manager.
Dewey, what part of this looks like authorities should be involved?
This guy has retracted part of his analysis based on comments, but tries to make a case that passwords and cookies in the http headers are more likely to be exposed than keys. Remember, http-auth is still used a lot. http://blog.erratasec.com/2014...
Oh, who am i kidding? We'll just pretend everything's okay again after most people have patched the hole.
Don't worry, there are plenty of other holes.
"First they came for the slanderers and i said nothing."
I'm not sure where all the frenetic press came from. Surely there have been other vulns that are more severe? Like maybe one of these?
"First they came for the slanderers and i said nothing."
Can you please tell me where the passwords are in this memory dump ...
Have you ever seen a real exploited piece of data?
These are taken from Yahoo production servers, a day or two ago:
http://cdn.arstechnica.net/wp-...
http://cdn.arstechnica.net/wp-...
Can you guess where the password is, now? (And those didn't even take that many tries)
I have not seen actual SSL private keys floating around just yet, but given that the original researchers said they managed to get private keys from their own servers, I think it is reasonable to assume that some production servers must have already leaked them.
If OpenSSL is (as quite a few people who know what they are talking about have claimed) poorly written and hard to maintain, why no-one has tried to come up with a simple, easy to evaluate solution.
Or is SSL/TLS really that hard to properly implement?
Me too. I also want to know what company Dreamchaser works for. Dreamchaser's infuriating condescension is why so many people despise picking up the phone and calling the IT dept. for help ("Yeah, sure, I'll call the helpless desk and they'll fix my problem. Ya, you betcha."
Or maybe Dreamchaser does all his banking with paper checks.
Suppose SourceForge is/was vulnerable (I don't know that that's the case...I opened a ticket to find out).
Suppose a developer's login credentials were grabbed before SourceForge reacted and closed the hole.
Great. Now a bad character can upload malware as the latest release for any of the compromised developer's SourceForge projects.
Yeah...chew on that.
I honestly don't know what you're talking about. There's been a vulnerability disclosed. Fixing it is trivial. Regenerating your keys is (or should be) trivial. End of story.
Yes, this vulnerability is scary, and even more scary thing is that there are probably other vulns that bad in the wild, and most likely plenty of them. But this is over.
When I first saw Stuxnet and the extent of this shit, I lost all confidence in online data, for good. Heck, Stuxnet even infiltrated an offline network. Heartbleed is shit compared to this. The point is that everything that is online can be breached. End of story. We closed one door yesterday, I'm sure there are still 100 others open. So you see? No big deal really.
Write boring code, not shiny code!
If you learned anything from Stuxnet, you would no that no data is secure whenever it's online. No data at all. Stuxnet had zero-days for all OSes that noone knew about before it was discovered, and not just one of them. Chances are your system is already compromised and nobody even knows about it. And if it is not, it could be at any time. We closed a door with Heartbleed, but there are countless doors still open, just waiting to be discovered.
Write boring code, not shiny code!
I looked at the code. The design is AMAZINGLY bad. I think it's probably a deliberately bad design that was created to hide the exploit. Probably the feature itself was designed for the exploit, but the exploit was kept out of the first version of it so that the code wouldn't be under scrutiny when the exploit was added.
I read that that part of the protocol (the ping - probably a custom add-on for open ssl) was designed by the same guy who implemented it.
It's a ping which:
1) is an unnecessary feature in itself.
2) giving it a payload is also unnecessary
3) allowing that payload to be longer than necessary to stamp, number and keep track of packets is malicious. 1 byte would be enough to be useful (if hard to use) 8 bytes should be more than enough for anyone! 65530 some odd bytes max? WTF? Who in their right mind allows that in a ping? It's designed as AT LEAST a secret side channel!
4)giving that payload a "size" field that's redundant (since the packet HAS a size) is malicious and allowed the programmer to trust a bullshit size field instead of using the actual size. And there is the anatomy of snuck in exploit - make a redundant field which should never be trusted or used - then pretend to trust it. God, no programmer in their right mind!
If anyone is in charge of openssl they should exclude the guy who wrote that and rewrite everything he's touched.
Or, Openssl should be considered trash and replaced in every package.
none of those are even in the same ballpark of criticality compared to this vulnerability. All of them combined don't even come close.
Forgot to write the source for the images: http://arstechnica.com/security/2014/04/critical-crypto-bug-exposes-yahoo-mail-passwords-russian-roulette-style/
What really needs to be done with certs? Do they really need to be revoked and reissued? Must the re-issued cert have a new issue date? At least one service provider I work with has claimed they somehow reissued their certificate, but it has the same, old issue date and a different signature. Is this enough?
It's a bit more than that. There's no way to be sure data wasn't already leaked before you got the patching in - you know there will be hopefuls scanning the entire internet for vulnerable hosts and grabbing everything they can. That means you have to assume your SSL cert is compromised, which means generating a new one and getting it signed. And anything else the compromised process might have in memory space - SSH auth keys, user password database, that sort of thing. It's just annoying. There's nothing really difficult, but if you've got a few servers it could take up a significant amount of time re-securing everything.
You can't unsend that data, but perfect forward secrecy means that old data can't be decrypted even if the SSL key leaks, and new data can only be decrypted with an active MITM.
...if only people would actually turn it on.
Of course, this particular vulnerability is even worse than just exposure of on-wire traffic. It also exposed potentially anything in memory for the past two years, including the things you didn't even want to send to other people -- and it exposed them to anybody on the internet, not just people in a position to capture all your traffic. Patching your copy of OpenSSL is certainly trivial, but dealing with all the rest of the fallout from this is most definitely not.
Not "easy" to fix, but "simple" to fix. Digging a lake with a spoon is simple, but I wouldn't say easy.
all the more reason to have encrypted memory
Donald 'Duck' Dunn: We had a band powerful enough to turn goat piss into gasoline.
The latest version of Mint is Petra (16). Olivia is 15. You'd think criticaly security updates would be backported at least.
May the Maths Be with you!
Perhaps today is a good day to run IIS!
...if Microsoft has released something with a bug like this. Somehow I doubt there'd be so much analysis. To me this demonstrates that open != secure necessarily; how long has anyone been able to read the source-code here and even known about it for months? Let's just all agree that open-sourcing code is no guarantee of any kind of quality.
throw new NoSignatureException();
Whichever way you look at it, this isn't a great day for open source software though is it. And I speak as an advocate, i.e. recommending using Linux over Windows for servers, because of, you know, security. Can't help feeling a little bit embarrassed at the moment.
The only way to get private keys is to attack a BSD system before anyone else makes a request. At least that's the only report I've found that was successful. Not likely to be practical in the real world.
"First they came for the slanderers and i said nothing."
That's only true if you believe company's secret keys are all being compromised, which they aren't.
"First they came for the slanderers and i said nothing."
my "unwilling to wade in the bulls" take on this affair is that some part of ssl on the outward face of that service is bleeding large chunks of raw memory, in response to a trivial attack.
i'm not clear on on is which parts of memory are bleeding..
only ssl services memory?
or all memory from all services in the vicinity?
or all memory in general?
i'm hearing that sll privates are amonst the things that are leaking, but can anyone please clearly define which parts of my various systems have been wide open to the world, these past two years and more?
some people are saying ssh is not affected..
this is important to me - i don't care about ssl, and consider the bulk of the applications it supports to be trivial, but i cannot work without ssh, and am not aware of any viable alternatives.
i've always assumed that ssh was built on top of ssl - as in why reinvent the wheel - but when i went digging, trying to sort out which versions i have been running, and what i need to update, etc, i seem to be finding that ssh is actually standalone, apparently?
perhaps built on similar principles, but by differerent groups, and never merged - or do i have a bent picture?
apparently i need to learn more about these things - in matters like this i prefer to be able to read the source than listen to pundits pound the keys..
i am one, so i know, don't argue with me.
anyway - hello - if anyone can please enlighten or point at the histories of both open ssl and ssh for me, it would be a great help,
and if anyone could please point at, or repeat the answer to my question about what exactly has been exposed - thank you.
No, you actually have to fix the code to add bounds checking, or download a new version of OpenSSL (which probably gets you other fixes as well, unless you were already running the latest version.)
Recompiling OpenSSL with the proper flag isn't enough to do the job - there are people who've done that and had problems keeping OpenSSL stable on their platforms, and more importantly, that still doesn't stop the Heartbleed attack from causing trouble. You need to get the code not to try to fetch memory beyond the appropriate object's array bounds, though OpenSSL should also default to using malloc()/free() instead of rolling its own badly.
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
After updating my openssl in Gentoo Linux, I noticed that there is a USE flag for 'tls-heartbeat'. So, I have now removed that use flag from all of my Gentoo installs.
Is there any useful reason to keep heartbeat around?
Ops, I shuld have usd the prevuwe but in.
Damn. No mod points. Somebody mod the parent post up please. This is pertinent and informative.
friends don't let friends teleport drunk