Slashdot Mirror

← Back to Stories (view on slashdot.org)

One Month Later: 300,000 Servers Remain Vulnerable To Heartbleed

Posted by Soulskill on from the server-security-hipsters-don't-follow-the-crowd dept.

DavidGilbert99 writes: "The Heartbleed Bug cause widespread panic from internet users around the world worried their sensitive information was being targeted. While system administrators were warned to patch their systems, a security researcher notes that 300,000 servers remain vulnerable to the heartbleed flaw a full month later. He said, 'Last month, I found 1-million systems supporting the "heartbeat" feature (with one third patched). This time, I found 1.5-million systems supporting the "heartbeat" feature, with all but the 300k patched. This implies to me that the first response to the bug was to disable heartbeats, then later when people correctly patched the software, heartbeats were re-enabled. Note that only OpenSSL supports heartbeats, meaning that the vast majority of SSL-supporting servers are based on software other than OpenSSL.' A developer at Vivaldi Technologies AS also pointed out that a significant number of server administrators botched their response, going from safe to vulnerable."

23 of 60 comments (clear)

  1. Re:fp by TWX · · Score: 3, Funny

    Woo - first :p

    If this is what you have to do to be happy, my heart bleeds for you...

    --
    Do not look into laser with remaining eye.
  2. We can't patch yet... by Anonymous Coward · · Score: 2, Funny

    ...because we're waiting for vendors to issue patches. Quit whining about the new, latest security hole, because there have been numerous bad security holes in the past, and they will continue to exist in the future.

    1. Re:We can't patch yet... by XanC · · Score: 2, Insightful

      You should be fired.

    2. Re:We can't patch yet... by Penguinisto · · Score: 4, Informative

      ...because we're waiting for vendors to issue patches.

      1) who is the "we" you refer to? 9 times out of 10, there are workarounds, ranging from shutting off the heartbeat feature in OpenSSL to parking an SSL proxy host or load-balancer (depending on application) in-between your affected box and the rest of the planet. And yes, it's that fucking important.

      2) The "new, latest security hole" in this instance can turn your company's reputation and sales into rancid mush should you get compromised, and in this case, there's no easy way to catch it before they get in. Oh, and don't ask about the potential for lawsuits that a data breach can generate from pissed-off customers.

      3) If a vendor hasn't coughed up a fix by now? Stop using the product, and/or learn enough about it to wedge in your own fix until you can replace the product with something whose vendor is more responsive.

      4) Sibling isn't entirely flamebait... a competent sysadmin is more than just a keyboard button actuator - he/she should have enough technical mojo to cook up a means to help protect his career and his company in cases like this. If one of my admins told me what you just wrote without providing solid proof that no workaround exists, I'd sit him down and ask him if he really wants to continue his career as a sysadmin.

      --
      Quo usque tandem abutere, Nimbus, patientia nostra?
    3. Re:We can't patch yet... by jader3rd · · Score: 2

      3) If a vendor hasn't coughed up a fix by now? Stop using the product, and/or learn enough about it to wedge in your own fix until you can replace the product with something whose vendor is more responsive.

      Sometimes it takes more than a month to replace an in place product. Also, just because you're running machines that have OpenSSL on them doesn't mean that anyone in the company has any idea about how to compile code. Right now my father's company is dealing with heartbleed. They take equipment from vendors, and rents the equipment out to customers with servicing contracts. Apparently part of the contracts they have with the vendors is that they're not allowed to do software patches at all. Apparently it was industry standard. About four days into emergency heartbleed meetings my father said that that policy has been changed for all future orders.

  3. Certificate extortion by Scutter · · Score: 5, Interesting

    What would help is if there were some certificate system that didn't rely on extortion or exorbitant prices. I know several admins that mitigated the hole but couldn't replace their certificates either because the signer charges a ridiculous revocation fee (I'm looking at you, StartSSL), or because the cost of cutting and signing new certificates was too high. We need a better trust system.

    --

    "Tell me doctor, with all of your defenses, are there any provisions for an attack by killer bees?"
    1. Re:Certificate extortion by complete+loony · · Score: 2

      We need a standard for publishing public keys in DNS, then signed by your registrar and secured via DNSSEC (or similar, I'm not claiming to be an expert on exactly what level of security DNSSEC provides). I mean we already trust the DNS system to some extent to handle proof of ownership, extending that system to give some form of cryptographic authentication at the connection level makes perfect sense to me. DNS registration is a weak link in the certificate chain already.

      Of course a different name resolution system wouldn't hurt too.

      --
      09F91102 no, 455FE104 nope, F190A1E8 uh-uh, 7A5F8A09 that's not it, C87294CE no. Ah! 452F6E403CDF10714E41DFAA257D313F.
    2. Re:Certificate extortion by Scutter · · Score: 2

      >Yup, twenty-five whole dollars. That's the price of several Big Macs, with fries!. Shameless what some CAs will charge.

      (Not defending the CA racket here, but $25 isn't really that much when they give the certs out for free. In any case why revoke them, just replace them with a new, free cert. Yes, I know someone can spoof the server using the old cert, but if you want to save the $25...).

      That's $25 per certificate. That may sound cheap to you, but it's not cheap to everyone and especially not when you may have several (or dozens) to replace. In any case, revocations should be free. Also, StartSSL won't let you cut a new certificate for a host while an unrevoked certificate exists for that host, so you either pay them to revoke it, wait until it expires, or change the hostname (or move to a different CA, I guess). All three are unacceptable scenarios, IMHO.

      --

      "Tell me doctor, with all of your defenses, are there any provisions for an attack by killer bees?"
    3. Re:Certificate extortion by Njovich · · Score: 2

      If you have a site where an attacker would have bothered with the elaborate process of getting the private key, and then do MITM attacks with it on users, and it would actually matter, you wouldn't have used StartSSL in the first place, and $25 would be absolutely nothing for you.

      Hint: not you

    4. Re:Certificate extortion by Penguinisto · · Score: 2

      Can't you just get one CA signed certificate and use that to sign all the other certificates in your organization yourself?

      Well, until that 'master cert' gets compromised, in which case whoever stole it can immediately turn it into a certificate printing machine...

      You sure you want that headache?

      --
      Quo usque tandem abutere, Nimbus, patientia nostra?
    5. Re:Certificate extortion by Penguinisto · · Score: 2

      None of which has any bearing on my original point, which is that we need a better and more secure way of applying security to web servers that isn't reliant on the good graces of a third party (either through their schedule of fees or through their procedures and policies).

      I agree on the concept as posted, well sorta. Right now it's a choice between a SSL cert saying "trust me" (self-signed certs), or "trust him" (CA-generated certs.) A better way would be nice, but on a practical level that better way would be damned hard to implement, and I don't even want to know what kind of overhead it would add to the typical user session. I mean, do you really want your local bank handing out RSA keyfobs to each customer just so that you can bank online with a better feeling of security? From an Ops and Support standpoint, it would be the stuff of nightmares.

      GP does have a point though - on a *practical* level, if you have a great enough need for security, you spend for it. If you think $25 is costly, try getting a cert from Thawte or VeriSign - those bastards are downright confiscatory on their pricing, and many institutional customers demand you only use those vendors or similar when you build something for them.

      --
      Quo usque tandem abutere, Nimbus, patientia nostra?
    6. Re:Certificate extortion by PAjamian · · Score: 2

      In any case why revoke them, just replace them with a new, free cert.

      What is the point in replacing a cert if you aren't going to revoke the old one? Replacing the cert doesn't solve anything if the old one is still valid and usable.

      --
      Windows is a bonfire, Linux is the sun. Linux only looks smaller if you lack perspective.
    7. Re:Certificate extortion by diamondmagic · · Score: 2

      It was called DNSSEC Stapled Certificates. Was -- Chrome removed it.

      See also RFC 6698.

      Note you can already do this with SSH keys, where's the implementation for TLS for HTTP?

      --
      Wonder what the public key field is for?
  4. 12 of those are mine. by hawguy · · Score: 3, Interesting

    We have are 12 dev/test servers that I didn't bother to patch because they'll be decommissioned in a month or two along with the rest of the datacenter they are in, and even though they support SSL connections (with a disposable cert from our private CA), they are generally only used with HTTP and have no private data to protect, and are almost completely unused now.

    If someone wants to spend time trying to steal the server's private key or steal user data from the server, that's fine with me, I'd rather have them spend time on my disposable server than someone's real server.

  5. Safe to vulnerable by jrumney · · Score: 3, Interesting

    A developer at Opera Software also pointed out that a significant number of server administrators botched their response, going from safe to vulnerable.

    I'm guessing a lot of these are going to be people running older stable distributions that were running versions of OpenSSL from before the bug was introduced. On hearing the news, they may have quickly looked to see if there was an upgrade, seen that there was not, and either installed a newer rpm/deb they found somewhere, or downloaded the latest source they could find (in a source repository for their distro perhaps) and built from source. Meanwhile the patch for the bug had only just been checked into git.

    1. Re:Safe to vulnerable by Sir_Eptishous · · Score: 2

      I almost made that mistake.
      Then I thought I should gather the version info of all my servers and check those against what is ACTUALLY vulnerable, instead of just upgrading to upgrade.

      --
      We play the game with the bravery of being out of range
  6. The SSL problems. by jellomizer · · Score: 4, Insightful

    I am not a systems administrator (I am a software designer, when I do administration it requires a lot of trail and error.), I do however have to setup an SSL site once every few years. And because of the rarity of this action this is one of those jobs that are difficult to do, compared to other jobs. Sure if your web browser is installed via an Apt-get you are good. However there are times where you need to install it manually, and then you fight and tinker until SSL works, when it does work, your tendency is not to tinker with it anymore.
    The issue with Heart Bleed is that it effects open SSL, one of the trouble maker libraries, that require more then just the Basic make config & make & make install.

    Now there are a lot of sites setup my armature system admins, many who are less technical then I am, who will get it going and let it run. There isn't any enterprise architecture, the web site is running on a single PC with a single hard drive, chances are the hard drive had already died, and the site is just running from active memory.

    --
    If something is so important that you feel the need to post it on the internet... It probably isn't that important.
  7. Re:Browser plug-in by XanC · · Score: 2

    FoxBleed is one. It doesn't block the connection, but it lets you know about the problem.

  8. Let me expose my ignorance... by swb · · Score: 4, Insightful

    As I understand this, a vulnerable server can expose its private SSL key to an attacker. With this private key, I can decrypt all of its encrypted SSL traffic.

    This correct so far?

    Now, as I understand this so far, having the private key is great, but I need to be able to MITM the connection to decrypt anything.

    How hard is this? At the transport layer, this would require snooping the network connection of the server; someplace locally on the LAN (easiest, port mirror, maybe) or at the ISP (harder, maybe less likely).

    The other option would be some kind of DNS spoofing/vulnerability/cache poisoning, redirecting all the server traffic to a system I controlled and then piping it back out. How likely is this?

    1. Re:Let me expose my ignorance... by Qzukk · · Score: 3, Insightful

      a vulnerable server can expose its private SSL key to an attacker

      A vulnerable server can expose anything in the server process's memory space to the attacker. Obligatory: http://xkcd.com/1354/

      --
      If I have been able to see further than others, it is because I bought a pair of binoculars.
    2. Re:Let me expose my ignorance... by Dagger2 · · Score: 3, Interesting

      As I understand this, a vulnerable server can expose its private SSL key to an attacker. With this private key, I can decrypt all of its encrypted SSL traffic.

      As already mentioned, it's anything in the server's memory. Or the client's, since Heartbleed affects clients too.

      Now, as I understand this so far, having the private key is great, but I need to be able to MITM the connection to decrypt anything.

      It depends whether the connection is using perfect forward secrecy or not. If it's using PFS, then you need an active MITM to grab the session keys, so you can't decrypt old captured traffic and you need to keep your MITM up for new traffic. If there's no PFS, then all traffic ever sent with a given SSL cert can be decrypted with access to that cert's private key. All you need is to passively sniff it, then store it for later on the off-chance you ever get (or crack) the key.

      (I'm going to write a small essay on this, because it's important but very poorly documented on the web.)

      Given that, you'd think PFS would be common, but according to this study it's only available on 60-70% of web servers (they don't give a precise number, just 60% that support DHE and 18% that support ECDHE, but those two sets overlap), of which 80% prefer to use cipher suites without PFS, so about half of webservers either don't support PFS or typically won't use it. Slashdot doesn't, for example. Neither does microsoft.com. I guess that's just the homepage, but then windowsupdate.microsoft.com doesn't use it either. It's not supported on outlook.com's web, IMAP, POP3, or SMTP servers. addons.mozilla.org and marketplace.firefox.com also join the club, but their main website and the Firefox update sites do PFS at least. I couldn't find a Google property that didn't do PFS.

      And on top of that, of those sites that do use it, 99.3% use 1024-bit DH parameters, which essentially lowers the length of their RSA keys to 1024 bits (which affects the 80% of sites with 2048-bit or longer RSA keys).

      If you want to make sure you're actually using PFS, and with decent DH parameters, you generally need to make sure to configure it. Apache does this for you automatically from 2.4.7 onwards (before that, it'll use PFS but only with 1024-bit DH parameters). A lot of other software requires being fed DH parameters manually -- for instance, Courier's IMAP/SMTP servers, ZNC, ircd-hybrid etc. (And when was the last time you configured DH parameters for a server?)

      You can check if any given connection supports PFS by looking at the cipher suite in use. If it starts with DHE or ECDHE, it has PFS. (The "E" at the end stands for ephemeral; if it says DH, ECDH, or doesn't mention either of those, then there's no PFS). You can check with e.g. CipherFox in Firefox, or using the openssl command-line tools:

      $ openssl s_client -connect www.debian.org:443 | grep Cipher
              Cipher : DHE-RSA-AES256-GCM-SHA384

      If you point it at servers you use regularly, you'll probably be pretty depressed at the results. I know I was when I was making that list above...

  9. Re:Browser plug-in by crow · · Score: 3, Informative

    Thanks! I see there's also one for Chrome:

    https://chrome.google.com/webs...

    I just hope these plugins are trustworthy.

  10. Re:Security Fatigue by tlhIngan · · Score: 2, Insightful

    Perhaps a lot of server administrators are simply tired of dealing with the unending farce that constitutes modern internet security, and have simply decided to give in. What's the use in spending time and effort on security measures which frequently fail, sometimes spectacularly so in the case of heart-bleed. In particular, what's the point of protecting customer data if organizations like the NSA can simply walk in and take it, or if you're already selling it en-masse to marketers.

    I suspect a lot of them are like that. Or rather, the devs are too lazy. One of the reasons is simple - the 30% that Apple, Google, Microsoft, Valve etc., take for product distribution. Why do that when you can just set up your own Linux server, write a bit of code to get personal information and then use Paypal to handle the money?

    Of course, such systems remain vulnerable because it's too expensive to maintain (you want to write product software, not maintain the damn website) - forgetting part of that 30% is so you don't have to hire someone to keep up to date on security patches and other flaws.

    Meanwhile your customer's data remains vulnerable, and most likely since it was hacked together, lacks any detection to know if it was broken into and data stolen.

    Yes, a lot of people feel the likes of Google/Apple/Microsoft/Valve/Shopify/Amazon/etc take too much money as their "cut" and want it all for themselves, without realizing it's actually a lot of ongoing full-time work, and the services take that cut so you can concentrate on more important things.