Slashdot Mirror


Verisign Certificate Expiration Causes Multiple Problems

We had to do a little sleuthing today. Many readers wrote in with problems that turned out to be related. A certificate which Verisign used for signing SSL certificates has expired. When applications which depend on that certificate try to make an SSL connection, they fail and try to access crl.verisign.com, the certificate revocation list server. This has effectively DOS'ed that site, and Verisign has now updated the DNS record for that address to include several non-routable addresses, reducing the load on their servers. Some applications affected include older Internet Explorer browsers, Java, and Norton Antivirus (which may manifest itself as Microsoft Word being very slow to start). Hope this helps a few people, and if you have other apps with problems, please post about them below.

34 of 360 comments (clear)

  1. Re:Who needs them? by grub · · Score: 5, Informative


    Self-signed certificates are fine for Joe-Hobby website, but when you're about to enter a credit card number online it's assuring to see that the SSL cert is signed by a real organization and not "l33t_d00d@hotmail.com"

    --
    Trolling is a art,
  2. If people are getting errors coming to your site.. by nharmon · · Score: 5, Informative

    saying that your certificate is expired or not yet valid...except that it is...you need to go here.

  3. Fixed this today... by heironymouscoward · · Score: 4, Informative

    On one of our customers' systems (IIS). Turns out they had already installed the new Verisign intermediate certificate but had not removed the old one. IIS happily used the old one...

    Lesson: if the certificate expired yesterday, remove it from IIS and then reboot the thing.

    --
    Ceci n'est pas une signature
  4. Windows Explorer by thedillybar · · Score: 4, Informative
    I noticed this happening yesterday on my WinXP machine. After clicking Start->Programs and right-clicking on any icon, c:\windows\explorer.exe attempts to connect to crl.verisign.com [198.49.161.200], port 80.
    As the article states, this also resolves to some unroutable IPs:
    198.49.161.205
    198.49.161.206
    10.0.0.1
    10.0.0.2
    10.0.0.3
    64.94.110.11
    198.49.161.200
    Windows Explorer also appears to freeze (at least temporarily) if a firewall (or presumably a lack of Internet connection) prevents this from being made. It's possible, however, that if crl.verisign.com will not resolve, it will not freeze as it will if it resolves but cannot connect. Unfortunately, this is still a problem even if you have an Internet connection because of the stability (or lack thereof) of the Verisign site.
    1. Re:Windows Explorer by Politburo · · Score: 2, Informative

      I've never heard of this, and wouldn't trust only one post on slashdot to prove it to me, like you just did.

    2. Re:Windows Explorer by Zloopy · · Score: 2, Informative

      Did a little test and came up with this:

      When right-clicking on a directory in Explorer, the hour-glass shows up for like 10 seconds, and the firewall complains about Explorer wanting to access the internet. Turning it off, I notice that a connection to 64.94.110.11:80 is made.

      That IP resolves to:
      Search results for: 64.94.110.11

      Internap Network Services PNAP-05-2000 64.94.0.0 - 64.95.255.255
      VeriSign/Network Solutions PNAP-LAX-VERISI-RM-13
      64.94.110.0 - 64.94.110.255

      If I turn off Check for revocation in IE Advanced settings, the delay is gone and nothing shows up in the connection log.

  5. Also problems with Oracle by jgerry · · Score: 2, Informative

    Well, not the Oracle database directly... But Oracle sent out a memo that certain Oracle products (Oracle Wallet Manager, in particular) would simply cease to function properly until the user upgraded their Verisign certificate(s).

    I can't find ANY info on Oracle's website about this, though. The memo was sent to Oracle Premium Support customers but I don't know if the info has been generally distributed.

    Woops!

    1. Re:Also problems with Oracle by BMarkmann · · Score: 3, Informative

      It can be found here.

  6. Re:Duke Nukem by Valegor · · Score: 2, Informative

    I have installed and still occasionally play the dos version of Duke Nukem(and of course doom) on an XP machine. I just had to change the compatibility mode on the executable. Compatibility mode is the only reason I upgraded to XP from 2000.

  7. Oracle notified me of this yesterday... by Perrin7 · · Score: 3, Informative

    I received the following email yesterday: Oracle Corporation has been notified by Sun that the set of VeriSign Class 2 and Class 3 Certificates used in Oracle products will be expiring on January 7, 2004. Please review MetaLink Doc 260332.1: Expiration of VeriSign Class 2/Class 3 Certificates on Jan 7,2004 for detail information.

  8. Verisign isn't the only game in town by justMichael · · Score: 4, Informative

    I use Instant SSL cheap, good service and I haven't seen any compatibility issues.

    1. Re:Verisign isn't the only game in town by justMichael · · Score: 3, Informative

      "Trusted by 99.3% of current Internet users"

      Nope, it's a funny number, but it seems to be some kind if industry norm.

      I really don't think I should disclose how big my transactions are to this company. It's really none of their business.

      Actually you don't. What this does is provides a sort of insurance to the consumer. See here.

      It's just peace of mind for the consumer, that says that if I/you rip them off as an InstantSSL customer, InstantSSL will guarantee any fraudulant transaction up to the amount of your cert.

  9. Re:Who needs them? by Anonymous Coward · · Score: 2, Informative

    Thawte - cheaper than Verisign, much easier to work with them, and will work fine in any 4.0+ browser.

  10. Workaround to Explorer problems by BigJavaGeek · · Score: 5, Informative

    Because of the crl problems, Explorer has been acting slowly doing some seemingly unrelated activities. Copying or right-clicking on folders often is followed by a several second hang. To workaround, deselect "Check for publisher's certificate revocation" under the Advanced setting for IE (even though it is not IE running, that's where the setting should be changed). After this, no more Explorer hangs. Hope this helps someone. If you know why Explorer is checking crls for anything when doing a copy operation on files, please post.

    1. Re:Workaround to Explorer problems by JoeShmoe · · Score: 4, Informative

      I think you missed something in the blurb about this problem. The problem is Norton Antivirus, not Explorer. Norton is probably doing some kind of check on its virus signature files by validating their signature. This function is probably being handled by IE as the default browser function, which is getting hung up on the unroutable revocation site.

      So, to clarify, when you try to do a file operation, like copy, Norton intercepts the operation so it can check the file for a virus, then gets itself held up while waiting for IE to tell it if the signature is valid so it can check for that virus. End result is that Explorer never gets an answer from Norton and the operation hangs. Ditto for Word and other applications Norton watches closely.

      I too had this same problem on one of two Dell laptops. One used the default McAfee ScanShield that came with it, the other had been reloaded with Norton Anti-Virus. That machine had all sorts of crazy errors, such as Word hanging during opening, hanging when you right-clicked a file, hanging when you tried copying files.

      The system also had ooodles of pending updates from Microsoft that had been downloaded but not installed. I'm willing to bet one of them was a root server update or similar. Of course, the problem could be on Norton's end, meaning they need to update the security cert on their server? I'm not sure exactly how it works.

      - JoeShmoe
      .

      --
      -- I wonder which will go down in history as the bigger failure: the War on Drugs or the War on Filesharing
  11. Re:Who needs them? by LostCluster · · Score: 5, Informative

    There's software out there so anyone can sign a certificate. Who needs the suits at Verisign?

    Because a cert signed by you is useful for nothing more than "This conversation is encrypted, and I say I'm me." A cert signed by a Verisign translates to "This conversation is encrypted, and Verisign says I'm me."

    What good is that? Well, not much among geeks, we don't trust Verisign further than we can throw them, but we're depending on them to keep this silly DNS thing going. However, web browsers are set with a default list of trusted "Certificate Authorites" who are allowed to sign certificates. Companies who are on those lists can sign a certificate that'll work without errors, anybody else's certificate will prompt a message indicating that the name's right, the time's valid, but the issuing authority isn't on the list of authorities you trust. (You can manually add a new authority if you want... but try convincing users to do that!)

    The problem is, so many cheapskates have now signed their own certificate that the bogus authority error isn't stopping users since it's so common when nothing's really wrong. As a result, we're seeing a lot of look alike sites use SSL to get the padlock to come up, and users not being phased by the red-flag alerts that this doesn't seem to be the site they think it is.

  12. Re:Slow Word by dablob · · Score: 2, Informative

    To get Word and Excel to start working again:

    Open Nortons Control Panel - this might take
    a few minutes while it is broken but it
    will come up eventually. Under the Miscellaneous
    Section of Anti Virus, deselect the Enable Office
    Plug-in.

    That will not fix any general slowness in Norton,
    but it will allow you to read your Word/Excel
    documents.

  13. Re:Who needs them? by KlomDark · · Score: 5, Informative

    Uh, Thawte is owned by Verisign, smart guy...

    But they are a lot cheaper for some reason... Go figure...

  14. Its happening on most servers. by Steepe · · Score: 5, Informative

    Very nice of them to.. I don't know.. let someone know before today. We spent a ton of staff time this morning trying to figure out why we could connect to our servers but not the payment engines via ssl. 4 hours later we figured it out.

    Couple of nice links.

    http://sunsolve.sun.com/pub-cgi/retrieve.pl?doc= fs alert%2F57436
    http://www.verisign.com/support/ven dors/exp-gsid-s sl.html

    --
    Just three more hours seapeople and you can finally take me away from this crappy God Damned planet full of hippies
  15. Interesting side effect... by Anonymous Coward · · Score: 1, Informative

    If you have enabled "Make this directory available off-line" in W2K or later, Windows will try to access the crl server whenever you delete a file... thus adding to the self-DDOS.

  16. Re:null routing Certificate Revocation List Server by KlomDark · · Score: 2, Informative

    Depending on how you have your server configured, it either means you are accepting revoked certificates, or are UNABLE to accept ANY certificates.

    The default for some web servers is that if the CRL is unavailable, it will reject ALL presented certs.

  17. CRL scalability by d_engberg · · Score: 2, Informative


    I'm guessing that this Denial of Service effect is largely due to the known scalability problems with X.509 CRLs. In a mature Public Key Infrastructure (PKI), about 1 in 6 certificates is revoked. A CRL is around 20-30 bytes in length for every revoked certificate.

    That means that if you've issued 250,000 certificates, you can expect to have a CRL of about 1MB.

    This aggregate information isn't bad for some back-end processing, but when a lot of clients try to grab the CRL, you can quickly saturate even a high-end 100Mbps hosted server farm.

    Virtually every serious large-scale PKI (including VeriSign and Microsoft) is moving to OCSP to replace CRLs since each client will retrieve ~1kB per status request rather than a full 1MB CRL.

  18. CA certs in Java by VC · · Score: 3, Informative

    There is a file in the JDK called cacerts.
    (find . -name cacerts is your friend), this contains the certificates Java uses when initiating ssl connections.
    As of yesterday Sun was still shipping java with the expired 3a certificate.
    The way to include the new 3a certificate is to use the keytool command.
    The format is somthing like: keytool -v -keystore cacerts -import newcert.pem
    The default password for java's cacerts file is "changeit"
    VC
    ps how many geek points do i get for fixing this last week?

  19. Re:Who needs them? by badzilla · · Score: 2, Informative

    Free six-month certificates - these really work, at least for recent versions of IE. I have one installed on the SSL server in my garage. Issued by some good people in Barcelona.

    IPSCA

    --
    "Don't belong. Never join. Think for yourself. Peace." V.Stone, Microsoft Corporation
  20. Re:null routing Certificate Revocation List Server by benwb · · Score: 2, Informative

    Except for the fact that Verisign owns Thawte.

  21. Re:Who needs them? by Ben+Hutchings · · Score: 4, Informative

    Self-certificates are worthless except when distributed through an existing secure channel. Without a proper certificate, all I know is I'm encrypting the session key with someone's public key, but I don't know whose it is. I might as well send the contents in the clear.

  22. There are alternatives to Verisign... by rufey · · Score: 2, Informative
    I used to work for one of VeriSign's competitors in the PKI world, and there are other options other than going to VeriSign. However, there were only two that I could find today on the net. Some of the others I knew about apparently don't exist anymore.

    beTRUSTed, which recently purchased Baltimore's CyberTrust and OmniRoot businesses. I used Baltimore's certs all the time to avoid VeriSign.

    Digital Signature Trust, a subsidiary of Identrus. I've used their TrustID certs to avoid giving money to VeriSign as well.

    Both of the above certificate authorities have their roots in the most current IE and Netscape/Mozilla browsers. Digital Signature Trust does a lot of stuff with banks (being owned by Identrus, which was created by a bunch of banks).

  23. Re:Who needs them? by greenhide · · Score: 2, Informative

    I haven't tried them personally

    I have, and we are now actually a reseller for them (although we only "resell" it to the people we host). ChainedSSL (Equifax in Astroturf) has been working hard to switch us over to their certificates. They're trying to spread a bunch of FUD because the InstantSSL certificates have a root that is owned by Baltimore, which has just been bought out. But InstantSSL has much better browser compatibility (something like 99% of all browsers vs. Equifax's 95%).

    They generally have very fast turn around, usually you can get the certificates that day if you have your documents in order.

    The nice thing is that once you're a reseller, you become responsible for the the validity of the seller, which means that certificates are issued as soon as you submit them.

    --
    Karma: Chevy Kavalierma.
  24. Re:Update... by falcontx · · Score: 2, Informative

    I suppose I should have linked. Here is a link:

    http://securityresponse.symantec.com/avcenter/down load/pages/US-N95.html

    falcontx

  25. Re:Who needs them? by Anonymous Coward · · Score: 3, Informative

    It is easier and less detectable to sniff a connection than it is to intercept and modify all data flowing over the connection. Thus a self signed cert is better than nothing, but it does indeed have obvious security failings.

  26. My company was affected... by retro128 · · Score: 2, Informative

    I work at a CNC machine shop and the app that sends programs to the machine broke today because of that. I would have never heard about it if it wasn't for my brother in law, who works for a company running the same application.

    The fix was as follows: Open Internet Options, click Advanced tab. Under Security turn off both Check for Server Certificate Revocation and Check for Publisher Certificate Revocation. I think this fix should work for other apps that are affected by the same problem...Thought I'd pass it along.

    On a side note, it's pretty scary that this has happened to begin with. What I had to go though was pretty minor since the problem was on one machine, but what about an entire enterprise with an app installed on 1000's of computers that were broken because of this? Because of all this ridiculous "signed app" nonsense, not only are you down, but through proxy Microsoft made you dependant on one of the biggest bastardized companies I know...Verisign. Don't expect this problem to fix itself in a timely manner.

    If this is a sign of things to come, Palladium will bring Hell on earth.

    --
    -R
  27. You've misunderstood "certificate" (easy to do). by smcv · · Score: 2, Informative

    A certificate is so named, because the signer has CERTIFIED the holder to be trustworthy.

    You'd think so, wouldn't you? Unfortunately for the sanity of anyone using a certificate architecture, you're wrong.

    The certificates issued by Verisign and other Certifying Authorities are more "proof of ID" than anything else; the CA makes no assertions about the trustworthiness of the owner, they just assert that the public encryption key you've just been sent belongs to the same people who own the server you're connecting to.

    A typical CA certificate as used in SSL, translated into English:

    "We hereby certify that the following RSA key [...] belongs to the owner of shopping.example.com. Signed, Verisign."

    When your browser connects to https://shopping.example.com, the server sends you its certificate, and the browser checks Verisign's signature on that certificate. If the server proceeds to steal your credit card number, subscribe you to undesirable mailing lists, etc., that's between you and example.com; it's only Verisign's fault if it turns out they issued a wrong certificate.

    PGP uses the same principle: when you sign someone else's key, the statement you're "signing" is something like this:

    "The following public encryption key [...] belongs to Joe Bloggs ; I have met Joe and verified the photo on his passport. Signed, pclminion."

    GnuPG (and probably PGP) never talks about certificates, only about signatures.

    If that certificate is later used to commit a felony, say, credit card fraud, then YOU could be held legally liable, because YOU CERTIFIED that this guy was trustworthy. You were negligent in failing to find out that he wasn't.

    The only way you could be held responsible is if it turns out that you were so sloppy about checking Joe Bloggs' ID that you were actually negligent; (i.e. didn't check it at all, or accepted an obviously fake form of ID, or something); in most jurisdictions digital signatures aren't legally binding anyway.

    Anyway, this is what the trust mechanism in PGP is for.

    [Digression: You can build up a "web of trust" by saying things like:

    - I trust [... some people ...] so if one of them says he's confirmed Joe Bloggs' identity, that's good enough for me; (full trust)

    - these other people: [...] I don't trust so much, but if three different people all say they've confirmed Joe's identity, I'll believe that they're not all conspiring against me, so that's OK too; (partial trust)

    - everyone else either I don't know, or I know but don't trust, so I'll ignore what they say when I make my decisions.

    (These trust values are a private decision, there's no reason to reveal them to the world.)

    end digression]

    If you incorrectly sign someone's key, and a third party gets hurt as a result, you could easily argue that it's that third party's fault for trusting your opinion.

    Incidentally, you can emulate the "certifying authority" model in PGP by giving full trust to Verisign, Thawte et al, and no trust to anyone else. This is a painfully limiting model compared with the full web of trust, though; to me it looks as though the whole mechanism was designed to make money for certifying authorities.

  28. Re:Not the first Verisign CRL certificate problem by meat.curtains · · Score: 3, Informative

    EVERY SINGLE CUSTOMER who renewed their Global/Secure Site Pro SSL certs within the last thirteen months were told, when they received their certs that they also had to update their intermediates. They were given an address to get the intermediate, and instructions. They were told this would happen.

    This is not true, at least for Verisign resellers, like Trustwise in the UK. I renewed two global certs 5 months ago and was not told.

  29. Re:Not the first Verisign CRL certificate problem by Stephen+Samuel · · Score: 2, Informative
    It's unbelievable that Verisign which claims to be in the business of Internet security and SSL/TLS digital certificates - the dominant company with 95%+ market share - could let their Root Certificate Authority expire, then force its users to effectively patch their systems by importing the new certificate for the root CA after the fact. That's just bad engineering.

    That's not such a big shock... As somebody else pointed out, root certs NEED an expirey date. What throws me is that Verisign seems to be acting like this broadsided them. How many million people using their certs, and crl.verisign.com resolves to two IP addresses??? I figure that they've got enough money coming in off of this business that they should have been able to afford to put a machine on a good number of major networks out there. I mean, aren't things like this why people are supposedly paying them $150+ a pop for certs?

    The other thing to do to aleviate this problem would have been in software design. If software is designed to go automagically looking for replacement certs, it should be designed to go on a random date before the cert expires.. That way the network hit would have been distributed over the few months instead of over the last few hours.

    --
    Free Software: Like love, it grows best when given away.