Slashdot Mirror


US Amazon.com Website Down For Over 1 Hour

CorporalKlinger writes "CNET News is reporting that Amazon's US website, Amazon.com, has been unreachable since 10:30 AM PDT today. As of posting, visiting www.amazon.com produces an 'Http/1.1 Service Unavailable' message. According to CNET, "Based on last quarter's revenue of $4.13 billion, a full-scale global outage would cost Amazon more than $31,000 per minute on average." Some of Amazon's international websites still appear to be working, and some pages on the US Amazon.com site load if accessed using HTTPS instead of HTTP."

22 of 228 comments (clear)

  1. But... by Shikaku · · Score: 5, Informative

    It works just fine for me right now.

    Also now you are Slashdotting it!

  2. (partially) works for me... by nathana · · Score: 2, Informative

    I was just about to post saying that I had no problems getting to the site. I hit Amazon's home page, and it came up just fine for me...the first time. I was about to hit submit until I decided to also try navigating around the site a bit, log into my account, etc.; so I went back to try, and ran into the problem.

    So, it seems to be working...at times.

  3. HTTPS works by Anonymous Coward · · Score: 1, Informative

    just change the URL from HTTP to HTTPS and it works

    so only port 80 servers are down

  4. Re:OH NOES by Jason+Levine · · Score: 2, Informative

    Not to nitpick or anything, but at $31,000 per minute, an hour outage would cost $1,860,000, not $31,000.

    --
    My sci-fi novel, Ghost Thief, is now available from Amazon.com.
  5. Re:DNS Issue by geekgirlandrea · · Score: 1, Informative

    So, because some loser can set up a few records in a completely different domain, this is supposed to somehow have some effect on Amazon's DNS?

  6. Re:do a whois. Looks like DNS got pwn3d. by Em+Adespoton · · Score: 4, Informative

    That whois lookup says absolutely nothing... I could add amazon.com.myserver.net as a dns record too, and it would have nothing to do with the lookups for amazon.com. The trick is to use whois to see what IP address www.amazon.com currently points at.

    However, as has been pointed out, HTTPS works, so it's defininitely not a DNS issue. More likely someone along the chain corrupted a pooling link to the main http server and it propogated. I've done the same thing on apache2 servers in the past and had the same result; https still works fine, but http returns an error on key pages.

  7. Re:do a whois. Looks like DNS got pwn3d. by quazee · · Score: 5, Informative

    Of course, it's not a hack.

    A fully-qualified DNS domain name ends with a dot, so you should type 'whois amazon.com.' instead.
    Those "hacked" results you are getting are just bogus amazon.com.foo.bar. subdomains.

    --
    throw new SuccessException("Sig read successfully");
  8. Re:DNS Issue by geekgirlandrea · · Score: 2, Informative

    It's a giant cube farm, and their code is like some sort of crawling horror of reanimated spaghetti which long ago swallowed up and devoured all documentation. And then there's the deployment system. As I mentioned in another comment on this article, it can't deal with dynamic libraries. When I left, it was a real and immediate issue how we were going to keep a certain product's dependencies small enough that it would be able to *link* in a 32-bit virtual address space. The linker was up to something like 2.8 GB of working set.

  9. Re:So, it finally happened... by AuMatar · · Score: 5, Informative

    Well, if you worked there in the last 2 years you'd know that the giant monolithic app is dead and not mourned. I drew the short straw and had to sit in all day on the con call when they were taking it down area by area.

    --
    I still have more fans than freaks. WTF is wrong with you people?
  10. Anonymous Coward by Anonymous Coward · · Score: 2, Informative

    I would login to post this but I'm afraid of losing my "mole".

    I received word about 30 minutes ago that Amazon has been the victim of a DDoS attack this morning. At first, their Ops team didn't realize they were under attack and thought it was a traffic spike related to a promotion, but after about an hour of throwing hardware at the surge they realized what it was. And once they tubed the source IPs in the botnet another crop of zombies showed up.

    It looks like they are getting a handle on it now as things are better. Bad day to work in Ops at Amazon I guess. I'm cracking a beer in your honor now, fellas. Good luck.

  11. Re:So, it finally happened... by SatanicPuppy · · Score: 2, Informative

    I've never seen an NDA that would restrict someone from saying, "It was a bunch of big programs that took a long time to load."

    Seriously. She didn't even say what language or what platform they're running on, which is more useful and easy for even a non-employee to figure out.

    --
    ad logicam Claiming a proposition is false because it was presented as the conclusion of a fallacious argument.
  12. Re:So, it finally happened... by Anonymous Coward · · Score: 4, Informative

    GP meant a single ~1GB binary, not 900 million binaries. See Obidos
    GP is approximately 3 years out of date. See Gurupa

    Since I can't give any details directly, I'll let wiki do it.

  13. Re:DNS Issue by DanielT_xOOx · · Score: 2, Informative

    whois information has nothing to do with DNS. You should use whois amazon.com -h whois.networksolutions.com to get proper info. What you saw is a result of wildcard search in wrong whois server.

    The DNS servers for a domain name are announced in root DNS servers - and there, everything is fine. For example, dig NS amazon.com @a.gtld-servers.net return correct DNS servers: udns1.ultradns.net. and udns2.

    However, dnsreport show lots of errors with nameservers:
    http://private.dnsstuff.com/tools/dnsreportsmpl.ch?domain=amazon.com

  14. Re:AWS and EC2 by dave420 · · Score: 5, Informative

    Not particularly. Their S3 and EC2 services are completely seperate from their webserver. All throughout this outage, S3 and EC2 have been running flawlessly, as usual. If anything, this is a great reflection on how resilient their clusters are.

  15. Down again by jnana · · Score: 2, Informative

    It was just down for me again at 15:08 PST (same "service unavailable" HTTP error), after it had been working again for a while, so they clearly have not completely resolved whatever the issues were.

  16. Re:So, it finally happened... by Otter · · Score: 2, Informative
    Fuck up a dynamic library and you fuck everything. Fuck up one of those 900M programs and you've fucked 1/900M'th of everything.

    I think her point was that there's one gigantic binary, not an enormous number of tiny ones.

  17. Amazon declares outage is over. Light on details. by CorporalKlinger · · Score: 5, Informative

    CNET has updated the post to include a statement from Amazon.com that the outage is over. The total downtime was something like 5 hours. From the CNET follow-up article:

    "But as to the explanation, the company only hinted that its complicated computing infrastructure was, unsurprisingly, a culprit.

    'Amazon's systems are very complex and on rare occasions, despite our best efforts, they may experience problems. We work to minimize any disruption and to get the site back as quickly as possible," the company said, declining to comment further.'"

  18. Re: if you don't like the GNU linker use Gold by sodul · · Score: 2, Informative

    Gold is about 5 times faster than the regular GNU linker. It will only work on x86 code (64bits included) and ELF targets (linux/solaris)

  19. Re:This will surely help by SavvyPlayer · · Score: 2, Informative

    And for good measure ...

  20. Re:DNS Issue by Anonymous Coward · · Score: 1, Informative

    grandparent's info is years out of date.

  21. Re:This will surely help by sideswipe76 · · Score: 2, Informative

    Actually, we had to ADD hosts to our vips just to accomodate the slashdotting!

  22. Re:This will surely help by sideswipe76 · · Score: 2, Informative

    Actually, we did notice. Particularly, performance.amazon.com needed to have hundreds of host added to account for the added slashdot traffic