Slashdot Mirror


US Amazon.com Website Down For Over 1 Hour

CorporalKlinger writes "CNET News is reporting that Amazon's US website, Amazon.com, has been unreachable since 10:30 AM PDT today. As of posting, visiting www.amazon.com produces an 'Http/1.1 Service Unavailable' message. According to CNET, "Based on last quarter's revenue of $4.13 billion, a full-scale global outage would cost Amazon more than $31,000 per minute on average." Some of Amazon's international websites still appear to be working, and some pages on the US Amazon.com site load if accessed using HTTPS instead of HTTP."

68 of 228 comments (clear)

  1. Beer on the server? by Lilith's+Heart-shape · · Score: 5, Funny

    I guess somebody spilled beer on the servers? I had no idea the guys from FARK also ran Amazon.

    1. Re:Beer on the server? by The+Dobber · · Score: 3, Insightful

      Isn't the Loss number a bit misleading? Wouldn't the typical Amazon shopper see the site down, figure there has been a problem and return at a later time?

      It's not like there are a lot of alternatives out there. Sure, some specialized places might fill part of the bill, but once you've become accustomed to Amazon, you more or less stick with em.

  2. But... by Shikaku · · Score: 5, Informative

    It works just fine for me right now.

    Also now you are Slashdotting it!

    1. Re:But... by Anonymous Coward · · Score: 2, Funny

      If amazon can be slashdotted then she is not the brawd I thought she was.

  3. This will surely help by moderatorrater · · Score: 5, Funny

    I'm sure the sysadmins appreciate Slashdot sending thousands of requests their way while they're site's already down. While we're at it, maybe we should find someone with a papercut and start squirting lemon juice all over them.

    1. Re:This will surely help by sloth+jr · · Score: 4, Insightful

      It's really not all that difficult to survive a slashdot pounding for commercial web shops, even for dynamic content. Generally speaking, a popular link is going to generate perhaps 500k views a day for a day and some.

      Only exceptions would be if there was a lot of heavy content being served on each page turn, saturation of one's uplink is a possibility - 10Gb links to the backbone aren't that common as yet, and CDNs like Akamai helps alleviate a good portion of that traffic.

      My totally unsubstantiated guess is there was some DNS fooage that directed sites to a down cluster or possibly a screwed up CDN leg, but I'll be interested to see what's truly up.

      sloth jr

    2. Re:This will surely help by Goaway · · Score: 5, Interesting

      You think traffic from SlashDot would even be noticeable on Amazon's servers? You have some delusions of grandeur there.

    3. Re:This will surely help by DrHanser · · Score: 5, Insightful

      Digg sends far more traffic to a site than Slashdot does (obviously it wasn't always this way). And digg's traffic isn't particularly noteworthy to a site of any reasonable size. (Say, Ars Technica, nevermind amazon.)

      Yahoo Buzz, on the other hand, sends *huge* amounts of traffic, noticeable to sites like, again, Ars but again no disruptions of service*. But I doubt that amazon would even hiccup. If you think slashdot would even be a blip on amazon's radar, you have some serious delusions about 1) slashdot's size 2) amazon's size or 3) both.

      * According to one of the devs.

      --
      What is humor if not pain tempered by time?
    4. Re:This will surely help by afidel · · Score: 3, Insightful

      Slashdot has dual 10gig link's I'm sure Amazon's got multiple OC192's or a couple OC768's. Hmm, but on further research it looks like the may only have OC48's, at least for EC2.

      --
      There are 4 boxes to use in the defense of liberty: soap, ballot, jury, ammo. Use in that order. Starting now.
    5. Re:This will surely help by SavvyPlayer · · Score: 2, Informative

      And for good measure ...

    6. Re:This will surely help by sideswipe76 · · Score: 2, Informative

      Actually, we had to ADD hosts to our vips just to accomodate the slashdotting!

    7. Re:This will surely help by sideswipe76 · · Score: 2, Informative

      Actually, we did notice. Particularly, performance.amazon.com needed to have hundreds of host added to account for the added slashdot traffic

  4. Patents by QuantumRiff · · Score: 5, Funny

    Wait until a patent comes out for: "Taking a web presense offline, to generate discussion about the web presense, thereby increasing awareness about the site." Also, sucks to be the guy that stepped on the surge protector laying on the floor....

    --

    What are we going to do tonight Brain?
    1. Re:Patents by Samus · · Score: 5, Funny

      The Twitter guys will have prior art.

      --
      In Republican America phones tap you.
  5. So, it finally happened... by geekgirlandrea · · Score: 5, Interesting

    Believe me, if you've seen the code that runs that site, it's impressive it runs as well as it does. Try to imagine 900M static binaries that take almost an hour to link because of some tiny little code change, because they can't be fucked to make their deployment system deal with dynamic libraries reasonably.

    1. Re:So, it finally happened... by geekgirlandrea · · Score: 5, Interesting

      Um, I used to work there. Believe it or not, there are some people here with real jobs and stuff.

    2. Re:So, it finally happened... by Anonymous Coward · · Score: 5, Funny

      Okay mythical female Slashdot reader, with your unicorn and leprechauns friends chasing chupacabres out of the Amazon codebase. We believe you.

    3. Re:So, it finally happened... by geekgirlandrea · · Score: 5, Funny

      Yeah, it wasn't like that at all. There used to be a few chupacabras, but I think the shoggoths ate them all, and those leprechauns never did any work. They'd just wander around all day, picking people's pockets and teleporting away.

    4. Re:So, it finally happened... by AuMatar · · Score: 5, Informative

      Well, if you worked there in the last 2 years you'd know that the giant monolithic app is dead and not mourned. I drew the short straw and had to sit in all day on the con call when they were taking it down area by area.

      --
      I still have more fans than freaks. WTF is wrong with you people?
    5. Re:So, it finally happened... by pclminion · · Score: 4, Interesting

      Believe me, if you've seen the code that runs that site, it's impressive it runs as well as it does. Try to imagine 900M static binaries that take almost an hour to link because of some tiny little code change, because they can't be fucked to make their deployment system deal with dynamic libraries reasonably.

      Fuck up a dynamic library and you fuck everything. Fuck up one of those 900M programs and you've fucked 1/900M'th of everything.

      What does Amazon's back end compile for? If it's Linux, that's an issue right there. The GNU linker has pathological behavior when linking large numbers of static libraries. I work on a relatively small (~1 million line) codebase and it takes about ten minutes to link. Link it on another platform (e.g. Solaris) and it links in about five seconds.

      The problem isn't the huge number of libraries. The problem is that the linker blows.

    6. Re:So, it finally happened... by SatanicPuppy · · Score: 2, Informative

      I've never seen an NDA that would restrict someone from saying, "It was a bunch of big programs that took a long time to load."

      Seriously. She didn't even say what language or what platform they're running on, which is more useful and easy for even a non-employee to figure out.

      --
      ad logicam Claiming a proposition is false because it was presented as the conclusion of a fallacious argument.
    7. Re:So, it finally happened... by Anonymous Coward · · Score: 4, Informative

      GP meant a single ~1GB binary, not 900 million binaries. See Obidos
      GP is approximately 3 years out of date. See Gurupa

      Since I can't give any details directly, I'll let wiki do it.

    8. Re:So, it finally happened... by Otter · · Score: 2, Informative
      Fuck up a dynamic library and you fuck everything. Fuck up one of those 900M programs and you've fucked 1/900M'th of everything.

      I think her point was that there's one gigantic binary, not an enormous number of tiny ones.

    9. Re:So, it finally happened... by Joe+U · · Score: 3, Funny

      It's a much more robust, albeit distributed, architecture Well, obviously not as robust as you're making it out to be.
    10. Re:So, it finally happened... by bark · · Score: 2, Interesting

      The post on the google blog had a reply from the person who did Gold (see the link to Gold a few posts back) had this to say:

      Ian Lance Taylor said...

      ralph: The main difference in gold is that it was designed from the ground up to work for ELF. The GNU linker was designed to work for a.out and COFF.

      ELF conceptually requires three passes over the object files, a.out and COFF require two. A version of the third pass was hacked into the GNU linker by using a backpatch system on the symbol table, in which the GNU linker makes some decisions when it first reads the object file, and then undoes those decisions when appropriate after seeing all the objects. The backpatching causes the GNU linker to traverse the symbol table multiple times; this is very expensive in a large link. Reducing the number of symbol table traversals is probably the most significant change.

      A couple of smaller but significant changes can be found on my blog:

      Multi-threading.

      C++ templates avoid byte swapping.

      April 7, 2008 9:03 AM

  6. Re:How is this news? by felipekk · · Score: 5, Insightful

    Because this represents 31k USD every minute.

  7. D&D did it. by Silicon+Jedi · · Score: 5, Funny

    One of the top sellers on Amazon is the D&D 4th Edition Core Rules giftset. It apparently is only shipping to some pre-orders. The geeks are freaking out and untintentionally DoS'ing Amazon.

    1. Re:D&D did it. by OrangeTide · · Score: 4, Funny

      I sure am glad you didn't take his statement seriously. Would have been pretty silly to argue a point against that.

      --
      “Common sense is not so common.” — Voltaire
  8. (partially) works for me... by nathana · · Score: 2, Informative

    I was just about to post saying that I had no problems getting to the site. I hit Amazon's home page, and it came up just fine for me...the first time. I was about to hit submit until I decided to also try navigating around the site a bit, log into my account, etc.; so I went back to try, and ran into the problem.

    So, it seems to be working...at times.

  9. Re:We're sorry... by felipekk · · Score: 5, Funny

    Hmmmm... sounds like they are being DoS'ed by MediaDefender.

  10. Re:Analogy by rhombic · · Score: 2, Funny

    Considering how big Amazon is, it's more like using a fire hose on an aircraft carrier, I think.

    --
    1984 was supposed to be a warning, not an instruction manual.
  11. D&D by The+Aethereal · · Score: 2, Funny

    4th edition D&D books came out today. Coincidence?

  12. Re:OH NOES by Jason+Levine · · Score: 2, Informative

    Not to nitpick or anything, but at $31,000 per minute, an hour outage would cost $1,860,000, not $31,000.

    --
    My sci-fi novel, Ghost Thief, is now available from Amazon.com.
  13. How much lost? by robo_mojo · · Score: 3, Insightful

    "$31,000 per minute"

    Even if accurate, that's assuming everyone who sees the error message will go somewhere else to buy their books.

    I imagine some people would just wait to buy the book from amazon later when it is up again (probably very soon).

  14. Somebody write a book about this please by davidwr · · Score: 2, Funny

    I hope it's better than The Cuckoos Egg but I wouldn't know, I couldn't place my order for Stoll's book.

    --
    Knowledge is how to play a game, intelligence is how to win, wisdom is knowing what game to play.
  15. Re:How is this news? by Anonymous Coward · · Score: 5, Insightful

    Because this represents 31k USD every minute. That assumes that everyone who would have bought something doesn't just try again when the site's back up. Nevermind that the number quoted is talking about a global outage -- this is just a partial outage.
  16. Re:How is this news? by mixmatch · · Score: 5, Insightful

    Exactly, except that not everyone that would have purchased their products in those 60 minutes will buy elsewhere. They hour they came back online they could make 1.9 x typical USD per minute. That and the fact that this is not really a holiday season of any sort, so sales are likely nowhere near the peak rates they reach around Christmas, New Years, etc...

  17. Re:do a whois. Looks like DNS got pwn3d. by felipekk · · Score: 3, Funny

    Looks like you are the one who got pwn3d.

  18. Re:Analogy by UncleTogie · · Score: 2

    Better comparison, but couldn't you work a car into it somehow?

    Sure! How 'bout:

    It's like trying to break a car window by shooting spitballs at it....

    --
    Don't tell me to get a life. I'm a gamer; I have LOTS of lives!
  19. Re:do a whois. Looks like DNS got pwn3d. by Em+Adespoton · · Score: 4, Informative

    That whois lookup says absolutely nothing... I could add amazon.com.myserver.net as a dns record too, and it would have nothing to do with the lookups for amazon.com. The trick is to use whois to see what IP address www.amazon.com currently points at.

    However, as has been pointed out, HTTPS works, so it's defininitely not a DNS issue. More likely someone along the chain corrupted a pooling link to the main http server and it propogated. I've done the same thing on apache2 servers in the past and had the same result; https still works fine, but http returns an error on key pages.

  20. They Think I'm a Robot by Bryansix · · Score: 4, Funny

    When I try I get to a page that says they think I'm a robot and I don't have access to see their website.

    Well I think THEY are the robot. I don't know if I can win this argument...

    1. Re:They Think I'm a Robot by Kelson · · Score: 3, Funny

      How do you feel about the fact that that you think they are the robot?

  21. Re:do a whois. Looks like DNS got pwn3d. by quazee · · Score: 5, Informative

    Of course, it's not a hack.

    A fully-qualified DNS domain name ends with a dot, so you should type 'whois amazon.com.' instead.
    Those "hacked" results you are getting are just bogus amazon.com.foo.bar. subdomains.

    --
    throw new SuccessException("Sig read successfully");
  22. Re:OH NOES by navygeek · · Score: 3, Funny

    Hush, the troll ran out of fingers and toes.

    There, there little troll. Please continue your nonsensical rant.

  23. The problem isn't Amazon . . . by UnknowingFool · · Score: 4, Funny

    It's all you people not typing in the write web address. Try it again. Make sure you put in the umlaut correctly. What do you mean there's no umlaut in Amazon.com? *Unplugs toaster and plugs back in Amazon's server* Wait 5 minutes and try again. --BOFH

    --
    Well, there's spam egg sausage and spam, that's not got much spam in it.
  24. Re:DNS Issue by geekgirlandrea · · Score: 2, Informative

    It's a giant cube farm, and their code is like some sort of crawling horror of reanimated spaghetti which long ago swallowed up and devoured all documentation. And then there's the deployment system. As I mentioned in another comment on this article, it can't deal with dynamic libraries. When I left, it was a real and immediate issue how we were going to keep a certain product's dependencies small enough that it would be able to *link* in a 32-bit virtual address space. The linker was up to something like 2.8 GB of working set.

  25. Re:OH NOES by VENONA · · Score: 4, Insightful

    That reasoning doesn't really work for me.

    You'd have to factor in the ratio of income from the
    US site v others (UK, etc.). IMHO, the US site is likely to be more profitable than others. You'd have to plow through an annual report to really know, and factor that in.

    The larger flaw, though, is that you're subtracting one minute, when the title states > 1 hour. That implies going on A couple of million US$ in losses, which is significant, as investors don't know the reason, and caution would indicate that it could be recurring, such as the problems SalesForce has had. That hit their stock prices, etc.

    The Amazon outage is more complex--TFA indicates that some of their services were unavailable for different amounts of time, etc. What are those service worth? All anyone has is a number--from CNET. Did they do anything like a real analysis, reading quarterly reports, etc? No, by long odds. Amazon does application hosting. What customers were affected, what percentage of the business is involved, and what do CxOs of large clients think?

    The odds are actually quite good that many people give a crap. Investors (and CxOs) don't like uncertainty. It wouldn't surprise me to find some Wall Street analyst(s) making calls. Maybe it was an outage on a critical replication server, problem identified, fixed, and will provably never happen again. But maybe not. We'll see.

    --
    What you do with a computer does not constitute the whole of computing.
  26. $31,000 per minute! by DirePickle · · Score: 5, Insightful

    would cost Amazon more than $31,000 per minute on average.
    Because obviously if someone tries to buy something and Amazon is broken for an hour, they're just going to not-buy it or buy it from a competitor. Because you definitely can't wait an extra hour to place an order when it'll take 2-10 days for the product to get shipped to you anyway.
    1. Re:$31,000 per minute! by Omnifarious · · Score: 3, Insightful

      Because obviously if someone tries to buy something and Amazon is broken for an hour, they're just going to not-buy it or buy it from a competitor. Because you definitely can't wait an extra hour to place an order when it'll take 2-10 days for the product to get shipped to you anyway.

      Well, they will frequently come back, yes. But the site being down also affects consumer confidence in a big way and that will make fewer people likely to go to the site.

      So, using the metric of exactly how much you sell in a given time period is likely inaccurate, but I suspect the actual impact is higher, not lower.

    2. Re:$31,000 per minute! by Otter · · Score: 2, Insightful
      But the site being down also affects consumer confidence in a big way and that will make fewer people likely to go to the site.

      C'mon, how many people are really going to stop buying from Amazon because their website was down for a few hours on June 6, 2008?

  27. thinks I am a robot by hloo · · Score: 3, Interesting

    tried to access it from holland just now, got this message: We're sorry! You have been denied access to this feature because we believe you violated the terms, conditions, rules, guidelines or policies of our site in the past. If you believe we have taken this action in error, you may contact us at ad-help-us@amazon.com. We apologize for the inconvenience. Frequently Asked Questions Q: Why am I seeing this page? A: This page is usually shown when we believe that the request is coming from a robot or other automated source of requests. If you are not a robot please contact us immediately by emailing ad-help-us@amazon.com and we will reinstate your access to our website.

  28. Anonymous Coward by Anonymous Coward · · Score: 2, Informative

    I would login to post this but I'm afraid of losing my "mole".

    I received word about 30 minutes ago that Amazon has been the victim of a DDoS attack this morning. At first, their Ops team didn't realize they were under attack and thought it was a traffic spike related to a promotion, but after about an hour of throwing hardware at the surge they realized what it was. And once they tubed the source IPs in the botnet another crop of zombies showed up.

    It looks like they are getting a handle on it now as things are better. Bad day to work in Ops at Amazon I guess. I'm cracking a beer in your honor now, fellas. Good luck.

  29. AWS and EC2 by DrHanser · · Score: 2, Interesting

    A bit strange, the people wondering why this is news. Amazon provides the backend for a number of web services with their EC2 and AWS platforms. This is going to make third parties seriously consider whether or not they want to trust Amazon with their business.

    That is yet another reason why this is Real News(tm).

    --
    What is humor if not pain tempered by time?
    1. Re:AWS and EC2 by dave420 · · Score: 5, Informative

      Not particularly. Their S3 and EC2 services are completely seperate from their webserver. All throughout this outage, S3 and EC2 have been running flawlessly, as usual. If anything, this is a great reflection on how resilient their clusters are.

    2. Re:AWS and EC2 by DrHanser · · Score: 2, Insightful

      I know it's been running fine, I happen to use AWS.

      But for business purposes, that fact isn't going to matter much to a PHB. What a PHB is going to remember is "Gee, didn't they have a serious outage a little while ago... better use something else!" Even if the best solution is, in fact, AWS + EC2.

      Perception is more important than reality in business, unfortunately.

      --
      What is humor if not pain tempered by time?
  30. Get better Amazon, we love you! (T_T) by __aailob1448 · · Score: 2, Interesting

    Amazon: A credit to Jeff Bezos. I love Amazon prime, I enjoy my Kindle, I like the prices and the one click purchases and the mp3 previews and the look inside the book and the no-bullshit mp3 store (which I don't use) and the useful reviews and the decent recommendations, etc ! Amazon almost never leaves a bad taste in my mouth and keeps innovating with features that are actually not RETARDED or HOSTILE to me! ZOMG!

    Amazon is as good as eBay-Paypal is evil. Both are outstanding products but one is loved and one is hated.

    Sooo...in the time that I wrote this post, Amazon lost enough money to sustain me my entire life. That's sad.

  31. Cost of outage by sugarmotor · · Score: 3, Interesting

    a full-scale global outage would cost Amazon more than $31,000 per minute on average.
    I don't trust this; some people may buy later if there is an outage, no?

    Stephan

    --
    http://stephan.sugarmotor.org
  32. Great move! by PontifexPrimus · · Score: 2, Funny

    [...]thereby increasing awareness about the site." Yeah, I never heard of those ah-mah-zon guys before... what do they sell, anyway? Warrior women?
    --
    -- Language is a virus from outer space.
    1. Re:Great move! by fireboy1919 · · Score: 4, Funny

      Yes, they do.

      Which is cheesy awesome.

      --
      Mod me down and I will become more powerful than you can possibly imagine!
  33. Re:Analogy by ColdWetDog · · Score: 5, Funny

    Where is BadAnalogyGuy when you need him!!!!

    Out in the back, working on his car.

    --
    Faster! Faster! Faster would be better!
  34. Re:do a whois. Looks like DNS got pwn3d. by Chapter80 · · Score: 2, Funny

    Who's the ass-clown that marked this informative?

  35. Re:DNS Issue by DanielT_xOOx · · Score: 2, Informative

    whois information has nothing to do with DNS. You should use whois amazon.com -h whois.networksolutions.com to get proper info. What you saw is a result of wildcard search in wrong whois server.

    The DNS servers for a domain name are announced in root DNS servers - and there, everything is fine. For example, dig NS amazon.com @a.gtld-servers.net return correct DNS servers: udns1.ultradns.net. and udns2.

    However, dnsreport show lots of errors with nameservers:
    http://private.dnsstuff.com/tools/dnsreportsmpl.ch?domain=amazon.com

  36. Re:HTTPS works by pimpimpim · · Score: 2

    yeah, swell, let's have amazon's servers encrypt all traffic, that'll reduce their server load.

    --
    molmod.com - computing tips from a molecular modeling
  37. Re:How is this news? by Dachannien · · Score: 2, Funny

    Slashvertizement at work. You know, because nobody's ever heard of Amazon.com before.
  38. Down again by jnana · · Score: 2, Informative

    It was just down for me again at 15:08 PST (same "service unavailable" HTTP error), after it had been working again for a while, so they clearly have not completely resolved whatever the issues were.

  39. Re:Analogy by CarAnalogy · · Score: 5, Funny

    No, that's me you're referring to.

  40. Amazon declares outage is over. Light on details. by CorporalKlinger · · Score: 5, Informative

    CNET has updated the post to include a statement from Amazon.com that the outage is over. The total downtime was something like 5 hours. From the CNET follow-up article:

    "But as to the explanation, the company only hinted that its complicated computing infrastructure was, unsurprisingly, a culprit.

    'Amazon's systems are very complex and on rare occasions, despite our best efforts, they may experience problems. We work to minimize any disruption and to get the site back as quickly as possible," the company said, declining to comment further.'"

  41. Re: if you don't like the GNU linker use Gold by sodul · · Score: 2, Informative

    Gold is about 5 times faster than the regular GNU linker. It will only work on x86 code (64bits included) and ELF targets (linux/solaris)