Slashdot Mirror

← Back to Stories (view on slashdot.org)

Amazon's Move Off Oracle Caused Prime Day Outage in One of its Biggest Warehouses, Internal Report Says (cnbc.com)

Posted by msmash on from the closer-look dept.

Amazon is learning how hard it can be to move off of Oracle's database software. From a report: On Prime Day, while the e-retailer was dealing with a major website glitch that slowed sales, the company was also dealing with a technical problem in Ohio at one of its biggest warehouses, leading to thousands of delayed package deliveries, according to an internal report obtained by CNBC. The problem was in large part due to Amazon's migration from Oracle's database to its own technology, the documents show. The outage underscores the challenge Amazon faces as it looks to move completely off Oracle's database by 2020, and how difficult it is to re-create that level of reliability. It also shows that Oracle's database is more efficient in some aspects than Amazon's rival software, a point that Oracle will likely emphasize during this week's annual OpenWorld conference in San Francisco.

12 of 130 comments (clear)

  1. Really? by willaien · · Score: 4, Insightful

    Was it just a regular outage that could have happened to anyone, or something very specific to their own infrastructure?

    Just because a change was made at some point in the past, you don't get to just assume that everything would have been fine if Change X or Y hadn't been made. Oracle isn't a silver bullet.

    1. Re:Really? by Mr+D+from+63 · · Score: 4, Insightful

      Was it just a regular outage that could have happened to anyone, or something very specific to their own infrastructure?

      Just because a change was made at some point in the past, you don't get to just assume that everything would have been fine if Change X or Y hadn't been made. Oracle isn't a silver bullet.

      This, and the obvious risk of issues anytime you make such a large change. You fix them and move on. "thousands of delayed packages" sounds like a blip for Amazon. Bad weather can do that.

    2. Re:Really? by GameboyRMH · · Score: 4, Funny

      Oracle is a silver bullet if your wallet is made from werewolf fur!

      --
      "When information is power, privacy is freedom" - Jah-Wren Ryel
    3. Re:Really? by lgw · · Score: 5, Informative

      Was it just a regular outage that could have happened to anyone, or something very specific to their own infrastructure?

      Just because a change was made at some point in the past, you don't get to just assume that everything would have been fine if Change X or Y hadn't been made. Oracle isn't a silver bullet.

      I have some contacts at Amazon and can shed some light on this. Normally, Amazon retail prioritizes "Prime Day prep" above all else. Every team must prove they can stand up to the spike in load, and fill out lots of paperworks demonstrating they did adequate diligence. Rumor is that Prime Day was actually started as a way to do this exercise twice a year (and thus get better at it), rather than only for Christmas shopping.

      However, this year is different. Moving off Oracle has been made the first priority of every retail team (well, every one that uses Oracle in any way, which is most). No doubt that shift in priorities is what's at play here: given the thousands of teams, it's no surprise that some team somewhere dropped the ball given the conflicting priorities.

      So it's less about "Oracle was a silver bullet" and more about "changing stuff you don't usually change".

      --
      Socialism: a lie told by totalitarians and believed by fools.
  2. Bad things will happen to you! by dj245 · · Score: 5, Funny

    Oracle: Don't you dare change to a competing product. Bad things will happen to you.

    --
    Even those who arrange and design shrubberies are under considerable economic stress at this period in history.
    1. Re:Bad things will happen to you! by ilsaloving · · Score: 5, Funny

      Apparently we need a +1 Ominous moderation.

  3. I think Oracle sees the writing on the wall... by Darlok · · Score: 5, Interesting

    Between Java and their Enterprise platforms, if Oracle spent as much time listening and responding to their customers as they spent threatening them, they might be in a far better position today. Any major platform transition is going to have problems unless you're exceptionally lucky. There's just too many moving parts in Enterprise systems for humans to get everything right on the first try. Oracle won't tout all of the problems people have moving ONTO their software from a competitor, but that transition pain happens too.

    Every year that goes by, it seems like Oracle is in a more tenuous position, despite their increased revenue. They've already lost the SME space -- I don't know of a single company anywhere in our client base, or within my sphere of influence, that still uses Oracle software. Organizations are bumping up against the limits of NetSuite -- the costs to integrate 3rd-party or industry-specific components, compared with other ERPs, are turning out to be more significant than expected. So we have clients and vendors migrating ERPs over time.

    Oracle is becoming the Comcast of the software world. They treat everyone like crap, but were so deeply embedded that they were hard to dislodge. With every passing year, that is less true, and I think Oracle knows it. Unfortunately, they seem to be choosing to double-down on the "treat everyone like crap" strategy, rather than actually fixing the systemic problems that might eventually sink them...

    --
    Notice: Your mouse has been moved. Windows will now restart so this change can take effect.
    1. Re:I think Oracle sees the writing on the wall... by ctilsie242 · · Score: 5, Interesting

      The funny thing is that Oracle could get back into many peoples' good graces. If they offered ZFS under the GPL and allowed it to become part of the default Linux kernel, this would be one of the biggest enterprise issues that would get solved.

      Similar if they opened up a lot of their Solaris IP, instead of letting it die a slow death. Zones and LDOMs would be quite useful in Linux, even with it duplicating existing hypervisor functionality.

  4. Re:Their own technology? by jeff4747 · · Score: 5, Informative

    https://en.wikipedia.org/wiki/...

    They're developing their own technology because of implementing RDS. IIRC, RDS was originally a customized MySQL, and then they implemented Aurora.

  5. Re:MongoDB is webscale by AlanBDee · · Score: 5, Insightful

    I don't understand why Oracle even exists given my experience with it.

    Because it's a damn good database. The question isn't about it's capabilities, it's whether it's worth the cost. As for their other products I agree with you; it's way too sluggish. But I believe Amazon was just using their database.

    Now Amazon moving away from Oracle is a good thing; as servers get faster and the open source alternatives get better Oracle's database is losing it's foothold. I for one won't be sad to see that happen.

  6. Re:Their own technology? by Hulfs · · Score: 5, Informative

    Look up Amazon Aurora.

    They've basically created new a DBMS that runs on top of their cloud infrastructure and is optimized for their EBS (elastic block storage). They have Postgres and MySQL flavors of the database, both of which utilize the actual DB "engines", Amazon has written their own storage backends and added a bunch of other optimizations to the codebase (they've made most messaging asynchronous where possible). Because of the use of the actual database engines they claim 100% compatibility for both Postgres and MySQL. We use the MySQL flavor and haven't run into any compatibility issues with SQL queries or stored procs. Because of the performance optimizations inherent in how it was designed to run in their cloud, we were able to significantly reduce the amount of CPU/RAM utilized to run our application and still retain similar throughput - in essence, we were able to use a smaller RDS instance size, thus reducing our costs.

    One of the really nice things about it is virtually instant (and faultless) replication due to the way they rely on EBS itself to replicate data, rather than through a replication system sending queries (or binary data) to another remote system.

  7. Comment removed by account_deleted · · Score: 5, Insightful

    Comment removed based on user account deletion