Slashdot Mirror

← Back to Stories (view on slashdot.org)

Recurly's Backup Mess Takes Days to Clean Up

Posted by samzenpus on from the best-practices dept.

A cascading hardware outage struck subscription payment provider Recurly last week, and that started a long example in how not to manage critical infrastructure. From the article: "Last Monday, the payment provider suffered an intermittent hardware failure, which prevented the company from processing either payments or refunds. The company says it serves over 1,000 customers, including Adobe, BrightCove, and Fox News Radio, processing recurring payments for subscriptions. By Friday, the company still hadn’t completely straightened out the mess, providing updates to customers using payment gateways such as Authorize.net and LinkPoint/First Data."

7 of 21 comments (clear)

  1. Reminds me of Authorize.net by Mr.+Kinky · · Score: 4, Funny

    This case reminds me of our payment processor Authorize.net in 2009, when a fire took down the whole network and infrastructure for many days. It was only solved when one of the guys over at Authorize.net literally

    1. Re:Reminds me of Authorize.net by MetalliQaZ · · Score: 5, Funny

      He would have finished the story but he had a cascading hardware failure that took out his network...

      --
      "Here Lies Philip J. Fry, named for his uncle, to carry on his spirit"
  2. I would've been leery of... by Anonymous Coward · · Score: 5, Funny

    ...a service provider named Recurly in the first place.

    Same goes for any provider named Relarry, Remoe or Reshemp either for that matter.

    1. Re:I would've been leery of... by Alien+Being · · Score: 2

      I'm Honest Moe, that's Honest Shemp, and that's... that's Larry.

  3. No backups by Anonymous Coward · · Score: 3, Interesting

    This is a perfect example of redundancy not being the same as backups. They had redundant encryption devices, but the failure of one rolled over into the other. They had no backups (that's right, none at all) that they could restore from. From what they've told us, they intend to resolve this issue by adding more redundancy.

    Yes, really.

    1. Re:No backups by Anonymous Coward · · Score: 3, Funny

      They should have used RAID.

    2. Re:No backups by tlhIngan · · Score: 3, Informative

      This is a perfect example of redundancy not being the same as backups. They had redundant encryption devices, but the failure of one rolled over into the other. They had no backups (that's right, none at all) that they could restore from. From what they've told us, they intend to resolve this issue by adding more redundancy.

      Correction, they have no backups of the keys that the encryption accellerators used. End result is now they have a bunch of encrypted data, with little in the way of being able to recover it because the keys used are lost or corrupted.

      Sounds like they need to be hacked and their information "liberated" so they can recover it :).