Recurly's Backup Mess Takes Days to Clean Up

← Back to Stories (view on slashdot.org)

Recurly's Backup Mess Takes Days to Clean Up

Posted by samzenpus on Monday September 10, 2012 @06:58AM from the best-practices dept.

A cascading hardware outage struck subscription payment provider Recurly last week, and that started a long example in how not to manage critical infrastructure. From the article: "Last Monday, the payment provider suffered an intermittent hardware failure, which prevented the company from processing either payments or refunds. The company says it serves over 1,000 customers, including Adobe, BrightCove, and Fox News Radio, processing recurring payments for subscriptions. By Friday, the company still hadn’t completely straightened out the mess, providing updates to customers using payment gateways such as Authorize.net and LinkPoint/First Data."

21 comments

Min score:

Reason:

Sort:

Reminds me of Authorize.net by Mr.+Kinky · 2012-09-10 06:58 · Score: 4, Funny

This case reminds me of our payment processor Authorize.net in 2009, when a fire took down the whole network and infrastructure for many days. It was only solved when one of the guys over at Authorize.net literally
1. Re:Reminds me of Authorize.net by MetalliQaZ · 2012-09-10 07:00 · Score: 5, Funny
  
  He would have finished the story but he had a cascading hardware failure that took out his network...
  
  --
  "Here Lies Philip J. Fry, named for his uncle, to carry on his spirit"
2. Re:Reminds me of Authorize.net by Anonymous Coward · 2012-09-10 07:43 · Score: 0
  
  He accidentally the post.
3. Re:Reminds me of Authorize.net by carlos92 · 2012-09-10 08:31 · Score: 1
  
  Literally what? The suspense is killing me!
4. Re:Reminds me of Authorize.net by maxwell+demon · 2012-09-10 08:54 · Score: 1
  
  Yeah, that never could happen with me, because I
  
  --
  The Tao of math: The numbers you can count are not the real numbers.
5. Re:Reminds me of Authorize.net by tstrunk · 2012-09-10 09:25 · Score: 1
  
  I know that technician! His name was Candlejack, right?
  When he came to
6. Re:Reminds me of Authorize.net by Anonymous Coward · 2012-09-10 10:32 · Score: 0
  
  Used to work for a large payment processor and outages were pretty common. We'd regularly have our gateway fail or transactions not process at all every month. Something as simple as not checking the UPS batteries monthly in the data center to make sure they were still good caused the last outage... Pretty common practice to half-ass everything, they don't care about supporting the customers just getting their percentage off your transactions.. Credit Cards biggest scam ever, if you run a business and take CC they will bleed you dry with all the fees and other interchange bullshit. You'd be surprised how many merchants have trouble running credit cards and regularly double authorize charges on people's cards. Nothing like having your money being held hostage for a number of days or weeks...
7. Re:Reminds me of Authorize.net by arglebargle_xiv · 2012-09-10 18:07 · Score: 1
  
  Pretty common practice to half-ass everything, they don't care about supporting the customers just getting their percentage off your transactions..
  A friend of mine runs a networking services company who got called into a medium-sized payment processor a few months back to upgrade a server, about an afternoon's work. After several months of 10-12 hour days he's now got them up to the level where they're about quarter-arsed. With another few months' work they'll be at the level of half-arsed. When he described the original setup he found I thought he was making it up, it was just fail layered upon fail layered upon fail, like something a bunch of drunken geeks have invented as a joke to see how dysfunctional a collection of systems and networking you could make that would still appear to work most of the time.
8. Re:Reminds me of Authorize.net by cusco · 2012-09-11 03:39 · Score: 1
  
  Makes me glad that I pay cash for everything possible.
  
  --
  "Think about how stupid the average person is. Now, realise that half of them are dumber than that." - George Carlin
Too big to fail? by Anonymous Coward · 2012-09-10 07:22 · Score: 0

I'm a little glad we aren't so big that if our colocation network access fucks up we end up on slashdot.
I would've been leery of... by Anonymous Coward · 2012-09-10 07:23 · Score: 5, Funny

...a service provider named Recurly in the first place.
Same goes for any provider named Relarry, Remoe or Reshemp either for that matter.
1. Re:I would've been leery of... by Anonymous Coward · 2012-09-10 08:58 · Score: 0
  
  Don't forget Rejoe you insensitive clod!
2. Re:I would've been leery of... by Alien+Being · 2012-09-10 13:37 · Score: 2
  
  I'm Honest Moe, that's Honest Shemp, and that's... that's Larry.
No backups by Anonymous Coward · 2012-09-10 07:51 · Score: 3, Interesting

This is a perfect example of redundancy not being the same as backups. They had redundant encryption devices, but the failure of one rolled over into the other. They had no backups (that's right, none at all) that they could restore from. From what they've told us, they intend to resolve this issue by adding more redundancy.
Yes, really.
1. Re:No backups by Anonymous Coward · 2012-09-10 08:34 · Score: 3, Funny
  
  They should have used RAID.
2. Re:No backups by tlhIngan · 2012-09-10 09:18 · Score: 3, Informative
  
  This is a perfect example of redundancy not being the same as backups. They had redundant encryption devices, but the failure of one rolled over into the other. They had no backups (that's right, none at all) that they could restore from. From what they've told us, they intend to resolve this issue by adding more redundancy.
  Correction, they have no backups of the keys that the encryption accellerators used. End result is now they have a bunch of encrypted data, with little in the way of being able to recover it because the keys used are lost or corrupted.
  Sounds like they need to be hacked and their information "liberated" so they can recover it :).
3. Re:No backups by Anonymous Coward · 2012-09-10 09:48 · Score: 0
  
  Backups are not backups if you can't recover from them. They needed to have copies of data, separated from their production environment, along with a copy of encryption keys to access those backups. My understanding is that they didn't even have snapshot backups, let alone encrypted ones with keys backed up separately.
4. Re:No backups by Anonymous Coward · 2012-09-10 15:41 · Score: 0
  
  I don't think your grasping his joke :)
Another Leafycaust by Anonymous Coward · 2012-09-10 08:25 · Score: 0

It was due
Coitainly! by Anonymous Coward · 2012-09-10 10:15 · Score: 0

Nuyk nuyk nuyk.