Ma.gnolia User Data Is Gone For Good

miller60 writes "The social bookmarking service Ma.gnolia reports that all its user data was irretrievably lost in the Jan. 30 database crash that knocked the service offline. Ma.gnolia founder Larry Halff recently discussed the crash and the lessons to be learned from Ma.gnolia's experience. A lesson for users: don't assume online services have lots of staff and servers, and always keep backup copies of your data. Ma.gnolia was a one-man operation running on two Mac OS X servers and four Mac minis."

Comments

Mac reliability

[Anonymous Coward]

Crashing Macs? That's unpossible!

Re:Mac reliability

[beelsebob]

Yes, because they don't come with apache and php pre-installed, only a ticky box away from running.

Seriously, do people still not realise that OS X is just UNIX with a pretty UI?

Re:Mac reliability

[Idiomatick]

In the same sense that horses and monkeys are JUST mammals. Doesn't mean that they share THAT much in common...

Re:Mac reliability

[Blakey Rat]

Yeah, but that's exactly the surprising part. Why would you pay Apple $3000 for a xserve running Apache and MySQL, with a crappy service contract (no next-day service, no on-site service-- I've looked into it), when you could buy an equivalent Dell server for $2100, running the exact same Apache and MySQL, and get a next-day and on-site service contract?

Anyone who buys an xserve is an idiot.

Re:Mac reliability

[blhack]

Even IBM does better, and they suck.

One morning I came in and was looking at the logs. SMART was reporting that one of the disks in one of the servers was going to go bad soon. Not 15 minutes after i even noticed this in the logs, an IBM tech was there with a fresh one ready to replace it.

How? The server called home, told IBM about the error, and they disbatched a tech immediately.

If that "sucks", your service must come with free hookers or something.

In other news

[jetsci]

Facebook was recently brought down when their hamster keeled over and ceased powering their Amiga.

Re:In other news

[elrous0]

In that case, the death of Mr. Furry was a much greater loss than the data.

Re:In other news

[palegray.net]

I think it's a tragedy that the first thing that came to mind reading your post involved a blood-spattered "furry" (as in a dude wearing some fluffy costume). The Internet has done terrible things to my mind.

Needless loss

[hattig]

Argh, why not just add a backup or replication database on one of the spare Mac Minis?

That way you would have needed a complete server farm disaster to mess things up irretrievably.

Re:Needless loss

[eln]

A simple periodic dump to an external hard drive would have at least been something. I know that small-time operations shouldn't be expected to have robust backup schemes, but if your primary purpose is to store other people's data, the FIRST thing on your mind should be how to back it up. Once you lose someone's data, they'll never use anything you put out again, and they'll tell all their friends not to either.

Re:Needless loss

[Ephemeriis]

Argh, why not just add a backup or replication database on one of the spare Mac Minis?

That way you would have needed a complete server farm disaster to mess things up irretrievably.

Replication gives you redundancy, much like RAID does. It lets you survive a hardware failure or two. It is not a backup. If the building burns down, or a tree falls on your server room, or lightning fries everything you are still screwed.

What they needed was a backup. A tape, or removable HDD, or a flash drive, or a CD, or something that can be taken out of the building on a regular basis. Once a day, once a week, once a month... Whatever.

Then, no matter what happens to your live hardware, you've got a backup you can restore from. Buy some new hardware, throw your backup at it, done!

Re:Needless loss

[Blakey Rat]

If you read the transcript, that's what they were doing, a simple firewire DB dump.

The problem is that they never tested the backups, and they didn't keep versioned backups. So they'd been backing-up the corrupted database for awhile before the site finally crashed for good. When it crashed, they only had the corrupted database backup. Additionally, the DB server was on RAID but of course the corrupted DB would just get saved to both HDs, so that's no good in a situation like this.

Basically, when the site crashed, he had three copies (2 RAID, 1 backup) of the data: all corrupt. The guy wasn't totally retarded when it came to backups, just 80% retarded.

Who the hell is Ma.gnolia

[Jailbrekr]

And how can they be slashdot worthy when they are a social networking site with ONLY a half a terabyte of data? In short, who cares?

Re:Food for Stallman

[ZeroPly]

Stallman's argument is more that cloud services are almost always non-open. He does not have a per se objection to cloud services - and if you were to reveal all your source code and protocols, I doubt it would be objectionable to him.

Of course it's impossible to free cloud services in the sense of modification and distribution, but if the source is open you have the chance to make your own.

Finally! A privacy solution!

[Captain Spam]

All right, let me get this straight: First you people bitch and moan when Facebook says they'll save user data forever. NOW you people bitch and moan when this site loses user data forever! You're never happy, are you?!?

Re:Lesson?

[Knowbuddy]

Lessons such as "Regularly monitor and maintain backups like [any] business should?"

I love it when people say things like this. It shows me that they've never actually had to set up an enterprise backup strategy. I'm certainly not defending the Ma.gnolia guys, but I also can't stand it when people are on a shakier soapbox than they realize.

I'm sorry, but when you are used to the whiz-bang-pretty of Web2.0, the state of enterprise-level backups is horrifically archaic and dismal. And, btw, given the size of today's hard drives and databases, for pretty much all intents and purposes "Enterprise" == "More than one computer with more than just a few files on a drive".

Compare and contrast: a 1 TB hard drive will run you roughly $100. Do you know how much it then costs to backup that TB?

• LTO-4 tapes, 800GB each, $50-$150 each tape plus roughly $2500 for the drive. Figure 2 tapes/day * 10 days backups = 20 tapes * $100 = $2000 in tapes alone. Congrats, that 1 TB just cost you $4500 in enterprise backups ... not to mention the time involved each day in doing a backup. You might save yourself some time and money by doing incrementals ... but then you have to balance that risk with complexity of backups and difficulty in restores.

• NAS is trickier. The cheap NAS solutions, sub $1000 such as Buffalo and LaCie, aren't going to get you much more than a TB or two. And at that point, are you really any better off than the RAID solution? Maybe, maybe not. As you start to scale into IBM or Dell solutions, you are almost immediately beyond a $2500 price point before you even get to hard drives. Oh, and don't forget the cost of a gigabit switch so that it doesn't take you days to do a single backup.

• iSCSI? Seriously? Not an option for SOHO businesses.

Then there's backup software to contend with. It's not just as simple as "go buy a copy of BackupExec" -- there's different licensing for databases, and network backups, and whatever arcane rules they want today. I'm a PC guy so I can't talk much about Enterprise-level Mac backup solutions, but I can without a doubt say that Time Machine is not one of them.

It's even more dismal when it comes to Open Source solutions. Have you ever actually tried to setup Bacula? It may be the 600lb gorilla of OS backup solutions, but it's still a royal pain. And to the "just set up a cron job for rsync" guys, c'mon, really? Good luck with that.

So, please, let's dispense with the thought that backups are easy. Backups really suck. Hard. That's why so many people want to think of RAID as a backup solution -- because the step from one hard drive to two or three is easy, but the next step is much farther away than you think.