Ma.gnolia User Data Is Gone For Good

miller60 writes "The social bookmarking service Ma.gnolia reports that all its user data was irretrievably lost in the Jan. 30 database crash that knocked the service offline. Ma.gnolia founder Larry Halff recently discussed the crash and the lessons to be learned from Ma.gnolia's experience. A lesson for users: don't assume online services have lots of staff and servers, and always keep backup copies of your data. Ma.gnolia was a one-man operation running on two Mac OS X servers and four Mac minis."

Comments

Food for Stallman

[Rinisari]

This bad news is delicious food for Stallman's argument against "cloud" services.

Re:Food for Stallman

[ZeroPly]

Stallman's argument is more that cloud services are almost always non-open. He does not have a per se objection to cloud services - and if you were to reveal all your source code and protocols, I doubt it would be objectionable to him.

Of course it's impossible to free cloud services in the sense of modification and distribution, but if the source is open you have the chance to make your own.

Re:Food for Stallman

[Anonymous Coward]

In response to your sig... Never let an officer see you swallow something that he may reasonably believe to be evidence. Really. You will regret it. Swallowed != lost.

Re:Food for Stallman

[hobo sapiens]

Well, that and the fact that many cloud services destroy your privacy. RMS argues that we are being shortsighted to trade our privacy for "kewl!"

He's right, you know! About everything.

http://www.guardian.co.uk/technology/2008/sep/29/cloud.computing.richard.stallman

Needless loss

[hattig]

Argh, why not just add a backup or replication database on one of the spare Mac Minis?

That way you would have needed a complete server farm disaster to mess things up irretrievably.

Re:Needless loss

[qoncept]

Or back it up, like, once a day, or week, or ever, to a flash drive or something. That's a lesson that's already been learned, and it's common sense. I'm terrible about backing up my own data (anything I've lost and recovered is usually something that just happened to be on a remote web server somewhere, coincidentally, because it was always intended to be on the web). But all of my websites, with other users' data, are backed up. It doesn't take a very complex scheme or much thought. A cron job to dump your database and tar your web structure and then copy it to a different location.

I definitely have my doubts that someone who could make this mistake is all that capable "lessons learned."

Re:Needless loss

[eln]

A simple periodic dump to an external hard drive would have at least been something. I know that small-time operations shouldn't be expected to have robust backup schemes, but if your primary purpose is to store other people's data, the FIRST thing on your mind should be how to back it up. Once you lose someone's data, they'll never use anything you put out again, and they'll tell all their friends not to either.

Re:Needless loss

[Ephemeriis]

Argh, why not just add a backup or replication database on one of the spare Mac Minis?

That way you would have needed a complete server farm disaster to mess things up irretrievably.

Replication gives you redundancy, much like RAID does. It lets you survive a hardware failure or two. It is not a backup. If the building burns down, or a tree falls on your server room, or lightning fries everything you are still screwed.

What they needed was a backup. A tape, or removable HDD, or a flash drive, or a CD, or something that can be taken out of the building on a regular basis. Once a day, once a week, once a month... Whatever.

Then, no matter what happens to your live hardware, you've got a backup you can restore from. Buy some new hardware, throw your backup at it, done!

Re:Needless loss

[Anonymous Coward]

Small time operations should be expected to back up just the same as large ones. I wrote simple routines to backup my db nightly, compress it, upload it and on the receiving end decompress it and restore it. If any step fails it emails me. I check it manually every week and save backups for 2 years (quite a bit of data but for legal reasons it's necessary).

The whole setup took me maybe a day to get working. There is NO excuse for not having backups.

I lost one of my primary servers on a sunday at around 5pm. Had everything working by the next morning on my backup servers. Never was I so happy that I had good backup and implementation/restoration plans.

Re:Needless loss

[metamatic]

Except cron+tar isn't sufficient. You need versioning. Otherwise if your database is corrupted and you don't notice immediately, your backup gets corrupted automatically.

I back up my web sites using cron + rsync + rdiff-backup.

Re:Needless loss

[Blakey Rat]

If you read the transcript, that's what they were doing, a simple firewire DB dump.

The problem is that they never tested the backups, and they didn't keep versioned backups. So they'd been backing-up the corrupted database for awhile before the site finally crashed for good. When it crashed, they only had the corrupted database backup. Additionally, the DB server was on RAID but of course the corrupted DB would just get saved to both HDs, so that's no good in a situation like this.

Basically, when the site crashed, he had three copies (2 RAID, 1 backup) of the data: all corrupt. The guy wasn't totally retarded when it came to backups, just 80% retarded.

Who the hell is Ma.gnolia

[Jailbrekr]

And how can they be slashdot worthy when they are a social networking site with ONLY a half a terabyte of data? In short, who cares?

Re:Mac reliability

[jetsci]

So umm...I have a confession...

I had no idea anyone actually used Mac's as servers. Sure, I bet you can get apache running or something but I didn't realize anyone had. Therefore, this is my first bit of exposure to this idea of Macs as servers and its all negative!

Woe is me.

Lesson?

[The Moof]

discussed the crash and the lessons to be learned

Lessons such as "Regularly monitor and maintain backups like and business should?"

Lessons Learned??

[ACK!!]

Like frickin' having a backup? Isn't that one of the first things you ever learn if your business relies on computers + userdata?

Re:Food for Fault-Tolerance

[RobBebop]

It's food for any argument against any web service that doesn't publish it's reliability information or publicize the data for what types of mechanisms it has in place in case of disasters like a corrupt database, fried motherboard, or busted hard drive.

There's a design methodology that's used by NASA for manned missions: Any individual component should be able to fail without compromising the mission. Of course, in the last few decades we've seen 2 out of 5 Shuttles go ka-boom! so obviously this NASA guideline isn't enough and it's *REALLY* hard to prevent failure when a perfect storm of multiple systems experience failure at the same time.

So if anything, I'd say this is an argument that supports robust, reliable, fault-tolerant design rather than just kludging a half dozen systems together and calling it a "web service".

Ma-who?

[Anonymous Coward]

One man operation which doesn't make backups, sounds fairly typical to me, remind me again why this is Slashdot worthy?

What's with the OS X users losing data?

[PrimeWaveZ]

I mean, just because a few medium-profile sites running on Macs have experienced a failure causing data loss doesn't make them unique. Every OS and every type of hardware will, at some point, experience a failure. It's the PEOPLE that make the failure a problem, and it sure looks like this tard was a problem.

Who the hell doesn't back up their data? Seriously? This is "Slashdot worthy" because some hapless Mac user lost their data. BOO FUCKING FAIL. Move on.

Re:Mac reliability

[beelsebob]

Yes, because they don't come with apache and php pre-installed, only a ticky box away from running.

Seriously, do people still not realise that OS X is just UNIX with a pretty UI?

Hardware RAID isn't magic, mirrors aren't archives

[argent]

LH: "The server was RAID. Its disk was RAID, so that's one of the things we're looking at. But it was a software RAID, so if it's a filesystem problem then... that's not gonna do any good because the the errors were RAIDed as well."

Since the file system and database were corrupted, it wouldn't matter if it was hardware RAID or software RAID. That's not the problem at all, the problem is there was no archival backup, and their only backup was a file sync... that replicated the database errors on the backup.

To backup a database, you dump it in a serialized form, or maintain a serialized form of the data in parallel with the database.

Free service

[tsstahl]

And the users got what they paid for.

Simple as that.

The flip side is that this guy's service will probably be the MOST reliable going forward.

Of course he should have had reliable backups; now he is the poster child for backups. Remember, nobody pays you for backups, only for restores.

Backup Testing?

[skyriser2]

Ouch... Isn't part of a backup strategy to sometimes attempt a recovery from a backup, on a test system?

Re:Mac reliability

[Anonymous Coward]

And here I thought OS X was just BSD with the Mach kernel and some fancy API.

Re:Mac reliability

[SatanicPuppy]

Mac servers are pretty. They do okay, they have nice swanky data enclosures, and the form factor is roughly the same as anyone elses.

It's just whether or not you want to use OS X. I disagree that OS X is "just unix," however. It's not even "just linux" or "just bsd". OS X has it's own warts, and while it may be stable and friendly, I'd rather have a real *nix running on less pretty hardware.

The best use I've ever had for the big Mac servers is running as a file server in a windows/mac environment. If you still have any pre-OS X machines around, that's about the only way to get them all on the same machine (If you say windows mac volume, I'm mailing a dead fish to your house).

Otherwise, you know, you can install apache, whatever, but it's not any different from using a regular linux server in terms of increased functionality, and there are some significant OS update issues that can cause problems. Mac updates are of the all or nothing school, and they WILL break stuff, so you need to be careful.

Re:Food for Fault-Tolerance

[bsDaemon]

NASA guideline isn't enough and it's *REALLY* hard to prevent failure when a perfect storm of multiple systems experience failure at the same time.

I'm not saying that saving Apollo 13 wasn't hard, or an extremely great accomplishment, however I am going to say "slick and pretty" (the shuttle) is generally the opposite of "robust" or "fault tolerant." Slick and pretty is also usually more expensive.

The basic, non-pimped xserve is $2999. An identically configured node from eRacks, running your choice of BSD (the default on these quad-core Xeons seems to be OpenBSD) or Linux, $1894 -- leaving you with plenty of room in the budget to build a bigger, badder node, or replace one when it fails.

I suppose the point i'm trying to make is that if you're going for function over form (Apollo capsul), it easier to plan for more contingencies on the same budget you'd otherwise be spending on gee-whiz factor (shuttle).

Re:Mac reliability

[meadowsoft]

I think you are missing the importance of a disaster recovery plan, with backups, for any mission critical hardware, regardless of vendor. Why didn't Mag have any sort of backup plan that was tested? Clustered hardware does not equal a backup plan - thanks for trying there.

Was there in fact a schedule of backups of the operational system? This seems like a rubber band and duct tape operation to me.

Re:Food for Fault-Tolerance

[NormalVisual]

obviously this NASA guideline isn't enough and it's *REALLY* hard to prevent failure when a perfect storm of multiple systems experience failure at the same time.

Neither the Challenger nor the Columbia represented simultaneous multiple failures. They *did* represent cascade failures that should have been planned for, but weren't.

Re:Mac reliability

[Anonymous Coward]

Why would you pay Apple $3000 for a xserve running Apache and MySQL...when you could buy an equivalent Dell server for $2100, running the exact same Apache and MySQL

You wouldn't. It's a "right tool for the job" situation and XServes aren't the right tool for running Apache and MySQL. They have the flexibility to run Apache and Mysql, which is nice if you buy them for some other purpose and then either no longer need them for that purpose or find that you have spare capacity and want to use it that way. But if you're buying a server dedicated to tools available for Linux, then the XServe is probably not the best option.

Anyone who buys an xserve is an idiot.

Or someone who needs a server that does things that a Linux server can't. There is software that's designed to run on OS X servers. And, despite Apple's efforts to make OS X desktops integrate well in either Unix or Windows networks (NFS, Kerberos, SMB, Active Directory, ect), there are things that an OS X server can offer network with OS X machines. If you need OS X desktops, an OS X server has definite uses. If you need to any of those applications or if your network is primarily OS X desktops, you buy an XServe. XServes are not the best solution to put in a data center and use to run a website. But that doesn't mean there aren't reasons to buy one. And buying one for a purpose that makes sense doesn't make you an idiot. What makes you an idiot is buying the wrong tool for the job. That, and making overly-broad generalizations.

Re:Mac reliability

[SSCGWLB]

Wait, you mean apples makes all their own hardware? Really?

No, they use CPUs from Intel, hard drives from WD/Seagate/Maxtor/whoever, graphics chips from nVidea/ATI, etc.

So, no, you do NOT get what you pay for hardware wise. You are getting something identical or very similar to everybody else, from the same manufactures, you are just paying a little more.