Ma.gnolia User Data Is Gone For Good

miller60 writes "The social bookmarking service Ma.gnolia reports that all its user data was irretrievably lost in the Jan. 30 database crash that knocked the service offline. Ma.gnolia founder Larry Halff recently discussed the crash and the lessons to be learned from Ma.gnolia's experience. A lesson for users: don't assume online services have lots of staff and servers, and always keep backup copies of your data. Ma.gnolia was a one-man operation running on two Mac OS X servers and four Mac minis."

Comments

Mac reliability

[Anonymous Coward]

Crashing Macs? That's unpossible!

Re:Mac reliability

[jetsci]

So umm...I have a confession...

I had no idea anyone actually used Mac's as servers. Sure, I bet you can get apache running or something but I didn't realize anyone had. Therefore, this is my first bit of exposure to this idea of Macs as servers and its all negative!

Woe is me.

Re:Mac reliability

[BJZQ8]

http://www.apple.com/xserve/

Re:Mac reliability

[beelsebob]

Yes, because they don't come with apache and php pre-installed, only a ticky box away from running.

Seriously, do people still not realise that OS X is just UNIX with a pretty UI?

Re:Mac reliability

[Anonymous Coward]

annd zunes were never cool. and no one has them...

Re:Mac reliability

[Idiomatick]

In the same sense that horses and monkeys are JUST mammals. Doesn't mean that they share THAT much in common...

Re:Mac reliability

[neokushan]

I want to say something here, but I get the feeling that no matter what I say, I just wont be herd.

Re:Mac reliability

[SatanicPuppy]

Mac servers are pretty. They do okay, they have nice swanky data enclosures, and the form factor is roughly the same as anyone elses.

It's just whether or not you want to use OS X. I disagree that OS X is "just unix," however. It's not even "just linux" or "just bsd". OS X has it's own warts, and while it may be stable and friendly, I'd rather have a real *nix running on less pretty hardware.

The best use I've ever had for the big Mac servers is running as a file server in a windows/mac environment. If you still have any pre-OS X machines around, that's about the only way to get them all on the same machine (If you say windows mac volume, I'm mailing a dead fish to your house).

Otherwise, you know, you can install apache, whatever, but it's not any different from using a regular linux server in terms of increased functionality, and there are some significant OS update issues that can cause problems. Mac updates are of the all or nothing school, and they WILL break stuff, so you need to be careful.

Re:Mac reliability

[FishWithAHammer]

Are any of the free BSDs or Linux variants certified Unixes?

(Honest question, I don't know.)

Re:Mac reliability

[Blakey Rat]

Yeah, but that's exactly the surprising part. Why would you pay Apple $3000 for a xserve running Apache and MySQL, with a crappy service contract (no next-day service, no on-site service-- I've looked into it), when you could buy an equivalent Dell server for $2100, running the exact same Apache and MySQL, and get a next-day and on-site service contract?

Anyone who buys an xserve is an idiot.

Re:Mac reliability

[idontgno]

That'd be gnus to me.

Re:Mac reliability

[Blakey Rat]

Fine; what company do you trust? HP? IBM? Replace "Dell" with them, and my example still applies. The fact is, *every* server vendor can do better than Apple. Even IBM does better, and they suck.

Oh, and BTW, all servers will have hardware problems from time-to-time. When that happens with your Dell, HP, IBM server, the guy is there in his truck in 4 hours. When that happens to your Apple server, you're SOL.

Re:Mac reliability

[meadowsoft]

I think you are missing the importance of a disaster recovery plan, with backups, for any mission critical hardware, regardless of vendor. Why didn't Mag have any sort of backup plan that was tested? Clustered hardware does not equal a backup plan - thanks for trying there.

Was there in fact a schedule of backups of the operational system? This seems like a rubber band and duct tape operation to me.

Re:Mac reliability

[DittoBox]

Except its the same hardware...well no, that's not true. You can get a Dell with actual hardware RAID when you're stuck with software RAID on an Xserve.

Furthermore Dell also has a 4-hour onsite 24/7 support package if I'm not mistaken.

I love my MacBook and the OS X desktop experience but you simply can't use an Xserve on business critical operations.

Re:Mac reliability

[DoofusOfDeath]

(If you say windows mac volume, I'm mailing a dead fish to your house).

Why, so it will attract the penguins?

Re:Mac reliability

[Anonymous Coward]

But that's more of a PR thing than anything. If I raise cows in the pasture behind my hose then they aren't "USDA Certified Organic" or any other such thing, but that doesn't really change what they are - it just means the haven't been inspected an labeled by some committee.

Same with Mac OS X being "Unix". It's more of a stamp of approval than anything.

Re:Mac reliability

[Anonymous Coward]

Why would you pay Apple $3000 for a xserve running Apache and MySQL...when you could buy an equivalent Dell server for $2100, running the exact same Apache and MySQL

You wouldn't. It's a "right tool for the job" situation and XServes aren't the right tool for running Apache and MySQL. They have the flexibility to run Apache and Mysql, which is nice if you buy them for some other purpose and then either no longer need them for that purpose or find that you have spare capacity and want to use it that way. But if you're buying a server dedicated to tools available for Linux, then the XServe is probably not the best option.

Anyone who buys an xserve is an idiot.

Or someone who needs a server that does things that a Linux server can't. There is software that's designed to run on OS X servers. And, despite Apple's efforts to make OS X desktops integrate well in either Unix or Windows networks (NFS, Kerberos, SMB, Active Directory, ect), there are things that an OS X server can offer network with OS X machines. If you need OS X desktops, an OS X server has definite uses. If you need to any of those applications or if your network is primarily OS X desktops, you buy an XServe. XServes are not the best solution to put in a data center and use to run a website. But that doesn't mean there aren't reasons to buy one. And buying one for a purpose that makes sense doesn't make you an idiot. What makes you an idiot is buying the wrong tool for the job. That, and making overly-broad generalizations.

Re:Mac reliability

[SSCGWLB]

Wait, you mean apples makes all their own hardware? Really?

No, they use CPUs from Intel, hard drives from WD/Seagate/Maxtor/whoever, graphics chips from nVidea/ATI, etc.

So, no, you do NOT get what you pay for hardware wise. You are getting something identical or very similar to everybody else, from the same manufactures, you are just paying a little more.

Re:Mac reliability

[period3]

and it comes with a free black turtleneck!

Re:Mac reliability

[blhack]

Even IBM does better, and they suck.

One morning I came in and was looking at the logs. SMART was reporting that one of the disks in one of the servers was going to go bad soon. Not 15 minutes after i even noticed this in the logs, an IBM tech was there with a fresh one ready to replace it.

How? The server called home, told IBM about the error, and they disbatched a tech immediately.

If that "sucks", your service must come with free hookers or something.

Re:Mac reliability

[ThrowAwaySociety]

a crappy service contract (no next-day service, no on-site service-- I've looked into it)

Not very hard, apparently.
http://www.apple.com/server/support/

You get 24/7 telephone and email support with 30-minute response. For hardware repairs, Apple-certified technicians provide onsite response within four hours during business hours and next-day onsite response when you contact Apple after business hours.

Re:Mac reliability

[nxtw]

1. OS X is certified Unix. I don't care if you disagree. You can disagree with gravity too if you want, but it will still keep your feet on the ground.

This is meaningless today. Most Unixlike systems today are not certified Unix systems.
OS X has some significant differences from traditional Unixlike systems and Linux - not necessarily disadvantages:

• OS X has a Mach-based microkernel (XNU) and a completely different driver model (I/O Kit)
• OS X's VM (virtual memory) system performs poorly, and swapping cannot be safely disabled. If you do disable swap, the OS acts poorly when it thinks it's run out of memory - it grinds to a halt. Linux by default will kill the process using the most memory if it runs out - this is configurable. Also, since Linux's VM system is configurable enough that you can control just how much is swapped and how much is cached, you can set Linux so that swap is only touched when it is really needed.
• OS X is huge - it's much larger than a minimal installation of a server Linux distribution. The GUI is not an optional component.

3. There is increased functionality above a "regular linux server". Take xgrid for instance. Many of the tasks it does it can do work only with mac software.

You can certainly add grid computing software to other operating systems. OS X is missing some functionality that a "regular linux server" may have. Even when considering third-party software, there are many things that can be done in Linux but not in OS X.

• OS X is somewhat lacking clustering ir high availability fatures. For example, a "regular linux server" running CentOS has DRBD, for distributed block devices, and GFS, a cluster file system. It doesn't have application failover of the type provided by Windows's built-in clustering or by Linux-HA; OS X only has IP failover.
• OS X has few server virtualization options - just Parallels. VMware ESX supports Windows, Linux, and Solaris. Commercial Xen distributions support Windows and Linux; paravirtualized versions of BSDs and OpenSolaris are available too. Solaris, FreeBSD, and addons to Linux provide effective kernel virtualization. Microsoft's Hyper-V runs Windows and can even run SUSE (and eventually RHEL.)
• OS X has no built-in or even officially supported iSCSI initiator. There is one free closed-source initiator, but commerical support is not offered - making it not very suitable for server use. Microsoft has a supported iSCSI initiator that works with Windows 2000 and up, and it's included in 2008 and Vista. iSCSI initiation is supported by many Linux distributions, including Red Hat and Suse.

Mac updates are not all or nothing. What in the world do you even mean by that? They have point updates, but so does linux. This doesn't mean they don't have updates for individual OS components too. There was a security update just the other day that wasn't "all or nothing."

Mac OS X security updates certainly are "all or nothing" - you have to install all of the patches included in the package or install none of them. Each package includes many fixes, and sometimes they break things. The updates are not available as individual pkacages. You cannot select which updates are applied to the system.

RHEL/CentOS has point releases, but there are plenty of individual package updates in between (to fix bugs, compatibility, and security issues.) Individual package updates are released when they are ready, not as part of a large security update bundle or a monthly schedule.

Re:Mac reliability

[drinkypoo]

It means you don't have to pay the performance penalty that netatalk has from resource fork handling since HFS+ is a native file system.

So if you're dealing with lots of small files with both forks, you're going to pay a penalty. What that says to me is that there are certain limited cases in which you might still need to use an Apple fileserver for performance, but in almost every real-world case where you actually have macs as clients, is it really an issue?

In other news

[jetsci]

Facebook was recently brought down when their hamster keeled over and ceased powering their Amiga.

Re:In other news

[elrous0]

In that case, the death of Mr. Furry was a much greater loss than the data.

Re:In other news

[palegray.net]

I think it's a tragedy that the first thing that came to mind reading your post involved a blood-spattered "furry" (as in a dude wearing some fluffy costume). The Internet has done terrible things to my mind.

Needless loss

[hattig]

Argh, why not just add a backup or replication database on one of the spare Mac Minis?

That way you would have needed a complete server farm disaster to mess things up irretrievably.

Re:Needless loss

[qoncept]

Or back it up, like, once a day, or week, or ever, to a flash drive or something. That's a lesson that's already been learned, and it's common sense. I'm terrible about backing up my own data (anything I've lost and recovered is usually something that just happened to be on a remote web server somewhere, coincidentally, because it was always intended to be on the web). But all of my websites, with other users' data, are backed up. It doesn't take a very complex scheme or much thought. A cron job to dump your database and tar your web structure and then copy it to a different location.

I definitely have my doubts that someone who could make this mistake is all that capable "lessons learned."

Re:Needless loss

[eln]

A simple periodic dump to an external hard drive would have at least been something. I know that small-time operations shouldn't be expected to have robust backup schemes, but if your primary purpose is to store other people's data, the FIRST thing on your mind should be how to back it up. Once you lose someone's data, they'll never use anything you put out again, and they'll tell all their friends not to either.

Re:Needless loss

[Ephemeriis]

Argh, why not just add a backup or replication database on one of the spare Mac Minis?

That way you would have needed a complete server farm disaster to mess things up irretrievably.

Replication gives you redundancy, much like RAID does. It lets you survive a hardware failure or two. It is not a backup. If the building burns down, or a tree falls on your server room, or lightning fries everything you are still screwed.

What they needed was a backup. A tape, or removable HDD, or a flash drive, or a CD, or something that can be taken out of the building on a regular basis. Once a day, once a week, once a month... Whatever.

Then, no matter what happens to your live hardware, you've got a backup you can restore from. Buy some new hardware, throw your backup at it, done!

Re:Needless loss

[Blakey Rat]

If you read the transcript, that's what they were doing, a simple firewire DB dump.

The problem is that they never tested the backups, and they didn't keep versioned backups. So they'd been backing-up the corrupted database for awhile before the site finally crashed for good. When it crashed, they only had the corrupted database backup. Additionally, the DB server was on RAID but of course the corrupted DB would just get saved to both HDs, so that's no good in a situation like this.

Basically, when the site crashed, he had three copies (2 RAID, 1 backup) of the data: all corrupt. The guy wasn't totally retarded when it came to backups, just 80% retarded.

Who the hell is Ma.gnolia

[Jailbrekr]

And how can they be slashdot worthy when they are a social networking site with ONLY a half a terabyte of data? In short, who cares?

Re:Who the hell is Ma.gnolia

[KDingo]

At least if you had privacy concerns with them, you have nothing to worry about now.

Re:Food for Stallman

[ZeroPly]

Stallman's argument is more that cloud services are almost always non-open. He does not have a per se objection to cloud services - and if you were to reveal all your source code and protocols, I doubt it would be objectionable to him.

Of course it's impossible to free cloud services in the sense of modification and distribution, but if the source is open you have the chance to make your own.

lesson #1

[petes_PoV]

on the 'net you can't tell the major corporation from the kid in a garage

lesson #2, trust no-one with your data

lesson #3 disaster recovery capability only exists after it's been tested

lesson #4 backups are useless unless you can prove you can recover from them

Lesson?

[The Moof]

discussed the crash and the lessons to be learned

Lessons such as "Regularly monitor and maintain backups like and business should?"

Re:Lesson?

[Knowbuddy]

Lessons such as "Regularly monitor and maintain backups like [any] business should?"

I love it when people say things like this. It shows me that they've never actually had to set up an enterprise backup strategy. I'm certainly not defending the Ma.gnolia guys, but I also can't stand it when people are on a shakier soapbox than they realize.

I'm sorry, but when you are used to the whiz-bang-pretty of Web2.0, the state of enterprise-level backups is horrifically archaic and dismal. And, btw, given the size of today's hard drives and databases, for pretty much all intents and purposes "Enterprise" == "More than one computer with more than just a few files on a drive".

Compare and contrast: a 1 TB hard drive will run you roughly $100. Do you know how much it then costs to backup that TB?

• LTO-4 tapes, 800GB each, $50-$150 each tape plus roughly $2500 for the drive. Figure 2 tapes/day * 10 days backups = 20 tapes * $100 = $2000 in tapes alone. Congrats, that 1 TB just cost you $4500 in enterprise backups ... not to mention the time involved each day in doing a backup. You might save yourself some time and money by doing incrementals ... but then you have to balance that risk with complexity of backups and difficulty in restores.

• NAS is trickier. The cheap NAS solutions, sub $1000 such as Buffalo and LaCie, aren't going to get you much more than a TB or two. And at that point, are you really any better off than the RAID solution? Maybe, maybe not. As you start to scale into IBM or Dell solutions, you are almost immediately beyond a $2500 price point before you even get to hard drives. Oh, and don't forget the cost of a gigabit switch so that it doesn't take you days to do a single backup.

• iSCSI? Seriously? Not an option for SOHO businesses.

Then there's backup software to contend with. It's not just as simple as "go buy a copy of BackupExec" -- there's different licensing for databases, and network backups, and whatever arcane rules they want today. I'm a PC guy so I can't talk much about Enterprise-level Mac backup solutions, but I can without a doubt say that Time Machine is not one of them.

It's even more dismal when it comes to Open Source solutions. Have you ever actually tried to setup Bacula? It may be the 600lb gorilla of OS backup solutions, but it's still a royal pain. And to the "just set up a cron job for rsync" guys, c'mon, really? Good luck with that.

So, please, let's dispense with the thought that backups are easy. Backups really suck. Hard. That's why so many people want to think of RAID as a backup solution -- because the step from one hard drive to two or three is easy, but the next step is much farther away than you think.

Lessons Learned??

[ACK!!]

Like frickin' having a backup? Isn't that one of the first things you ever learn if your business relies on computers + userdata?

What's with the OS X users losing data?

[PrimeWaveZ]

I mean, just because a few medium-profile sites running on Macs have experienced a failure causing data loss doesn't make them unique. Every OS and every type of hardware will, at some point, experience a failure. It's the PEOPLE that make the failure a problem, and it sure looks like this tard was a problem.

Who the hell doesn't back up their data? Seriously? This is "Slashdot worthy" because some hapless Mac user lost their data. BOO FUCKING FAIL. Move on.

"Private relaunch?"

[mr_mischief]

"Gee, Bob, we have the proof that this thing works. Why don't we sell it already?"

"Well, Bill, nobody wants to buy it and grandfather in all the whining freeloaders and their data."

"It's too bad we can't just drop all the data and start fresh."

"Well, why not, Bill? All we have to do is say it's been lost and can't be recovered. We can tell the buyer what's actually happening so they don't think we're total IT rejects who couldn't figure out a data retention policy."

"That's why I like working with you, Bob. You always have a way around the problem."

Have fun with it. The names have been changed (one changed anyway and one added), well, because it probably has nothing to do with reality. It sure is fun to ponder, though.

Transcript of interview

[Chalex]

Rather than watch the video or download the 23MB MP3, you can read the full transcript here:

http://ratafia.info/post/78915439/transcript-and-commentary-for-whither-magnolia

I can read much faster than I can listen.

Finally! A privacy solution!

[Captain Spam]

All right, let me get this straight: First you people bitch and moan when Facebook says they'll save user data forever. NOW you people bitch and moan when this site loses user data forever! You're never happy, are you?!?

Hardware RAID isn't magic, mirrors aren't archives

[argent]

LH: "The server was RAID. Its disk was RAID, so that's one of the things we're looking at. But it was a software RAID, so if it's a filesystem problem then... that's not gonna do any good because the the errors were RAIDed as well."

Since the file system and database were corrupted, it wouldn't matter if it was hardware RAID or software RAID. That's not the problem at all, the problem is there was no archival backup, and their only backup was a file sync... that replicated the database errors on the backup.

To backup a database, you dump it in a serialized form, or maintain a serialized form of the data in parallel with the database.

Re:Finally! A privacy solution!

[chaim79]

You have a 5 digit UID and you are just realizing this now?!?

!equivalent

[WiseWeasel]

Mac OS X Server runs a host of services, particularly for managing Mac OS X clients, that you won't find on any other OS, so there are reasons to get a Xserve in particular; web serving just is not one of them.

Comment removed

[account_deleted]

Comment removed based on user account deletion

Re:What is this "UNIX" you speak of...

[Dekortage]

Apple is 1% hardware and 99% Marketing. Not too much they do can't be done on a Dell or HP. They just make it appear to do it better/slicker/faster, that's all.

I'd peg it at 10% hardware, if not more. The internal hardware layout of Apple's desktop towers borders on beautiful. Beats Dell and HP hands down.

And, while its hardware failures tend to be more spectacular, I've generally found Apple hardware to be more reliable than any of the Wintel vendors. (...speaking as someone who has been supporting computers since before MS-DOS or the Mac...)