Why Mirroring Is Not a Backup Solution

Craig writes "Journalspace.com has fallen and can't get up. The post on their site describes how their entire database was overwritten through either some inconceivable OS or application bug, or more likely a malicious act. Regardless of how the data was lost, their undoing appears to have been that they treated drive mirroring as a backup and have now paid the ultimate price for not having point-in-time backups of the data that was their business." The site had been in business since 2002 and had an Alexa page rank of 106,881. Quantcast said they had 14,000 monthly visitors recently. No word on how many thousands of bloggers' entire output has evaporated.

Comments

DUH!

[Anonymous Coward]

DUH!

Re:DUH!

[djupedal]

As if millions of voices suddenly cried out in terror, and were suddenly silenced.

Re:DUH!

[larry bagina]

I pity the fool who hahas?

Re:DUH!

[hesiod]

We can only hope they remain silent.

Re:DUH!

[severoon]

Journalspace CTO: We don't need an expensive off-site backup solution b/c we mirror all of our data real-time. It's genius!

-entire database gets overwritten-

Journalspace CTO: Ohhhhhh...now I get it.

Re:DUH!

[NickFitz]

What about archive.org?

Ah, apparently not... :-D

Re:DUH!

[jcuervo]

What we've got here is failure to administrate.

Re:DUH!

[darthflo]

As if millions of voices constantly cried out in angst, and were finally silenced.

Fixed that for you. ;)

Re:DUH!

[lgw]

If your tape is readable the day you make it, it will be readable for many years. Tape (especially half-inch tape) doesn't really go bad over time - 20 year shelf life is common.

The reason you hear the "oh noooes, my tapes aren't readable" horror stories is because people are too lazy to verify the tapes at the time of creation, and the tape drives go silently bad over the years. Schedule a verify after every backup (and you don't need to verify everyhting you wrote, just a sample) and you won't have a problem with "bad tapes". Well, except maybe with QIC tape; what garbage.

Re:DUH!

[Kyril]

Amen to this -- at Fermilab we had a setup with lots of 8mm tapes, and we thought the MTBF was awfully high (several were failing each week), much more than the 30,000 hour MTBF specified...until we realized it was 30,000 hours with a 5% duty cycle, or 600 hours of use. 600 hours divided by even a dozen tapes is 50 hours, about 6 8-hour days and these were in use up to 24x6... The system also let us empirically confirm the single-bit error rate of DRAM, something on the order of 1 in 10^13 bits at the time.

Hot spare, hot swap, hot plug...that's how you gotta do it when you have so much hardware on hand that failures need to be planned for rather than prevented.

Again a frost post to a red story

[Anonymous Coward]

While this mirrors previous comments, it's not really a backup solution.

When is backing up *not* an option?

[wandazulu]

Mirroring, RAID, grid, whatever. At some point, you want your data safe and secure on something not physically attached to any power source.

Re:When is backing up *not* an option?

[Anonymous Coward]

Incremental backups to tape every night, full backup at the weekend. Tapes must be stored off-site at a proper storage location. Got lots of data and a small backup window? Get a faster tape drive and a tape robot. It costs money, but you data costs more.

This is at a minimum people. Come on!

Re:When is backing up *not* an option?

[z_gringo]

nightly dumps of the database and rsync of the data directories to servers in different locations should be adequate. If you have lots of data, I don't see how tapes are really going to do the daily backup jobs.

backing up nightly to a large mirrored NAS and a periodic copy to a removable device seems like a good way to go these days. I haven't used tapes for years.

Re:When is backing up *not* an option?

[postbigbang]

Nope.

Mirrors are fine, just snapshot them and store them offsite regularly. Do delta backups as needed but close-in for fast restoration.

There is no rational justification for tape anymore, what with the cost per TB stored on hard disks now under $130, total $$. Random accessibility unless you're stalling a subpoena, is just mandatory on backup media.

Re:When is backing up *not* an option?

[ba_hiker]

how 'bout this though.. hot-swap mirrored drives. pull 1/2 of the mirror at any time to make a backup. replace the pulled drives with blanks. keep a short stream of backup drives, say 8 or 9. drives are cheap. store in well padded metal boxes, offsite.

Re:When is backing up *not* an option?

[z_gringo]

NAS devices are cheaper and faster now. Lower end removable drives are not much more expensive than tapes, and they are a lot faster and easier to manage.

Re:When is backing up *not* an option?

[Wdomburg]

Even accepting your price that's a cost of about 12.7 cents per gigabyte and you can get 800GB native LTO-4 tapes for about $50, which comes out to about 6.3 cents per gigabyte.

But quoting costs for desktop grade SATA drives severely understates the true cost. For any non-trivial site installation you're talking near-line rated drives, drive caddies, storage shelves and additional SAN fabric. Then price out the additional power, cooling and rack space. Then price offsite shipping and storage for the bulkier, heavier and more delicate disk option.

Mirroring has its place. Snapshotting has its place. And backups to stable media still has its place too.

Re:When is backing up *not* an option?

[Trixter]

Amen, and why I just love ZFS (or any filesystem that supports instant snapshots). I use mirroring to cover drive failures, and I use weekly snapshots as backups. Once every three months, I offline to external disk. Minimum cost, more than reasonable protection.

Re:When is backing up *not* an option?

[Jon_E]

that's where trixter needs to zfs send/recv the snapshots to an offsite location (and probably roll snapshots more frequently)

Re:When is backing up *not* an option?

[Trixter]

That's not my company's policy, that's *my* policy. I can take a 3-month hit to my personal data. AND YET MY LAX PERSONAL POLICY WOULD HAVE SAVED JOURNALSPACE.

My *company's* policy is daily offsiting. Expensive, but very many of our locations could become a smoking hole in the ground and we'd still be able to restore and operate.

Re:When is backing up *not* an option?

[Wdomburg]

Fine. Get the cartridges, but what about the capital cost minus depreciation of the drive? What about random access?

Random access is why snapshots also have their place. :) Archival backups and nearline backups solve different sets of problems.

Now weigh those against an inexpensive jbod frame with a 2gb FC backplane.

What kind of capacity are we talking. For a small site you can pick up a little 2U unit that'll store 6.4TB uncompressed for under $5k. Or if you're running a larger site you can snag a 4U unit with two drives for about $15k that'll handle 30.4TB with optional expansion to 60.8TB native.

What's the write speed of LT vs a tasty little GB SAS drive?

120MB/sec per drive without compression. And now that you've talking about SAS drives your per TB cost is hopelessly optimistic. Even OEM packaged terabyte SAS drives are going to run you about a quarter a gigabyte, which is now four times the media cost of an LTO-4 solution.

Rackspace? You can put a dozen into about 4U.

So about 12TB in 4U compared to the 30TB unit I mention above.

Cooling? Although I'll grant you green cost, the random accessibility out-classes the seek time and tape insertion by a human cost dramatically.

Have you never heard of a tape library?

Stable media? Tape? Sometimes.

Properly handled tape is incredibly stable.

Shelf space?

If you're doing off-site storage, that's going to be an issue regardless of what media you're using. And as I pointed out, tape is far more compact and far lighter than disks.

No need to use tape anymore. Get out of the reality distortion field, but do the right thing by testing what you have and doing drills to ensure that whatever you have, works and is a procedure understood by all.

I'm not the one dismissing an entire class of technology while demonstrating ignorance of its costs and benefits.

Re:When is backing up *not* an option?

[SmurfButcher Bob]

I'm not sure what planet you're on, but I wish the rest of us were there with you.

Backup media should be and must be transported offsite every freakin day. You'd do that with a hard disk? Or more correctly, you'd do that with a STACK of hard disks? Or is your building fire, flood (including broken sprinkler pipes), gas leak, and drunken-truck-driver proof.

Re:When is backing up *not* an option?

[mabhatter654]

can you restore a RAID with different hardware? With LTO3 tape I have several drive choices.. notably I can by a NEW drive and know the tape will work even 3-4 years out. What happens when the maker of your RAID solution moves on and wants to send you next year's model? Will the encryption and striping still line up on different hardware made by a different company?

Re:When is backing up *not* an option?

[mabhatter654]

can a stored disk drive sit on a shelf for a year or two of being tussled about and still work? Tapes have nearly all the moving parts external and replaceable in the drive, not the media. A cardboard box of backup tapes will survive storage pretty well.. a box of disk drives not so much.

Re:When is backing up *not* an option?

[rrohbeck]

Anybody who uses disk based backup for a while finds out that it needs to be augmented by tape sooner or later. A disk based subsystem gets full pretty soon once you get used to the convenience. If you continue to buy disk drives it gets really expensive so people find that the best of both worlds is D2D2T. This reduces the size and speed of the tape subsystem you need, but doesn't make it obsolete. You need offsite storage anyway.

Re:When is backing up *not* an option?

[mabhatter654]

again with the power requirements! To keep even month and year would require a massive amount of extra hardware and power, not to mention people to tend it. I'd agree it's super good at recovery but how much more value are you getting versus tapes in a safety deposit box?

Re:When is backing up *not* an option?

[Darkk]

Not too bad of an idea, least there is duplicate data at a different location which is better than what these guys are proposing.

Ideal thing is to have the duplicated data be MILES from the main DC site but it's not always practical when you have large volume of data to replicate and backup. Yes I know all about rysnc and smart data replication but still nothing like good old fashioned complete full backups at a expense of time. That always seems to work well for most people.

Re:When is backing up *not* an option?

[dbIII]

I'm not talking in the abstract here - the tapes was on nine inch reels with the data in SEGD format and now the data is now on the system. Nice set of assumptions you have there, but I was using a real example.

Disks are still crap for archival storage and for most of us they are still crap for daily offsite backups. It would be nice to have a second site and a really fast, cheap way to get terabytes of data to it - but for me the expense of that is a lot more than taking a box of tapes to a storage shed. Eventually those daily tapes become archival tapes when projects finish. To cap it all off, most of the incoming data where I work comes in on tape. DVDs are utter crap when you want to be sure a lot of data can actually be read at the endpoint and only hold 4.5GB anyway. Portable USB drives are replacing tapes for transport but they are just as slow to read and fragile to transport, and even then you get the time consuming crap of spending twenty minutes going around the building to find someone with the same sort of USB drive because it came without cables.

The final point is I really do not want to have to maintain the systems to have a couple of hundred terabytes of storage when the working set is well under 5 terabytes and there would be a lot of repition in that few hundred terabytes of recovered tapes. It then also raises security problems to ensure client data is not easily accessable by other clients. Disks are not the best for archival storage. One of the reasons I was using tapes from 1982 is that nobody was interested in the area the data came from between that data and this year. Disk space is still too expensive to keep two and a bit copies of something big for more than twenty years :)

Tapes are often annoying and I sopmetimes wish we could be rid of them but there are many circumstances where they are useful - paticularly archives, transport and backups. While I've been happily using rsync to take daily snapshots of important stuff for years and distributed it to machines in different buildings it still cannot always replace tape. I've had too many dead hard drives or unreadable DVDs (plus they are too small) to seriously consider them as archival storage. As for transport - why are USB hard drives so horribly slow? The ease of random access is irrelevant when you want a lot of files from the disk so LTO and SDLT still win there.

Re:When is backing up *not* an option?

[lgw]

Your setup sounds *completely* vulnerable to a single malicious employee (with the right passwords). Typical engineer: protect against multiple failure modes, and disregard malice.

You don't have a backup until you seperate the data from the ability to destroy that data. Of course, there is WORM disk storage that meets that need, but that's far more expensive than tape (though handy for meeting auditing requirements).

Re:When is backing up *not* an option?

[WuphonsReach]

Agreed. Tape makes a lot of sense for high-volume applications.

It's just being crowded out of the low-end market by ever larger and ever cheaper hard drive sizes. Tape costs would have to drop by about a factor of 4 (or more) to compete in the lower end of the market where 100 tapes is a lot.

(If I could backup 800GB for $10, that would be much more of a no-brainer decision. The cost-advantage would be high enough to pay for the expensive tape drive. And $50 LTO-4 tapes are a lot better then back when a lot of large-capacity tapes cost $100 each.)

Dear Every Corporate Tool in the Universe:

[yttrstein]

And that's why your IT department actually needs funding. Sleep tight.

Re:Dear Every Corporate Tool in the Universe:

[TubeSteak]

And that's why your IT department actually needs funding. Sleep tight.

They've had the site live for 6 years.
This wasn't a lack of funding, it was just sheer stupidity.

6 years and nobody ever thought it'd be a good idea to back everything up to dvd or an external hard drive. HTML compresses really well in case they didn't know.

Re:Dear Every Corporate Tool in the Universe:

[yttrstein]

Stupidity IS a lack of funding. Pay the salary of someone smart enough to handle your data correctly if you have no interest in becoming smart yourself. Simple.

Re:Dear Every Corporate Tool in the Universe:

[Vanders]

Apparently they only had the two drives: one mirroring the other. They had one single drives worth of data. Just think how little data that really is, for a moment. They could have bought a second hand tape drive from eBay and a box of new tapes for a couple of hundred dollars and had a complete backup solution going in one afternoon.

For want of a nail, the shoe was lost.
For want of a shoe, the horse was lost.
For want of a horse,the rider was lost.
For want of a rider, the battle was lost.
For want of a battle, the kingdom was lost,
And all for the want of a horseshoe nail.

Re:Dear Every Corporate Tool in the Universe:

[Volante3192]

Never underestimate the beancounter's desire to save every cent possible. If your site's working perfectly fine, well, what's the point of having backups? Seriously, I see this happen all the time with small businesses. "Oh, it's never failed before, why do we need backups?" Then the server implodes.

Course, they then get pissed at us for not preventing it, but what do they expect us to do, shell out for a tape drive with our own cash? I think not.

Re:Dear Every Corporate Tool in the Universe:

[mrchaotica]

Hell, they could have spent $50 on a USB hard drive (i.e., half-assed it) and been better off!

Re:Dear Every Corporate Tool in the Universe:

[slugtastic]

I'm more surprised that the site lived for 6 years without back-up. That's pretty hardcore.

Re:Dear Every Corporate Tool in the Universe:

[modmans2ndcoming]

Screw that!! IT Departments are cost centers and have absolutely no benefit to the bottom line of a company... none at all... nope.

Re:Dear Every Corporate Tool in the Universe:

[Kjella]

Being too stupid to recognize your own shortcomings is also a form of stupidity. Or hubris, whichever is more appropriate.

Re:Dear Every Corporate Tool in the Universe:

[bb5ch39t]

Absolutely correct! Why management here is drooling over how much they money could save if they just didn't need the damn IT department. And all those damn desktop computers! Why, life was much better back in the days of paper ledgers and pencils! Sigh - if only we could have the perfect company. One which only has high level managers and none of the "riff raff" that infects them. Oh! Wait! That's Congress. And they have a monopoly.

Re:Dear Every Corporate Tool in the Universe:

[digitalgiblet]

Pay the salary of someone smart enough to handle your data correctly if you have no interest in becoming smart yourself.

The first step is admitting you are stupid. That is hard for most people. Of course today they are having NO trouble making that cognitive leap...

Re:Dear Every Corporate Tool in the Universe:

[slushdork]

Maybe they should have used this backup strategy, although this one looks more like this...

Re:Dear Every Corporate Tool in the Universe:

[sjames]

A USB drive is an excellent non-archival backup. Two or more in rotation is even better. That plus a decent RAID for the primary storage will cover most data losses. Even better if the drive goes home with the admin at night.

Re:Dear Every Corporate Tool in the Universe:

[troll8901]

Even better if the drive goes home with the admin at night.

Would the admin be tempted to look at other people's data?

Re:Dear Every Corporate Tool in the Universe:

[teg]

Never underestimate the beancounter's desire to save every cent possible.

That's contrary to my experience. Other expenses have been skimped on occasionally, but just mention the word "backup" and the funding was there.

Re:Dear Every Corporate Tool in the Universe:

[sjames]

If so, he can do so anyway.

Re:Dear Every Corporate Tool in the Universe:

[mabhatter654]

Don't send tapes home with Admins... send them to the bank to be put into a safety deposit box with the days checks if you have to. Admins don't want tapes in their home, it's a corporate security risk and the admin WILL forget to bring some back because one or two is no big deal... until they're not at your company anymore. I know I wouldn't do that because I wouldn't want to be the guy who's laptop bag gets ripped off with customer data on tapes inside it. It's just bad mojo waiting to happen.

Treat data media just like the companies cash money.

rm -rf /

[corsec67]

rm -rf /

That is one reason why mirroring isn't a backup, and why backups should ideally be off-line.

Re:rm -rf /

[Piranhaa]

C:\>rm -rf /
'rm' is not recognized as an internal or external command,
operable program or batch file.

Everything's still running here...

Re:rm -rf /

[Anonymous Coward]

Ha! I store all my data in directory names!

Re:rm -rf /

[Poltras]

$ del C:\*.* /s /q /y
-bash: del: command not found

Guys...

Re:rm -rf /

[dfdashh]

Judging by your OS, not for long

Re:rm -rf /

[TheRaven64]

What happened to deltree?

Re:rm -rf /

[mabhatter654]

operators make mistakes... the higher up they are paid the more likely they are to do it. Take the manager covering for somebody on vacation... they jump in and mistype something... boom! data gone. Happens to the best of us... the really good ones have ways in place to make sure mistakes have a way of being undone.

how many AS400 operators typed PWRDWNSYS *IMMED and got a surprise!

Ouch

[scubamage]

We do data hosting, and I can't imagine how catastrophic that would be. Jebus. Let this be an ultimate example of why numerous backups are needed. Always. Without question.

Re:Ouch

[conureman]

Or even one, stale, backup.

Re:Ouch

[jabithew]

This story put the fear of god into me. The first thing I did since reading it is to back up the website I admin (for my dad) locally. I'd always assumed our host would have good backup, but that seems naÃve now.

Re:Ouch

[slugstone]

Working at several hosting places I would say,you are correct. Never trust a hosting service backup. I always told our customers to never trust our backup. Sometimes backups just never happened. They are not high on the list of things to keep working.

Excellent!

[GravityStar]

Excellent! We can use their demise as yet another cautionary tale.

Re:Excellent!

[El Torico]

Excellent! We can use their demise as yet another cautionary tale.

Ironically, it's more useful than the entire collection of blogs that they stored.

Re:Excellent!

[TwilightXaos]

Yes.

But you wouldn't think everyone would catch on...

Mirroring is not intended as a data backup

[zaibazu]

It is an inexpensive protection against a total harddisc failure, but effective at this part. A software going rogue or a user deleting the wrong files can't be helped by it.

That's what backups are for

[MBCook]

It's really unfortunate that this happened. If they had simply had a backup snapshot of the DB they could have restored it. RAID only saves you from disk failures. It doesn't work on OS/user failures.

Unfortunately this is the kind of thing you tend to learn from experience (either yours or someone else). It's very easy to think "RAID 1 = disks are safe".

Just like a database cluster wouldn't have saved them. A clustering database can save you from load, or you can swap servers if a disk goes bad. But when someone issues "DELETE * FROM..." the other cluster nodes start to happily run the same thing and now you have 2 (or 3 or 10 or...) empty database boxes.

I hope those bloggers had a backup of some sort of their own.

Re:That's what backups are for

[ergo98]

Unfortunately this is the kind of thing you tend to learn from experience

Even a moment of thought would have made it abundantly clear that this was not a backup situation, and it certainly should not require loss to pound it into someone's head.

They were clearly way over their heads if they thought this protected them from anything other than a single drive failure. More likely they were entirely aware of the risk, but decided to wing it anyways.

Re:That's what backups are for

[MBCook]

My guess (and this is a guess, I'd never heard of the site before yesterday) is that this is some guy who started his own little site and it got bigger and bigger. Basically he never designed the backup, the system was just slowly pieced bigger and bigger until it got to it's current state.

The comments in the messages from the site's operator about the cost of the drive recover and thinking both drives just died at once indicate to me that this site was basically a hobby for him and he isn't experienced as an admin.

Re:That's what backups are for

[mzito]

Ah, it totally depends on the type of database cluster. For example, with Oracle, if you're using Oracle DataGuard, even in synchronous replication mode you can define an "apply delay" - basically, "Don't acknowledge this commit until it is written locally, and copied and acknowledged on the remote side, but don't actually apply the transaction for two hours"

That way, if someone does a delete * from blogs;, it will be reflected immediately on the production, but you've got a nice window to sort it out.

Plus, if you've got database flashback turned on, you can simply say, "Flash my database back to what it looked like before someone was an idiot", and all your data comes back.

These features are expensive in Oracle, but they can be very useful when you actually need them.

El Oh El

[greymond]

That's all I can say at this. I'm really surprised that with all the users they had, they are so quick to say "everything is gone and we're giving up" instead of just starting over and maybe implementing protocol that would make sure this doesn't happen again.

Re:El Oh El

[kurtmckee]

I'm really surprised that with all the users they had, they are so quick to say "everything is gone and we're giving up"

Considering how complete and unrecoverable the loss is, they have no idea who their users are. The accounts would have to be recreated from scratch, but who would try? Their users have no reason to ever trust them again. Journalspace would have a difficult time wooing back their original users, and no new user would seriously consider using them.

Bowing out is the only recourse, but I'm glad they're considering releasing their source code.

Re:El Oh El

[spuke4000]

Indeed. Everyone knows that when you drive your company into the ground through incompetence you don't give up! You go to Washington to get your bail out. That's the American way.

Re:El Oh El

[Chris Pimlott]

Their post said that only the task-specific server for data was hosed. If Journalspace offered paid services, then their billing system should still have all their customer's details.

Thank you

[ari_j]

This is fascinating and altogether newsworthy. I had never before thought of this. I am very pleased, indeed, that kdawson engaged his most finely-honed editorial faculties to post this article to the front page, as it is not only stunning and fascinating in substance but also rather eloquently written.

Inconceivable?

[LSD-OBS]

I do not think it means what you think it means.

How hard is it to remember:

[computersareevil]

Mirroring: High availability
Backups: High reliability

The rules of backups

[Anonymous Coward]

The rules of backups:

1. Backup all your data
2. Backup frequently
3. Take some backups off-site
4. Keep some old backups
5. Test your backups
6. Secure your backups
7. Perform integrity checking

Re:The rules of backups

[Anonymous Coward]

1. Do not talk about backups
2. DO NOT TALK ABOUT BACKUPS

Re:The rules of backups

[Anonymous Coward]

1. Backup all your data
2. Test your backups
3. Backup frequently
4. Test your backups
5. Take some backups off-site
6. Test your backups
7. Keep some old backups
8. Test your backups
9. Secure your backups
10. Test your backups
11. Perform integrity checking
10. Test your backups

Every company I've worked at has had a backup plan. Exactly zero have had a recovery plan.

To many shops think HA==DR

[uncledrax]

It's more an issue that some people think that HA == DR.. which obviously this story reminds us that it is not the same thing.

Mirroring / RAID == HA.. if one of your HDDs let the smoke out, you still don't incur downtime. If you have a hot-spare, you're even better.. all it does it let you have alittle time to correct the
issue (ie: "It can wait until morning").

Also, one other very important thing.. mirroring doesn't prevent/restore data corruption. If you're mirroring your rm -rf (as pointed out by Corsec67 below), your RAID will happy do what it does.. and span your command to all your disks.... Congrats, you just successfully gave yourself HA to your disk erasing! :]

Backups are DR.. If your RAID croaks.. your SOL if you don't off-machine backups. If you accidently nuke your disks with an rm or something, you can still go back and restore data.. sure you'll likely loose -some- data, but -some- is better then all in this case.

Re:To many shops think HA==DR

[xyphor]

DR is Disaster Recovery

HA is High Availability

Re:To many shops think HA==DR

[cbiltcliffe]

I tried Googling, but the only results I got were a medical office in Chinatown.....

Re:To many shops think HA==DR

[The Blue Meanie]

People who care about their data and their business know what they mean.

Although, at my particular shop, we use the term "BC" instead of "HA".
BC = Business Continuance (HA = High Availability)
DR = Disaster Recovery

BC = "Looks like we just lost a drive in the array. Better replace that right away." or "Oops, broke one of the multiple fibers to the SAN. Where's the spare again?"
BC also applies to our load-balanced clusters of web servers and application servers that allow for the offlining or loss of entire machines without losing functionality. You need more than your data existing on media to Continue Business - you and your customers need to be able to GET to it somehow.

DR = Your building just burned to the ground, taking every single piece of furniture, equipment, paper, and magnetic media inside along with it. Now what?
Please note that the coolest, slickest, snapshotted NAS with terabytes and terabytes of awesome cheap SATA storage in it is worth exactly JACK in this scenario if it's in the same building as the source material. Offsite backups are not optional, and offsite storage of hard drives isn't exactly the easiest thing to do.

Re:stunned silence

[conureman]

I am experiencing a strange phenomenon. The jaw-drop reflex has been popping my mouth open for several minutes and won't stop. If I focus I can close it, but then it pops open again. wow.

Only 2 drives?

[lalena]

Maybe I could understand that there might be issues with backing up live databases, and they didn't want to deal with it. Still not an excuse.
BUT, according to the site "the server which held the journalspace data had two large drives in a RAID configuration". Only TWO drives.
All they had to do was pull one of the drives, replace it, and lock up the original off site. In a couple of hours the drives would have been mirrored again.

To the HR department

[squeegee_boy]

Important note: don't hire the IT dude with Journalspace.com on his resume.

Re:To the HR department

[DaveV1.0]

The only problem with that idea is that it may not have been the IT guy's decision to save money by not having a true backup system. I have seen companies skimp on backup systems because they thought their RAID system was enough.

Re:To the HR department

[ergo98]

A better backup solution needn't cost much, or even anything. Simply FTPing to your own home machine on occasion would have been a millionfold improvement (given the popularity metrics, I don't think this was like a staffed operation or anything. Just a guy or two)

Re:To the HR department

[squeegee_boy]

OK, you hire him first. I think he's available.

A lesson for admins, and users too

[gzipped_tar]

No doubt this incident is the result of the admin's fault. He's been confusing mirroring and backup and carried on the mistake until it's too late, as pointed out in other comments.

Now what about a user's angle? The morale is you can never think your data is safer when it's "in the cloud". If you value your blog and your readers, you *should* save a copy of your work as well as the readers' info, *locally*, somewhere you have control over.

There's no place like $HOME.

Re:A lesson for admins, and users too

[djmurdoch]

And a corollary to the parent's good advice: if you can't easily get a complete copy of your work, find another host. Manual one-by-one downloads don't cut it.

Re:Noobs. No, really.

[emag]

Even the greenest IT employee knows that mirroring is to protect against hard drive failure and not software corruption.

I only wish that were true. I've given up arguing with friends about this, who insist that their mirrors are good enough backups. I just stare at colleagues who think such, especially those who SHOULD know better. And I *know* coworkers are doing this @ work, too, and I'm just waiting for about 50TB of data to suddenly go missing...

No Archive.org either

[computersareevil]

They also purposely blocked archive.org via a robots.txt exclusion, so the bloggers can't use that to try and recover some of their blogs.

There is a denial going on

[hwyhobo]

In today's world where primary storage and protection storage are well-defined, and where entire industry grew around it (examples: NetApp, Data Domain), one is hard-pressed to understand the reason for such a debacle. The reading of the note referred to in the article leads me to believe, unfortunately, that Journalspace's IT department did not understand the difference.

It is sometimes considered a bad form to say something bad about fellow techies. We prefer to look for 'outside' causes. Still, to learn and avoid the same problems in the future, one has to admit his mistakes first. This paragraph from the Journalspace's page:

The value of such a setup is that if one drive fails, the server keeps running, using the remaining drive. Since the remaining drive has a copy of the data on the other drive, the data is intact. The administrator simply replaces the drive that's gone bad, and the server is back to operating with two redundant drives.

makes me believe there is a denial going on.

Re:There is a denial going on

[jbezorg]

The temptation for "But Macs can't fail!" bashing was strong, but I resisted. It did lead me to a question though. That is: Had they been mislead by the Mac culture? Could there been something in Apples ads or documentation that would lead them to this mistake?

The answer? No. At least not from Apple.

From page 32 of TFM: http://images.apple.com/server/macosx/docs/Server_Administration_v10.5_2nd_Ed.pdf

Defining Backup and Restore Policies
All storage systems will fail eventually. Either through equipment wear and tear, accident, or disaster, your data and configuration settings are vulnerable to loss. You should have a plan in place to prevent or minimize your data loss.

Re:There is a denial going on

[Lost Race]

It is sometimes considered a bad form to say something bad about fellow techies.

Yeah, right. If there's anything professionals love to do, it's talk trash about their peers. What's the first thing a computer guy says when you bring him in to fix a broken system? "My god, what idiot spec'd/built/installed/configured this piece of garbage? It's a miracle it ever worked at all!" Ditto every other kind of professional, from plumber to surgeon to architect to accountant.

(As such a professional, I often discover that the idiot I'm complaining about was me.)

Re:There is a denial going on

[hwyhobo]

Ditto every other kind of professional, from plumber to surgeon to architect to accountant.

My experience is the exact opposite, particularly when comes to a medical profession. It's like mafia, and no one dares speak ill of another 'made man', at least not on the record.

I worked for over a decade as an independent networking consultant, and some of the most daring statements I heard people make when criticizing someone else's design were of the kind "perhaps it is not the most ideal for your environment. Needs change quickly, and not everything can be foreseen". Being a loudmouth rarely buys you a lot of business. Even your clients don't want to see that. It's not in good taste, and then, there is always a possibility somewhere in their minds that if you speak that way of others, you may speak that way of them.

Someone needs to be FIRED

[spitek]

You pay your infrastructure people to maintain business, continuity I mean the tittle of this post made me go, "Really, no shit" That's like systems admin 101! If the admin was aware then the manager that didn't listen needs to be fired. If the manager listened and they are just run by retards then they got what they deserve. You'd think 17,000 visitors a month would be worth enough to do it right, in add revenue alone. The cost of a consumer machine running linux with a few TB's of SATA space - $1200 How much the company paid to have a system's admin play video games all day - $50,000 The cost of a 17,000 vistor a month site going down because they had no data base backups - Priceless.

Re:Someone needs to be FIRED

[macbuzz01]

I think they are all fired now.

Mirroring

[jav1231]

See mirroring is like...well a mirror. If you stand before one and stick a fork in your eye your mirror-image does the same. In real time. Analogies are there for a reason.

Re:Mirroring

[gEvil (beta)]

See mirroring is like...well a mirror. If you stand before one and stick a fork in your eye your mirror-image does the same. In real time. Analogies are there for a reason.

There's a major flaw in your analogy. See, if I stick a fork in my right eye, the mirror image will stick a fork in his left eye. Between the two of us, however, we still have one good left AND right eye. So ipso fatso, I have a complete backup.

Re:Just give up?

[IMarvinTPA]

The article says the data recovery company has found the drives wiped. There is no recoverable data.

It seems like the actual site failure was on the 23rd or so.

IMarv

PS, the internet archive was blocked by their robots, so there isn't even that to look at. http://web.archive.org/web/*/http://www.journalspace.com

Google cache diving

[Chris Pimlott]

Looks like at least some content is still in Google's cache, those looking to salvage their journals should act quickly.

You can limit google's search results to a particular site by using the "site:domainname.com" search term (example) and then click the "Cached" links of each result to see Google's copy.

There's also a Greasemonkey script for Firefox that can automatically add Google Cache links next to page links, so you can navigate from one cached page to another easier.

Re:No Archive.org either-Compound Foolishness

[Nom du Keyboard]

They also purposely blocked archive.org via a robots.txt exclusion, so the bloggers can't use that to try and recover some of their blogs.

This is just compound foolishness. I gather they did it in an attempt to control bandwidth costs since it's hard to imagine any other reason.

You need more than backups ...

[blowdart]

You don't just need backups. You need to TEST them. Having a backup run every night is nice and all; but if the tapes are unreadable and no error was reported, or if you're doing it wrong and the backup is corrupted and you only find out when you come to restore ....

Re:You need more than backups ...

[nabsltd]

The best way I have found to test the backup is to nuke the data and restore.

Seriously, if you know what files store the data (and that you are backing up), just stop services and rename a directory or two so the data is "gone". Then, restore from backup, start the service, and see how things look. Another good way is to restore the data to a VM that runs the same software as the production server. You can sandbox a simulation of the entire Internet inside a few VMs if you want, and test what happens.

I just did something similar when I upgraded the OS on a VM that runs a MySQL server:

1. Create and configure new VM
2. Stop services on old VM
3. Run backup on old VM
4. Stop old VM
5. Reconfigure new VM with correct IP, etc., and restart
6. Restore data to new VM from backup
7. Test

Basically, if things had gone poorly, I could just stop the new VM and revert back to the old one.

Re:You need more than backups ...

[mortonda]

Backups must be:

1) Automated - if you need human intervention, it will fail

2) Point-in-time - the system must be able to provide restores for a set of times, as fitting for the turn around on your data. A good default is: daily backups for a week, weekly for a month, and monthly for a year

3) TESTED: You must fully test the restoration process (if this can be automated, even better). Backups that you can't restore from a bare machine are worthless.

For better disaster recovery, backups should be:

4) offsite - if a fire or tornado hits, is the backup somewhere else?

5) easily accessible - how long will it take to get the restore going?

Yes, it is a backup solution

[PearsSoap]

Since when is Slashdot in the habit of disagreeing with Linus Torvalds?

Only wimps use tape backup: _real_ men just upload their important stuff on ftp, and let the rest of the world mirror it ;)

Re:Professionalism

[moderatorrater]

Adding the OSX comment and that a bug in their code is impossible is even lamer.

The drives were overwritten sector by sector on a machine that didn't have any of their code running on it. Their application couldn't have done it because it couldn't execute arbitrary code on that server. The "impossible" comment makes sense to me.

As for it being lame/unprofessional to name the possibilities, I disagree. He states the OS it was running on and said that it was either an OS problem or sabotage. There might be a few possibilities, but that about sums them up right there. He was being thorough and open; what's the problem with that?

Double Duh!

[Roger W Moore]

Since they apparently used OSX Server this is particularly bad. All they needed was a large enough USB attached disk and then to turn on Time Machine. Might not be the best solution for their needs but it is hard to imagine one which requires less effort.

Re:Double Duh!

[azav]

Or attach a 4 TB Drobo to it and then use Time Machine.

Then make a backup and test the restore.

Their admin is criminally incompetent.

Re:Double Duh!

[CarpetShark]

All they needed was a large enough USB attached disk

Correction: all they needed was a large enough, functional, external disk.

Finding functional external drive products isn't so easy, I've discovered.

Re:Double Duh!

[Anonymous Coward]

Not saying OS X is not pretty good on the desktop/laptop, just that for your servers you should use Linux or possibly Solaris or BSD, but not OS X or Windows.

So, you say they should use BSD (among other options) yet say they shouldn't use BSD or windows?

I do agree with your windows point, but as to your confusing BSD comment, i'd have to say you were right the first time and wrong the second.

And since i know this joke will go right over your head, heres a tip:
OS X is BSD

Re:Double Duh!

[gd2shoe]

OS X is not BSD in the same way that Ubuntu is not Debian (and to a much greater extent, might I add). Mull over that one for a while.

Re:Double Duh!

[MarkRose]

Not quite. Backing up a live database can be a bit tricky. By the time you finish copying part of the database, the first bit can change again. So you have to create a snapshot of some kind. And that has to be supported in the database setup (at the application or server level) in order for the backup to be in a consistent state. And you don't want your backup process to degrade site performance, either. So a simple file copy is totally inadequate.

A common solution is replication. Backup is then performed by creating a replication point on the slave database machine then taking a snapshot and copying that while while master database machine continues serving at full speed. Replication can then catch up when the backup is complete. Another advantage to having replication is duplication on the machine level -- if the master fails, go live to the slave with minimal to no downtime. Set both machines up in a master-master configuration and you can swap back and forth as needed, allowing live maintenance and backup with no performance degredation.

Re:Double Duh!

[MBCook]

*BZZZZT*

The GP was 100% correct. If you had kept reading, you'd see that the suggestion was to use replication so you can lock the DB into a consistent state while backing up. When the backup is done, the box starts replicating again. If you didn't have the backup box, you'd have to lock the production database while your backup was going on.

He was suggesting replication purely as a way to avoid having to pause the application during backup, not as the backup it's self.

Re:Double Duh!

[mksql]

> Backing up a live database can be a bit tricky.

Seriously? If your database of choice is a chore to backup while live, you need to rethink your choice.

Full or incremental backups should be a trivial operation, with support for intra-backup change capture only a little more effort (log shipping, replication, etc.)

Of all the reasons to lose data, "Backups are hard!" should not be in the list.

Re:Double Duh!

[mabhatter654]

Except if somebody issued a drop table... then the repliants get dropped to... nice faithful mirroring!!! Been there, done that.

A good solution is to use mirroring like this, and then take the replicant offline to do real, full backups without taking down the production box. Then you have a live copy if drives or processors go bad to bring up immediately, and a backup tape to cover boo boos like this one. I believe that's what parent is getting at.

Re:Double Duh!

[penguinstorm]

BSD is no longer BSD either. You need to pick your flavour, whichever one suits your poison.

Re:Double Duh!

[Feyr]

did they ever fix their multithreading issues, where the performance went to shit as soon as you got more than 4 threads going at once?

THAT is what puts me off os x server, not teh pretties

Re:Double Duh!

[Murpster]

Their admin is criminally incompetent.

Isn't that the same as saying "they had a Mac admin"?

Re:Double Duh!

[tomknight]

Here's what I've found being said on the topic:

"I think I can reveal this much: an EX IT guy didn't do something he was supposed to have done. This wasn't discovered until AFTER the disks crashed. So, there were probably other reasons for this guy being an EX, too. Anyway, the crash happened, the mistake was discovered but now too late to fix."

From: http://dorrie.de/F1/viewtopic.php?t=194&start=375&sid=a65b1d9b0fbcc3c3e8df874fc167d495

Re:Double Duh!

[darkonc]

"everybody knows that MACs don't need admins!"

<sigh...>

Re:Double Duh!

[JoelKatz]

"Either can do the job so one is a backup..."

Which one is the backup?

The whole point of a backup is that it is *stable*. Neither copy is stable, so there is no "backup on the hardware level". There are two active systems.

If you cannot restore an accidentally-deleted file from it, it's not a backup.

It is a serious mistake to use the term "backup" in relation to a RAID 0 array. There is only one correct way you can do that, "either disk can serve as a backup for the other, should its media fail".

Either disk can serve as a backup for the other *drive*. However, there is no backup copy of the data. It is *not* a backup solution. There is no backup.

There is, however, fault-tolerance. A media fault can be tolerated. But if the active copy of the data is corrupted, there is no backup.

Personal backups of online data

[RevWaldo]

This is why users should be able to easily back up their own data for any online service. If a service entrusted with your data provides no straightforward way to drop a copy of it onto your own hard drive, don't trust it. I'd go as far to say that any service that doesn't strongly recommend you keep your own backups shouldn't be trusted.

Do the big kahunas of the "Web 2.0" world give users that option? Gmail, Myspace, Facebook, Twitter etcetera ad nauseam?

Re:Google Cache to the rescue?

[LordSnooty]

I hope affected users are looking into this, I just did a search of a random JS blog and 2,000 entries were returned, all cached it would seem. So many people might be able to recover their work in a very painstaking manner.

Re:Just give up?

[girlintraining]

Since you're the only poster to reply without yelling "idiot" (thanks, btw) -- Zeroing the drive makes software recovery impossible. It doesn't make data recovery impossible. There are ways to read the offset data, though this is getting harder as magnetic densities increase every year. Ontrack data recovery specializes in that kind of thing. I've seen them do it. Granted, it's not a 100% thing -- you don't get back something that even resembles a filesystem. At least a third of it is uselessly garbled binary.

OS X Server

[DTemp]

The site was run on OS X Server... I think this may be indicative of the level of IT effort with the company. Look, *I* run an OS X Server... but *I* am a Biology major that knows approximately dick about the UNIX command line, and use it to run a server that I probably wouldn't be able to run any other way. I also have it backup nightly to a cheap NAS, archiving old backups, and I've tested a restore to make sure it works.

This is probably just a couple guys who ran a website in their spare time... not a huge IT effort that failed.

Re:Just give up?

[imsabbel]

Thats bullshit, and has been for decades.
Its a myth. Just learn about it. Even if we use our newest AFM, or XMCD microscopy, you wont see an overwritten byte in any drive of the last 5 years. And even the last decade is very doubtful (basically, since GMR drives are around).
There IS NO SPACE between tracks anymore. Bits are right next to each other. If you overwrite, nothing above the superparamagnetic limit is left.

Not even the NSA could get anything useful out of a single overwrite with zeros (well, except relocation sectors and other specialities that might compromise security, but doesnt help with a backup)

I remember Karl Rove had a JournalSpace....

[bodland]

I'm just sayin'

Darwin awards

[ZiggyM]

Are there Darwin awards for websites?

Archive/redo logs too

[DamnStupidElf]

ACID compliant databases use a log, much like a filesystem journal, that contains all the changes made to the database before those changes were actually written out to the main database storage. When you back up the raw database, you back up all the logs since at least the time you started backup up the raw files until the time the backup was finished, and when you need to restore the database you put the raw data back and then let the database replay the logs.

tape backs up perfectly....

[SethJohnson]

Back in the nineties, a friend of mine was backing his mac system up weekly with a tape drive. The thing is, he was using the same tape to repeatedly back up onto. One day he calls to tell me he needed some help recovering files on his hard drive after a crash. I asked, "What about the tape backups?" He said, "That thing backs up perfectly. The problem is, it doesn't restore at all."

Seth

But wait... They're hosting experts!

[bigtallmofo]

The company that runs Journalspace (or used to, anyway) is Lagomorphics. They will host your site for you...

http://www.lagomorphics.com/hosting/

At Lagomorphics, we're OS X hosting experts. We've been using the Mac mini and Xserve platforms for years, and we're proud to offer you the opportunity to use our colocation facility. Just send us your Mac mini, or let us provide the hardware.

Something is fishy here.....

[FlyingGuy]

For everything to be just gone and I mean LONG gone, then something besides a truncation or un-linking of the file had to occur.

Now I don't know all that much about the apple file system, but I would imagine it is like most file systems in that it links clusters and sectors of data together using some sort of allocation table, hash, b-tree or something.

Now unless they had file scrubbing turned on and the OS purposefully went out and overwrote every segment of the file with 01010101 and 10101010 then the vast majority of the data should still be there, at least I would think it would be. I mean even the nastiest revenge oriented guy, would have to be able to invoke some kind of program to do that.

I am assuming that it was an SQL database of some flavor. I don't know much about MySQL internals but I am pretty sure a

delete from table

simply goes through the index and marks pages deleted and does not physically go out and scrub ever page that has data on it. I know that is how Oracle works.

So this leaves me wondering about the data recovery house.... I they were doing a sector by sector read on the entire drive ( either of them ) they should till see all sorts of data on the disk. Now I don't know if the database compresses data on the fly ( some do, some don't) and I don't know if drive compression is an option on OS-X. If so, I can see where they would see just mostly larges amounts of compressed data ( making things VERY difficult if not impossible to recover, but baring that, most OS's have the hooks built in do simply do a sector by sector read of the storage device and although your binary data ( images and the like ) might be unrecoverable, you could probably get most if not all of the text.

Just a thought, but hey I might be crazy, it is just the hacker in me that brings these things to mind...

Re:Serves em Right

[FlyingGuy]

I see your point, but something about this does not pass the smell test.

To have nothing on the HD(s) then someone had to very very carefully wipe the entire disk by overwriting every block and sector that the data occupied, and that would have made whatever DB system shit its pants as it started seeing data disappear so it would have been really obvious, really fast that something was amiss and as you relate would have more then likely caused a kernel panic and or at least a core dump of the DB system.