Slashdot Mirror

I also wanted to point out that the difference between full and partial backups disappears when you do backups using file-level hard links as your backup solution.

Doing backups disk-to-disk with rsync, using the hard links option, the difference between partial and full backups disappears. All backups are full AND partial; you get the benefits of both.

We do an "incremental" backup daily, in that only the files changed in the interim transfer as part of the backup, and only the changes occupy additional disk space, but the result is a "full" backup in that we end up with a complete snapshot of the file system that can be copied or used on demand.

This is really the way to go, and our particular solution is free and open sourced long ago.

In the end, we decided (as we have with most other cloud services) that the whole idea didn't live up to the hype, and we opted to lease a dedicated server housed in someone else's data centre and we basically just do an automatic rsync from our normal servers to the back-up with suitable levels of encryption applied throughout.

It's a rather cost-effective to do this. Modest equipment, inexpensive, large consumer SATA hard disk in RAID arrays. With a decent power supply, you can pack 20 or 30 TB of storage into a commodity case and have a reliable backup solution.

We do this, too, for all our online assets. I released the scripts I use to manage the backups years ago as an open source project.

I haven't released changes for a while because... it works.

You roably want to see our galiant efforts to stop ID theft.

http://www.effortlessis.com/stopidtheft

A case in point is my own set of backup scripts (this is not) Backup Buddy

My primal instincts kicked in and I was ready to attack since I misread that as Bonzi Buddy.

But you forgot to say why!

Many times, developers will have a list of features that they figure are "1.0". They may not have reached all the features yet, but the features developed thusfar may be very stable.

A case in point is my own set of backup scripts (this is not) Backup Buddy. I've been using them for years, they work very well, stable even with very large sets of data. (Well into the TBs currently, managing over 100 backup sources in 24 hour rotation)

But I don't consider them "1.0" yet because I always envisioned a handy-dandy web interface for managing backup rotations, verifying backups (currently working) and recovering files 1-by-1 securely. So, I edit config files. (aw shucks)

I second that motion....

We do something similar with rsync, backing up about 6-8 TB of data. We have php scripts that manage it all and version the backups, keeping them as long as disk space allows. Heck, you can even have a copy of our scripts free of charge!

With these scripts, and a cheap-o tower computer with huge power supply and mondo cheap, SATA drives, we manage to reliably backup a half-dozen busy servers off-site, off-network, to a different city over the Internet automagically every night.

Yes, more information is needed, blah blah blah. But it's definitely a feasible idea.

It took me about 10 minutes to create this simple web-page would could conceivably be used to steal identifying information. It would take a few hours to add stuff like the ability to run credit cards, and simulate a faux "Your identity was not found".

This website was easy to make using a free template found online. With the exception of the target page for all the links, it would easily pass the "sniff test" for many people. It looks friendly! It's got a kid and a butterfly on it! The news stories are current! (copy/paste from google news for "Identity Theft") Feel free to check it out. Total time spent was about 10-15 minutes. (I purposefully put in a few spelling/grammar mistakes, just to exaggerate my point)

So I hack up a spam engine, log in via some open wifi hotspot, and I have a business overnight? ID theft is much, much easier than we all think. And we want to believe that this guy isn't also doing it?

Even if solid-state drives are expensive as hell and not much better than current mechanical/magnetic hard drives right now, I don't expect them to stay that way so this is a step in the right direction.

The fact that SSD devices can compete with Hard Disks today shows not just excellent growth, but purely awe-inspiring growth. Despite being a much smaller marketplace than the magnetic HD marketplace, SSD storage has almost caught up with magnetic Hard Drives.

To show what an incredible accomplishment this is, you need to really understand exactly what this graph actually means.

It shows how hard disk capacity has grown since 1980. Yeah, it's gotten bigger every year... whoopdie doo, right? Notice that this is a logarithmic graph. Each line is 10x the line before, so you really don't see the significance of this, so I rewrote the graph in a "real" scale.

What previously looked like a smooth, predictable growth actually represents a cliff of growth. Capacity has grown so fast that it's been a challenge to find uses for this much storage. We've had to re-invent the meaning of what is a computer in order to make use of so much new found power - over and over, and over again.

And yet, despite having a dramatically smaller marketshare, much less R&D, SSD storage has managed to all-but catch up to this fast-moving target. This isn't just cool, it's incredible. Every year, SSD drives get a little closer to parity with their spinning cousins.

I have an 8 GB thumb drive, but I also still have a couple 1 and 3 GB drives from a few years back on the shelf. This kind of growth is simply astounding!

I have a generic P3 with 9 HDDs in it totaling just shy of 4 TB of data. (most of the drives are 500 GB) They are all ATA drives, so I have a couple of ATAPI cards (with two IDE ports on each) giving me 8 drives. The 9th drive is a junker drive on the MB IDE port from way back when that only holds the O/S and the backup scripts. (CentOS 4.x) For whatever reason, it won't boot from the PCI IDE cards, and I don't care to fight it.

Works a real treat for nightly offsite rsync backups coordinated with the backup scripts I wrote some years ago, which manage backups, automatically running every night, use hard links to save disk space, and automatically use as much of the disk array space as possible for as many backups as there is space to support.

One advantage of disk-to-disk backups is that you can recover a file or two from backups with almost no hassle. We've actually incorporated this capability natively into our application allowing any of our users to recover any file to any of the available previous versions with a mouse-click. They love it!

Your post got me to thinking about using color instead of blank space to separate text.

I put together some prototypes based on this, trying to maintain the word density but still making it easier/faster to read.

Results surprised me a bit - using color to differentiate lines works quite well after a very short adjustment stage! Try it out - the results might surprise you!

Addressing both issues:

1) Off-site backups - that's why you back up offsite! Set up a backups-server (it doesn't need to be fast, it will be I/O bound) with a pile of cheap, big, slow HDD. Set up as RAID 5 or RAID 1 - your preference.

Then use a tool like Backup Buddy or Backup PC to back up the files.

It's automatic. It's off-site. It works with good-sized data sets. (I'm managing just over a TB these days, a number that's growing fast. I'm able to do unattended offsite backups every 24 hours, and I have several months worth of backups on file that I can access instantly)

2) Rotating thru tapes - tools like Backup Buddy and/or BackupPC take care of that, too! Assume you have 200 GB of "Enterprise Data" to backup. So you get a TB or so in your backup array, (say, 4 350 GB disks, RAID 5) and use one of the above tools. You'll see months worth of backups out of that.

Without swapping tapes, hiring anybody to take them "off site" and for just a few hundred bucks in cheap, IDE disks. Can tape do that, too?

oooh look at you with your loaded questions.

no i didn't think it was polished enough, and then i saw there were other more mature tools that do the same thing (dirvish).

I might mention that I discuss the plusses and minuses of dirvish on my related projects page and still get an e-mail every week or so about Backup Buddy.

If it's good enough for you, it's good enough for somebody else. Release your stuff! If only because you might have thought of a feature that I might like! It's this cross-fertilization that makes OSS evolve so quickly and so effectively...

Yes, we have reverted to the Windows 95 technique where we shut all processes down, and display a screen that says:

"Please Turn Off Your Computer (Aaaarrrrgghhh Matey)"

The reason for this was more simple than you may have thought - many computers circa 1995 still had manual ("AT" style) power supplies that couldn't be turned off by the Operating System.

In fact, I *still* have one computer in the house that uses an AT power supply - an ancient AMD K6-2 running Linux that's been upgraded with almost 2 TB of HDD that's used simply to backup other systems with a backup script I wrote when I couldn't find anything else better.

I've now gotten almost a decade of service out of that machine - it's BIOS date is (if I recall correctly) in 1997!

Is it just me, or does this article make you feel like you're already overdosed on something?

Moz on Fedora, the lettering is just all goofed up and quite hard to read...

we've NEVER had a hard drive failure ... Of course the question is "What brand/type?" Unfortunately I don't know. But when I buy my own computer (this one was a gift) I will be sure to find out.

Manufacture is almost irrelevant. Every one, it seems, has a bad run from time to time. Spend more to get "server" grade SCSI, and you'll get better performance. But, I doubt your father is doing that.

But with only 5 computers in the house, what's to go wrong? I've deal with nearly an order of magnitude more hardware in the past 3 years, and have had only 1 HDD failure, not including a bad buy from a cheeseball distributor who sold me bum hardware)

PS: Knock on wood...

Anyone doing anything more than that on the web has to know something besides PHP since doing anything complex in PHP simply isn't very easy at all.

Having done some rather large and complex projects with PHP, your comment leaves me very curious.

What complex thing(s) is/are difficult to do in PHP? (I'll draw the line at stuff like rendering 3D, since the language clearly isn't meant for stuff like that)

I've had excellent results

1) Developing semi-distributed, (borrowing a buzzword) RIA type application using PHP-GTK.

2) Read headers from MP3s

3) Forked it into a daemon to process TCP socket calls with a home-rolled protocol,

4) Parsed Apache and Sendmail log files,

5) Run system administration,

6) Build a large-scale backup system

and much more, as well as the usual "I built a weblication using LAMP". I fail to see where PHP is particularly limiting in general programming...

If its IO bound, these may function just fine.

One of the servers I manage is a "backup" server that incrementally backs up numerous other hosts using software I wrote (in part) Backup Buddy.

Basically, it's an old AMD K6-2 450 (yes, it's so ancient it's even AT instead of ATX) with a few PCI IDE controller cards and a crapload of IDE HDDs. CPU performance is irrelevant - it's all I/O bound, rsync over SSH.

It does the job wonderfully, and has for a long time. I have many months worth of backups of all important hosts and data on this "backup" server..

No doubt fixing tha link, http://www.effortlessis.com/contact.html is one of the things you have to get done. ;-)

For offsite backups, rsync is your friend. I wrote a nifty wrapper for rsync (in PHP) to manage backups of multiple systems and accounts with options so that unchanged copies of a file only hardlink to a single file.

I manage about 500 GB of data in backups over a 1.5 Mb line with this tool. I can "rollback"
files on the servers to any of a number of points in the past several months, since this tool does versioning. It works very, very well, and I have a complete, updated, off-site image of all the servers I manage every 24 hours!

Well, sorta my own.

I'm currently battling with computer3g and have been documenting their mis-steps, one after another.

I bought a hard drive from Computer3g aka ViewMicro and had one DOA. They refused to honor their 7-day DOA warrantee, and I found out that the drives weren't even supported by the manufacturer!

It's (to me) a case of fraud, and it goes on from there, if you're interested.

One of the things that's particularly endemic to the Slashdot community is the "black/white" point of view - the idea that something is secure, or not, it's white or it's black.

But that's not how security is! It's all shades of grey, and the darker the shade of grey, the worse off things are.

Nothing is ever bulletproof, and seldom is anything ever wide-ass open to the world. It's somewhere in between.

I have a remote-desktop package integrated with one of my apps. It makes for very easy tech support, and I've got it built right into the menu system of my most popular application, so that customers using my software package have access to instantaneous, high-quality tech support.

To prevent users from popping up on my development system anytime they have a question, I put a password in place. It requires a small, 4-digit numeric code, and it changes every day.

By slashdot standards, this is terrible security. It's numeric. No letters, just numbers. The code changes every day, but only based on the day of year. It can easily be predicted, if one has any understanding of the underlying, otherwise very simple algorithm used to guess these numbers.

Anybody with a packet sniffer could crack it with one support session.

But, in this case, it really doesn't matter. The worst that will happen is that your computer's desktop will appear on my screen without my Windows VM.

You could DOS me with 10,000 VM screens, but it would take a very short amount of time for me to block the port number for the VPN and kill that.

So, what's the purpose for improving security? It's secure enough. And that's the point. Many people around here will have a cow if something is potentially crackable, while sitting behind physical locks that can be compromised with an expired credit card.

Gosh! Somebody could pull out their credit card, slide it through the gap between the door and the jamb, and break into your home!

In a black/white world, your home would only be considered safe if it had 1/4 inch steel plate exterior, and locks that the NSA would have serious trouble with.

In the real (shades of grey) world, a deadbolt and a solid-core door is usually good enough, and people live with the odds. Heck, even in the worst ranked neighborhood, you have about a 3.5 to 4 percent chance of getting burgled in a given year. (http://www.ojp.usdoj.gov/bjs/glance/burg.htm) I almost never lock my back door, and I've never had a problem with it.

That's good enough security for most, as evidenced by the fact that the most important issue was national security or "the war in Iraq" in the recent election. (http://www.rasmussenreports.com/Issue%20Clusters_ Election%20Night.htm)

Notice that individual household crime isn't even on the list (unless you include the 6% "domestic issues", despite the relative insecurity of the average home.

Brought home to me by the book "Secrets and Lies" by Bruce Schneier, this world is not a black and white world. Relative risk must be evaluated, and the equation must be brought to something we can all live with.

PS: Link to sites with A tags appears to be broken on slashdot. I tried numerous times to post links to the aforementioned sites and could not do so.

Your site is unreadable.

Fix your site so it's readable in Mmozilla, please?

The P2P solution either requires users to have their cable modem pegged and nearly unusable for 60 days (80gb is the current best laptop drive size, most cable modems max out at 128kbps up), or that they backup only a fraction of their hard drive.

My home network is on a 1.5 Mbit ADSL line. I backup data remotely using Backup Buddy. (which I wrote, based on rsync)

When i deploy a new server, it typically takes 1-3 days to do the first backup. Once that's done, it takes far, far less than that per night per server to keep everything up. Every morning, it's done. the usual process per server is about 30-45 minutes.

First time sucks. After that, it's much, much more manageable.

I provide a high-uptime guarantee for my hosted clients. I don't actually host all that much information, just a few gigabytes of heavily databased information. As such, I get few hits, but each hit is very, very important.

1) Primary server is configured with IDE RAID level 1, two drives mirroring each other in realtime, 80 GB each.

2) Hot failover server on a different network, different city, with the same size drives as the primary system. (Backup server is not raid, tho) This is for failover in case of severe emergency.

3) Network backups performed at a 3rd offsite location using rsync over ssh. The scripting I use (in PHP) is available at effortlessis.com/backupbuddy . (though I need to update the current release) This is a cheap, low-end dedicated system with big, cheap IDE drives (~ 400 GB) that just backs them up.

It's incremented going back about 1.5 months. (At any point, I can roll back the system to any point as far back as 1.5 months)

With this setup, any two systems can fail completely and I'll still have virtually no data loss.

From your web page, it seems that your business depends completely on products that run on Windows; your job depends on people believing in M$ products. It is difficult to take your comments very seriously. Your web page says Highest levels of security but I do not see any information (other than a suggestion to download a M$ security patch for IE) on security; did I miss something?

If you look at our website you'll find a web-based Contact Manager software accessable from (virtually) all the devices you mentioned above.

It's tested with Konqueror, IE, and Mozilla, and is known to work with a number of Palm-based devices, including the Handspring Treo cellular phone. /Shameless Plug