Slashdot Mirror
Online Website Backup Options?
pdcull writes "I can't be the only person on the planet who has this problem: I have a couple of websites, with around 2 GB of space in use on my hosting provider, plus a few MySQL databases. I need to keep up-to-date backups, as my host provides only a minimal backup function. However, with a Net connection that only gets to 150 Kbps on a good day, there is no way I can guarantee a decent backup on my home PC using FTP. So my question is: does anybody provide an online service where I can feed them a URL, an FTP password, and some money, and they will post me DVDs with my websites on them? If such services do exist (the closest I found was a site that promised to send CDs and had a special deal for customers that had expired in June!), has anybody had experience with them which they could share? Any recommendations of services to use or to avoid?"
Rather than "posting DVDs" I'd go for something like Amazon's S3 and just dump the backup to them. Here is a list of S3 Backup solutions that would do the job.
I've personally moved away from hard-media as much as possible because the issue on restore is normally about the speed to get it back on the server and its there that online solutions really win as they have the peering arrangements to get you the bandwidth.
An Eye for an Eye will make the whole world blind - Gandhi
It seems like the only problem with your home computer is FTP. Why not use rsync, which does things much more intelligently - and with checksumming, guarantees correct data?
The first time would be slow, but after that, things would go MUCH faster. Shoot, if you set up SSH keys, you can automate the entire process.
Oh, you're not stuck, you're just unable to let go of the onion rings.
Presumably, much of that 2 gig of data is static, so perhaps you could look into minimisation of exactly *what* you need to back up? It might be within the realm of your net access.
I record my sleeptalking
We use http://www.bqinternet.com/
cheap, good, easy.
I 100% agree with NerveGas on the rsync suggestion. I use it in reverse to backup my laptop to my hosting provider.
Here's the one thing to remember in terms of rsync. It's going to be the CURRENT snapshot of your data. Not a big deal, except if you're doing development and find out a week later that changes you made to your DB have had unintended consequences. If you've rsynced, you're going to want to have made additional local backups on a regular basis so you can roll back to one of those snapshots prior to when you hosed your DB. Apologies if that was obvious, but rsync is the transfer mechanism. You'll still want to manage archives locally.
Seth
$5 / month hosted VPS on linux = awesome!
Or simply use rsnapshot. However whatever backup solution you use, make sure to create dumps of your databases as backing up the database files while they are in use will give you backup files you cannot restore from. If you backup your database dumps, you can exclude your databases files from the backup.
either setup a rsync client in your home machine, or have someone backup your site for you (read: http://www.rsync.net/)
And if you want to have rsync at your windows machine, follow the instruction here: http://www.gaztronics.net/rsync.php
By doing rsync, you can turn on and off the backup process, it will pick it up where it was. And when you have full copy of your entire site in your local drive, next time, it will only be the incremental backup of it.
And for mysql, you can use mysqldump (http://dev.mysql.com/doc/refman/5.0/en/mysqldump.html)
yeah, i say rsync and mysql replication would work nicely. of course you have to look into them and decide if they meet your needs, but i think you'll find it's probably good enough.
You may have to use extra tools to break your archive into seperate chunks fitting Gmail's maximum attachment size, but I've used Gmail to backup a relative small (~20mb) website. The trick is to make one complete backup, then make incremental backups using rdiff-backup. I have this done daily with a cron job, sending the bz2'ed diff to a Gmail account. Every month, it will make a complete backup again.
And a seperate Gmail account for the backup of the mysql database.
This may be harder to do with a 2GB website, i guess, since Gmail provides atm about 6GB of space which will probably last you about 2 months. Of course you could use multiple gmail accounts or automated deletion of older archives...
But seriously, 2GB isn't too hard to do your from own PC if you only handle diffs. The first time download would take a while, but incremental backups shouldn't take too long unless your site changes drastically all the time.
Wow! So you are asking somebody to download your website's home folder and database, look at the password and private information of members, and deliver you dvd that is ready to be restored with rootkit along?
After looking at the available options, I decided that there was nothing which met my criteria for convenience, efficiency, and security. So I decided to create my own.
I'm looking for beta testers: http://www.daemonology.net/blog/2008-05-06-tarsnap-beta-testing.html
Tarsnap: Online backups for the truly paranoid
rsync to get the data, cp -al to keep snapshots. I've been using this for years to manage TB of data over relatively low-speed links. You'll take a hit first-time (so kick it off at night, kill it in the morning, and the next night just execute the same command and it'll eventually catch up, then cp -al it, then lather rinse, repeat. This page: http://www.mikerubel.org/computers/rsync_snapshots/ has been about for years. Use it!
Yes. rsync is lot better, I always use it. I personally use rsync with -z option to compress and decompress the files on the fly, which improves speed a lot, most of files being text files.
Many hosting providers do not have this option and not even sftp. :-/
So that makes that you are stuck with FTP or need to change hosting provider, which is also not always an option.
Don't fight for your country, if your country does not fight for you.
Send me your FTP details and some cash and I'll...backup...your data.
Unbelievable Backup Software, BackupPC, it uses Rsync, and will solve all your troubles, it's truly amazing backups/restore solution, check it out ..
all the best!
Riaan
...because if you are and you're planning to sent personal data (there can't be many 2GB web sites that contain no personal data at all) on DVD through the mail, you might want to look at recent pronouncements from the Information Commissioner. A large fine could be waiting for you if you go down that route.
I would seperate the content. First there is the MySQL part. Export it on a daily basis (or more often). You can export it as a whole or only those parts that you desire. Make a php page for each thing you desire to download and protect it however you like.
Then point lynx to it to download the file.
The content is another matter. To update my sites I use sitecopy What I do is make the site localy and when I am ready, I run sitecopy and it will upload the site.
As I do incremetial backups localy, I do have the previous version there.
If this is not an option, it should not be too hard to use sitebackup to, uh, backup the site.
Put all this in a script and crontab should do the rest.
Don't fight for your country, if your country does not fight for you.
Ok, since it doesn't seem to be a big website, it's probably on a shared host, so more complex backup software isn't an option in most cases.
Php backup scripts should work on any shared host, try looking at these scripts: http://www.absoft-my.com/pondok/sitebackup.php http://phpclasses.betablue.net/browse/package/3585.html
I use a product called SquirrelSave:
http://www.squirrelsave.com/
which uses a combination of rsync and SSH to push data to the backup servers. The client is currently only for Windows at the moment, but with promises of a Linux and OS X version coming soon.
It generally works quite well - WinSCP is included to pull data back off the servers.
Have you considered changing hosting providers to one that offers a better backup solution? There are hosts out there that take several hourly, daily and weekly httpdocs backups along with daily mysql dumps and binary logs to provide point-in-time database backups. I know this because I work for a hosting provider that does just this: http://www.ayudahosting.com.au/faqs/hosting/ (see the last FAQ).
As for posting DVD's, I'd suggest contacting your hosting provider and asking them if they would be willing to do it for you every month or so. Getting your hosting provider to do this for you eliminates the need to transfer the data etc. We often get requests like these and are always more than happy to assist.
One thing a lot of people forget when they propose backup systems is not just how quickly can you take the backup, but how quickly do you need it back?
A sync to your own PC with rsync will, once the first one's done, be very fast and efficient. If you're paranoid and want to ensure you can restore to a point in time in the last month, once the rsync is complete you can then copy the snapshot that's on your PC elsewhere.
But you said yourself that your internet link isn't particularly fast. If you can't live with your site being unavailable for some time, what are you going to do if/when the time comes that you have to restore the data?
I use dropbox.. and even though it's in beta, you get 10GB of space for free :D
http://getdropbox.com/
I'm in agreement that an rsync based offsite backup solution is always a great idea. rdiff-backup or duplicity is the way to go.
That being said, proper backups is a must that any web host should provide. I used to use dreamhost and they did incrementals and gave you easy access to them. Some time ago I outgrew shared hosting and went to slicehost which offers absolutely awesome service and although backups cost extra, they do full nightly snapshots, and it's all easy to manage (restore, take your own snaps, etc) via a nice web interface.
Seriously, take your money where your mouth is: find a better host -- AND do your own offsite rsync based backups.
I use rsync on a few dozen systems here, some of which are over 1TB in size. Rsync works very well for me. Keep in mind that if you are rsyncing an open file such as a database, the rsync'd copy may be in an inconsistent state if changes are not fully committed as rsync passes through the file. There are a few options here for your database. First one that comes to mind is to close or commit and suspend/lock it, make a copy of it, and then unsuspend it. Then just let it back up the whole thing, and if you need to restore, overwrite the DB with the copy that was made after restoring. The time the DB is offline for the local copy will be much less than the time it takes rsync to pass through the DB, and will always leave you with a coherent DB backup.
If your connection is slow, and if you are backing up large files, (both of which sound true for you?) be sure to use the keep-partial option.
One of my connections is particularly slow and unreliable. (it's a user desktop over a slow connection) For that one I have made special arrangements to cron once an hour instead of once a day. It attempts the backup, which is often interrupted by the user sleeping/shutting down the machine. So it keeps trying it every hour it's on, until a backup completes successfully. Then it resets the 24 hr counter and won't attempt again for another day. That way I am getting backups as close to every 24 hrs as possible, without more.
Another poster mentioned incrementals, which is not something I need here. In addition to using a version of rsync that does incrementals, you could also use something off-the-shelf/common like retrospect that does incremental but wouldn't normally work for your server, and instead of running that over the internet, run it on the local backup you are rsyncing to. If you need to go back in time a bit still can, but without figuring a way to jimmy in rsync through your network limits.
I work for the Department of Redundancy Department.
In you use wordpress for your site, you can use blogreplica.com, an online blog backup service which was created with this specific goal in mind. blogreplica.com connects to your blog using XML-RPC and retrieves all the content to its servers where you have full access to it any time. Maybe this works for you
NSA.gov?
NSA: We backup your data so you won't have to!
How it works:
First, edit each page on your website ab add the following meta tags: how-to, build, allah, infidels, bomb (or just any of the last three, if you're in a hurry).
On the plus side, you don't need to give them your money, nor your password.
On the minus side ... there is no minus side (I mean, who needs to travel anyway?)
Posting anonymously as I have moderated in this thread (that, and they already know where I live).
Free service with great support for MacOS, Linux and Windows. Features 2GB free disk space, WebDAV, SFTP, HTTPS and more. Also has a nice easy-to-use picture viewer for sharing photos. https://mydisk.se/web/main.php?show=home&language=en
Consider Manent (http://trac.manent-backup.com , freshmeat entry: http://freshmeat.net/projects/manent). It can currently back up local directory to a remote repo, so you can easily set it up to run at your server to back up to your home, and in the future it will be able to back up an FTP directory. :)
It is extremely efficient in backing up a local repository. A 2GB working set should be a non-issue for it. I'm doing hourly backups of my 40-G home dir.
Disclaimer: I am the author
Other posters mentioned rsync (which I agree with). You might also look into rdiff-backup. Another backup option is SVN. The nice thing about these options is that once you get the initial (0-level) backup they will only upload the changes. So, assuming that you are not updating things all the time, any of these options will work well even over 150KB connection.
... his slow internet connection, and wants to pay something to not have to move files over his slow internet connection.
How about:
- Pay for a hosting provider that DOES provide real backup solutions....
- Pay for a real broadband connection so you CAN download your site....
As with most things that are 'important'...
Right, Fast or Cheap - pick two.
paintball
Just at the same time this story hits slashdot, I see a bunch of stories on reddit/digg about 10/20/100 ways to back up your web site!!!111 What an AMAZING coincidence. There couldn't possibly be any business interests in this.
OK, I keep hearing "use rsync" or other software. What about those of us who use shared web hosting, and don't get a unix shell, but only a control panel? Or who have a shell, but uncaring or incompetent admins who won't or can't install rsync? I know the standard slashdot response is "get a new host that does" but there are dozens of legitimate reasons that someone could be saddled with this kind of web host.
Shutting down free speech with violence isn't fighting fascism. It IS fascism!
rsync.net --- online backup provider with sshfs/sftp/webdavs/scponly support
Sorry for the self plug, but this just seems silly. Your web host should be backing up your website and offer you restorations. I guess this isn't a standard feature any more. But it is at Suso. We backup your site and databases everyday. And can restore them for you for free.
If you are subscribed to say Box.net, using Conduit is one possibility. Making a compressed tar.gz backup with rsync shouldn't take that long--it's the file transfer that will at 150K. If you cron a daily/weekly unattended overnight backup with rsync over ssh that downloads (sftp -b batch) to your pc, that might be best. Your backup needs to take into consideration any database back-end. I just use mysqldump to output the entire database to a gzipped file. Lot's of examples on the net. Google is your friend. Good Luck. Dietrich T. Schmitz Linux IT Consultant www.dtschmitz.com
Get a shell and set up a chron job to ftp your data at reasonable involves. Many ISPs that provide data center services also offer back up solutions. Did you do any investigation before posing this question?
If you are using rsync you could also buy space from rsync.net. Very cheap drive space and you can then rsync from your webserver to another location.
nt
If you are willing to consider pay services, I have had great experience using a free service (colotraq) that helps get you exactly what you are looking for when it comes to internet/telecom/etc services. They are like lending tree or real estate agent - they don't do the stuff them self, they just have some giant b2b marketplace. And - No, I do not work for them, just really have done well with them.
As many stated before I think you have better options. Many backup services can give you "Delta", or "Incremental" backups. For each backup only data that has changed gets uploaded. The advantages to this are lower bandwidth, and the ability to have restore points. If you check around there are many companies that offer this.
... or give a look at Dirvish. It uses rsync and keeps full snapshots using hardlinks for unachanged files. Works like a charm for me.
If you keep your backups for one month, S3 costs about $300 per TB. That's not a bad price for offsite backup that's easily accessible from both your main and disaster recovery servers.
price list
Reduce, reuse, cycle
I've been using rsync.net for more than a year, and it works great. I back up four websites from one server -- files and MySQL databases -- each night, each week, and each month, and only once did the backup not work as planned. Good tech support, too.
Greg Raven
As long as there's any left, I'll take mine first.
Hi everyone, I didn't mention in my question that where I'm living (Rio de Janeiro slum) there aren't that many options for internet access. Also, as all my sites are my very much not-for-profit, I'm limited as to how much I can spend on my hosting plan. I've been using Lunarpages for a number of years now, and generally find them very good, although if I stupidly overwrite a file, or want to go back to a previous version of something, I'm out of luck. As I am a lazy (read time-challenged) individual, I tend to use Pmwiki and maintain my sites online, hence my need for regular, physical backups. Anyway, thanks everyone for your help, I still can't help thinking that somebody must be able to make a pile of cash offering such as service to non-techie site owners...
From the BSD camp (or at least one of the developers), there's TarSnap, which offers very high encryption and confidentiality, and also incremental backups (via snapshotting).
-- Sig down
Here in the UK we use an online backup company called Perfect Backup. You just install a bit of software on each machine and it backups according to your own schedule. The best thing is, it does a binary diff of each file and only sends the changed parts of the file so conserving bandwidth. It's pretty configurable. The pricing seems pretty good too compared to some other providers. It's more expensive than people like Carbonite, but then this is a *business* grade product with support for things like Exchange, SQL Server etc and if you ring them, they pick up the phone after a couple of rings!
So that makes that you are stuck with FTP
wget --mirror?
LFTP's scripted system allows mirroring the backup and only getting files that have changed. With some server-side scripting to dump database diffs it wouldn't be hard to make a FTP backup solution that only downloaded changed files.
I drink to make other people interesting!
no text
duplicity (for the paranoid) or rdiff-backup and rsync.net.
Btw, ftp is a plain text protocol, why do people still use it over the internet?
From a site slashdotted on another post today:
;)
A solution using rsync + a smart trick with hardlinks to provide changes history.
He also uses loopback filesystems to save the backup on windows servers keeping file permissions/ownership. Definetely worth taking a look:
http://ttsiodras.googlepages.com/backup.html
this sounds like a very good solution. Okay, it would still be a pain over a slow connection if you need to restore, but assuming your provider's backup can be used as a starting point, it should be fine
What is your FTP login and bank account information? I will require a cashiers check for $5000 for a processing fee, an afterwards will deposit 5 million dollars in an escrow account in your name. My e-mail address is formerdeceasednigerianoilminister@hotmail.com
How about getting a hosting provider that does backup? I've been using pair.com for my sites for several years. They have about daily or twice-daily recent snapshots on the same server (that you can access yourself if you need to), then on-site backup, then off-site backup. As far as I know they don't ship those to customers, but this doesn't look to me like a very big risk. I haven't had to use any of the backups, and I think they haven't had any (big) loss of data since they went online more than ten years ago, as far as I can infer from reading the user groups. Of course you are at the mercy of the provider should a disaster happen, but is this really that much risk if they manage it properly?
:) http://promote.pair.com/direct.pl?vad1.com+63035
Shameless $0.2 plug: if you get an account at pair, sign up through this affiliate link
17779 eligible voters in a district, 17779 'vote' as one. This is Russia.
In order to properly facilitate a backup schema that will work for you, there are some important questions I would ask as a consultant to start offering suggetions.
There are more questions but that will give us a start in helping you get a real solution that will help YOU out.
I looked into this recently. There are a lot of commercial offerings. However, the only thing I found was 1) FLOSS, 2) had S3 support out of the box, 3) had a storage format that was documented and simple enough to restore without the software or even the docs, and 4) didn't use "mystery crypto" was some software called "brackup" from Brad Fitzpatrick of LiveJournal and memcached fame ("brackup" = "brad's backup.")
Brackup is written in perl using good OO practices and is very hackable. The file format is non-binary and very simple. I was able to figure it out and restore some files by hand without looking at the documentation (just an S3 browser, a copy of GPG and my key, and a text editor.) It uses GnuPG to encrypt unattended backups without risking the secret keys.
One cool thing about Brackup is it has a "plugin" architecture for special handling of certain filetypes. It comes with a plugin for handling MP3's that backs up the ID3 tags separately from the sound data. That way if you retag your library it only has to backup the new metadata and not the entire MP3.
It's not commercial quality and it's not for "mere mortals." It's something that Brad wrote to scratch his itch and some other people have been hacking on it. It's been working well for me and meets my requirements better than anything else so I'm recommending it to other hackers.
It's in CPAN...
I hope you have a better plan if you ever need to do a full restore in anger. It's all right spending days backing up in small chunks, but if your data ever goes south, it's going to take at least as long to restore it all again. In the mean time, your web site/application/business is flat on its back.
Its called archive.org.
Intron: the portion of DNA which expresses nothing useful.
I can normally resist this, but this is too much:
"the closest I found was a site that... had a special deal for customers that had expired in June!"
What about customers that died in May? Are they screwed again?
Do they expect much repeat business from the recently departed?
Is this a way to get around the memory erasure of the River Lethe (in Hades)? A way around the memory erasure during Buddhist reincarnation? If so, how would we know they successfully restored their memories? (I guess we would know the reincarnated retained their memories in a few years, when they start talking again.)
Thank you, thank you. The fish is very good tonight, and don't forget to tip your waitress!
For every complex problem there is an answer that is clear, simple, and wrong. -- H L Mencken
There are many sites that give you tons of storage for backing up files, with various ways of storing the data, many of which are free. Google and http://www.adrive.com/ are two that come to mind. No need to deal with your slow connection.
https://mozy.com/?code=UXW2GB
"Beware of he who would deny you access to information, for in his heart, he dreams himself your master."
rsync is your best option, ftp is the work of the devil for backups. if you have time and don't mind, in your backup directory, create a directory for monday tuesday wednesday... to sunday, then in your script, have your rsync target path change based on the day of week. One hint if doing it, make the current day directory, rsync to it, then dup it 6 times right away (for the other 6 days of the week. Good luck, been using this method for 7 years now ;) and you'll always have 1 week of backups to fall back on.
Syncback is a great piece of software and you can set it up to just back what has changed so you don't have to worry as much about slow connections.
http://www.2brightsparks.com/
They even have a freeware version.
We have online backups that will satisfy your needs. Online and rsync based, hosted in a datacenter. Send Us an email to info@dattabank.com.ar
but you're a tool for using ftp.
Something like 12-15 years ago someone I was familiar with had a DirecPC Satellite down link and a meager 14.4 / 28.8 dial-up connection for up link. He was a WAREZ op of sorts, and lets just say you didn't push out validated clean WAREZ with a dial-up connection. He had mentioned to me that he used FTP clients that could establish remote FTP sessions from one FTP server to another FTP server. These GUI clients allowed him to do the simple, select and drag copying we all appreciate, but the data never hit his network link or computer. This was all done in the internet cloud so to speak.
It stands to reason that this same FTP client package should still be available today, and that you would be able to backup your data by having multiple hosting providers, and just have a folder on each provider with the other sites backups.
Sorry I don't know the FTP clients that supported this, but I did witness the process once or twice.
Regards,
Ryan Pritchard
Fun Extends All Basic Life Expectancies
It's not only the client -- the servers must support it, and unfortunately, not too many do. Good idea to try, though!