For maintaining a high-integrity archive of documents, try Boar. It can even version control huge documents, like movies and photos. http://www.boarvcs.org/
True, versioned file systems are nothing new. But how do you create backups of those file systems so that the file history is backupped as well? And can you easily verify and update those backups? Versioned file systems are nice, but they are not the solution to the archiving problem.
All of them, if you ask me. Storage is cheap, and as long as you have a good versioning system (like Boar) so that the different versions doesn't get in your way, I see no reason to delete anything. Imagine that you rescale your favorite image so that you can send it to a friend in a mail. You accidentially hit "save" and replace the original image. Everything looks fine at a glance. Years later you want to use that picture for a large poster, and find that it is 640x480 now. And the original is lost, since you used rotating backups.
I'd like to push my own project for a moment, as I designed it to handle just this problem. It is called Boar, an open source project providing "Simple version control and backup for photos, videos and other binary files". The philosophy is that because human errors are just as problematic (or more) than hardware problems, version control is necessary for all vital data, be it code or baby pictures. And when you have all your files in one large, nice repository, Boar makes it easy to create and maintain _verified_ copies on external HDDs or whatever.
(I mentioned my project in an earlier answer as well. Sorry for repeating myself, but I really think this project is the solution for a lot of the problems people are talking about here...)
You are looking for Boar, an open source project providing "Simple version control and backup for photos, videos and other binary files". The philosophy is that version control is necessary for all vital data, be it code or baby pictures. And when you have all your files in one large, nice pile, Boar makes it easy to create and maintain verified copies on external HDDs or whatever. Splitting your data on a bunch of DVDs is a sure way to bring chaos to your files.
Disclaimer: I'm the author of Boar, and I think that absolutely everyone who values their files should use it. Or something similar, although I haven't found anything else that fits my needs.
Bup seems to be a great backup program, while Boar aims to be a full vcs. There is certainly some overlap between these types of applications, but in general, backups will sooner or later be replaced by more recent backups, and it is up to you to make sure that you don't accidentially delete any files, because you will have a limited amount of file history. Now, bup seems to allow for efficient backups, so perhaps you don't need to (cannot?) purge old backups. However, I like vcs-like workdir concept, and a commit and update command to keep workdirs synchronized between different computers. Also, boar allows for efficiently making or updating verified copies of the repository, which makes it easy to maintain good backups. But granted, the distinction in general between a good backup tool and a vcs is blurry.
Yes, I hear that a lot... True deduplications seems to be important for lots of people, and boar will gain "true" data deduplication quite soon, it's on the roadmap. (Even if I doubt that most people actually need it... what do they do? Do they obsessively edit their exif headers all day?)
In the meantime, boar already has a trivial form av deduplication in that it only stores identical files once. This is actually quite useful, as it allows for the cheap creation of "views" of the data. I can keep all my pictures in one place, including the blurry ones, and then have another directory containing only the nice ones, ready for a slideshow. Also, I have often experienced the problem that I have two _almost_ identical copies of some large file tree. Which one to keep? Often, I'd just keep both and wait for a rainy day to sort it out (which never comes). This was one of the things that drove me to create boar. Nowadays, I will just import both trees into my boar repo and delete the originals. Duplicate files will only be stored once in the repo, so the benefit will be almost as good as if I had taken the time to look through all those files, but with zero effort. Maybe I'll merge those similar directories some day, but until then, they'll at least not clutter up my harddrive.
This problems keeps popping up more and more often, as people collect more and more data... I don't have the answer to the question on how to keep everything indexed and searchable, but I do have the answer to the question on how to safely version control and store/backup such large amounts of data data... A little project I have called Boar. I quote from the project front page:
"BOAR aims to be the perfect way to make sure your most important digital information, like pictures, movies and documents, are stored safely.
* BOAR prevents data loss due to human or machine error * BOAR makes it possible for you to restore any or all of your files from any point in time. * BOAR makes it easy to maintain verified backups of your data, including file history. * BOAR will make it much more likely for your digital heirlooms to reach your grandchildren some day.
If you are familiar with vcs software such as Subversion, you might think of boar as "version control for large binary files". But keep reading, because there is more to it."
I have thought long and hard about this problem. The result - a project I call "BOAR". Big Object ARchive... or something like that. Quite seriously, I think there is a huge gap in data security between backups programs and version control, and BOAR is my attempt to fill that gap. People keep saying that they are using ZFS or Dropbox or whatever and seem to be quite content, but I think they are missing the point. You also need a container for your precious information, that allows you to verify and ensure that your data is consistent and complete. BOAR aims to be that container. I quote from the project front page:
"BOAR aims to be the perfect way to make sure your most important digital information, like pictures, movies and documents, are stored safely.
* BOAR prevents data loss due to human or machine error * BOAR makes it possible for you to restore any or all of your files from any point in time. * BOAR makes it easy to maintain verified backups of your data, including file history. * BOAR will make it much more likely for your digital heirlooms to reach your grandchildren some day.
If you are familiar with vcs software such as Subversion, you might think of boar as "version control for large binary files". But keep reading, because there is more to it."
This is exactly the reason why I started working on my own archiving / version control application. I use it to store my most important large files, like photos and videos. It uses a completely transparent, very simple repository format. It's quite powerful, and at the same time, even if the original software is unavailable, any programmer will be able to extract the files from the repository in half an hour. It's a little bit like a version control system (like subversion, git, etc), but suitable for large files. It is named "boar" (Big Object ARchiver... or something like that). Check it out on google code, http://code.google.com/p/boar/ Help and/or suggestions are very welcome!
I'd like to take this opportunity to push my very own brand new project for a VCS for large binary files. Most VCS:s so far has been developed for code. Nothing wrong with that, but there are many other kinds of data that also deserves a safe, version controlled repository they can call home. My tool is named "boar", it's written in python and I need users and help to make it perfect.
To quote the project page: "Boar aims to be the perfect way to make sure your most important digital information, like pictures, movies and documents, are stored safely. If you are familiar with vcs software such as Subversion, you might think of boar as "version control for large binary files". But keep reading, because there is more to it..."
Slashdot Mirror
For maintaining a high-integrity archive of documents, try Boar. It can even version control huge documents, like movies and photos. http://www.boarvcs.org/
True, versioned file systems are nothing new. But how do you create backups of those file systems so that the file history is backupped as well? And can you easily verify and update those backups? Versioned file systems are nice, but they are not the solution to the archiving problem.
I call it a beta only because I'm still adding features. It is stable. If you had read the whole paragraph you quoted, you'd have noticed that.
So far I've prioritized testing, basic features and robustness, but yes, a GUI would be nice.
Help is certainly appreciated. The project is quite mature though, so it can be used right now.
All of them, if you ask me. Storage is cheap, and as long as you have a good versioning system (like Boar) so that the different versions doesn't get in your way, I see no reason to delete anything. Imagine that you rescale your favorite image so that you can send it to a friend in a mail. You accidentially hit "save" and replace the original image. Everything looks fine at a glance. Years later you want to use that picture for a large poster, and find that it is 640x480 now. And the original is lost, since you used rotating backups.
Save everything, I say.
I'd like to push my own project for a moment, as I designed it to handle just this problem. It is called Boar, an open source project providing "Simple version control and backup for photos, videos and other binary files". The philosophy is that because human errors are just as problematic (or more) than hardware problems, version control is necessary for all vital data, be it code or baby pictures. And when you have all your files in one large, nice repository, Boar makes it easy to create and maintain _verified_ copies on external HDDs or whatever.
The project page is on google code at http://code.google.com/p/boar/
(I mentioned my project in an earlier answer as well. Sorry for repeating myself, but I really think this project is the solution for a lot of the problems people are talking about here...)
You are looking for Boar, an open source project providing "Simple version control and backup for photos, videos and other binary files". The philosophy is that version control is necessary for all vital data, be it code or baby pictures. And when you have all your files in one large, nice pile, Boar makes it easy to create and maintain verified copies on external HDDs or whatever. Splitting your data on a bunch of DVDs is a sure way to bring chaos to your files.
The project page is on google code at http://code.google.com/p/boar/
Disclaimer: I'm the author of Boar, and I think that absolutely everyone who values their files should use it. Or something similar, although I haven't found anything else that fits my needs.
Bup seems to be a great backup program, while Boar aims to be a full vcs. There is certainly some overlap between these types of applications, but in general, backups will sooner or later be replaced by more recent backups, and it is up to you to make sure that you don't accidentially delete any files, because you will have a limited amount of file history. Now, bup seems to allow for efficient backups, so perhaps you don't need to (cannot?) purge old backups. However, I like vcs-like workdir concept, and a commit and update command to keep workdirs synchronized between different computers. Also, boar allows for efficiently making or updating verified copies of the repository, which makes it easy to maintain good backups. But granted, the distinction in general between a good backup tool and a vcs is blurry.
Yes, I hear that a lot... True deduplications seems to be important for lots of people, and boar will gain "true" data deduplication quite soon, it's on the roadmap. (Even if I doubt that most people actually need it... what do they do? Do they obsessively edit their exif headers all day?)
In the meantime, boar already has a trivial form av deduplication in that it only stores identical files once. This is actually quite useful, as it allows for the cheap creation of "views" of the data. I can keep all my pictures in one place, including the blurry ones, and then have another directory containing only the nice ones, ready for a slideshow. Also, I have often experienced the problem that I have two _almost_ identical copies of some large file tree. Which one to keep? Often, I'd just keep both and wait for a rainy day to sort it out (which never comes). This was one of the things that drove me to create boar. Nowadays, I will just import both trees into my boar repo and delete the originals. Duplicate files will only be stored once in the repo, so the benefit will be almost as good as if I had taken the time to look through all those files, but with zero effort. Maybe I'll merge those similar directories some day, but until then, they'll at least not clutter up my harddrive.
This problems keeps popping up more and more often, as people collect more and more data... I don't have the answer to the question on how to keep everything indexed and searchable, but I do have the answer to the question on how to safely version control and store/backup such large amounts of data data... A little project I have called Boar. I quote from the project front page:
"BOAR aims to be the perfect way to make sure your most important digital information, like pictures, movies and documents, are stored safely.
* BOAR prevents data loss due to human or machine error
* BOAR makes it possible for you to restore any or all of your files from any point in time.
* BOAR makes it easy to maintain verified backups of your data, including file history.
* BOAR will make it much more likely for your digital heirlooms to reach your grandchildren some day.
If you are familiar with vcs software such as Subversion, you might think of boar as "version control for large binary files". But keep reading, because there is more to it."
Please check it out at google code: http://code.google.com/p/boar/
I have thought long and hard about this problem. The result - a project I call "BOAR". Big Object ARchive... or something like that. Quite seriously, I think there is a huge gap in data security between backups programs and version control, and BOAR is my attempt to fill that gap. People keep saying that they are using ZFS or Dropbox or whatever and seem to be quite content, but I think they are missing the point. You also need a container for your precious information, that allows you to verify and ensure that your data is consistent and complete. BOAR aims to be that container. I quote from the project front page:
"BOAR aims to be the perfect way to make sure your most important digital information, like pictures, movies and documents, are stored safely.
* BOAR prevents data loss due to human or machine error
* BOAR makes it possible for you to restore any or all of your files from any point in time.
* BOAR makes it easy to maintain verified backups of your data, including file history.
* BOAR will make it much more likely for your digital heirlooms to reach your grandchildren some day.
If you are familiar with vcs software such as Subversion, you might think of boar as "version control for large binary files". But keep reading, because there is more to it."
Please check it out at google code: http://code.google.com/p/boar/
This is exactly the reason why I started working on my own archiving / version control application. I use it to store my most important large files, like photos and videos. It uses a completely transparent, very simple repository format. It's quite powerful, and at the same time, even if the original software is unavailable, any programmer will be able to extract the files from the repository in half an hour. It's a little bit like a version control system (like subversion, git, etc), but suitable for large files. It is named "boar" (Big Object ARchiver... or something like that). Check it out on google code, http://code.google.com/p/boar/ Help and/or suggestions are very welcome!
I'd like to take this opportunity to push my very own brand new project for a VCS for large binary files. Most VCS:s so far has been developed for code. Nothing wrong with that, but there are many other kinds of data that also deserves a safe, version controlled repository they can call home. My tool is named "boar", it's written in python and I need users and help to make it perfect.
To quote the project page: "Boar aims to be the perfect way to make sure your most important digital information, like pictures, movies and documents, are stored safely. If you are familiar with vcs software such as Subversion, you might think of boar as "version control for large binary files". But keep reading, because there is more to it..."
Check it out! (pun intended)
http://code.google.com/p/boar/