Slashdot Mirror
Do-It-Yourself Internet Archiving?
A moron asks: "Web pages change and disappear all the time. For legal and historical purposes, I need to have accessible archives of the websites I maintain. I'm basically looking for a do-it-yourself version of Internet Archive's Way Back Machine which provides a simple versioning system and accessibility through web interface. Is there already software that does this? If not, what ideas does Slashdot have to make such a system possible? How should it work? What existing tools can be used together to make a workable system?"
"There are all sorts of tools out there that will archive web pages, and each have other necessary features such as making links relative. I don't always have filesystem access to pages, so tools that rely on such access won't work. There are some obvious tools that do part of the job such as:
But grabbing pages is only part of my, and I suspect many other peoples needs. The other pieces include intelligently archiving the pages, and making them accessible. If a page or a page element hasn't changed, there is no need to store multiple copies. The archives need to be easy for end users to navigate, search, and link."
ARCDIR = `date +%y%m%d` /var/www/archives
cd
mkdir $ARCDIR
cd $ARCDIR
wget -r http://mysite.com
Add error-checking and season to taste.
If you want to be more efficient like the poster wanted, you could easily have it always fetch to the same directory and just use cvs to check in. This eliminates duplicate storage. There are many free web-based CVS browsers out there with date searching and similar features. Might not be quite as nice as the wayback machine, but it definitely does the job for free.
A lot of folks are doing a simple version of the above to maintain SCO mirrors so there's to be no history erasing before the trial. God bless you all -- it will make the case that much stronger for us.
this
I highly recommend that you check out w3mir - which was found after a quick search on CPAN (The Comprehensive Perl Archive Network). I particularly like w3mir due to it's ability to compare against existing copies of your local mirror - which is more of what you're looking for. Using this in conjunction with a simple shell script (to tar and mv files, as so desired - hooked to a cron job) will create your very own, automated Internet Archive.
-- Kleptotherapy: Helping those who help themselves.
Whoops, didn't read the extended article.
Put each site in a CVS repository. Check it out, wget the live site, check the copy of the live site back into the CVS repository.
I think wget is the way to go, perhaps with the "-m -k" flags, and then check the whole directory tree into CVS using `date +%Y%m%d` as your version number.
I have wanted for a while now to be able to run a command "every time a file is created or updated" in a tree. I know how to do this on a per-directory basis, but I would love to run a command on the changed-file itself whenever an update occursas far as I know, there is no way to cause a "trigger" of this sort. Due to this limitation, I've instead been running the same command on every file on the server each morning. Yes, it's a cron-controlled script and I don't need to touch it, but having it scan and change attributes on every file every day seems sub-optimal.
Any solution which works for this would probably work for this guy too (file news.php has been updated, cp to news-md5-date.php)
-- 'The' Lord and Master Bitman On High, Master Of All
If you have console access to the machines (or can at least make a script), CVS could be a viable solution. Just maintain a central CVS server and have the websites do CVS commits when timestamps on files change. On the other hand, this might not really work if you have dynamic content.
No comment.
If you're solely maintaining static sites, just keep copies of the site as published.
A fine is a tax you pay for doing wrong and a tax is a fine you pay for doing all right.
Sourceforge is open source, why not go d/l that. YOu can use CVS as an easy way to switch around and do upgrades. YOu can develop a site, then run upgrade via cvs and if something unexpected breaks, downgrade via cvs. Once you get the infrastructure in place things like that would be a breeze.
/* oops I accidentally made a comment, sorry */
Bloggers are acutely aware of this problem, they link to pages that change or are moved to paid archives, they call it "linkrot." I've started to provide a .pdf capture of linked articles on my blog, as well as the original link (which I usually take down if I notice it's disappeared).
I like Adobe Acrobat for this job, you just point it at a URL, tell it how many levels you want to archive, and go. You can even archive externally linked pages if you uncheck "stay on same server," or you can select other options like "Archive Whole Site."
You may want to consider something that caches the pages as they are displayed. This will add overhead and doesn't scale, but would allow you to keep a copy of the pages as they were dispayed. You could atleast use it for a subset. For example you use JSPs to serve up pages from a DB but the resulting page is different depending on parms to the page. wget isnt' going to capture all of this, so when the page is generated you write it out with a timestamp ( you build some intellegence so the page only gets written once a month or something) then you archive all of the written out pages and you have a copy of the site as it appeared. Another reseason this may be a good idea is because if you upgrade somesoftware* (in this case Java) You will be able to the as it was rendered under Java 1.3.1 and not how it renders under 1.4.2. Finally if you turn this feature on while debugging, you can really help the developers out, b/c now instead of hearing from the users "My page didn't work, it was weird somehow" You actually have a copy of what was sent to thier browser.
I use teleport pro.
-------
Support Indy Music. Buy
How about using Heritrix, the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler?
"Web Users Should Not Engage in Promiscuous Browsing" --CERT
You could always extract the site with adobe acrobat and have a 'distilled' in tact, usable site in a single file.