Slashdot Mirror

← Back to Stories (view on slashdot.org)

Do-It-Yourself Internet Archiving?

Posted by Cliff on from the combatting-the-slashdot-effect dept.

A moron asks: "Web pages change and disappear all the time. For legal and historical purposes, I need to have accessible archives of the websites I maintain. I'm basically looking for a do-it-yourself version of Internet Archive's Way Back Machine which provides a simple versioning system and accessibility through web interface. Is there already software that does this? If not, what ideas does Slashdot have to make such a system possible? How should it work? What existing tools can be used together to make a workable system?"

"There are all sorts of tools out there that will archive web pages, and each have other necessary features such as making links relative. I don't always have filesystem access to pages, so tools that rely on such access won't work. There are some obvious tools that do part of the job such as:

But grabbing pages is only part of my, and I suspect many other peoples needs. The other pieces include intelligently archiving the pages, and making them accessible. If a page or a page element hasn't changed, there is no need to store multiple copies. The archives need to be easy for end users to navigate, search, and link."

2 of 29 comments (clear)

  1. In five lines or less... by Mr.+Darl+McBride · · Score: 5, Informative
    For a small site with complete backups, make a script:

    ARCDIR = `date +%y%m%d`
    cd /var/www/archives
    mkdir $ARCDIR
    cd $ARCDIR
    wget -r http://mysite.com

    Add error-checking and season to taste.

    If you want to be more efficient like the poster wanted, you could easily have it always fetch to the same directory and just use cvs to check in. This eliminates duplicate storage. There are many free web-based CVS browsers out there with date searching and similar features. Might not be quite as nice as the wayback machine, but it definitely does the job for free.

    A lot of folks are doing a simple version of the above to maintain SCO mirrors so there's to be no history erasing before the trial. God bless you all -- it will make the case that much stronger for us.

    1. Re:In five lines or less... by Mr.+Darl+McBride · · Score: 5, Informative
      A quick caveat:

      If archiving SCO or other such pr0n sites, or if you have no-robots policies set on your own site that you're archiving, you'll need to tell wget to be a little rude. He needs to go where robots aren't meant to go. I figure if you were going to visit every page yourself anyway, it's not so impolite. And besides, robots.txt is for other people. You know... the ones we make ride the back of the internet.

      To accomplish this: cat >>~/.wgetrc "robots = off"