Slashdot Mirror

← Back to Stories (view on slashdot.org)

Wikipedia Community and Internet Archive Partner To Fix One Million Broken Links on Wikipedia (wikimedia.org)

Posted by msmash on from the internet-is-beautiful dept.

More than one million formerly broken links in the English Wikipedia have been updated to archived versions from the Wayback Machine, thanks to a partnership between the Internet Archive, and volunteers from the Wikipedia community, and the Wikimedia Foundation. From a blog post: The Internet Archive, the Wikimedia Foundation, and volunteers from the Wikipedia community have now fixed more than one million broken outbound web links on English Wikipedia. This has been done by the Internet Archive's monitoring for all new, and edited, outbound links from English Wikipedia for three years and archiving them soon after changes are made to articles. This combined with the other web archiving projects, means that as pages on the Web become inaccessible, links to archived versions in the Internet Archive's Wayback Machine can take their place. This has now been done for the English Wikipedia and more than one million links are now pointing to preserved copies of missing web content. What do you do when good web links go bad? If you are a volunteer editor on Wikipedia, you start by writing software to examine every outbound link in English Wikipedia to make sure it is still available via the "live web." If, for whatever reason, it is no longer good (e.g. if it returns a "404" error code or "Page Not Found") you check to see if an archived copy of the page is available via the Internet Archive's Wayback Machine. If it is, you instruct your software to edit the Wikipedia page to point to the archived version, taking care to let users of the link know they will be visiting a version via the Wayback Machine.

8 of 21 comments (clear)

  1. The links will be reverted as "not notable" by Anonymous Coward · · Score: 3, Informative

    Wikipedia, reverting knowledge since 2001.

  2. Awesome by edittard · · Score: 2

    Somebody wrote a perl script!

    --
    At the bottom of the /. main page it says 'Yesterday's News'. Well they got that right.
  3. Re:IMPORTANT INFORMATION, PLEASE REREAD by D00MSlayer · · Score: 4, Funny

    Have you ever eaten human? They're GRRRRRREEAAT!

  4. Not good enough by stevel · · Score: 2

    I have found many cases on Wikipedia where the links are broken but the correct content exists at a different URL. This auto-archive system would bypass that and perhaps prevent ever recognizing that the link target still exists. This is especially an issue for links to corporate and government pages where someone periodically gets the bright idea to reshuffle the web site's organization and doesn't put in permanent redirects.

    1. Re:Not good enough by skids · · Score: 3, Interesting

      By the same token, links could be taken over by bad actors at any time, or might delete content that was relevent to the reason they were linked in the first place.

      I kinda wonder if Wikipedia should *only* link to wayback content (just with a one-click option for a live/updated link and maybe an option to perform an edit to update it to a more recent wayback copy), because it is more in the spirit of the wiki audit-trail. Of course, that would probably require adding more resources at wayback.

      --
      Someone had to do it.
    2. Re:Not good enough by stevel · · Score: 4, Interesting

      That's a problem across the whole web and, at least the deletion part, happens more often than you'd think. When I've updated links on Wikipedia, I note that it not only asks for a CAPTCHA but alerts editors to the change, in case the change was malicious.

      I think the motivation is good, but the implementation (as I understand it) could be better. Perhaps what is needed is to add a Wayback link alongside the original one. Does Wikipedia have a process for human review of broken links? In the cases I've found, replacement links can be found quickly for content that just moved.

  5. Those links will still break by alexo · · Score: 4, Informative

    Because the Internet Archive applies robots.txt rules retroactively.

    1. Re:Those links will still break by brewthatistrue · · Score: 3, Interesting

      yes, even if a domain squatter gets the domain.

      Additionally, their interpretation of robots.txt is questionable.

      It was meant to prevent automated crawlers, not human-requested fetches, yet often the web archive will disallow me from archiving a page because of robots.txt.

      This is one reason I often will archive to both http://web.archive.org AND archive.is.

      Archive.is explains its robots.txt policy in its FAQ.

      http://archive.is/faq#Why_does...

      > Why does archive.is not obey robots.txt?

      > Because it is not a free-walking crawler, it saves only one page acting as a direct agent of the human user. Such services don't obey robots.txt (e.g. Google Feedfetcher, screenshot- or pdf-making services, isup.me, )

      People have asked about this on the archive.org forum but I haven't read them all to see if there are any good answers.