Slashdot Mirror

← Back to Stories (view on slashdot.org)

Internet Archive Says It Has Restored 9 Million Broken Wikipedia Links By Directing Them To Archived Versions in Wayback Machine (archive.org)

Posted by msmash on from the better-web dept.

Mark Graham, the Director of Wayback Machine at Internet Archive, announces: As part of the Internet Archive's aim to build a better Web, we have been working to make the Web more reliable -- and are pleased to announce that 9 million formerly broken links on Wikipedia now work because they go to archived versions in the Wayback Machine.

For more than 5 years, the Internet Archive has been archiving nearly every URL referenced in close to 300 wikipedia sites as soon as those links are added or changed at the rate of about 20 million URLs/week. And for the past 3 years, we have been running a software robot called IABot on 22 Wikipedia language editions looking for broken links (URLs that return a '404', or 'Page Not Found'). When broken links are discovered, IABot searches for archives in the Wayback Machine and other web archives to replace them with. Restoring links ensures Wikipedia remains accurate and verifiable and thus meets one of Wikipedia's three core content policies: 'Verifiability.'

5 of 40 comments (clear)

  1. Re:how are these links broken in the first place? by Anonymous Coward · · Score: 2, Informative

    It's not like somebody broke the links by editing Wikipedia. Websites disappear all of the time.

  2. robots.txt to block Wayback Machine by tepples · · Score: 5, Informative

    Until the domain's new owner sets up a robots.txt, causing Wayback Machine to retrospectively block public access to the archived copy of a document. See debate about this a year and a half ago.

    1. Re:robots.txt to block Wayback Machine by drinkypoo · · Score: 5, Informative

      Until the domain's new owner sets up a robots.txt, causing Wayback Machine to retrospectively block public access to the archived copy of a document. See debate about this a year and a half ago.

      Except they don't do that any more, unless the domain's new owner explicitly blocks the internet archive's user agent. A disallow * policy is now ignored.

      --
      "You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"
  3. Re:how are these links broken in the first place? by ZorinLynx · · Score: 5, Informative

    Link rot.

    Even websites that have been around for decades experience it, because they change the structure of their site, breaking links to articles that might even still be available.

    If you follow a CNN link from 15 years ago, it probably won't work.

    It's a bit scary to think how much of our history we're losing to link rot and archive.org is doing their best to fight it. They are awesome people.

  4. Re:how are these links broken in the first place? by Anonymous Coward · · Score: 2, Informative

    This is about external links, not wikilinks.