Slashdot Mirror
Wikipedia Community and Internet Archive Partner To Fix One Million Broken Links on Wikipedia (wikimedia.org)
More than one million formerly broken links in the English Wikipedia have been updated to archived versions from the Wayback Machine, thanks to a partnership between the Internet Archive, and volunteers from the Wikipedia community, and the Wikimedia Foundation. From a blog post: The Internet Archive, the Wikimedia Foundation, and volunteers from the Wikipedia community have now fixed more than one million broken outbound web links on English Wikipedia. This has been done by the Internet Archive's monitoring for all new, and edited, outbound links from English Wikipedia for three years and archiving them soon after changes are made to articles. This combined with the other web archiving projects, means that as pages on the Web become inaccessible, links to archived versions in the Internet Archive's Wayback Machine can take their place. This has now been done for the English Wikipedia and more than one million links are now pointing to preserved copies of missing web content. What do you do when good web links go bad? If you are a volunteer editor on Wikipedia, you start by writing software to examine every outbound link in English Wikipedia to make sure it is still available via the "live web." If, for whatever reason, it is no longer good (e.g. if it returns a "404" error code or "Page Not Found") you check to see if an archived copy of the page is available via the Internet Archive's Wayback Machine. If it is, you instruct your software to edit the Wikipedia page to point to the archived version, taking care to let users of the link know they will be visiting a version via the Wayback Machine.
By the same token, links could be taken over by bad actors at any time, or might delete content that was relevent to the reason they were linked in the first place.
I kinda wonder if Wikipedia should *only* link to wayback content (just with a one-click option for a live/updated link and maybe an option to perform an edit to update it to a more recent wayback copy), because it is more in the spirit of the wiki audit-trail. Of course, that would probably require adding more resources at wayback.
Someone had to do it.
That's a problem across the whole web and, at least the deletion part, happens more often than you'd think. When I've updated links on Wikipedia, I note that it not only asks for a CAPTCHA but alerts editors to the change, in case the change was malicious.
I think the motivation is good, but the implementation (as I understand it) could be better. Perhaps what is needed is to add a Wayback link alongside the original one. Does Wikipedia have a process for human review of broken links? In the cases I've found, replacement links can be found quickly for content that just moved.
yes, even if a domain squatter gets the domain.
Additionally, their interpretation of robots.txt is questionable.
It was meant to prevent automated crawlers, not human-requested fetches, yet often the web archive will disallow me from archiving a page because of robots.txt.
This is one reason I often will archive to both http://web.archive.org AND archive.is.
Archive.is explains its robots.txt policy in its FAQ.
http://archive.is/faq#Why_does...
> Why does archive.is not obey robots.txt?
> Because it is not a free-walking crawler, it saves only one page acting as a direct agent of the human user. Such services don't obey robots.txt (e.g. Google Feedfetcher, screenshot- or pdf-making services, isup.me, )
People have asked about this on the archive.org forum but I haven't read them all to see if there are any good answers.