Broken Links No More?

johndoejersey writes "Students in England have developed a tool which could bring the end to broken links. Peridot, developed by UK intern students at IBM scans company weblinks and replaces outdated information with other relevant documents and links. IBM have already filed 2 patents for the project. The students said Peridot could protect companies by spotting links to sites that have been removed, or which point to wholly unsuitable content. 'Peridot could lead to a world where there are no more broken links,' James Bell, computer science student at the University of Warwick, told BBC News Online. Here is another story on it." See also the BBC story.

Comments

Can someone say "Bad Idea Jeans"?

[Anonymous Coward]

There are two parts to this tool, one of which is bad quite and one of which is quite good.

First, replacing links. This is a rather quite bad idea. Here's why, with an example.

In general, we can all agree that the technology behind Google is pretty impressive. It has its own "More Pages Like This" feature, which we can assume is at least somewhat similar to this one. Complex content analysis amoung billions of pages, to determine which are similar and which are different.

So, suppose we had a link to Major League Baseball, www.mlb.com on our page. And suppose, for whatever reason, that their site went away (perhaps a few more players' strikes?).

Well, what does Google suggest as a replacement? Check it out here.

First the National Football League (NFL), then the National Basketball Association (NBA), and then the National Hockey League (NHL). Followed by the ESPN sports network, and NASCAR racing.

Obviously if wanted to link to a site about baseball, all of those (other than ESPN) are really entirely irrelevant.

But if we wanted to link to a site about professional sports organizations, all of those (other than ESPN) are QUITE relevant.

Can this software know our intent?

Hardly.

You really have to question the ability of machines to select relevant links.

The situation is this: If someone goes to the trouble to manually create links in the first place, those should not be automatically changed to other sites that some computer program thinks may be related. Links shouldn't be inserted automatically; if someone needs more information on something you haven't linked to, they can use a search engine. And then your company isn't liable to look idiotic by linking to irrelevant sites.

Now, the other aspect of this product.

Removing dead or changed links is quite another matter. Automated removal of links is a great idea and quite useful. For example, consider when someone's domain name expires and it is taken over by a porn site. It'd be great to have a program that automatically removes links to it from your site. Like this tool, this could be based on a percentage of changed content--if the content changes significantly, remove the link quickly and automatically. If the content changes some intermittent amount, flag the link as needing review by the webmaster.

But in those both case, the software should present the webmaster with a list of such questionable links, those it has removed from the site temporarily, and then allow the webmaster to select replacement links.

Manually. With relevance.

Re:Can someone say "Bad Idea Jeans"?

[GigsVT]

The "related" search isn't what you should be looking at.

Try this.

Re:Can someone say "Bad Idea Jeans"?

[Anonymous Coward]

You probably bookmark your A/C posts so you can slip back and check them. You're not fooling anyone, you know.

Re:Can someone say "Bad Idea Jeans"?

[TheRealMindChild]

Actually, I think something pretty simple that should happen (in the context of search engine links, not links in general) is that when a visitor searches google, and I click one of the returned links, google should do something like queue that perticular page for respidering. Make it so a page cant enter that queue more then say once a week, and you will find that we come up with less and less broken links.

While this might seem like a LOT for google to be doing on the backend, I would have to think that a majority of the public ends up visiting the same 5-10% of the the internet each day (number pulled from my ass, but an educated guess at least).

Re:Can someone say "Bad Idea Jeans"?

[Anonymous Coward]

heh.... and what if "whitehouse.gov" for whatever reason doesn't respond ... the third thing on the list is ...

Re:Can someone say "Bad Idea Jeans"?

[dwerg]

Sadly I can't get into details, but their not using technology like the 'related' functionality in Google. They try to find the document that was previously on the other end of the dead link, so the link will never be rewritten to something vaguely related to the original document.

The reason why they want to replace the links manually is because some webmasters have to manage thousands op pages and don't want to press the 'ok' button every time the system detected a change.

Re:Can someone say "Bad Idea Jeans"?

[kzinti]

Obviously if wanted to link to a site about baseball, all of those (other than ESPN) are really entirely irrelevant.

Yep. Because Major League Baseball has strong conceptual similarity to several other concepts: the game of baseball, professional sports, American culture, and others. Granted some are more specific than others, but that's a pretty tough judgment call that depends on the context in which the original link occurred. If I link to mlb.com from a site about baseball, then it means something different than if I link to mlb.com from a site about American culture, and if mlb.com goes dead, then the link would have to be replaced in different ways for the different sites. So this link-replacement software has to be smart about both the destination and the source (context) of a link in order to replace a dead one properly.

I wouldn't go so far as to say this has to be done manually. In theory, software could handle it, but I've never seen software smart enough to automate the task for a such a broad information source as the Internet.

(Disclaimer: no, I have not RTFA; just reacting to an interesting post.)

Re:Can someone say "Bad Idea Jeans"?

[AndroidCat]

That's Mr. Patented Bad Idea Jeans to you! (I'm afraid to RTFA to see how trival their patents might be.)

Re:Can someone say "Bad Idea Jeans"?

[shawn(at)fsu]

After RTFA it doesn't seem like a fair comparison to say it's like google's "related" or Verisign '
"product", this looks like a technology a webmaster would use on there own site. It also gives them they option to accept the suggestion or not. This could be really good for corporation with large intranet sites as webmaster leaves documents constanly get moved etc.

I think had the original poster read the article they wouldn't have gone of half cocked. IBM must also be somehwat confident that this is new technology or else they wouldn't have filled two patents for it.

Re:Can someone say "Bad Idea Jeans"?

[Jotham]

This goes off the assumption that it just uses the link name or address to find the page.

Basically when a page is indexed by a search engine such as google, the first step is to create a document vector from the document based on the repetition of words (terms) and how common these words are (ie. list of TFIDF values -- term frequency * inverse document frequency).

Anyway, this document vector is what is compared against by the search engine to find matches (which is how google can return results is 0.14 seconds). It also acts as a extremely good (and small) statistical representation of the document.

I'd put money on the fact that the program calculates and keeps a record of the document vectors of these linked pages. When re-run, it then can tell how much a linked page has changed from last time it was checked (as mentioned in the article) and if a page has moved it can refind it with a lot of precision. (finding a 100% match if the page has just been moved but the content hasn't changed at all).

Quite clever, and if this was used, you could basically totally reshuffle your web's directory structure, re-run the tool, and have all your links back in place...

Great

[gowen]

Peridot could lead to a world where there are no more broken links,

... just links that don't got where the author intended. Gee, thats ... just great.

Hang on. On similar lines, I've a great idea. Suppose I type a nonexistent hostname into my browser. Wouldn't it be good if the DNS server just gave me its best guess instead of an error message. Or some kind of Site Finding search engine. That'd be even better than ... :)

Re:Great

[rokzy]

I agree it's a bad idea and is imo looking in the wrong direction.

I want things to be LESS tolerant of mistakes, not more. this is why the web is so fucked up. when people can get away with absolute shit, why produce anything better than shit?

Re:Great

[waterford0069]

mean while at Verisign...

"Hey guys, we have grass roots support, check out slashdot!"

Re:Great

[tannnk]

Hey... We had this kind of features on internet before

I liked this one:

[underpar]

A team in the Netherlands built an application that listens to contact centre conversations, picks out relevant keywords and automatically prompts the call centre agent with possible answers.

Does this app take the form of a paper clip? Because that would be a great idea!

Semantic Web?

[jarich]

Wouldn't this idea work a lot better with semantic web markup attached to links and also to intranet pages?

well

[Anonymous Coward]

I think the link is broken... :)

No more broken bookmarks...

[greppling]

...would be good enough for me. I find it really annoying how many of the bookmarks I don't use often are broken after about a year or so.

Re:No more broken bookmarks...

[TrentL]

Why would that be a good thing? If the page I originally bookmarked is gone, I want an error message, not a redirection to something similar.

What if the page is deleted, not changed

[alta]

My biggest problem is when I follow a link to a website that's no longer there. Yeah, moved pages happen, but I don't think they happen as often as deleted pages, expired domains, deleted websites, etc.

Re:What if the page is deleted, not changed

[troon]

That's what the 410 Gone HTTP response header is for. If only admins would use it more...

Take this with a grain of salt

[blankman]

This sounds a little like SiteFinder from Verisign. Click a broken link and isntead of a helpful error message you get whatever content IBM thinks is appropriate. Certainly this could be useful, but it could also end up as just another vehicle for advertising.

Re:Take this with a grain of salt

[liquidsin]

The big difference here is that in the case of SiteFinder, Verisign had control over where you ended up for basically the entire internet. This seems like it would be the type of thing that would run as an Apache mod that would get invoked when a 404 gets returned, and so would only affect that particular site. There's a big difference between going to www.linuxdistro.org/whats_new.html and getting redirected to www.linuxdistro.org/whatsnew.php like this would probably do, and going to www.linuxdistor.org (typo intentional) and having Verisign redirect you to www.microsoft.com because they're getting paid to advertise.

Re:Take this with a grain of salt

[julesh]

If you read the article (the BBC one, which is the only link in there with any relevant information) you'll find that's not how it works. It alerts the webmaster and suggests a replacement, rather than randomly "fixing" other people's pages.

Not Entirely New

[terrencefw]

I've seen lots of site that return search results based on bits of the broken link instead of 404's.

Suppose you have broken link http://somesite.com/foo/bar.html, some sites return a list of search results from within 'somesite.com' matching 'foo' or 'bar'. Quite clever, and much more useful than a plain old 'page not found' error.

This just takes that one step further by doing the searching at the referring end instad.

The Slashdot use?

[makomk]

I actually considered whether it would be possible to write some code to detect if linked-to content has been replaced. The reason I was interested was to make it impossible for someone to put up a copy of a slashdotted page, link to it in a posting, and then substitute it for a copy of goatse once they'd been moderated up.

I decided it'd be too hard for software to decide whether a change was significant. I wonder how this software does it - presumably, you can change the threshold?

More info...

[Guano_Jim]

You can read more information about this process here.

worrying

[TwistedSpring]

"Peridot could lead to a world where there are no more broken links". Yes, it could. Peridot could also lead to a world where broken links are not manually and intelligently spotted and repaired, but automatically repaired. Automatic resolution of what a link "ought" to point to is never going to be accurate (look at search engines), and could make a company website a minefield of confusion and frustration for the user.

Only time will tell, I suppose.

Re:worrying

[FearUncertaintyDoubt]

Imagine a case where a broken link is pointed to another link, which later itself becomes a broken link, and so on...it might even be possible that somehow the chain loops back on itself at some point. One thing I've realized in my career is that if you handle an error too gracefully, no one bothers to fix it. I prefer to have errors cause enough of a problem that there is feedback to fix it.

Can I just have a web site that, you know, works?

[grasshoppa]

Websites need to be useful before I start caring about broken links. I can think of any number of sites that started off with the best of intentions, but never quite live up to being useful.

From bad layout, to missing options, to obscure names for common links, it seems that people are actively trying to hide crap from the end user, making their website utterly worthless.

Can we devise a tool that fixes this problem first?

Well that sounds perfectly dreadful

[Illserve]

Some algorithm cruising through my website, rearranging files as it sees fit?

Sounds like a recipe for utter disaster in the worst case, and a source of mildly embarassing incidents at best.

How about this algorithm just report dead links to a human instead of trying too hard to be clever?

This sounds like someone had to come up with a final project, and settled on this one.

yawn

[FinestLittleSpace]

Look, I'm not being a troll or flamebait or whatever, but seriously, I've had enough of this fucking pipedream chasing crap that gets posted to BBc news and then swiftly chucked up on slashDot.

The whole BBc News Technology section reminds me of the 'Tomorrows World' program when it was in full swing, saying how everything could be 'the next big thing' and that we'd likely se eit on shop shelves and in every home 'in a year'. Why do these people never learn that so much of this is just press release bullshit?

I'm gonna rant more, so sorry. It's just ridiculous... why do we always have to have these blopdy 'next big things'? Why can't people actually look at things rationally (say, as a geek not a mother of 2 who's never touched a computer) and think 'shit, that's not gonna get very far'.

All these posts are so ednlessly flawed, yet we still get news items with titles like 'THE END OF SPAM FOREVER?', 'LINUX COULD KILL WINDOWS IN A MONTH', 'COULD WE SAY GOODBYE TO THE INTERNET AS WE KNOW IT?'... it's all sensationalist bullshit that *might* happen, but not in the press-release inflated year like they always claim.

And... ?

[Doesn't_Comment_Code]

Maybe I'm being overly naive, but checking for broken links doesn't seem all that spectacular to me. It wouldn't take long to write a script to find all the broken links on a page.

The only parts that seemed worth while are replacing the links automatically, and testing if links are relevant.

I'm not so sure I'd trust a computer to do those things though. I'd much rather have the links flagged and checked by a human.

Re:And... ?

[pipingguy]

It wouldn't take long to write a script to find all the broken links on a page.

Just use Xenu's Link Checker.

Re:And... ?

[julesh]

It doesn't only find broken links -- it also alerts you if the content changes substantially. This sounds very useful to me.

CMS

[Anonymous Coward]

Any good Content Management System should already take care of any internal broken links automatically, or notify the webmaster so he'll be able to take care of it manually (in the case of page deletion, etc).

The only kind of people who'd go out of their way to use this software, probably have already use some sort of CMS.

It will work, but that isn't good, here is why

[tod_miller]

A link points to document X.

If document X moves, and the link is invalid, a search for the link might actually find document X, and therefore, you have your benefit, and you would have saved a 404.

However - if a document becomes deprecated and deleted, then how can you assume the link is valid?

Or indeed, if the document has no relevant substitute.

A genealogy providing a link to another Willian Wallace wouldn't be good news if the original page went missing.

A better system is automated 404 alerting to the webservers administrator.

A bad link gets hit, bam, what document, from where. You can work things out intelligently, not automatically.

I think this is silly, perhaps grasping at straws, I see no reason why we would replace all our links to google 'I feel lucky' searches, so why do something like this?

This is the essence of what they have, and all they have done is coulded the search IP field (which is important) with 2 more patents, again increasing costs and endangering open source innovation, the true innovative playing field.

Of course, I could be wrong.

Re:It will work, but that isn't good, here is why

[GeorgeH]

Ideally, cool URIs don't change, but in the real world they do.

If document X moves and the link is invalid, you should be serving an HTTP 301 Permanent Redirect and well behaved user agents will update their bookmarks, and well behaved content management systems will update their code. If document X is gone, you should be serving an HTTP 410 Gone.

Ideally, 404 is supposed to mean that the web server has never heard of the file in question before, but in the real world...

Re:It will work, but that isn't good, here is why

[rho]

This isn't a one-time, forever-and-ever-amen technology. You start with an automatic link-checker and link-fixer. Then you add features like "list all the changes so an editor can filter the results", then you move to "direct potential changes to a team of experts", and so forth and so on. The idea is pretty good. When you're a Big Company with a huge website with thousands of links, having this automagic tool is a lot better than having (unprofessional) dead links.

I, personally, hate dead links with a passion. And, usually, I can devise a Google search that will give me the new home of the old link--often nothing's changed other than the server. A tool that does this for me is useful. Sure, there's plenty of issues that need to be looked into, but that's what we used to call "the Next Version".

It's easy to nitpick this. Seeing the technology, and then seeing how it can be improved in its next iteration is what separates a visionary from a Slashdot howler monkey.

Obligatory RTFA

[acvh]

this isn't about replacing links on the internet as a whole... it's about replacing links on your company website, or at least reviewing those links.

not everything that happens in the world is an attempt by big brother to steer internet traffic to verisign or microsoft.

Not New at All

[Nazmun]

Spyware/Adware and IE already give you search results and links. The only difference is that this automatically places you at a different link without a choice.

Slashdotted

[genneth]

Damn you slashdotters!!! I work at IBM and the intranet server is down! I can't believe you've managed to cause the automatic load-balancer to kill the intranet in favour of a slashdot article.

Damn you!!

And purple hatstands

In other developments....

[tod_miller]

some over funded jumped up interns have developed a high tech, method and software and system to stop the slashdot effect.

Each webserver will return a redirect to a google cache lookup for itself if the load sever gets too high.

1: Stupid idea
2: Patent
3: Wait 'til someone nudges at your generously worded patent
4: happily license this unrelated technology to keep thier VC peeps in the green.

Simple solution

[mirko]

ErrorDocument 404 script.pl
Where script.pl parse the wanted URL and ask an indexing engin to find the most relevant page associated with the query...

Vulnerability?

[darkmeridian]

Remember Google-hacks at http://johnny.ihackstuff.com/? Basically, since Google effectively snoops millions of servers, you can use this information to break into servers and get information. Having an internal feature that connects broken links to real pages may be orders of magnitudes worse. What if I imaginatively "linked" to a made-up URL to see what's on your servers? This could be bad news if it's effectively done.

Prior Art

[Bill Dimm]

They think there is something to patent here? Seems like there should be prior art all over the place. At MagPortal.com we have been using software to repair our links to articles for years.

Rather then finding new relavent links...

[Lifix]

How about this, lets find a better way to eliminate bad links. Have a bot scan your companies, web site, and every time you find a link to an outside source, save that page to your servers, if the link gets screwed up, you can replace it with a link to the saved web page in your server until you can do something about it.

This would not work with large web sites, but if it is just a link to a how-to guide or something small like that this would work.

No thanks

[stratjakt]

I'd prefer a more helpful 404 page, maybe with some links to the homepage or main sections of the site on it.

Sort of a "cannot find hello.jpg, click here to go back to the main page".

My point being, if the document I'm looking for is not there, I want to know it's not there. I don't want to read something else, thinking it's what I meant to read.

Usually when I'm googling around and clicking stuff I'm looking for the answer to some coding or computer related problem. I don't want to click on a link for "configuring Samba 3.0 with AD support", and wind up on a "Configuring Samba 2.2 with LDAP" and waste my time following bad advice.

German readers...

[dukoids]

may want to take a look at the master's thesis of Nils Malzahn (from 2003, in German) to see (in detail) how this actually can work:

http://www-ai.cs.uni-dortmund.de/DOKUMENTE/malzahn _2003a.pdf

Basically, the thesis evaluates different methods to build a kind of "finger-print" of a page. The finger print is used to find the page with google if it is gone, or has changed significantly.

The internet wayback machine was used to learn distinguishing disappeared pages from pages changing slightly over the time.

implications

[theMerovingian]

Wonder where it would send me if www.hotmail.com were down?

*shudder*

(disclaimer: no, I didn't actually look to see what's on that site)

A better solution from the BOFH

[tomhudson]

The BOFH has a better solution http://www.theregister.com/2004/09/17/bofh_2004_ep isode_31/

"Ladies und Gentlemans, I present to you... The Newsmaker!" the PFY chirps happily, waving his hand at his squid plug-in.

"Which does?"

"Give me a news headline, anything, no matter how ridiculous!"

"Scientists discover intelligent life in Redmond!"

>clickety< >clickety< >click< >clickety< >tap< >tap< >clickety< ... >click<

"Right, now Google for it!"

I dutifully fire up Google, bash in Redmond and Intelligence, and roger me senseless with a full height drive if the first 10 hits don't point show up the headline I've just created, pointing at Time Warner, Yahoo News, all the greats...

"Interesting - injecting false links into Google to point at news sites. I like it!"

"Ahem," the PFY interrupts. "Click on one of the links."

I do so, and grab that hard drive for a second go if the site concerned doesn't come up with the headline in question!

"You hacked the news site?"

"Not at all! I used the base idea behind banner blocking to remove the lead headline of a news site and insert my headline instead. You can even add a picture if you want, but obviously only for things that are possible to prove."

"So will this work for all the news sites listed?"

"Oh yes. And more importantly, the various search sites as well. So no matter what common search engine you use, the proxy discards the first 100 matches and inserts 100 of its own 'matches' instead."

Honestly, which of the two is more deserving of patent protectin as an "innovative, non-obvious invention"?

SED?

[Kenja]

So, they've invented SED? Cause thats what I've been using for years to replace old/broken links. A simple script using the netsaint/nagios service tests can check if a link is still good and then build a list of bad ones to be replaced by script number two using SED.

Re:Can I just have a web site that, you know, work

[grasshoppa]

Alright, I gotta know, how is this a troll?

I don't mind being modded down, but seriously, I am at a loss here.

Instead

[Dr. Stavros]

Just use the W3C's link-checker.

been there, done that.

[Quickening]

For those running a real browser, just make this a link, preferably in your personal tool bar.

javascript:Qr=document.URL;if(Qr=='about:blank') {v oid(Qr=prompt('Url...',''))};if(Qr)location.href=' http://web.archive.org/web/*/'+escape(Qr)

Now when I click on a link that isn't there, I select my Archive search button and it shows me the Wayback Machine's history of that link. Of course it works only if the url hasn't been modified by the server. If it has it's another couple steps (copy link, ^T, archive search, paste url in pop-up dialog)

Google "I'm Feeling Lucky"

[Flamefly]

You could create your links using googles im feeling lucky feature, assuming it was just a generic link site looking for interesting sites rather then specific articles.

e.g:
http://www.google.com/search?hl=en&ie=UTF-8&q=News +For+Nerds&btnI=Google+Search

And voila, you'll site will take you to the most popular related site to news for nerds, automagically, if slashdot died one day, another site would take it's place in the google rankings. FF.

Changed is worse in some situations

[phorm]

This is quite often true in respect to sites/companies with large webpages and hence lots of links. One company I used to work in the internet/intranet division for kept links to several partners' webpages. When one of those partners let their domain expire, it was bought out by a pr0n company.

You can imagine how much the staff enjoy the content on the new page... and the IT Security folks especially as the proxy was suddenly giving them lots of nice warnings about workers' viewing inappropriate conduct (probably due to the nasty popups, etc).

scary enough...

[Rev.LoveJoy]

FrontPage has been able to "Scan your web site for broken links" since it first came out in ... what 1997?

Clippy indeed, must be a slow news day,
- RLJ

Firefox extension

[malx]

On a slightly related note, a Firefox extension that searched links ahead and removed the link rendering for those that return a 404 might be handy (albeit fairly evil).

On a less related note, I've long been disappointed that some 300 series status codes in HTTP are so under-exploited, both by clients (e.g. automated bookmark management) and people running web sites.

Actually a complex process

[smagruder]

I periodically run dead link checking software to perform this function with regards to my bookmarks, some of which I publish on my web sites.

There are many things that happen to links, such as redirects, but to conclude that a link is down because you get a 4xx or 5xx HTTP response is extreme. Sometimes sites go down for a period of time for various reasons. Such a link replacement process would need to have some kind of forgiveness mechanism. Further, sometimes links move elsewhere without the benefit of redirects--this replacement process therefore shouldn't replace links with "related" content, but the same content that's moved to another spot.

The bottom line is that the replacement process requires a step in the process where a human being reviews link change recommendations.

Coining a nickname for this technology...

[SuperKendall]

The "I'm feeling lucky" link.

network down

[Pragmatix]

Of course, if your links happen to go to a network that is experiencing a temporary outage, this tool would wreck havoc.

Soon the target network would be back up, but all your links would be lost and randomly changed to something less useful. Good Invention!

Hey micheal RTFA

[ZosX]

From the fucking summary.

Broken Links No More? "Students in England have developed a tool which could bring the end to broken links. Peridot............Peridot could lead to a world where there are no more broken links,'"

I'll troll to hell for this, but I could care less, and I have no problems standing up for what I say. This is terribly irresponsible journalism. No fucking where in the summary does it mention intranet or corporate websites. A world would be pretty global, would it not? Again, the headlines and summaries are getting totally out of sync with the actual articles. Nothing gets edited and people are bitching all over the place about what essentially has amounted to stories (hah, an actual story on slashdot) that are ads. I mean come on. Product announcements versus real news?

I'm sure for as many people that read this site (and pay the bills, as well as *ahem* salary) that the editors must surely get a great deal of story submissions every day. Hell, I'm sure that many people wouldn't actually mind some fucking content once in a while other than links to other stories. The site hasn't been changed in what, like years now? Are the "editors" that busy that they can't even hire someone to actually double check their stories for dupes and errors?

Mod me to hell for this, but I have merely summarized what many have bitched about for months and months and months now, and frankly, while I love the humor and the insightfulness that many share here, the "editors" are really getting lazy these days. Feel free to correct me, I mean I'm sure all those "nothing to see here" and downtimes are starting put a cramp in the afternoon quake matches, so I guess they could be working on getting the site running better, right?

I feel sorry for the subscribers.

zosX

A better idea

[pdamoc]

Maybe it would be better if that smart program replaced the links with links from:
archive.org
or maybe google cache.
Then ofcourse it has to be smart enough to know it did that and replace the links back with the originals if they come online.
Sometimes "broken links" can recover.

predecessor: robust hyperlinks

[pangloss]

There were two fellows at UC Berkeley (Phelps and Wilensky) who implemented the idea of "fingerprinting" web pages at least as far back as 2000. It was a non-trivial fingerprinting (i.e. not just MD5 hash of a web page).

As far as I know, they haven't done any more recent work on this and the software is only available via archive.org.

A paper

I gather that the IBM effort is different in significant respects, but it certainly employs ideas from Phelps & Wilensky.