On Finding Semantic Web Documents

Anonymous Coward writes "A research group at University of Maryland has published a blog describing the latest approach for finding and indexing Semantic Web Documents. They have published it in reaction to Peter Norvig's (director of search quality at Google) view on the Semantic Web (Semantic Web Ontologies: What Works and What Doesn't): 'A friend of mine [from UMBC] just asked can I send him all the URLs on the web that have dot-RDF, dot-OWL, and a couple other extensions on them; he couldn't find them all. I looked, and it turns out there's only around 200,000 of them. That's about 0.005% of the web. We've got a ways to go.'"

Comments

It's not about the filename

[Simon Brooke]

It's not about the filename extension (if any), silly. It's about the data. Valid RDF data may be stored in files with a wire range of extensions, or even (how radical is this?) generated on the fly.

What matters is first the mime type (which is most likely application/xml or preferably text/xml), and the data in it.

Oh, and, First Post, BTW.

Re:It's not about the filename

[crschmidt]

Preferably application/rdf+xml . Anything else is not appropriate for RDF-serialized triples. text/xml and application/xml are both wrong for this kind of data.

This will become more important as resources are represented in multiple ways, for tools to consume: they ask for a specific type, and fallback may fall the wrong way if people start telling their webservers that RDF is something it's not.

Re:It's not about the filename

[Anonymous Coward]

Outstanding first post!

Re:It's not about the filename

[old_guys_can_code]

I work at one of the few places that crawls billions of URLs each month, and I observed exactly the same thing as Peter. There just isn't that much xml/rdf/daml/owl on the web. At the point when we had crawled 6 billion URLs, I found only 180,000 URLs that had a mime type or extension to indicate that they were machine-readable metadata.

The reason is something that people in the semantic web community are loathe to talk about - that there isn't enough incentive for people to create metadata that they put out for others to read. When we write web pages or blogs, we are able to express ourselves to other humans, but when we put out data there is no clear incentive (economic or otherwise) to justify the effort. This is probably why there is so little metadata being published.

If you wish to dispute the small amount of data, feel free to put up a web server showing a million URLs of metadata created by others.

Semantic?

[Anonymous Coward]

I used to love their Norton Utilities.

What about...

[Apreche]

What about all the pages that are .rss but are actually rss 1.0, those are rdf-based. And what about all the rdf which is in the comments of .html files and others? My creative commons license is rdf, but its inside a .html file. Sure, we do have a long ways to go, but the semantic web is bigger than a few file extensions findable by google.

What's all this...

[Anonymous Coward]

What's all this about finding Semetic web documents... Oh... Never mind.

Yahe

[Anonymous Coward]

HTML are semantic. It's all in the

body

unexpected?

[AnonymousCactus]

Without a large number of widely used tools out there that make use of semantic information there won't be that much content designed for them...and without content designed for them the tools won't exist and certainly won't be widely used. Currently it's more of an academic exercise - if we somehow knew what all this information on the web actually was, what could we do with it? More interesting it seems then are approaches at bypassing the markup by hand and do something equivalent automatically.

Solution without a problem?

[faust2097]

Semantic web stuff if cool and all but I honestly don't believe that it will ever really take off in any meaningful way. For one, it takes a paradigm that people know and understand and adds a lot of complexity to it, both on the user end and the engineering end.

Plus a lot of the rah-rah booster club that's grown up around it sound a whole lot like the Royal Society folks in Quicksilver who keep trying to catalog everything in the world into a 'natural' organization.

What it basically comes down to for me is that it seems like a great framework for single-topic information organization but at a point we need to keep our focus on the actual content of what we're producing more than the packaging. For this to be ready for prine time the value proposition needs to move from a 30-minute explanation involving diagrams and made-up words ending in '-sphere' to something even less than an "elevator pitch" like 2 sentences.

Re:Solution without a problem?

[Anonymous Coward]

" For one, it takes a paradigm that people know and understand and adds a lot of complexity to it,"

but what if i leverage this paradigm with action orientated out of the box solutions?

surely this sea change couppled with changing the onion, will serve to generate open door mindshare in this post 9/11 society...

no?

Re:Solution without a problem?

[amembleton]

No

Re:Solution without a problem?

[Anonymous Coward]

reification baby, that's all you need to know.

say it with me

"reification"

ooh yeah. it means "to make into a thing".

"reification"

mmm, gotta love them RDF buzzwords.

the spam problem

[hostyle]

Who cares about the semantic web or any new web technology if its going to be deluged by spam within 5 days of deciding to use it, and thus becoming unusable / untrustable as a resource. Deal with the spam problem, then come back to me about these great new technologies that are vulnerable to it.

Re:Plasma TV Lesson

[LMAOff]

Dude! Its got to be the funniest thing Ive heard so far. Here is an idea for your friend to save him some embarresment. Ask him to turn on the tv hook up the laptop and pause some vidoe (non-porn) that way he can cover up the burn in... actually burn the remaining stuff so they cant make out Man! My buddies are laughing their A** off too.. Good one will remember for a long time to come.

Wow

[KillerDeathRobot]

I thought I knew what these articles were supposed to be talking about, but it turns out I had no clue.

Re:Wow

[Anonymous Coward]

I thought I knew what these articles were supposed to be talking about, but it turns out I had no clue.

Summary:
The semantic web is all about putting useful, very well defined meta-data all over the place so that doing useful things will be much easier. The google dude says it isn't working because:
1. People can't agree on what the meta-data should be
2. Meta-data will be abused anyway.

It relies too much on people getting together, making rules, and sticking to the rules.

Re:Plasma TV Lesson

[jahmike]

Hahahahahah .... this shit is funny ahhahahahaha I wonder if he will have nightmares?

Wait, what's does Semantec...

[Spy Handler]

Norton Antivirus got to do with this web technology?

Re:Slashdot lies, opinions, and half-truths

[hostyle]

When you can honestly show me how any sane person can support the RIAA's stance on anything - yet remain a money-gouging, price-fixing cartel who pay their representants (the artists) pittance - I may actually listen to you.

LiveJournal and other weblogging services

[crschmidt]

Every user of a LiveJournal-based website running recent code has a FOAF file. Let's look how many users that is:

* LiveJournal.com: 5751567
* GreatestJournal.com: 717406
* DeadJournal.com: 474435
* Weedweb.net: 22650
* InsaneJournal.com: 12970
* JournalFen.net: 7629
* Plogs.net: 7086
* journal.bad.lv: 4530

(This list is most likely incomplete.)

In addition to this, every Typepad user has an account: according to the 6A merger stories, that's another million users. Add in the RDF from all the Typepad RSS files, and that's another 1 million.

All Wordpress blogs have a feed, located at /feed/rdf or /wp-rdf.php, which is in RDF. Movable Type comes preinstalled with an RSS 1.0 feed. Each of these has at least a couple thousand users.

So, we've got, just as a guess, about 9 million RDF files out there in the blogging world alone. Throw in a hell of a lot of scientific data, and everything on RDFdata.org, and you start to get an idea that the world is a lot more Semantic Web enabled than you seem to think it is.

Re:LiveJournal and other weblogging services

[Da_Weasel]

About 75% of those that signed up for those various blogging services have never actually posted a single entry in their blog. So the actual numbers is more like 2.2 million of so. Even with a devistating hit like that it's still 10 times more that the number stated in the article though....lol...and its still just the bloggers alone.

Re:LiveJournal and other weblogging services

[zangdesign]

So, we've got, just as a guess, about 9 million RDF files out there in the blogging world alone.

Care to venture a guess as to how many of those actually contain useful information? Really, who cares if Melanie in Oshkosh really, really loves Justin Timberlake, or Winthorpe in Des Moines really, really wants people to sign up so he can get an Ipod?

Furthermore, once you start tying all this information together, doesn't that just make the work for corporate data miners just that much easier?

Of course, you could salt in a bunch of useless, random data, which of course, means that the whole shooting match is useless.

Wow, a lot is missing from this survey!

[Chris Croome]

A few sites I have worked on that are run by MKDoc are listed in their top 500, since MKDoc generates a RDF metadata file for every HTML document, but the biggest and most interesting are missing, I expect that there are perhaps several hundred times more RDF documents out there than they have found...

Hey Michael....

[Anonymous Coward]

How's censorware.org doing?

OT Re:Slashdot lies, opinions, and half-truths

[ezzzD55J]

I know it's offtopic (and trollish), but..

* Slashdot editors are abusive. We all remember The Post.

Anyone know what he's talking about here?

Re:Slashdot lies, opinions, and half-truths

[Ghostgate]

Any company ending in AA is evil. Especially if it doesn't want you distributing its works without paying for it. Somehow, this mindset is supposed to make sense.

That's harsh, man! I have nothing against the AAA. Why, just last week they came and changed my tire when I had a flat and was without a spare.

Anyway... did you say anything else, or was that pretty much it? Oh, yeah! Almost forgot:

Slashdot is dead.

Wow! That's pretty big news. But has Netcraft confirmed this???

Don't worry, I'm laughing with you, not at you. No... no, really.

I guess they forgot to read the TOS

[dracocat]

From the article: For each new site S we encounter, we give Google the narrower query 'filetype:owl site:S', which will often end up getting some additional results not included in the earlier query.

From the Google TOS: You may not send automated queries of any sort to Google's system without express permission in advance from Google.

I am serious. These researches just used a lot of resources from Google that they had no permission to use. Researchers especially should try to be good citizens on the net and not do tons of automated querying to websites without permission--especially when it is specifically prohibited.

Google has spent a lot of time and money to get the information that they wanted; and when asked for copies of it google didnt give it to them--so instead they just took it without permission.

I would call that stealing, except I wont because that will start a whole other thread thelling me that information cannot be stolen.

My point is, if you want to do research, at least play by the rules that you are given. It may take longer and require more work, but that seems better than using information that you dont have permission to use.

Re:I guess they forgot to read the TOS

[Anonymous Coward]

Well, this has probably resulted in fewer than 10 thousand Google queries -- several for each unique site on which we've seen a semantic web document -- over several months. We do this to generate seed URLs for our own crawlers. That doesn't sound like a problem to me. I'm sure it's noise, and a very still, quiet noise, in Google's transaction stream.

Re:I guess they forgot to read the TOS

[Anonymous Coward]

That doesn't sound like a problem to me.

The problem is that you've clearly broken their terms of service. You are taking advantage of their resources without their permission. In case you don't get it, that's WRONG.

How hard is it to get permission from Google? Are you really that lazy??

Re:I guess they forgot to read the TOS

[Anonymous Coward]

I happen to know the members of the ebiquity lab personally, I was a grad student in another lab at UMBC. Just to clarify what the person who wrote this post completely made up, the ebiquity people actually had an account with Google to use the web APIs for use in Swoogle. I know this because at the time they were working on it, Google was having some sort of technical problem and they couldn't register for a license key. So they asked around if anyone had a valid key that they weren't using and that ebiquity could borrow until they were able to register for their own key. They weren't sending automated queries without Google's express written consent, they were using the Google API to do the queries and they were not abusing the system because they took the initial 1000 query results they got, did some processing on their own, and then issued narrower queries to gain additional information. It doesn't specify that you can only run one query, it just says you can't get back more than 1000 results for a single query. I'm not sure how you can call this "stealing" or imply that it might be stealing, given that Google provides the information free of charge to anyone who types in a query on their site. So while you raise a good point that researchers should not circumvent rules in order to accomplish their research goals, in this particular case you haven't a clue what you're talking about and you shouldn't try to tarnish the reputation of well known researchers in the AI community.

Re:I guess they forgot to read the TOS

[dracocat]

I must have been mislead by statements such as:
We never did get any help from Google. -- Tim Finin
and statements like: Please do not write to Google to request permission to "meta-search" Google for a research project, as such requests will not be granted. -- Google.

You may be right, they may have had permission, but all I see is complaints against google for no cooperation--which I would count special permission to use their database as being.

At any rate, if I misunderstood the Tim Finin, then I do apologize--but I do have to say that the way they talk about working around Google's restrictions and their lack of help sure makes it sound like they have never received any such permission.

Re:I guess they forgot to read the TOS

[Finin]

Perhaps I should have clarified how Google didn't help. I had asked Peter Norvig if we might be able to get all of the .owl, .rdf, .rdfs, etc files. He said he'd check, but we never heard back. This was during the early stage of the runup to the IPO, so I was neither upset nor surprised. If Google started helping all of the web hackers in the world, they'd never get anything done. Besides, it gave us a new problem to solve and left us with a ice warm feeling after we did. I'm pretty sure our use of the Google API was in line with their TOS and didn't stress their systems any more than a random user with a bad caffeine habit.

Re:I guess they forgot to read the TOS

[Anonymous Coward]

the google api by definition allows automated querying of google, i would suppose the terms of service are for automated querying of the browser accessible service.

Re:I guess they forgot to read the TOS

[claes]

I would call that stealing, except I wont because that will start a whole other thread thelling me that information cannot be stolen.

Too late! In that case Google is the bigger thief with all the cached content it has stored and provides through its own servers.

Re:I guess they forgot to read the TOS

[/dev/trash]

Nah. This would be a valid example of Fair-Use, that everyone here likes to bandish about.

Re:Plasma TV Lesson

[funkify]

No, you should offer to attempt to "repair" it yourself then do what LMA suggests YOURSELF. If this works, charge him $$$. Of course, I don't know enough about plasma screens to know if it would really work or not. You might try pausing on an all-white screen.

blah blah blah semantic blah blah blah

[Anonymous Coward]

That's about 0.005% of the web. We've got a ways to go.

I dunno about you, but I'm not going to do this to any of my data, unless I'm forced to (i.e., my editor saves it that way, or Firefox 5.0 doesn't read it otherwise).

So don't hold yer breath.

Re:Slashdot lies, opinions, and half-truths

[hostyle]

So you don't deny that they are "money-gouging, price-fixing cartel" then? I guess we're in agreement then ...

Two sentences, eh?

[misuba]

You're on.

1) A simple human- and machine-readable schema is defined for marking up descriptions of items for sale or wanted.
2) Google learns how to read them, thereby putting eBay, Craigslist, and other sundry companies out of business and putting your data back in your hands.

Okay, so the second sentence is a bit of a run-on, and this use case has a whole lot of hairy details I'm leaving out. But the possibilities are pretty exciting nonetheless.

Re:Two sentences, eh?

[cmowire]

The problem is, that doesn't require the semantic web or any sort of semantic technologies.

A simple well-formed XML document will suffice and be simpler to write. And if you *really* want to make it fit into the semantic web, you can provide an XSLT file that translates it to RDF. The problem is that RDF is wounded by having an incredibly ugly syntax.

Furthermore, the simple model of posting XML or RDF documents and google not only magically finding them but "putting the data back in your hands" is flawed. Part of the reason why eBay, Craigslist, etc. all exist is because there is just enough checks and balances in place that makes them not useless. Craigslist has distributed flagging and requires email addresses, eBay has accounts and a reputation system. Sure you can create reputation systems, but in order to make that work, you need to be able to pick one easily and preferably automatically. So you need.. ehrm.. a reputation system for reputation systems...

The problem is, not only do you need to be able to do an "elevator pitch" 2 sentences of a useful app, you also need to make it not fall over due to abuse, scaling laws, etc.

I think the biggest problem is that you can't trust metadata blindly. And most of the big "semantic web" stuff assumes that you can, or that figuring out trust can be "solved".

And there is metadata that's trusted. EXIF tags are trusted, simply because there's no benefit in lying. RSS is provisionally trusted simply because the user picks if they want to syndicate or not -- again, you are doing it to draw people back and lying doesn't help.

The problem is the entire semantic web movement is doing damage to its possibilities for the future. Because RDF and OWL and RDF Schema and such are all such thick and hard-to-grasp items and also because it's a buzzworded hyped technology, stuff that could be written to enable semantic technologies isn't.

See, I have my doubts that you can really write applications that, through the magic of ontologies and other stuff, can make intelligent relations between data in such a way that the total time spent in making an ontology, a schema, and whatever else necessary to make useful things happen is less than it would have taken to write a simple application that does what you want it to.

But, really, if it can be made to work, it needs to scale. There needs to be data. Sure FOAF and RSS 1 use RDF. But OPML, RSS 2, and Atom don't.

The problem is, the Semantic web folk need to quit complaining about there not being any semantic-accessible data or applications and start figuring out how to get around that.

My current thesis is that if you really want to turn the Internet into a tuple space, the fact that there isn't much current data in RDF isn't insurmountable, it's just nobody's willing to accept that you can't just force people to do everything around RDF at this point in the game.

Longwinded rant, but if you go back up to the top of the post, there's the rest of my argument. If you don't have RDF-format data, can't send out jack-booted thugs to force people to make RDF-format data, and need some of it to make things work properly, you need to figure out how to generate it.

Which there's plenty of stuff to draw from. Convert Atom, RSS 2, OPML, and other data to RDF representations and you've increased your space. You can generate more from webpages by parsing HTML for meta tags, links, and stuff. You can query google for "What's Related" and such.

The problem is, nobody's bothered to work on a tool to make tuple-spidering code to generate tuples for RDF. They aren't even trying to come up with halfway-point guidlines about making existing and new documents more able to eventually be converted to RDF.

Instead, semantic web folks are just complaining about how there's nothing to work with.

Re:Two sentences, eh?

[faust2097]

But I like craigslist better than I like Google. And the quality of that data in my hands is dependent upon the internet community at large AKA those people who write gay erotic fan fiction about Star Trek characters.

Besides, Google's pagerank has been owned by 'optimizers' for years. I don't trust them any more than any other commercial enterprise.

And FOAF is the closest the 'blogosphere' has come yet to physically jerking each other off.

Re:Two sentences, eh?

[mike_sucks]

"The problem is, that doesn't require the semantic web or any sort of semantic technologies."

You're right of course, but for any such initative to be successful, it needs to use a standard (or at least widely-known and stable) format/grammar/etc so that thrid-party systems can understand your data.

This is where RDF, OWL and the other semamtic web technologies come into it. Why invent another system when there is already one there?

"The problem is that RDF is wounded by having an incredibly ugly syntax."

No, you're confusing the XML syntax with the model. RDF isn't the XML format, that's just one way to serialise an RDF graph. You also have the n-triple format and others. Ideally, XML serialised RDF would never be hand-written, it _is_ a pig to do so. But is a convenient way to dump an RDF graph in such a way that it can be reliably machine read (which is the whole point of the semantic web).

Notice also how OPML, RDF2 and Atom are never generated by hand? Given the format is best generated by a computer and best consumed by a computer, what's the problem with the format?

It would be nice if there were one canonical way to serialise the graph, thus making processing with tools that aren't RDF-aware easier (eg, an XSLT processor), but I don't think that is a show-stopper.

"I think the biggest problem is that you can't trust metadata blindly. And most of the big "semantic web" stuff assumes that you can, or that figuring out trust can be "solved"."

So why are places like Ebay, Amazon and so on trusted? How is buying something directly via the Ebay web interface any different from buying it via Google, which picked up the same auction from EBay's RDF feed?

There's a lot of places where trust comes into it. Do you trust Google? Do you trust EBay? Do you trust the seller? The semantic web doesn't solve this problem, but it can make it much easier for you to locate the thing in the first place.

"And there is metadata that's trusted. EXIF tags are trusted, simply because there's no benefit in lying."

Right, so why not make the EXIF data available via RDF anyway? Even though it can't be trusted, at least I would be able to search for images that proport to be of a sunset taken between between 17:00 and 18:00, using a Canon IXUS II? That's more than I can do now.

"If you don't have RDF-format data, can't send out jack-booted thugs to force people to make RDF-format data, and need some of it to make things work properly, you need to figure out how to generate it."

Well, that's main problem the sematic web faces today, lack of tool support. Why doesn't Dreamweaver people to embed Dublin Core RDF into every document it produces? Why don't the endless numbers of slide-show gallery generators do the same for EXIF data?

Note however that this isn't a problem caused by the XML RDF serialisation format.

"The problem is, nobody's bothered to work on a tool to make tuple-spidering code to generate tuples for RDF."

Honestly, what's the point? It would be much more producive to refit authoring and content management tools so they produce RDF and search engines and the like to consume RDF. We'd me much better off.

Re:Two sentences, eh?

[SunFan]

1) A simple human- and machine-readable schema is defined for marking up descriptions of items for sale or wanted.

How does a typical shop keeper learn to do this and apply it to their wizard-made web page?

The second point about Google is fairly ripe for abuse. Meta tags in HTML were mostly rendered useless by porn sites, for example. Also, sites like eBay tend to concentrate useful information in useful ways, while feeding keywords to Google can often be frustrating for anything remotely generic.

Re:Two sentences, eh?

[cmowire]

The problem with RDF is that they are putting really technical terms on really simple ideas and nobody's done an especially good job of distilling it down to the most basic level in such a way that anybody can program it. It's not just that the format's ugly, it's that, as far as I can tell, the vast majority of folks who actually are in a position to output semantic information suffer eye-glaze-over when they try to understand RDF.

Furthermore, the problem is not just that you need to have the tools output the semantics, but you have to get people to put them in the document. When's the last time you've seen somebody fill in the various meta-information fields on a word document? So, sure Dreamweaver could put Dublin Core metadata there, but people won't use it. In fact, they'll probably complain if it's there because it'll either bloat the size of their documents or include information that they'd rather not have included -- like EXIF thumbnails and revision notes and things like that. Dublin Core won't get you anywhere on any of the search engines because the meta tags got too much search-engine-spam crap stuffed into them. Likewise, if there was an image gallery that gave an RDF representation of the EXIF tags, it would slow down load times, so unless there was a good advantage to using it that way, people would complain that crap they don't want is in the document. Remember, people are *surprised* that folks who use flickr and del.icio.us are actually adding metainformation of any sort to their personal stuff.

I don't implictly trust EBay or Amazon.com. However, I do trust that there's at least some modicum of crap-removal at play. Remember, the problem with the web isn't just finding stuff. It's finding stuff without finding crap.

The problem is that there's a log of different tools for building sites. There's quite a few weblogging systems (Slashcode, Drupal, Radio UserLand, Movable Type, etc) out there. There's tons of shopping carts. There's a variety of commercial CMS systems. There's frontpage and dreamweaver. There's stuff that nobody in their right mind would use anymore, except that they are too lazy to use something else. There's hand-coded stuff. Simply put, there's a lot of stuff generating pages. Most of the time, they can't even be bothered to make their pages work on Mozilla or be compliant to any sort of standard. About the only thing that can be said about the task of changing how most of the authoring and content management software in use outputs stuff is that it's easier than moving to IPv6.

No, folks have been moaning for several years now that we need to RDF enable stuff. It's not going to happen. The only road forward is to accept that and figure out how to have a semantic web without requiring huge buy-in before you've got useful apps.

And if there start to be some actual applications of semantic web technology, other than tools for generating the data and viewers that claim to be revolutionary but always seem to just display things as a directed graph, then people will start thinking about outputting RDF.

Re:Two sentences, eh?

[mike_sucks]

"as far as I can tell, the vast majority of folks who actually are in a position to output semantic information suffer eye-glaze-over when they try to understand RDF."

That's interesting, do you have some figures to back that up? I'm a generic web developer from a small city in a backwards country, and I get it. Are you saying that RDF is harder to learn then say, writing a POSIX or Windows application? A FPS? A Java web-app? A Linux kernel module? Because plenty of people do those things, every day.

"Furthermore, the problem is not just that you need to have the tools output the semantics, but you have to get people to put them in the document."

If you don't want to provide metadata, then fine, don't do it. But if you don't provide metadata, it doesn't matter if you don't provide it RDF or don't provide it some other format. Aren't you arguing against RDF, not metadata in general?

"So, sure Dreamweaver could put Dublin Core metadata there, but people won't use it. In fact, they'll probably complain if it's there because it'll either bloat the size of their documents"

If size is a concern, then you can provide the RDF in a separate file and use a HTML link element to point to it. In fact IIRC, for HTML documents and for XHTML served as text/html, this is the only correct way of embedding serialized RDF in a web page.

"I don't implictly trust EBay or Amazon.com. However, I do trust that there's at least some modicum of crap-removal at play. Remember, the problem with the web isn't just finding stuff. It's finding stuff without finding crap."

Right, so you have to trust an RDF-aggregation-based auction site or search engine as well. Maybe you'll trust the auction site because they have similar policies and procedures as EBay and Amazon, maybe you won't. Search engines have to deal with metadata spam in the same way they deal with keyword spam in the HTML document's body or with link farms. How is any of that that RDF's (or metadata in general's) problem?

"The problem is that there's a log of different tools for building sites."

There certainly is a lot of tools for building sites, but how is that a problem? If people want to publish metadata, they can choose a tool to allow them to do so, or to modify their existing tools and/or practices.

"No, folks have been moaning for several years now that we need to RDF enable stuff. It's not going to happen."

Err, it has already happened. Blogging has made RDF-based syndication a must-have feature for web based applications and has shown how the semantic web can make life a lot easier.

The question is, where is it going to spread to next?

Re:Two sentences, eh?

[cmowire]

*one* RDF-based technology (And only for part of the market) in the past several years means that we'll start having real semantic applications in 2105 or so.

Remember, there's nothing intelligent about syndication. It fits just as well into the "well formed web" as it does the "semantic web". All RDF does is make things much more verbose than it is otherwise. The whole point of the semantic web was so that I could view a RDF Site Summery file and have my web browser automatically figure it out and link to other places with it.

No, I don't have any facts to back up that your average web designer's eyes will glaze over. But then, your average web designer isn't thinking about RDF, so the message is either getting lost or ignored somewhere.

My theses are:
1) RDF, without a dramatic reduction of complexity, coupled with real applications that can only be done with RDF, not merely well-formed XML, is doomed for failure.
2) Trustworth metainformation is valuable. However, it's also extremely rare. Many of the "cool things" that the semantic web was supposed to enable require trusted metainformation.
3) Until there are real applications for Semantic Web technologies, people won't Semantic Web enable their software in any substantial way. Therefore, if you want the Semantic Web to happen you either need to find a way to make existing metadata RDF-accessible or you need to make one with existing technology.

Re:Two sentences, eh?

[mike_sucks]

"Remember, there's nothing intelligent about syndication. It fits just as well into the "well formed web" as it does the "semantic web"."

Sure, syndication is a specialization of the semantic web in general, but when you come down to it, that's all the semantic web is: machine readable information about a web-accessable resource. There's nothing special or crazy or obtuse going on, it's that simple.

"But then, your average web designer isn't thinking about RDF, so the message is either getting lost or ignored somewhere."

Maybe, but only because we are driven by what our customers want. But the demand is there, and growing. Many companies and government organisations want compusory metadata for their {Inter|intra|extra}net web sites. It is interesting to note that it is required that all Australian Government web sites provide DC metadata. That trend is going to continue.

"1) RDF, without a dramatic reduction of complexity, coupled with real applications that can only be done with RDF, not merely well-formed XML, is doomed for failure."

Heh. Well, I assert it isn't doomed to failure. I think we need to provide some evidence here.

"2) Trustworth metainformation is valuable. However, it's also extremely rare. Many of the "cool things" that the semantic web was supposed to enable require trusted metainformation."

As I said, it isn't about trusting the metadata, it is about trusting the source. If you trust the source, you trust the metadata. This situation isn't any different to what we have today with human-readable web pages.

Out of curiosity, what are these cool things that require technology based/provided trust?

"3) Until there are real applications for Semantic Web technologies, people won't Semantic Web enable their software in any substantial way."

There already is and they already do, as I said before.

"Therefore, if you want the Semantic Web to happen you either need to find a way to make existing metadata RDF-accessible"

Sure, HTML meta tags can be assumed to be statements about the HTML resource that contains them, Atom data may be able to be coerced to RDF as well, but you also have to have, say, a metadata-smart search engine that lets you search over this metadata. I.e. we still need better tool support.

" or you need to make one with existing technology"

Sorry, I couldn't grok that, make one what?

But how do I use this semantic data?

[sffubs]

Apart from RSS feeds, how can I use this data? I mean, I have RDF metadata available for pretty much every page on my website, but I haven't yet noticed anyone who actually reads it.

The semantic web seems like a good idea in principle, but I would really like to know just how I could use it in real life! Seriously, can anyone name a useful tool that relies on RDF feeds (again, aside from RSS-style stuff) or propose one that could? Perhaps if I saw a real application of the semantic web I would actually understand what RDF is actually all about.

Re:But how do I use this semantic data?

[crschmidt]

Check out http://crschmidt.net/semweb/ for info on some of the projects I've worked on which use the semantic web.

The most interesting one, in my opinion, is lorebot. Lorebot sits in a channel, and associates identified users to their FOAF files. Once it does this, it links them to a human readable description of HTMl about them, and, if possible, displays an image for them. Example output: online users, personal output.

There's also things like the FOAF or DOAP a matic: both of which take RDF and spit out a machine readable description. The Firefox plugins on my semweb page let you see in the corner of your browser when you have that information available.

There's more tools out there, but they don't tend to be as down to earth, because a lot of RDF data is in high-level stuff. The demonstrations are becoming a lot more usable though, and I expect that to continue over the next year.

Re:But how do I use this semantic data?

[Anonymous Coward]

The semantic web isn't for human consuption. It's meant to be written in a markup language, such as RDF or DAML+OIL, which can produce an ontology of knowledge out of all the existing information on the www, so that "intelligent" software agents can make use of the vast resource that the web is.

Re:OT Re:Slashdot lies, opinions, and half-truths

[britneys 9th husband]

He is talking about this comment.

There is additional background information and historical perspective available at the following sites:

Sllort's journal
Kuro5hin article

Just my opinion, but...

[crazyphilman]

I think the "Semantic Web" sounds great on paper, and is the next big thing in university research departments and etc, etc, BUT I don't think it's going to end up seeing wide use. Here are my reasons, basically a list of things that I as a web developer would hesitate on.

1. The Semantic web seems to require a lot of extra complexity without much "bang for my buck". If I build a page normally, all my needs are already met. I can submit the main web page to search engines, prevent the rest from being indexed, figure out how to advertise my 'page's existence... I'm pretty much set. The extra stuff doesn't buy me anything. In fact, I definitely would NOT want people being able to find information on my site without going through my standard user interface. I WANT them to come in through the front door and ask for it.

2. Let's say people start using this tech, which I imagine would involve all sorts of extra tagging in pages, extra metadata, etc. Now you have to trust people to A) actually know what they're doing and set things up properly, which is a long shot at best, and B) not try to game the system somehow. On top of that, you have to trust the tool vendors to write bug-free code, which isn't going to happen. What I'm saying is that all these extra layers of complexity are places for bugs, screw-ups, and booby traps to hide.

3. And, the real beneficiary of these sorts of systems seems to be the tool vendors themselves. Because what this REALLY seems to be about is software vendors figuring out a new thing they can charge money for. Don't write those web pages using HTML, XML, and such! No, code them up with our special sauce, and use our special toolset to bake them into buttery goodness! Suddenly, you're not just writing HTML, you're going through a whole development process for the simplest of web pages.

Maybe I'm getting crusty in my old age, but it seems that every single year, some guy comes up with some new layer of complexity that we all "must have". It's never enough for a technology to simply work with no muss and no fuss. Nothing must ever be left alone! We must change everything every year or two! Because otherwise, what would college kids do with their excess energy, eh?

Sigh... Anyway, no matter what you try and do to prevent the Semantic Web from turning out just like meta tags, the inevitable will happen. You watch.

Re:Just my opinion, but...

[l0b0]

The Semantic web seems to require a lot of extra complexity without much "bang for my buck". If I build a page normally, all my needs are already met.

How about the needs of the people actually using the page? If you don't care about the viewers, why bother putting it on the web?

I definitely would NOT want people being able to find information on my site without going through my standard user interface. I WANT them to come in through the front door and ask for it.

That sounds just like the kind of site I get pissed off at, when being redirected to the main page after finding the page I really want via Google. Forcing visitors to jump through hoops has never been popular.

Now you have to trust people to A) actually know what they're doing and set things up properly, which is a long shot at best, and B) not try to game the system somehow.

As a web developer, you probably already know what kinds of ugly designs there are out there. And yet, by some kind of magic, there are companies which create searchable indexes of these pages, and it just works. One of the benefits of this technology I expect to see in search engines shortly, is the possibility of semantic searches. How would you go about, today, looking for a bike magazine called "Encyclopedia" (I've tried)? Or research resultat relevant to your latest blog entry? Or the cheapest direct or indirect first class return ticket from London to New Delhi departing between one hour from now and 9 a.m. Thursday, with return between three and five days later, no smoking all the way?

Re:Just my opinion, but...

[crazyphilman]

"How about the needs of the people actually using the page? If you don't care about the viewers, why bother putting it on the web?"

Anything more complex than flat HTML is actually going to require the developer to retain some control over how the user views the pages. For example, take a page that allows you to submit an application online. The only appropriate place for a user to start is the start page of the application. NATURALLY I'm going to bounce you back to the beginning.

Anytime you try to do ANYTHING complicated, you're going to have to take control of the user's experience. This is as much for his benefit as yours, and it's a fact of life for web developers.

"That sounds just like the kind of site I get pissed off at, when being redirected to the main page after finding the page I really want via Google. Forcing visitors to jump through hoops has never been popular."

Tough nuts. The customer is not always right.

As for the rest, it's all possible with what we already have. Nothing you've described is aided or solved by the "Semantic Web".

Remember my comment about "gaming the system"? What prevents a porn site from creating a semantic web setup which says it's a bicycle site? In no time at all, you're wading through the same dreck you had with regular search engines.

The solution is NOT to create a "semantic web" but to improve YOUR SEARCH SKILLS. It's not my duty as a developer to hold your little hand and absolve you of the need to develop your skills. It's my duty to create web pages that do what they're supposed to do, every time, never breaking, never crashing, with consistent, repeatable, predictable results.

If someone like you gets bent out of shape because I'm enforcing the proper working of my page, well, tough. I don't need guys like you on my site anyway, you spend too much time trying to tinker around with it and thus, are a royal pain in the ass.

I guess You read the wrong TOS

[pavan0918]

The google TOS you are talking about is for the google website. We had used the google webservice api, please read the google api TOS .
Google api was built to allow automated queries so we were not "violating" the TOS.

So I think it is wrong on you part to comment on some one without having the full information.Ofcourse it may take longer and require more work, but that seems better than using wrong information.

Finding Nemo

[Anonymous Coward]

Any news on that down there?