Slashdot Mirror
The First Open Ranking of the World Wide Web Is Available
First time accepted submitter vigna writes "The Laboratory for Web Algorithmics of the Università degli studi di Milano together with the Data and Web Science Group of the University of Mannheim have put together the first entirely open ranking of more than 100 million sites of the Web. The ranking is based on classic and easily explainable centrality measures applied to a host graph, and it is entirely open — all data and all software used is publicly available. Just in case you wonder, the number one site is YouTube, the second Wikipedia, and the third Twitter."
They are using the Common Crawl data (first released in November 2011). Pages are ranked using harmonic centrality with raw Indegree centrality, Katz's index, and PageRank provided for comparison. More information about the web graph is available in a pre-print paper that will be presented at the World Wide Web Conference in April.
It looks like google.com is 4th, and bing.com is...
um...
...ok, does anybody know where bing.com is?
Koans and fables for the software engineer
Using old data, they rank digg above reddit.
Creative Commons? Why would that be in the top ten?
How can porn account for about 75% of all web traffic if the most common web sites are not listed on this report?
"Laboratory for Web Algorithmics together with the Data and Web Science Group of the University of Mannheim"
Two different universities are involved, check the links.
It's #1 in PageRank and Katz's index, #3 in Indegree centrality, What the hell is it? I went there, and I *still* don't know what it is.
well
164. slashdot.org
4072812. beta.slashdot.org
Profit!!!
Up top you have those web sites that have their fingers in damned near everything, because they are looking at "centralization" of the website. More and more websites are using videos, and who better than YouTube to host? Need to provide a way to search your website? Google has already done it for you. Need to update your 3 billion fans what you're having for lunch? Facebook and Twitter have you covered. I can't see the list from work, but I'd wager that Facebook is up there too, with their ever-present "like" buttons. What's surprising is Wikipedia, you'll only sometimes see a link to Wikipedia, even on discussions on Slashdot, they don't go out there and wave their hands saying "everybody link to me" like other sites do.
What about other aspects that would make a website "good"? Such as ease of navigation (find what you want in 5 clicks or less)? Size/amount of useful content? Number of external sites that link to their content?
If we included that sort of data, YouTube could potentially be far up there with Wikipedia. I would think Google and Bing would be ruled out entirely since by their very design they don't hold real data.
Being in the top 200 out of 100 million really isn't bad.
Yeah well that's before the BETA (fuck the BETA btw). /. rank will be 200 000 001
After BETA
http://www.nbcnews.com/id/15196982/ns/business-us_business/t/google-buys-youtube-billion/
There was a company looking at long-term growth instead of short-term profits - and they got both!
How nice of them to rank the problems of the internet for us.
Am I missing something? Does this site require Java or Silverlight or something. The page is very stark and there's no ranking shown. What am I missing? Did it get slashdotted?
If you look at the way they developed this list, it is closer to how Google ranks their searches. The metrics are scored on how many other pages link to the sites. For example, reddit and slashdot aren't high on the list because they link to other sites but very few link back. Creative Commons is in the top ten because everyone links there. It also explains why Myspace is so darn high.
I give it a solid 8 out of 10, based on the following:
Deductions given for trolls and things like this.
Taking guns away from the 99% gives the 1% 100% of the power.
...but said heathendom is too stupid to notice. If only you were willing to hang yourself more often, how entropy would drop! Death to faggots, to heathens, to you.
--
Sheshbazzar
It is now official. The Università degli studi di Milano has confirmed: Linux is dying.
One more crippling bombshell hit the already beleaguered Linux community when UNIMI confirmed that Linux's flagship domain, kernel.org, fell to a shocking #1797 in the Common Crawl rankings. You don't need to be the Amazing Kreskin to predict Linux's future. Its domain now ranks just behind Excite.com, the now-irrelevant search engine from the 1990s, which edges it out at #1796.
The glaring gap between Linux's ranking and the rankings of those in the vibrant, enterprise-ready world is in itself embarrassing enough: Apple #8, Microsoft #17, even Oracle #248. But what seals the coffin is that Linux has fallen behind even the notoriously moribund FreeBSD operating system in these industry-leading metrics, trailing it by nearly one thousand, five hundred positions.
10 PRINT CHR$(205.5+RND(1)); : GOTO 10
Amazon.com is not in the top 10, and MySpace is still in the top 20. Even more baffling, MySpace isn't even in the top 50 of any of the other rankings, so how did they come up with this score?
Taking guns away from the 99% gives the 1% 100% of the power.
Google are common accused of giving their own sites preferential results.
However this suggests not, with google Page Rank being generally lower than the "web data commons" result for the same sites, e.g. YouTube & Google
They need to put a caption on their results stating when the data for the ranking was last crawled.
It's important to keep in mind that this measures centrality, a measure of how interconnected a site is, not popularity. There are several extremely large self-contained communities that rarely get linked by other sites For instance, 4chan threads are deleted after a certain amount of time, meaning there's very little on the site to link to, which means it has low centrality; this is reflected by its position somewhere over 3000 on the linked page, much lower than its popularity in relation to many of the sites above it. I'm sure there are other examples, too, this is just the one that jumped out at me.
My own site is in the top 22,000,000! :-) ...and the bravenet version that was supposed to be shut down 10 years ago is still in the top 50,000,000... :-(
So... Are we going to toss the bottom 10% or the top?
Doesn't this work for Google?
"File to fit, pound to insert, paint to match" - Aircraft Maintenance 101
Whatever they are doing here does not reflect anything too useful (from my perspective). Source: I have a number of sites in the top 10,000 - and nothing here makes any sense. It doesn't correlate with any real world metrics I can see. ie: Sites that receive 140,000 visitors a day, and have millions of incoming links are showing up in the 1 million area, and sites of mine with little-to-no power are showing up in the top 100,000. Weird.
that page 10171778 wont be last for long.
This article and open rankings work is great, but...
The default ranking we show you is by harmonic centrality. If you want, you can find its definition in Wikipedia. But we can explain it easily.
Suppose your site is example.com. Your score by harmonic centrality is, as a start, the number of sites with a link towards example.com. They are called sites at distance one. Say, there are 50 such sites: your score is now 50.
There will be also sites with a link towards sites that have a link towards example.com, but they are not at distance one. They are called sites at distance two. Say, there are 80 such sites: they are not as important as before—we will give them just half a point. So you get 40 more points and your score is now 90.
We can go on: there will be also sites with a link towards sites that have a link towards sites that have a link towards example.com (!), but they are not at distance one or two. They are called sites at distance three. Say, there are 100 such sites: as you can guess, we will give them just one third of a point. So you get 33.333 more points and your score is now 123.333.
My intuition:
Incoming links with degree one should be allocated 1 point. *yep*
Incoming links with degree two should be allocated half of 1 point = 0.5 points. *yep*
Incoming links with degree three should be allocated half of 0.5 points = 0.25 points. *NOPE* It actually gets allocated 0.33 points.
This means degree ten links still get 0.1 point? 10 hops away and they're still showing up significantly? That measure is broken. 10 hops away should score vanishingly small... 0.5^(10-1) = 0.001 points is much more reasonable.
The measure shouldn't be 1/n (harmonic centrality), it should be 0.5^(n-1). I would love to see that set of rankings.
I thought you guys were boycotting this week. At least bring some facts to your discussion http://www.twst.com/update/388...
Is 1563649 a prime number?
Upon further reading (http://en.wikipedia.org/wiki/Centrality), methods that use an attenuation factor like I described are called Eigenvector Centrality, which Katz and PageRank are specific implementations of.
I would love to see that set of rankings.
I guess I have! :-)