Slashdot Mirror

With the huge number of postings on all the various forums, concerning this update, most people don't know where to start looking for information about the recent Google update. The following is an attempt to put down rationally (I hope) most of the information that is known and the (unproven) theories behind the update algo.

Introduction.

Starting on the 16th of November, a major shift in results was seen on Google. Veterans recognised that Google appeared to be doing a major update, not seen for many months, as reported first on WebMasterWorld who named it Florida, continuing the tradition of naming updates rather like hurricanes. In this case it was a hurricane! As was usual with many updates, there were moans and groans as people complained about their sites falling. Many people were unaffected (including us) but the symptoms of the sites being dropped were not usual. No penalties, such as PR0, seem to have been applied against pages that had fallen - yet none of the pages targeted at specific key phrases, particularly index/home pages, appeared in the top results for these search terms. Indeed some had dropped hundreds of places and, in some cases reported, off the scale. Yet these pages did appear for obscure phrases and were obviously still in the index.

It appeared to us and to several other respected names (though hotly disputed by others) that some sort of over-SEOd filter had been applied to check if overt SEO had been done for that particular phrase. It was as if Google were checking to see if external links to the site included the phrase, on-page optimisation was being done for the phrase and even if the domain included the phrase. If the density of the optimisation, both on and off the page, appeared too artificial, then a filter was tripped and down went the page - solely for that phrase.

Google had never looked favourably on abuse of their systems and many established SEOs looked upon this algo tweak as a way of Google getting rid of the abuses of links and stopping the scrambling for getting (and sometimes buying) links including your required anchor text from other high PR, but probably irrelevant to your subject, sites. It seemed to make sense.

On Friday, 21st November, Google decided to tighten the filter. All hell broke loose as tens of thousands of sites disappeared from positions they had held (in some cases) for years. We noticed some of our client sites plummeting for their major key phrase from being #1 to total invisibility. Yet this was only in highly competitive areas, not for their secondary phrases. These sites were, in most cases, not highly optimised, had not sought reciprocal links but had achieved their rankings through being on the web for 4 or 5 years. The bad news was that their company name and domain included the key phrase, sites (including directories) linking to those sites included the key phrase in their links and Google interpreted this as over-optimisation and down they plunged. In many areas all the top 20 ranking sites disappeared, including industry leaders, to be replaced by educational sites, news review sites, government sites, major shopping portals or directories. Something major had happened - but what?

The Facts!

Thousands of web pages have been suddenly demoted in the Google search results, primarily on the main commercial search terms for which they targeted their pages to be replaced by other sites who, in the main, referred to the search term obliquely. Several were the main shopping portals or business directories which gave listings for companies who may provide the services requested, many were not.

Very high-ranking authority sites seemed to be unfiltered.

The changes were starkly obvious on regional English language Googles where a regional filter was employed and there were less commercial sites with authority.

An example for Google UK is the search for the word shelving. On the

Ad blocking software provided by Norton (Symantec) is capitalism at its finest. On one hand Norton needs a way to encourage users to download their software. Providing an anti-banner tool is one way for Norton to add value to its product. On the other hand Norton can leverage their power against Ad networks. Ad networks can pay Norton a ransom to remove their tracking servers from Norton's anti-banner tool. It's a win win situation for Norton.

These anti-banner tools have been available for years however no one noticed because only smaller networks were affected. Tracking servers at x10, Performics, Mediaplex, Befree, and commission junction have been targeted for some time.

What makes Norton's anti-banner tool interesting today is Google involvement. Google is an Ad network (however most people fail to see this). Norton may have awoken a sleeping giant by targeting Google's servers. My guess is Google will pay Norton a fee. Google has too much at stake with Adsense to stand by while their ads are blocked. If Google does pay Norton then this justifies advertisings signicance. To those who think advertising does not add fuel to the internet, take a better look. In fact - just look at the top of your screen at the /. banner.

Here are more interesting threads on the subject:
webmasterworld
abestweb

(anyone else think it's ironic that the Google toolbar blocks popups?)

According to this link, AskJeeves and Look Smart are the same company...
so is this a marketing hype to keep Google stocks cheap for a hostile takeover??

On webmasterworld this very topic is discussed all the time (though mostly by search engine optimizers who apparently have nothing better to do with their time). If you can put up with the marketroids, it's actually a very useful website.

Alltheweb and Teoma seem to be Google's most credible challengers technology-wise, although Microsoft is also now developing its own search engine.

Google, seeing the risk, overhauled their search engine this summer--I wonder if anyone here has noticed the difference.

I've been asking around about this, and it's amazing how many people are just brushing it off as nothing. It is a serious issue for IP addresses that are being hit.

Here are some more posts on the topic, elsewhere. Note how some people just say "Oh, you are getting hits! Hits are good, no?".

http://www.webmasterworld.com/forum39/1435.htm

http://lists.jammed.com/incidents/2003/08/0369.htm l

http://www.derkeiler.com/Mailing-Lists/linuxsecuri ty/2003-08/0002.html

The blocking rules people suggest (see page five of the first link) don't work at my site, for some reason. Maybe it's because I only have access to .htaccess, not my own httpd.conf.

Google intentionally makes life difficult for people who try to duplicate the pagerank-checking feature of the Google Toolbar. I'm not sure why, but my guess is that it's to make it more difficult to discover Google's PageRank-calculating algorithm.

The word up - is that this isn't going to compete directly with the Google model, but something a bit closer to Inktomi and Overture.

If you're running Apache, and it looks like you are, you can avoid logging that crap (and minimize bandwidth and CPU waste) with this minor httpd.conf change. You can also block/ban email spiders (at least ones that report their agent name truthfully, which apparently is most of them) using the info at the same link.

Even more authorative
GoogleGuy saying its FUD :
"I think Andrew Orlowski is taking a comment and taking it in the direction that he wants to go. I would take that article with a grain of salt.
GoogleGuy, going for understatement. :) "

And so has GoogleGuy the google employee

They are creating a new blog index, but noone said anything about removing them from the main index. They havn't removed news from the main index.

I think Andrew Orlowski is taking a comment and taking it in the direction that he wants to go. I would take that article with a grain of salt.

GoogleGuy, going for understatement. :)

Q. Is it easier to maintain ACID principles with pure RAM?

A. Yes. This makes ACID almost trivial.

...

Q. Does writing the data to disk add some of the same problems of synchronization and ordering that led to the development of the ACID principles?

Q. If you use the disk only for backups, then things are much easier than before. It's when you store data on disk that you are still manipulating on disk that you need the ACID principles.

I'm confused. I actually haven't used MySQL much, and someone else can clarify its current ACID compliance. My application involves multiuser financial transactions. When making my DB selection a couple of years ago, at that time it was claimed that MySQL had some ACID deficiencies that made me nervous. I settled on PostgreSQL, which I'm very happy with.

But there's a lot more to ACID than just keeping RAM and disk in sync, and I don't see how RAM would make ACID that much easier, and certainly not "almost trivial". You still have all the transactional semaphores, record locking, potential deadlocks, rollbacks, etc. to worry about. In fact I don't see why you wouldn't just have the RAM pretend to be a disk and be done with it, since the disk version already has stable software. Then, if it is important to increase performance further, RAM-specific code optimization could be done over time, but slowly and carefully.

I'm sorry - I really don't want to get into a religious war here, but the interview didn't do much to bolster my confidence in MySQL for mission-critical financial stuff. Educate me.

Anyone see this new Microsoft robot crawling their websites? It's apparently legitimate, or at least acknowledged by Microsoft. Competition for Google?

Anyone see this new Microsoft robot crawling their websites? It's apparently legitimate, or at least acknowledged by Microsoft. Competition for Google?

google.com may be cool, but what good are they if they aren't even accessible? Google Down?

ok. i'll admit, i was quite astonished to see google returning HTTP 500 status codes.

From News.com and The Register, plus a big discussion at WebmasterWorld.

Yahoo is suing NCR, saying that Yahoo! is not violating NCR patents.

Webmaster World has the best forums I've seen for Google information.

The Google Weblog is also an excellent resource.

If you want to see Actual Google Inconsistencies, look at this page rank chart: Google Page Rank Discrepancy Chart yes, that really is the url, and yes, it's safe :)

Webmasterworld also has a discussion on this - link

Interesting to note that on their privacy page google does reserve the right to "[..] share information about you witha dvertisers, business partners, sponsors, and other third parties. However, we only divulge aggregate information about our users and will not share personally identifiable information with any third party without your express consent.

Ok, this is much clearer than "no statement". Now combine this with the objections expressed in this thread here, namely that google *does* place a *unique indentifier* (in form of a cookie) on every users computer.

To me this is worrying.

Over at Webmaster World there's some great discussions ongoing about how search engines like Google work. There's even a tiny bit of inside information, thanks to the regular posts in their Google forum from forum member Google Guy.

Over at Webmaster World there's some great discussions ongoing about how search engines like Google work. There's even a tiny bit of inside information, thanks to the regular posts in their Google forum from forum member Google Guy.

Yeah, there was no real information at all in the article. Go to WebmasterWorld.com if you want real info. Then you will learn a lot, like the latest news on the November 2002 update.

Google spends 3 or 4 days every month reorganizing their rankings. These changes reflect the fact that many sites change and new sites are added, in addition to alterations Google makes to their algorithm. Try reading here before you blame Google for your loss of traffic.

What a lot people dont seem to realise, is that the google toolbar is allowed (but apparently doesn't) to send back the URLs you visit, and toolbars (like alexa) and spyware do send back URLs you visit for indexing.

Furthermore, even if an engine like google didn't get the link from the toolbar, it could still get it from someones refererlogs.

If you don't want someone to read it - don't put it online.

Come on, Microsoft and bill gates have had bad terms associated with them at google for ages. Assuming that google changed this just for M$'s sake is ludicris.

If that was all they were worried about, they could simply have manually changed those searches to exclude MS (as they have done for people as small as Bernie Shiftman), that guy who spammed his resume around everywhere. Searches on his name would turn up pages bitching about him.)

Btw, you definitely deserve a +5 for plagiarizing this post verbatim. Well, except for the paragraph breaks, I guess.

Webmasterworld, the forum cited in the Wired article have a conference next week.

Looks like Google will be there .

:)

We who are editors at dmoz hold a lot of power right now. Its time for you to share in some of that power. Head over to dmoz and apply to edit your favorite category.

Can't decide where to apply?

• How about editing the category for the kind of car that you own.
• How about editing the category for your favorite video game.
• Or maybe you are interested in some Band, Actor, or Author.

Just because it's intentional, doesn't mean it's not a bug. The space-adding hack breaks code, breaks plain-text urls (exacerbated by the "Slashdot doesn't automatically turn URLs into links" bug), generally frustrates people, and can be fixed.

Slashdot's use of tables for layout is the only reason "page-widening trolls" exist. If Slashdot used a simple layout or a CSS layout, a single wide post would not cause other posts to wrap off of the screen. Slashdot's use of tables also makes IE users wait for the entire page to load before they can read the first comment, but the space-adding hack is the most visible result of using tables for layout. See also: More reasons to avoid using tables for layout rather than only using tables for tabular data.

You can read the Webmaster World article, "XHTML -- is now the time?" if you want to read a debate among professionals. There are many pros, primarily developers of small sites, that are advocating dropping NN 4 for XHTML Strict and CSS, but most developers aren't going that route.

They are developing XHTML 1.0 trans or HTML 4.01, maybe adding CSS to go foward. NN4 will be around for a while, and few people are willing to write them off simply to appease the standards gods.

In the real world, we build sites for human composition. We separate content from display with our databases and content management. HTML may be an inefficient way to get the data to the browser (XML+XSLT would be ideal, XHTML+CSS would be easier on the browser), but it works. The browser parsers are done.

Sure XHTML+CSS is easier on the browser, and that may help rendering issues. However, the reality is that old browsers will be with us for a while. Maybe in 5 years this will matter, but not until then.

Alex

You can read the Webmaster World article, "XHTML -- is now the time?" if you want to read a debate among professionals. There are many pros, primarily developers of small sites, that are advocating dropping NN 4 for XHTML Strict and CSS, but most developers aren't going that route.

They are developing XHTML 1.0 trans or HTML 4.01, maybe adding CSS to go foward. NN4 will be around for a while, and few people are willing to write them off simply to appease the standards gods.

In the real world, we build sites for human composition. We separate content from display with our databases and content management. HTML may be an inefficient way to get the data to the browser (XML+XSLT would be ideal, XHTML+CSS would be easier on the browser), but it works. The browser parsers are done.

Sure XHTML+CSS is easier on the browser, and that may help rendering issues. However, the reality is that old browsers will be with us for a while. Maybe in 5 years this will matter, but not until then.

Alex

That's what some people think anyway. Seems they've heavily de-emphasized Google in their searches, and in the past, they've used holiday weekends for similar switchovers. Time will tell...

Due to Slashdot's line length restrictions, lines of code over 50 characters long may not display correctly.

Isn't this something that can be fixed in Perl?

No. It can only be fixed by reducing the use of the <TABLE> tag and then turning off word-length restrictions.

One of the evil things about using tables for layout is that it forces you to use word-length restrictions on text content and width restrictions on image content. Tables expand when there is a single long word. Since all of the comments are in a single table, one 9000px-long word in a comment causes other paragraphs, even paragraphs in other comments, to wrap at 9000px instead of at the edge of the browser window. Without layout tables, the long word would still make a horizontal scrollbar appear, but other comments would wrap at the edge of the screen as if there had been no scrollbar-forcing comment.

For some other problems with table layouts, see my comment at webmasterworld. Note that tables are great for tabular data, but using them for layout at the same time makes them less useful for tabular data.

One other advantage of using CSS rather than tables is especially applicable to Slashdot: over a slow connection, users of older browsers such as IE 6 for Windows would be able to see the first comment without waiting for the rest of the comments to load. Mozilla can display comments one at a time despite the table-heavy layout, but last time I checked, it could only do so in Slashdot's light mode.

In Slashdot light mode, fixing the page-expanding-comment problem may be as simple as removing a single table tag. In heavy mode, it requires rewriting the layout to replace several layers of nested tables with divs and CSS. (Examples of existing light and heavy modes: light, heavy.) You can use the "block structure" web development bookmarklet to give each table a border (blue, green, or red depending on nesting level) if you want to see how the tables are nested without digging through the HTML source.

He GAVE SPECIFIC EXAMPLES of the "page-to-page" relationship being disproven, but here you are, the second tard bait coming to "set the record straight". Do a search on virtually anything, and a good portion of the results will come from aggregate sites: DO SOME TESTS, MORON. Exactly as he clearly stated: Whether the results are because of the site internally linking (inflating each sub-page), in the case of page-to-page linking, or it's site-to-site, THE RESULTS ARE THE SAME: Some random guy's Ford Transmission site becomes #1.

No details? They published the algorithm [stanford.edu] in 1999!

Gee, it's not like something could have changed in 3 years. Regardless, Google will not tell you your PageRank, but instead will obscure it as a bar graph. Why do you think that is? Many people are still unsure what the effects of domain names are in Google rankings, yet clearly they have a profound effect. PageRank is NOT A PUBLISHED ALGORITHM.

That's not even true. Google will return up to 1000 results in a search. Can this guy even count?

I will concur with that, however it depends on the search phrase. I've had several search phrases where Google seems to limit it to 750, and I presume he encountered the same thing.

There may be some confusion because the Google Toolbar, when viewing a page that hasn't been indexed, tries to "guess" what it's PageRank would be based on the site PageRank... but that's not "real".

If you want to know more about Google, the place to go is the Webmaster World Google forum.

Danny.

Internet Ninja, smart of you to point out the Google-Sets based visualization. I think that this dataset is potentially even more interesting then web-pages.

The only problem with Google-Sets, is that they were designed to be used with multiple search terms, not a single seed. Thus the data returned is often noisy, as can be seen by people questioning the links.

Wouldn't it be cool if Google's data were cleaned up enough that you could determine the exact degree of relatedness between any two concepts? That way, any area of interest could be mapped using a graphing tool like TouchGraph. And mapped in real time too, I am still getting around to generating a cronological record of how relationships between publications change over time.

My feeling is that this cleanup will happen, and visualizing the results will be the motivation for doing so. A text based list of similar items disguises the errors in the ranking because readers don't give that much importance to the order of appearance. A graph on the other hand can show hundereds of associations as opposed to 20 or so items that are usually shown in a list. Thus, a graph magnifies the errors in calculated similarity degrees, which makes it a powerful tool for improving the formulas involved in the calculations.

There is much more to be said about TouchGraph, but there will be more supporting material to say it later, so let me just finish with a couple of links.

Amazon has also released an API, and an TouchGraph powered Amazon Browser is in the works for that. Screen-shot
Work is being done to integrate the TGGoogleBrowser with the ODP and to represent hierarchical information using background colors.
More information about future devolompents is available via this thread on Webmaster World

Thanks, --Alex

Of those, only the DMOZ homepage validates. Even linux darling Google is full of nontrivial bugs on every page.

Instead of boosting the pigeonrank of any page that doesn't validate, I think it makes more sense to list a bunch of competing sites and say how well each one validates. Here are some sites that I think do it right:

Search Engine HTML Validation Results - list of major search engines and web directories, showing how many HTML errors the W3C validator finds on both the front page and the results of a search for "mp3 rippers". The table was posted on June 29. One site (dmoz.org) made its front page validate by July 3.

Financial Institutions and Mozilla Operability - list of many banks, saying how well Mozilla works with each bank. By concentrating on the practical "Can I use this site with Mozilla" rather than the ideal "Does this site validate", this site is more useful to users trying to decide which competitor to choose. It is therefore more powerful for getting the sites to fix themselves.

and for contrast:

Free Web Hosts - list of free web hosts and whether the host makes an uploaded web page stop validating. Doesn't have enough data for a table yet, so there's not much pressure on hosts to change. Uploads XHTML test pages rather than HTML 4.01 test pages, which seems like an odd choice to me.

I recommend you look at Webmasterworld there is a massive amount of knowledge there.
One of the guys from google even posts there on occasion.

The problem is when they step out of the box you gave them. If you want to load their page in a frame, it should show up and not throw a tempertantrum.

Look, I hate the intrusive, large ads, but fine. I choose to view a site our not, they are welcome to do whatever they want within the window.

However, do NOT try to disable my backbutton with screwy redirects that mess up my history (do a server-side 301 or 302 if you need to bounce me around, it's not my problem that you suck).

Do not do pop-ups, I gave you a window, use it. If you want more space, ask me to click on something. Pop unders, that's abusive. You don't get to hide ads for me, that's outrageous. Exit-pops are worse. If I hit back, go to another url, or close my browser, you're done. You have no right to harass me.

It's really a shame that MS and Netscape never really worked to make Javascript respect the user, but then, Microsoft has never shown any respect for their customers. Look at the recent Looksmart thing, the thread on webmasterworld shows what their puppet Looksmart is doing to screw over webmasters that paid $300 in good faith for a service that the two of them are rendering worthless.

Alex

The problem is when they step out of the box you gave them. If you want to load their page in a frame, it should show up and not throw a tempertantrum.

Look, I hate the intrusive, large ads, but fine. I choose to view a site our not, they are welcome to do whatever they want within the window.

However, do NOT try to disable my backbutton with screwy redirects that mess up my history (do a server-side 301 or 302 if you need to bounce me around, it's not my problem that you suck).

Do not do pop-ups, I gave you a window, use it. If you want more space, ask me to click on something. Pop unders, that's abusive. You don't get to hide ads for me, that's outrageous. Exit-pops are worse. If I hit back, go to another url, or close my browser, you're done. You have no right to harass me.

It's really a shame that MS and Netscape never really worked to make Javascript respect the user, but then, Microsoft has never shown any respect for their customers. Look at the recent Looksmart thing, the thread on webmasterworld shows what their puppet Looksmart is doing to screw over webmasters that paid $300 in good faith for a service that the two of them are rendering worthless.

Alex

Write a perl script using an automatic comment generator to post comments to all your favoirte weblogs and blogs (Not as hard to generate seemingly relavant comments as you think!)

Here's a recent post at Webmasterworld.com that accuses certain bots of spamming old-style guestbooks with ads.

Of course, I doubt it really helps the spammer that much, since I doubt Google gives much weight to guestbooks. On the other hand, guestbooks are easy targets, since I don't think most of them are actually being read by human beings.

Look, if you're interested in search engines, check out Webmaster World. You'll learn a lot more about search engines than reading Slashdot commentary about articles from the popular press.

These little games influence Google, but not on that many major search terms. It is MOSTLY useful for bubbling up bullshit results.

However, it is possible to spam Google, but not with what people are discussing here. BTW: amateur SEO is the best way to get your sites banned. Given that a google ban kills you on Yahoo as well, be careful. A site of mine that was making some pocket change (~$1500/mo) got the Google ban and when Yahoo changed their algorithm, BAM.

Do a search for "visa credit cards" without the quotes and see who you get. American Express doesn't have the words visa on their page (check the cache) but has many of their incoming links (from their SEO's domain farm no doubt) pushing it up for the phrase visa credit cards.

Note, do a search on 'credit cards' and you'll see Pay Pal in the top 10 without the words being on the page. Sufficient link-text can establish relevancy.

However, these blogger games are mostly silly and are simply inviting amateurs to get themselves in trouble.

The "dumb motherfucker" situation was obvious. With nobody optimized on that phrase, linking to a high PR page with that phrase would push it up. The rankings from google are mostly a matter of picking the most relevant results and sorting by PageRank (basically, sites with 2-3 toolbar PageRank lower may beat other sites by Google optimizing).

These are neat, and you should make your link text relevant and useful (people will like that too, no "click here" garbage).

Anyway, the Slashdot mental masturbation about Google's collapse because people can win worthless phrases is beyond rediculous.

Alex

Web site design needs a lot of different things, Information architecture & usability, HTML & XHTML, CSS & implementation bugs, search engine ideas and keyword research, Web server techniques & content management, deeziner discussion & tech discussion, good practices & sucky practices.

I could go on. My point is that you can either be a half-hearted jack-of-all-trades, or do the Web a favour and pick something, learn to understand it and collaborate with people who have complimentary skills.

Of course a Web site is no use if no one visits it. A link from the /. home page is a good start.

Calum

A known Google Tech says that, "Sometime in the next few days, I think we're going to put a promo line on our home page. It will say something like "Google does not show pop- up advertising." That just might raise the ante.

In the original message two weeks ago when Google announced the new beta reporting system over at Webmasterworld the Google tech in charge of the system said:

"Let me repeat that this is a test. I would not recommend trying to write a votebot. Remember the line from Indiana Jones--"choose wisely"? We have several interesting tricks to prevent vote stuffing, so I would choose not to try abusing it. Looks like this is also going to be a fun psychological experiment."