Googlebot and Document.Write

With JavaScript/AJAX being used to place dynamic content in pages, I was wondering how Google indexed web page content that was placed in a page using the JavaScript "document.write" method. I created a page with six unique words in it. Two were in the plain HTML; two were in a script within the page document; and two were in a script that was externally sourced from a different server. The page appeared in the Google index late last night and I just wrote up the results.

Comments

Nonsense words?

[Whiney Mac Fanboy]

An alert came in in the late evening of March 10th for "zonkdogfology", one of the words in the first pair

zonkdogfology is a real word:

zonk-dog-fol-o-gy zohnk-dog--ful-uh-jee
noun, plural -gies.

1. the name given to articles from zonk where the summary makes no sense whatsoever.

Serious question now - is the author of the article worried that the ensuing slashdot discussion will mention all his other nonsense words? I've no doubt slashdotters will find & mention the other words here, polluting google's index....

Re:Nonsense words?

[Anonymous Coward]

zonkdogfology is a real word:

It's a perfectly cromulent word, and it's use embiggens all of us.

The Results:

[XanC]

Save a click: No, Google does not "see" text inserted by Javascript.

Re:The Results:

[temojen]

And rightly so. You should be hiding & un-hiding or inserting elements using the DOM, never using document.write (which F's up your DOM tree).

Google Pigeon technolog

[sdugoten2]

The Google Pigeon is smart enough to read through Document.write. Duh!

If they weren't, then they're trying

[AnonymousCactus]

Google needs to consider script if they want high-quality results. Besides the obvious fact that they'll miss content supplied by dynamic page elements, they could also sacrifice page quality. Page-rank and the like will get them very far, but an easy way to spam the search engines would be to have pages on a whole host of topics that immediately get rewritten as ads for Viagra as soon as they're downloaded by a Javascript-aware browser. It's interesting to know the extent to which they correct for this.

Of course, there are much more subtle ways of changing content once it's been put out there. One might imagine a script that waits 10 seconds and then removes all relevant content and displays Viagra instead. Who knew web search would be restricted by the halting problem? I wonder how far Google goes...

Re:If they weren't, then they're trying

[gregmac]

You have to also remember though, that often the content generated dynmically is going to be of no use to a search engine, it will often be user-specific - there's obviously some reason it's being generated that way.

And if pages are designed using AJAX and dynamic rendering just for the sake of using AJAX and dynamic rendering.. well, they deserve what they get :)

Google request external JavaScript file?

[JAB Creations]

Check your access log to see if Google actually requested the external JavaScript file. If it didn't there would be no reason to assume Google is interested in non-(X)HTML based content.

Doesn't work; Good (kind of)

[The Amazing Fish Boy]

FTFA:

Why was I interested? Well, with all the "Web 2.0 technologies that rely on JavaScript (in the form of AJAX) to populate a page with content, it's important to know how it's treated to determine if the content is searchable.

Good. I am glad it doesn't work. Google's crawler should never support Javascript.

The model for websites is supposed to work something like this:

• (X)HTML holds the content
• CSS styles that content
• Javascript enhances that content (e.g. provides auto-fill for a textbox)

In other words, your web page should work for any browser that supports HTML. It should work regardless of whether CSS and/or Javascript is enabled.

So why would Google's crawler look at the Javascript? Javascript is supposed to enhance content, not add it.

Now, that's not saying many people don't (incorrectly) use Javascript to add content to their pages. But maybe when they find out search engines aren't indexing them, they'll change their practices.

The only problem I can see is with scam sites, where they might put content in the HTML, then remove/add to it with Javascript so the crawler sees something different than the end-user does. I think they already do this with CSS, either by hiding sections or by making the text the same color as the background. Does anyone know how Google deals with CSS that does this?

Re:Doesn't work; Good (kind of)

[VGPowerlord]

In actuality, it says "Use a text browser such as Lynx to examine your site, because most search engine spiders see your site much as Lynx would. If fancy features such as JavaScript, cookies, session IDs, frames, DHTML, or Flash keep you from seeing all of your site in a text browser, then search engine spiders may have trouble crawling your site." – Webmaster Guidelines, Technical Guidelines section, bullet point 1.

Re:How does document.write mess up your DOM tree?

[XanC]

If you're using document.write, you're writing directly into the document stream, which only works in text/html, not an XHTML MIME type, because there's no way to guarantee the document will continue to be valid.

In this day and age, document.write should never be used, in favor of the more verbose but more future-proof document.createElement and document.createTextNode notation.

google.com/?q=slashdotting+in+google+dollars

[kale77in]

I think the actual experiment here is:

• Create a 6-odd-paragraph page saying what everybody already knows.
• Slashdot it, by suggesting something newsworthy is there.
• Pack the page with Google ads.
• Profit.

I look forward to the follow-up piece which details the financial results.

Re:google.com/?q=slashdotting+in+google+dollars

[Scarblac]

Exactly, this is the typical sort of fluff that Digg seems to love. As far as I know, Slashdot had avoided this particular type of adword blog post crap until now.

Re:How does document.write mess up your DOM tree?

[jesser]

Perhaps more importantly, document.write can't be used to modify a page that has already loaded, limiting its usefulness for AJAX-style features.

I would make normal links, then use JS on top

[The Amazing Fish Boy]

So, what do you have to say about websites that have their entire user-interfaces built with content that gets filled by javascript asynchronously from a single html page?

If I understand you, you something like this: The site has two parts, a menu and content. When you click a menu item, rather than being taken to a new URL, it executes Javascript which fetches only the new content from the web server, then replaces the content section. So the URL doesn't change.

It's a nice improvement. Less bandwidth used, and a quicker interface.

Unfortunately, it's not often done right. The way I would do it is to first make the menu work like it normally would. Make each menu item a link to a new page. Then you apply Javascript to the menu item. Something like this:

// menuLink is the DOM element for each menu link.
// (i.e. get it from document.getElementById(), etc.)
menuLink.onclick = function() { getNewContent(); return false; }

(FYI, this is how I do pop-up windows, too.)

Putting it behind a login screen doesn't solve all the problems. You're right that it won't be searchable anyway, but people with older browsers or screen readers won't be able to access it.

I think Gmail actually offers two versions. One for older browser that uses no (or little?) Javascript, and the other which almost everyone else (including me) uses and loves. But I'm not sure how easy it would be to maintain two versions of the same code like that. I also don't think it's nice for the end user to have to choose "I want the simple version", though it may encourage them to update to a newer browser, I guess.

(Of course this is all "ideally speaking", I realize there are deadlines to meet and I violate some of my own guidelines sometimes. I still think they're good practices, though.)

If you want to see

[BrynM]

If you want to see through a search engine's eyes, open the page in Lynx. The funniest part about showing that method to another developer is when they think Lynx is broken because the page is empty. "It didn't load. How do I refresh the page? This browser sucks." Heh. Endless fun.

(method does not account for image crawlers)

Re:How does document.write mess up your DOM tree?

[hackstraw]

One of the most clever uses of document.write I've seen was something like: document.write("<--") YOU NEED JAVSCRIPT FOR THIS PAGE document.write("--&gt")