Slashdot Mirror
Google Books Makes a Word Cloud of Human History
An anonymous reader writes
"From Ed Yong at the Not Exactly Rocket Science blog: 'Just as petrified fossils tell us about the evolution of life on earth, the words written in books narrate the history of humanity. The words tell a story, not just through the sentences they form, but in how often they occur. Uncovering those tales isn't easy — you'd need to convert books into a digital format so that their text can be analyzed and compared. And you'd need to do that for millions of books. Fortunately, that's exactly what Google have been doing since 2004.' Yong goes on to explain that the astounding record of human culture found in Google Books offers new research paths to social scientists, linguists, and humanities scholars. Some of the early findings (abstract), based on an analysis of 5 million books containing 500 billion words: English is still adding words at a breathtaking pace; grammar is evolving and often becoming more regular; we're forgetting our history more quickly; and celebrities are younger than they used to be. You can also play with the Google Books search tool yourself. For example, here's a neat comparison of how often the words Britannica and Wikipedia have appeared."
AFAIK, Google Books doesn't do the sort of methodical OCR clean-up that Project Gutenberg does, so a lot of Google's digitized books have a a fair number of errors. It'd be funny to see what kind of blips this might creates in our extracted cultural history!
Interesting that it is case sensitive. Searching for "britannica,wikipedia" in lowercase, produces, for today, close to zero for brittanica, and 0.00005% for wikipedia, which is not far off the result for Wikipedia (with capital).
Putting these together, the case-insensitive comparison of brittanica and wikipedia has wikipedia already well ahead of brittanica, at around 0.00010% for britannica, vs 0.00013% for wikipedia.
Sometime around 1885, the very first Anonymouse Cowarde briefly tried writing about Slashdot, but apparently died off before his comments could be modded up.
Note that in the linked Brittanica / Wikipedia chart, Britannica appears higher due to smoothing being set as it is. Set it to a lower value, which gives a less pretty, more accurate chart, and Wikipedia is much higher by the present day.
Tell me, how are you proposing to measure the words and thoughts of those who did not take the time to put them down in a form that later generations could refer to?
Because if you have a time machine, I've got some business plans that could make us both filthy rich...
In Xanadu did Kubla Khan
A stately pleasure dome decree
I wish they had gone in the article into more depth about grammar changes, rather than just word forms. For example, sentence ordering, comma usage, and some various other grammar items would be more intriguing. I found the burnt/burned the most interesting comparison because it showed an example of two competing versions of a word.
Interesting idea, but as was stated in the article, there are definite limits to what this technique can study, and many are unconvinced of its value for more than highly limited problems.
"I may disagree with what you say, but I will defend unto the death your right to say it." -- Voltaire
Oh yeah, the only thing that ever matters is when a self-selected sample of writers puts words on paper. Nothing else matters.
I don't know that anyone besides yourself actually made that claim...
What is the percentage of humans who have lived? And what percentage of those humans got book deals
If we're talking about human history here, not many published authors actually had to get book deals. Those are a fairly recent occurrence.
and successfully negotiated the minefield to get not only published, but indexed by a 15-year-old company?
Google is indexing everything they can get their hands on. It isn't like you have to pay an entrance fee or anything.
Surely this is the sum of all human knowledge! How could it be otherwise? Oh, no, my anti-intellectualism is showing! How dare I question my betters?
The fact of the matter is that the important stuff is usually what gets written down.
Genealogies, religious texts, laws, business records, etc.
And even if it's fiction, it's generally a good indicator of what people care to read about. Lots of sex and scandal and whatnot.
Regardless of your opinion on the value of what gets written down... It isn't like we have a whole lot else to go by. We can't very well go back 1,000 years and just ask somebody what they think. We have to work with the records we have - be it written text, or the remains of a city, or statues, or whatever.
"Work is the curse of the drinking classes." -Oscar Wilde
Most of the actual hits there appear to be OCR-os for the word "suck" and "such", often due to the use of medial "s" that resembles an "f". The word "such" appeared on a page which was badly speckled.
Given that the word "suck" was often used in the expression "to give suck", many of those pages are quite hilarious ("she would not suffer the strange lamb to fuck"). I didn't see any actual "fucks" in the first few pages of hits.
I know that the word was known. Shakespeare made a sly reference to it in Merry Wives of Windsor. But I suspect it wasn't often set down on paper, at least not in the kinds of books that got preserved.
History isn't what really happened, it's what got written down. Everything else is evanescent (well, except for what archaeologists can dig up and reconstruct, which isn't much and not necessarily accurate -- and it only counts if they write it down). Mind, I'd be more impressed if Google were also tracking the content of every hieroglyph and cuneiform tablet ever found.
It will ever be thus, unless someone invents a time machine (or at least a time viewer).
-- Alastair
tl;dr
What one fool can do, another can. (Ancient Simian Proverb)
Oh oh, according to this graph, we're being overrun by vampires, and the slayers are dropping like flies :(
http://ngrams.googlelabs.com/graph?content=never+gonna+give+you+up&year_start=&year_end=&corpus=5&smoothing=0
"Ahh! I see you're in that indeterminate Schrodinger state where - oh, uh
http://corpus.byu.edu/coha
Corpus of Historical American English.
-- 400 million words, 1810s-2000s.
-- Allows for many types of searches that Google Books can't:
* accurate frequency of words and phrases by decade and year
* changes in word forms (via wildcard searches)
* grammatical changes (because corpus is "tagged" for part of speech)
* changes in meaning (via collocates; "nearby words")
* show all words that are more common in one set of decades than others
* integrate synonyms and customized word lists into queries
* etc etc etc
-- Funded by the National Endowment for the Humanities (NEH), 2009-2011.
Take a look at the "Compare to Google/Archives" link off the first page.
Which means, incidentally, that the trailing off of "fuck" at the beginning of the 19th century IS very interesting, for a different reason. It's watching the tail end of the use of the medial "s".
That's the kind of data that would have been really hard to gather any other way, unless the OCR were to distinguish between medial "s" and regular "s" in its results. There IS a Unicode for medial S, but most OCR doesn't go there.
So, we have a proxy for it: "suck" scanned as "fuck", which wouldn't otherwise appear very often. I should write a paper on it. Wouldn't "Use of 'Fuck' as a proxy for medial S" look great on my CV?