Site moved to http://www.peterpappas.com/2010/12/how-to-quantify-culture-google-ngram-viewer-explore-500-billion-published-words.html, redirecting in 1 second...

« Stop Worrying About Shanghai, What PISA Test Really Tells Us About American Students | Main | Design Your Website from the Bottom Up »

December 17, 2010

How To Quantify Culture? Explore 500 Billion Published Words With Google's Books Ngram Viewer

By now you must be aware that Google has been busy digitizing books - over 5 million are now available for free download and search. Recently Google Labs has made public a giant database of of names, words and phrases found in those books (along with the years they appeared). It consists of the 500 billion words contained in scanned books published between 1500 and 2008 in English, French, Spanish, German, Chinese and Russian. 

Google Labs has just posted the "Books Ngram Viewer" - a free online research tool that allows you to quickly analyze the frequency of names, words and phrases -and when they appeared in the digitized books. You type in words and / or phrases (separated by comma), set the date range, and click "Search lots of books" - instantly you get the results. Note: when "smoothing" is set to "0" the results will show raw data. Using a higher number produces an average - example "4" will give you four year running averages that will more readily display trends. 

In this graph I searched "horse, carriage, canal, train, steamship, bicycle, car, airplane" and set the date range to 1800 - 2000.  Link to this transport graph at Books Ngram Viewer The results offer some insights into when these new transportation terms found their way into print. 

Transport-1

I think Books Ngram Viewer has many interesting applications in the classroom. The first that comes to mind, is as tool to introduce the research method - form hypothesis, gather and analyze data, revise hypothesis (as needed), draw conclusions, assess research methods. Working in teams students can easily pose research questions, run the data, revise and assess their research strategy. Students can quickly make and test predictions. They can then present and defend their conclusions to other classroom groups. 

Using the Ngram viewer, will enable students to discover many insights which will require revisions to their research strategies - a great way to explore word usage, social context and statistics. Words have multiple meanings. In my transport example "car" appears in the graph long before the advent of the automobile. Was it used as railroad car? In contrast to newspapers, events and trends take time to find their way into books. "Pearl Harbor" does not reach a peak until 1945.

The frequency of occurrence scale is important (vertical Y-axis.) If you graph a high frequency word against a low frequency word(s), the low is reduced to a flat line at the base of the scale. (Abraham Lincoln and Marilyn Monroe) Remove the high frequency (Abraham Lincoln) and re-run the graph - the low frequency (Marilyn Monroe) will appear with more detail. For more specifics on how to search, click here.

[nGram even includes a "Rickrolling" Easter Egg.
Search for "
never gonna give you up"]

Update: Hat tip to Jean-Baptiste Michel of the nGram team who emailed me "In English, the data is good in 1800-2000, but not really before or after. Past that date, it looks like the composition of the corpus is changing; trends would indicate a shift in the corpus, not a shift in the underlying culture. So really, one shouldn't look at data past 2000 in English."

There are so many new free online tools that allow students to visualize and present information in new ways. For example,  see my post "Build Literacy Skills with Wordle." Let's start to collect classroom ideas for Books Ngram Viewer. If you find some interesting uses in the classroom, please post them as comment (with links). Here's a few more graphs to get you thinking.  Need ideas for nGrams? Click here.

Analyze societal values: "ex wife, ex husband"  
 Changing laws and social values?
Watch the change in the Y-axis scale - add "my ex" to the original graph.

Ex-1

Track trends: "latte, sushi, taco"
Link to graph 
Are these new food fads?

Latte-1

TrackBack

TrackBack URL for this entry:
http://www.typepad.com/services/trackback/6a00d8341d880253ef0147e0cc94fa970b

Listed below are links to weblogs that reference How To Quantify Culture? Explore 500 Billion Published Words With Google's Books Ngram Viewer:

Comments

Feed You can follow this conversation by subscribing to the comment feed for this post.

PHENOMENAL

Verify your Comment

Previewing your Comment

This is only a preview. Your comment has not yet been posted.

Working...
Your comment could not be posted. Error type:
Your comment has been saved. Comments are moderated and will not appear until approved by the author. Post another comment

The letters and numbers you entered did not match the image. Please try again.

As a final step before posting your comment, enter the letters and numbers you see in the image below. This prevents automated programs from posting comments.

Having trouble reading this image? View an alternate.

Working...

Post a comment

Comments are moderated, and will not appear until the author has approved them.