Googlebot, the web spider that crawls web pages for inclusion in Google’s search index, can sometimes run JavaScript code as well in addition to fetching text content from a web page.
Here’s why – my Google Analytics report for Web browsers says that 500+ visits to the site were made using Googlebot 2.1 browser which is actually the code for the user-agent for Googlebot.
These Googlebot visits can only be recorded in Google Analytics if the JavaScript code of Google Analytics on those web pages was executed by the Googlebot.
While it is possible that that some people may have changed the user-agent of their web browsers to mimic Googlebot, that is not the case here because even Statcounter recorded the Googlebot visits with more detailed information including the time it spent on the whole site for that session.
Am not sure how Googlebot determined web pages where it had to run the JavaScript code but this could be of some interest to AJAX based websites.
Find this article at: http://www.labnol.org/internet/search/googlebot-executes-javascript-on-web-pages/8040/
Tags: exclusive, feature, googlebot, javascript, Search
Reader Comments
Interesting but I think this bot in stats accounts for many times less than actual amount of scanning. It only shows 5 visits in stats in last month with actual visits daily and webmaster tools showing 250+ pages scanned every day.
I also have JS used evenly and equally on every page so no difference where it might want to run it. My guess is that’s some new test version of bot.
Written by Rarst on 03.23.09
Amit, this is very interesting. I see the same thing on my site, with all of the visits happening on 19 March. None before or since. They spent an average of 0 seconds on the site and only grabbed one page at a time. Using Urchin stats, I see that googlebot visited 2 additional times that day without triggering the javascript code.
Written by Tim Mayes on 03.23.09
Do you know what kind of Java Script code is it?
Written by Paul Sanduleac on 03.23.09
Its pretty amazing for me. I somewhere read that Googlebot does not traverse the Javascript, so its better not to write much Javascript for SEO purpose.
I am also thinking why not my website analytics doesnot show Googlebot as browsers that is used to visit my site.
Written by Suraj Shrestha on 03.23.09
I think this is an exception. The Googlebot was probably configured to run Analytics code for whatever reasons. I have never seen Googlebot run any Javascript code on any site.
Written by Kaushik on 03.23.09
Googlebot just sounds creepy.
Written by Quikboy on 03.23.09
Sounds strange. I think this is not the case. But may be I need to research more in bots.
Written by Mayur Somani on 03.24.09
I think googlebots execute javascripts in Ajax based websites. Let me check my website statistics.
Written by Royalblogger on 03.24.09
Yes Amit,it’s true. I too have experienced it. Whosamug.us, a widget that tells that how many people are online, sometimes shows a visit from Mountain View, USA. And I’m sure that is a Googlebot.
Written by Technology Blog on 03.24.09
“can sometimes” – wow, this sounds very accurate!
Btw the robot can parse out urls + there may be download accelerators with spoofed user agent strings.
Written by Xebeche on 03.27.09
I think you are very much right… I just saw analytics stats for my blog and it shows GoogleBot browser visiting my website, If it is executing JavaScript I would like to how it does. I do work in same trade and my profession include Monitoring Search Engine Activity and building spider I would like to know how GoogleBot executes JavaScript. Thanks for sharing
Written by Rajeev on 03.31.09