Google Search Scraper written in Google Scripts

G
Published in: Google Apps Script

Google search will block your IP address temporarily if you use a download utility like wget to download search pages for scraping Google. However, if you run the scraper on Google servers itself through Apps Script, the request won’t be blocked as the IP address of the scraper would be one of Google’s own IP address.

The other alternative is Import HTML, a Google Docs function that you can used to scrape Google search results inside a Google Sheet.

function scrapeGoogle() {

  var response = UrlFetchApp.fetch("http://www.google.com/search?q=labnol");

  var myRegexp = /<h3 class=\"r\">([\s\S]*?)<\/h3>/gi;

  var elems = response.getContentText().match(myRegexp);

  for(var i in elems) {

    var title = elems[i].replace(/(^\s+)|(\s+$)/g, "")
                        .replace(/<\/?[^>]+>/gi, "");
    Logger.log(title);

  }

}
Published in: Google Apps Script

Looking for something? Find here!

Meet the Author

Web Geek, Tech Columnist
A
Amit Agarwal

Amit Agarwal is a Google Developer Expert in GSuite and Google Apps Script. He holds an engineering degree in Computer Science (I.I.T.) and is the first professional blogger in India. Read more on Lifehacker and YourStory

Get in touch