Scrape Twitter Data with YQL

S
Published in: JavaScript - jQuery - Twitter

You can use YQL and JavaScript to quickly scrape tweets from Twitter search results on the client side. The tweets are returned as JSON that can be easily parsed using regular expressions.

The advantage with this approach is that it requires no OAuth (authorization) and even old tweets can be returned when the search request is combined with the since: and until: search operators.

YQL allows 2000 requests / per hour / per IP so you are also unlikely to hit the usage limit.

function fetchTweets(q) {

  var yql  = "http://query.yahooapis.com/v1/public/yql?q=select%20*%20from%20json%20where%20url%3D%22";
  var base = "https://twitter.com/i/search/timeline?f=realtime&src=typd&include_entities=0&q=";

  // Test the URL in YQL console to make sure it works
  var url  = yql + base + encodeURIComponent(q) + "%22&format=json";

   // Make synchronous AJAX request to yql
  var tweets = jQuery.ajax({type: "GET", url: url, dataType: 'json', async: false }).responseText;

  // Parse the JSON response
  var data = JSON.parse(tweets);

  // Return the HTML search results
  return data.query.results.json.items_html;

}
Published in: JavaScript - jQuery - Twitter

Looking for something? Find here!

Meet the Author

Web Geek, Tech Columnist
A
Amit Agarwal

Amit Agarwal is a Google Developer Expert in GSuite and Google Apps Script. He holds an engineering degree in Computer Science (I.I.T.) and is the first professional blogger in India. Read more on Lifehacker and YourStory

Get in touch