Making Twitter web service calls concurrently with Akka Futures

While I was working on improving how Sarah gets information from Twitter and other sources, I decided to take some of that code and hack together this example. It shows how to make three Twitter web service requests concurrently — and then wait for their results before moving on — with Akka Futures.

Before sharing the entire class, the cool Akka Futures stuff is in these lines of code:

 

 

// get an ActorSystem in scope for the futures 
implicit val system = ActorSystem("TwitterFutureSystem")

// get the desired info from twitter
val dailyTrendsFuture = Future { getDailyTrends(twitter) }
val usFuture = Future { getLocationTrends(twitter, woeidUnitedStates) }
val worldFuture = Future { getLocationTrends(twitter, woeidWorld) }

// wait for all of these before moving on
val dailyTrends = Await.result(dailyTrendsFuture, 5 seconds)
val usTrends = Await.result(usFuture, 5 seconds)
val worldTrends = Await.result(worldFuture, 5 seconds)

The three "get*Trends" methods happen to return strings, but they could return other object types.

Back to top

Future and Await

I think this is the right way to make three Future calls and then wait for the results, but I could be wrong. I tried to follow the example shown in the Composing Futures section of the Akka docs.

The important thing from my perspective is that all three of these calls need to work before I move on any further in my code, so I wait for each of them with the Await.result calls. Here's a quick note from the Await class Scaladoc on how the result method works:

Blocks the current Thread to wait for the given awaitable to have a result. WARNING: Blocking operation, use with caution.

@throws( classOf[Exception] )

I can confirm that the code works, and I get the desired results, which I've shown below. (It also explodes as expected when I turn off my Wi-Fi connection.)

Back to top

My main "driver" class

Given that brief intro, here's my complete main driver/test class:

import twitter4j.Twitter
import akka.dispatch.{ Await, Future }
import akka.util.duration._
import akka.actor.ActorSystem

object GetTrendsWithAkka extends TwitterBase {

  val propertiesFile = "/Users/al/Projects/Scala/ScalaTwitterScripts/src/twitter.properties"
  val woeidWorld = 1
  val woeidUnitedStates = 23424977
  val emailSubject = "Twitter Trends"

  def main(args : Array[String]) {
    populatePropertiesFromConfigFile(propertiesFile)
    val twitter = getTwitterInstance

    // get an ActorSystem in scope for the futures 
    implicit val system = ActorSystem("TwitterFutureSystem")

    // get the desired info from twitter
    val dailyTrendsFuture = Future { getDailyTrends(twitter) }
    val usFuture = Future { getLocationTrends(twitter, woeidUnitedStates) }
    val worldFuture = Future { getLocationTrends(twitter, woeidWorld) }

    // wait for all of these before moving on
    val dailyTrends = Await.result(dailyTrendsFuture, 5 seconds)
    val usTrends = Await.result(usFuture, 5 seconds)
    val worldTrends = Await.result(worldFuture, 5 seconds)

    println(buildRawStringForSarah(dailyTrends, usTrends, worldTrends))
  }
  
  def buildRawStringForSarah(dailyTrends: String, usTrends: String, worldTrends: String): String = {
    return "Daily trends:\n" +
    dailyTrends + 
    "\nU.S. trends:\n" +
    usTrends + 
    "\nWorld trends:\n" + 
    worldTrends;
  }

  def getDailyTrends(twitter: Twitter): String = {
    var sb = new StringBuilder
    val dailyTrends = twitter.getDailyTrends // ResponseList[Trends]
    val trends = dailyTrends.get(1)
    for (trend <- trends.getTrends) {
      sb.append(trend.getName + "\n")
    }
    sb.toString
  }

  def getLocationTrends(twitter: Twitter, loc: Int): String = {
    var sb = new StringBuilder
    val dailyTrends = twitter.getLocationTrends(loc)
    val trends = dailyTrends.getTrends
    for (trend <- trends) {
      sb.append(trend.getName + "\n")
    }
    sb.toString
  }

} // end object
Back to top

The TwitterBase class

As you saw in the code, that class extends my TwitterBase class. (This class may not make much sense in this example, so it may be helpful to know that I use it as the base class for a series of Twitter scripts I've created for Sarah and other purposes.)

import twitter4j.conf.ConfigurationBuilder
import twitter4j.Twitter
import twitter4j.TwitterFactory

/**
 * A base class to do the basic things for each script.
 */
class TwitterBase {

  // twitter
  var consumerKey = ""
  var consumerSecret = ""
  var accessToken = ""
  var accessTokenSecret = ""
  var twitterUsername = ""

  // email
  var emailSendTo = ""
  var emailFrom = ""
  var emailSmtpHost = ""

  def getTwitterInstance: Twitter = {
    val cb = new ConfigurationBuilder()
    cb.setDebugEnabled(true)
      .setOAuthConsumerKey(consumerKey)
      .setOAuthConsumerSecret(consumerSecret)
      .setOAuthAccessToken(accessToken)
      .setOAuthAccessTokenSecret(accessTokenSecret)
    return new TwitterFactory(cb.build()).getInstance
  }

  def populatePropertiesFromConfigFile(propertiesFilename: String) {
    val properties = Utils.loadPropertiesFile(propertiesFilename)
    consumerKey = properties.getProperty("oauth.consumerKey")
    consumerSecret = properties.getProperty("oauth.consumerSecret")
    accessToken = properties.getProperty("oauth.accessToken")
    accessTokenSecret = properties.getProperty("oauth.accessTokenSecret")
    twitterUsername = properties.getProperty("twitter_username")
    emailSendTo = properties.getProperty("email_send_to")
    emailFrom = properties.getProperty("email_from")
    emailSmtpHost = properties.getProperty("email_smtp_host")
  }

}
Back to top

The twitter.properties file

As you also saw, this code relies on a properties file. Here's my Twitter.properties file, without my secret Twitter keys. (Sorry, you have to get your own keys.)

debug=true
oauth.consumerKey=KEY1
oauth.consumerSecret=KEY2
oauth.accessToken=KEY3
oauth.accessTokenSecret=KEY4
twitter_username=YOUR_USERNAME
Back to top

Necessary libraries

You'll also need the Twitter4J and Akka libraries to make this work:

If putting all of that together is too much trouble, just leave a note in the Comments section below, and I can be a little less lazy and put a zip file out here, or create a Gist. (Reminder: You'll still need your own Twitter developer account to make this work.)

Back to top

The output

When I ran this code late on Monday, June 25, 2012, the printed output looked like this:

Daily trends:
#WhatReallyTurnsMeOn
#SolteraOtraVez
Lion King
#FavouriteMovieQuotes
Mufasa
HBO
Family Guy
Cristina
Simba
Hakuna Matata
Scar
Teresa
Project X
Magic Mike
Pânico
Motoqueiro Fantasma
Silvio Santos
Puerto Rico
Fantino
Ice Age

U.S. trends:
#SingleLadies
#RememberTheTime
Lisa Raye
#WWE
Love & Hip Hop
Kane
Stevie J.
Daniel Bryan
Let It Shine
Tiny

World trends:
#Raw
Stevie J
Single Ladies
#LHHATL
Joseline
#RememberTheTime
Mimi
Mean Girls
Lisa Raye
Love & Hip Hop

Again, I think this is the right approach, though I'm not 100% positive (I'm sure I'll find out as I experiment more with Sarah). If you're new to Scala, concurrency, and Akka Futures, I also hope it's helpful.

Back to top

Add new comment

The content of this field is kept private and will not be shown publicly.

Anonymous format

  • Allowed HTML tags: <em> <strong> <cite> <code> <ul type> <ol start type> <li> <pre>
  • Lines and paragraphs break automatically.
By submitting this form, you accept the Mollom privacy policy.