Improving Twitter search with real-time human computation

One of the magical things about Twitter is that it opens a window to the world in real-time. An event happens, and seconds later, people share it across the planet.

So an event happens, people instantly come to Twitter to search for the event, and we need to teach our systems what these queries mean as quickly as we can — because in just a few hours, the search spike will be gone.

How do we do this? We’ve built a real-time human computation engine to help us identify search queries as soon as they're trending, send these queries to real humans to be judged, and then incorporate the human annotations into our back-end models.