Twitter saw 752 percent growth in 2008 and 1358 percent in 2009. And that was just in Twitter.com visitors – 75 percent of Twitter traffic comes from 3rd party APIs. How did Twitter deal with the growth? Not that well in the
first year, when people were dressing up as the Fail Whale for Halloween. But Twitter was already scaling up back then; if they hadn’t, the service wouldn’t be the force is today.
"In the beginning, we did a lot of firefighting at Twitter,” said Twitter’s John Adams, lead engineer for all outward-facing services, during his Web 2.0 Expo talk, “Billions of Hits: Scaling Twitter.”
“We were trying to plan for the future,” Adams said.
What is also vital for a startup, Twitter learned, is data. Startups must be collecting and visualizing data as soon as they launch, Adams said.
“It’s a new world for admins,” Adams said. “People used to run systems ad hoc, but with real-time, you have to use analytics, track data, and see where things are trending. If you don’t collect data early, you will fail.”
“Once you collect all the data, you want to turn it into actionable information,” Adams said.
Ultimately, the lessons learned by Twitter’s engineers during the site’s launch and growth?
“Instrument everything. Cache as much as possible. Scale early. Don’t rely on memcache. Don’t use mongrel, use Unicorn,” Adams said to the audience of developers.