Over the last week or so I've been following a thread of blog posts that outline some of the issues that
Twitter is experiencing. For those of unfamiliar, Twitter is a "microblogging" site written in
Ruby on Rails. Over the last month or so they've seen some very serious growth. They're supposedly seeing over 11,000 request/sec which is putting a hurting on their servers. In
this interview, Alex Payne points out some limitations in Rails, most notably it's lack of built in support for talking to multiple databases, that are making it particularly hard for them to handle the increased load. As a result, several individual have put together plugins to make working with multiple read only slave databases possible.
Along with checking out the plugins mentioned above you may want to checkout the following posts that go into more detail on the problems.
What I find particularly interesting in the series of posts that talk about the scaling problems that Twitter is experiencing is that most of their problems are arising due to the database server getting nailed. They've been able to scale out their web tier but that doesn't help their poor database server in the least. I've been thinking about writing a series (or maybe just one) post on some of the things we've been thinking about regarding scaling our application. It's not written in Rails, and it doesn't need to handle 11,000 req/sec but it does need to handle large sets of data and be "hot ass fast". We're currently positioned well to handle scaling out our services tier quite easily, but we need to do some more work to figure out how best to scale out our data tier. We've had many discussions about how scaling out our services may not help since it will only increase the pressure on the database server. Anyway, hopefully I'll get a chance to post more about the things that we're thinking about in a future post.
Jumping back to the Twitter/Rails scaling problems...I'll be interested to see how things end up with Twitter, and Rails. I have a feeling that Rails will be better off when all is said and done, but will it be because of Twitter, or in spite of it?