Steve Eichert - performance http://iqueryable.com/ en-us Steve Eichert Mon, 16 Apr 2007 01:51:37 GMT newtelligence dasBlog 1.9.6264.0 steve.eichert@gmail.com steve.eichert@gmail.com http://iqueryable.com/Trackback.aspx?guid=3d716d5d-a594-4b4f-ae54-5f47928802c9 http://iqueryable.com/pingback.aspx http://iqueryable.com/PermaLink,guid,3d716d5d-a594-4b4f-ae54-5f47928802c9.aspx Steve Eichert Over the last week or so I've been following a thread of blog posts that outline some of the issues that Twitter is experiencing.  For those of unfamiliar, Twitter is a "microblogging" site written in Ruby on Rails.  Over the last month or so they've seen some very serious growth.  They're supposedly seeing over 11,000 request/sec which is putting a hurting on their servers.  In this interview, Alex Payne points out some limitations in Rails, most notably it's lack of built in support for talking to multiple databases, that are making it particularly hard for them to handle the increased load.  As a result, several individual have put together plugins to make working with multiple read only slave databases possible.

Along with checking out the plugins mentioned above you may want to checkout the following posts that go into more detail on the problems.
What I find particularly interesting in the series of posts that talk about the scaling problems that Twitter is experiencing is that most of their problems are arising due to the database server getting nailed.  They've been able to scale out their web tier but that doesn't help their poor database server in the least.  I've been thinking about writing a series (or maybe just one) post on some of the things we've been thinking about regarding scaling our application.  It's not written in Rails, and it doesn't need to handle 11,000 req/sec but it does need to handle large sets of data and be "hot ass fast". We're currently positioned well to handle scaling out our services tier quite easily, but we need to do some more work to figure out how best to scale out our data tier.  We've had many discussions about how scaling out our services may not help since it will only increase the pressure on the database server.  Anyway, hopefully I'll get a chance to post more about the things that we're thinking about in a future post.

Jumping back to the Twitter/Rails scaling problems...I'll be interested to see how things end up with Twitter, and Rails.  I have a feeling that Rails will be better off when all is said and done, but will it be because of Twitter, or in spite of it?
Scaling Twitter http://iqueryable.com/PermaLink,guid,3d716d5d-a594-4b4f-ae54-5f47928802c9.aspx http://iqueryable.com/2007/04/16/ScalingTwitter.aspx Mon, 16 Apr 2007 01:51:37 GMT Over the last week or so I've been following a thread of blog posts that outline some of the issues that <a href="http://twitter.com/">Twitter</a> is experiencing.&nbsp; For those of unfamiliar, Twitter is a "microblogging" site written in <a href="http://www.rubyonrails.com/">Ruby on Rails</a>.&nbsp; Over the last month or so they've seen some very serious growth.&nbsp; They're supposedly seeing over 11,000 request/sec which is putting a hurting on their servers.&nbsp; In <a href="http://www.radicalbehavior.com/5-question-interview-with-twitter-developer-alex-payne/">this interview</a>, Alex Payne points out some limitations in Rails, most notably it's lack of built in support for talking to multiple databases, that are making it particularly hard for them to handle the increased load.&nbsp; As a result, several individual have put together plugins to make working with multiple read only slave databases possible.<br> <ul> <li> <a href="http://drnicwilliams.com/2007/04/12/magic-multi-connections-a-facility-in-rails-to-talk-to-more-than-one-database-at-a-time/" rel="bookmark">Magic Multi-Connections: A “facility in Rails to talk to more than one database at a time”</a> </li> <li> <a href="http://revolutiononrails.blogspot.com/2007/04/plugin-release-actsasreadonlyable.html">ActsAsReadonlyable</a> </li> </ul> <br> Along with checking out the plugins mentioned above you may want to checkout the following posts that go into more detail on the problems.<br> <ul> <li> <a href="http://tomayko.com/weblog/2007/04/13/rails-multiple-connections">Twitter trouble</a> </li> <li> <a href="http://tomayko.com/weblog/2007/04/13/rails-multiple-connections">Twitter, Ruby, and Scaling</a> </li> <li> <a href="http://tomayko.com/weblog/2007/04/13/rails-multiple-connections">Rails and Scaling with Multiple Databases</a> </li> </ul> What I find particularly interesting in the series of posts that talk about the scaling problems that Twitter is experiencing is that most of their problems are arising due to the database server getting nailed.&nbsp; They've been able to scale out their web tier but that doesn't help their poor database server in the least.&nbsp; I've been thinking about writing a series (or maybe just one) post on some of the things we've been thinking about regarding scaling our application.&nbsp; It's not written in Rails, and it doesn't need to handle 11,000 req/sec but it does need to handle large sets of data and be "hot ass fast". We're currently positioned well to handle scaling out our services tier quite easily, but we need to do some more work to figure out how best to scale out our data tier.&nbsp; We've had many discussions about how scaling out our services may not help since it will only increase the pressure on the database server.&nbsp; Anyway, hopefully I'll get a chance to post more about the things that we're thinking about in a future post.<br> <br> Jumping back to the Twitter/Rails scaling problems...I'll be interested to see how things end up with Twitter, and Rails.&nbsp; I have a feeling that Rails will be better off when all is said and done, but will it be because of Twitter, or in spite of it?<br> <img width="0" height="0" src="http://iqueryable.com/aggbug.ashx?id=3d716d5d-a594-4b4f-ae54-5f47928802c9" /> performance;rails;ruby;twitter http://iqueryable.com/Trackback.aspx?guid=8b70eb2e-575a-4cec-a227-776e9b67c273 http://iqueryable.com/pingback.aspx http://iqueryable.com/PermaLink,guid,8b70eb2e-575a-4cec-a227-776e9b67c273.aspx Steve Eichert Recently we've been discussing ways to increase the performance of some performance critical areas of our application.  As such, a number of individuals have been doing research into various options.  Through our investigation we came upon Oracle's Timesten In-Memory database.  I was familiar with many other In-Memory databases, such as Firebird (SQLite?, VistaDB?), but up until yesterday I wasn't aware that Oracle had a product in the in-memory space.  While Oracle doesn't currently offer a .NET native provider for Timesten, TankardSoft does offer a commercial native .NET data provider for Timesten that supports the recently released version 7 of Timesten. 

Oracle Timesten In-Memory Database http://iqueryable.com/PermaLink,guid,8b70eb2e-575a-4cec-a227-776e9b67c273.aspx http://iqueryable.com/2007/04/10/OracleTimestenInMemoryDatabase.aspx Tue, 10 Apr 2007 11:09:19 GMT Recently we've been discussing ways to increase the performance of some performance critical areas of our application.&nbsp; As such, a number of individuals have been doing research into various options.&nbsp; Through our investigation we came upon <a href="http://www.oracle.com/technology/products/timesten/index.html">Oracle's Timesten In-Memory database</a>.&nbsp; I was familiar with many other In-Memory databases, such as <strike><a href="http://www.firebirdsql.org/">Firebird</a></strike> (<a href="http://sqlite.org/">SQLite</a>?, <a href="http://www.vistadb.com/">VistaDB</a>?), but up until yesterday I wasn't aware that Oracle had a product in the in-memory space.&nbsp; While Oracle doesn't currently offer a .NET native provider for Timesten, TankardSoft does offer a commercial <a href="http://www.tankardsoft.com/product.aspx">native .NET data provider for Timesten</a> that supports the recently released version 7 of Timesten.&nbsp; <br> <p> </p> <img width="0" height="0" src="http://iqueryable.com/aggbug.ashx?id=8b70eb2e-575a-4cec-a227-776e9b67c273" /> data;oracle;performance