Scaling Web Application Recommended Readings

Designing for scale is one of the greatest challenges when building when building web applications for the Internet. The huge scale of the Internet and the amount of potentials users requires applications to be able to handle huge amounts of data and traffic.

Today’s Internet applications has to be design with large scale in mind:

• It has to be able to accommodate increased usage
• It has to be able to accommodate increased data volumes.
• It has to be maintainable

While the need seems obvious, implementing a working solution seems is not trivial, and so we see a lot of new companies that fail to handle the load (Cuil, Twitter, ….)

Joining Nuconomy‘s ranks recently has opened me to the world of web scalability. And so, I’ve had to do quite a lot of reading the past couple of weeks.

I’ve compiled a list of the best resources I came across:

• The following presentation by Cal Henderson provides a detailed overview of common patterns and approaches when building application for high availability and scale (you might also want to check out his book his book) :

SlideShare Link
(removed their player embed as it was throwing internal exceptions. So much for SlideShare’s QA…)

• The High Scalability blog features lots of interesting content on the topic.
  Be sure to check its featured overviews over the technology behind the top Web 2.0 sites:
  • Amazon
  • Google
  • YouTube
  • MySpace
  • Twitter
• Another series of posts worth a reading is Tim O’Reilly series on Web 2.0 and Databases covering the data strategy of major Web 2.0 companies:
  • Google File System and BigTable
  • Flickr
  • MySQL
  • Second Life
  • Findory and Amazon
  • Craigslist
  • Blogline and Memeorandum
  • NASA World Wind
  • O’Reilly Research
• Google’s Paxos implementation. The Paxos algorithms is a set of protocols used for developing fault tolerant distributed databases. Google’s Paxos Made Live An Engineering Perspective (1052) is a very well written and easy to read paper, written by Google’s engineers, describing their experience implementing Paxos in the creation of Chubby, a system used by BigTable for keeping replicas consistent in case of failures:

We describe our experience building a fault-tolerant data-base using the Paxos consensus algorithm.
Despite the existing literature in the field, building such a database proved to be non-trivial. We describe
selected algorithmic and engineering problems encountered, and the solutions we found for them. Our
measurements indicate that we have built a competitive system.

…

We used the Paxos algorithm (“Paxos”) as the base for a framework that implements a fault-tolerant
log. We then relied on that framework to build a fault-tolerant database. Despite the existing literature on
the subject, building a production system turned out to be a non-trivial task for a variety of reasons:
While Paxos can be described with a page of pseudo-code, our complete implementation contains several
thousand lines of C++ code. The blow-up is not due simply to the fact that we used C++ instead
of pseudo notation, nor because our code style may have been verbose. Converting the algorithm into
a practical, production-ready system involved implementing many features and optimizations some
published in the literature and some not.

Got some more interesting scalability resources to share? feel free to leave a comment…