Scaling Web Application – Recommended Readings

Designing for scale is one of the greatest challenges when building when building web applications for the Internet. The huge scale of the Internet and the amount of potentials users requires applications to be able to handle huge amounts of data and traffic.

Today’s Internet applications has to be design with large scale in mind:

  • It has to be able to accommodate increased usage
  • It has to be able to accommodate increased data volumes.
  • It has to be maintainable

While the need seems obvious, implementing a working solution seems is not trivial, and so we see a lot of new companies that fail to handle the load (Cuil, Twitter, ….)

Joining Nuconomy‘s ranks recently has opened me to the world of web scalability. And so, I’ve had to do quite a lot of reading the past couple of weeks.

I’ve compiled a list of the best resources I came across:

  • The following presentation by Cal Henderson provides a detailed overview of common patterns and approaches when building application for high availability and scale (you might also want to check out his book his book) :

SlideShare Link
(removed their player embed as it was throwing internal exceptions. So much for SlideShare’s QA…)

We describe our experience building a fault-tolerant data-base using the Paxos consensus algorithm.
Despite the existing literature in the field, building such a database proved to be non-trivial. We describe
selected algorithmic and engineering problems encountered, and the solutions we found for them. Our
measurements indicate that we have built a competitive system.

We used the Paxos algorithm (“Paxos”) as the base for a framework that implements a fault-tolerant
log. We then relied on that framework to build a fault-tolerant database. Despite the existing literature on
the subject, building a production system turned out to be a non-trivial task for a variety of reasons:
While Paxos can be described with a page of pseudo-code, our complete implementation contains several
thousand lines of C++ code. The blow-up is not due simply to the fact that we used C++ instead
of pseudo notation, nor because our code style may have been verbose. Converting the algorithm into
a practical, production-ready system involved implementing many features and optimizations – some
published in the literature and some not.

Got some more interesting scalability resources to share?  feel free to leave a comment…

You may also like...

3 Responses

  1. August 13, 2008

    […] Scaling Web Applications – Recommended Readings (Eran Kampf) […]

  2. August 19, 2008

    […] Web: YUI 3.0 Preview Release 1 Scaling Web Application – Recommended Readings […]

  3. August 20, 2008

    […] Web Applications: Scaling Web Application – Recommended Readings […]