Designing for scale is one of the greatest challenges when building when building web applications for the Internet. The huge scale of the Internet and the amount of potentials users requires applications to be able to handle huge amounts of data and traffic.

Today’s Internet applications has to be design with large scale in mind:

• It has to be able to accommodate increased usage
• It has to be able to accommodate increased data volumes.
• It has to be maintainable

While the need seems obvious, implementing a working solution seems is not trivial, and so we see a lot of new companies that fail to handle the load (Cuil, Twitter, ….)

Joining Nuconomy’s ranks recently has opened me to the world of web scalability. And so, I’ve had to do quite a lot of reading the past couple of weeks.

I’ve compiled a list of the best resources I came across:

• The following presentation by Cal Henderson provides a detailed overview of common patterns and approaches when building application for high availability and scale (you might also want to check out his book his book) :

SlideShare Link
(removed their player embed as it was throwing internal exceptions. So much for SlideShare’s QA…)

• The High Scalability blog features lots of interesting content on the topic.
  Be sure to check its featured overviews over the technology behind the top Web 2.0 sites:
  • Amazon
  • Google
  • YouTube
  • MySpace
  • Twitter
• Another series of posts worth a reading is Tim O’Reilly series on Web 2.0 and Databases covering the data strategy of major Web 2.0 companies:
  • Google File System and BigTable
  • Flickr
  • MySQL
  • Second Life
  • Findory and Amazon
  • Craigslist
  • Blogline and Memeorandum
  • NASA World Wind
  • O’Reilly Research
• Google’s Paxos implementation. The Paxos algorithms is a set of protocols used for developing fault tolerant distributed databases. Google’s Paxos Made Live – An Engineering Perspective (1052) is a very well written and easy to read paper, written by Google’s engineers, describing their experience implementing Paxos in the creation of Chubby, a system used by BigTable for keeping replicas consistent in case of failures:

We describe our experience building a fault-tolerant data-base using the Paxos consensus algorithm.
Despite the existing literature in the field, building such a database proved to be non-trivial. We describe
selected algorithmic and engineering problems encountered, and the solutions we found for them. Our
measurements indicate that we have built a competitive system.

…

We used the Paxos algorithm (“Paxos”) as the base for a framework that implements a fault-tolerant
log. We then relied on that framework to build a fault-tolerant database. Despite the existing literature on
the subject, building a production system turned out to be a non-trivial task for a variety of reasons:
While Paxos can be described with a page of pseudo-code, our complete implementation contains several
thousand lines of C++ code. The blow-up is not due simply to the fact that we used C++ instead
of pseudo notation, nor because our code style may have been verbose. Converting the algorithm into
a practical, production-ready system involved implementing many features and optimizations – some
published in the literature and some not.

Got some more interesting scalability resources to share?  feel free to leave a comment…

Tags: Google, Internet applications, Scalability, Web 2.0, web scalability

WPF, Microsoft’s not-so-new-anymore UI technology offers new capabilities allowing both developers and designers to work together to achieve a stunning experience for their applications.

Power, however, does not come without complexity, and WPF does not provide a framework or a model to solve many of the problems faced by developers and designer when building an application:

1. Handling Rich Data Forms. Many applications, especially when it comes to enterprise applications, rely heavily on displaying and manipulating data. Fetching the data while keeping the UI alive and responsive is a complicated task that’s also hard to debug and requires an experienced developer doing the work.
Can we come up with a framework that will simplify data fetching?

2. Testability is a Requirement for Software Development Framework. Development organizations are no longer satisfied with simple reduction of costs for initial development and there’s a growing demand for frameworks and tool to facilitate a sustainable and agile development process.
Can we come up with a model that will allow writing tests for the application’s UI and behavior?

3. Metadata Driven User-Interface. WPF provides XAML as a meta-model for UI definitions. However there is no clear separation between metadata and code which is a mess when it comes to designer and developers working together.
Can we come up with a model to allow developers provide all the UI logic as closed building blocks that designer can just use in a plug-and-play manner?

Providing a Framework for Building Robust, Data-Driven UIs

The Model\View\Controller (MVC) architectural pattern has long been used by complex applications to present large amount of data to the user.
The pattern allows developers to separate the actual data (Model) from the user interface (View) and the business logic manipulating the data (Controller).

In the following set of articles I will present a variation of the MVC pattern tailored for modern UI development (in WPF) where we’d like the View to be the responsibility of a designer rather than a classic developer writing code.

I’ll be using the DataModel\ViewModel\View terminology to describe the pattern (although you may find the same pattern described using various other terminologies when browsing the net).

Introducing the DataModel\ViewModel\View Pattern

As mentioned earlier, the DataModel\ViewModel\View pattern is a variation of the MVC pattern. Its focus is on making the View, which is the actual UI presented to the user, the responsibility of a designer - a person who is generally more oriented towards graphics, art and interaction than to classic coding.

The design of the view should be done in a declarative form (XAML) using a WYSIWYG tool (Expression Blend).
In short, the actual UI is developed using different tools and languages by a person with a different skills set than business logic and data backend.

In order to understand the meaning behind the DataModel\ViewModel\View terminology lets look at the following diagram describing
typical architecture for our application’s presentation using this pattern:

The DataModel

The DataModel is defined exactly as the Model in MVC; it is the data or business logic that stores the state and does processing of the problem domain.
The DataModel abstracts expensive operations such as data fetching without blocking the UI thread. It can keep data “alive” fetching it periodically from source (example: stock ticket), merge information from several sources etc.
The DataModel is completely UI independent and pretty much straightforward to unit test.

The View

The View consists of visual elements and represents the actual user interface presented to the users (buttons, windows, graphics, etc.). It also defines interaction for keyboard shortcuts and other input devices .

The View is defined declaratively in XAML by the designer (usually using a tool such as Expression Blend).
Using such a declarative model makes it to harder to represent some state that the original  View from the MVC pattern was meant to deal with - this includes dealing with multiple modes of interaction (such as “view mode” and “edit mode”) that change the visuals and behavior of the controls.

This is where we make use of WPF’s advanced data binding mechanism. In a simple scenario we can simply bind the View to the DataModel and use binding expressions to perform one-way binding for display only values or two-way binding to allow editing values in the DataModel.

In most scenarios, however, only a small subset of the application’s UI can be bounded directly to the DataModel. This can be the case when the DataModel is a pre-existing class or data schema over which the application developer has no control. The values exposed by the DataModel are likely to require some processing in order to allow binding to UI elements. There may also be several complex operations that require code implementation and do not fit into the strict declarative-only definition for a View but are too application specific to be part of the DataModel (which we might not have control over).
We may also want to save some view state such as view mode (view\edit\etc.) or item selection etc.

To bridge this gap between the declarative View and the DataModel we define the ViewModel…

The ViewModel

The ViewModel bridges between the DataModel and the View and performs all the tasks mentioned in the previous paragraph.
The terms is meant to describe a “Model of a View” which basically means that the ViewModel abstracts all the behavior logic behind a specific screen (View) in the application.
The ViewModel include converters that can transform DataModel types into View types, Commands that can be executed the the View’s control and interact with the DataModel and general behaviors that can be attached to UI elements in the View.

Summary and Next Steps

The DataModel\ViewModel\View defines a simple yet powerful pattern allowing developers and designers to collaborate on building a robust, data-driver WPF UIs.

It allows separating the data layer from the view layer and the UI to support easier development of granular components that are also unit-testable.

To demonstrate how the various pattern components are developed and used we’ll be going over the development process of a stock ticker widget-like application dubbed Stocky (screenshot on the right) and see how this development pattern simplifies the creation of an otherwise quite complicated little application.

References:

• Dan Crevier’s series on data binding
• MCV at Wikipedia - http://en.wikipedia.org/wiki/Model-view-controller

Comments (6) from www.ekampf.com/blog/:

Tuesday, March 18, 2008 4:09:58 PM (GMT Standard Time, UTC+00:00)

In my company We developing a very big medical system with UI based on WPF.
We used a combination of the Model-View-Presenter and the DataModel-View-ViewModel introduced by Den Crevier’s.
looking forward to see your implementation.

Ran Trifon

Tuesday, March 18, 2008 4:52:31 PM (GMT Standard Time, UTC+00:00)

Well it’s pretty much the same…
The goal here is to summarize all the information into one place. Dan’s post are pretty short and straightforwards aimed at experienced developers and these post are meant to be more detailed.
I am going to post about topics he didn’t mention though…

Eran Kampf

Tuesday, March 25, 2008 3:41:31 PM (GMT Standard Time, UTC+00:00)

Nice post - I’m really looking forward to seeing where you go with this. I’ve just recently being trying to find some guidance on setting up an MVC/MVP framework in WPF. Dan’s series is great but I must admit that I really didn’t understand it all until I began my own implementation and things began to “gel”. Will be great to see another perspective on it.

Nigel Spencer

Tuesday, March 25, 2008 3:53:55 PM (GMT Standard Time, UTC+00:00)

Thanks Nigel,
Next post in the series is already available at http://www.ekampf.com/blog/2008/03/24/DevelopingARobustDataDrivenUIUsingWPFTheDataModel.aspx

Eran Kampf

Tuesday, March 25, 2008 6:38:44 PM (GMT Standard Time, UTC+00:00)

Eran:
Just wanted to say “keep up the good work”. between your work and Dan’s series of articles, I think I’m starting to get a handle on this. My one request is that I’d like to see how your DataModel interacts with the DataAccess Layer against SQL Server. Maybe just something against Northwind. I realize this might be outside the main scope but I think it would be interesting.
Sincerely,
Dale Williams

Dale Williams

Tuesday, March 25, 2008 7:34:38 PM (GMT Standard Time, UTC+00:00)

Hey Dale,
Thanks for the feedback
The 3rd post in the series will show a concrete DataModel example.
Since I was aiming to show how I build a Yahoo finance widget clone I was building the DataModel on that - keeping a stock data up to date (kind of like in Dan’s article).
However, once you see how the DataModel fetching is implemented it doesn’t really matter if the actual data is fetched via SOAP call, http, or a DB access so you’ll be able to implement a one-way binding to a data source of your choice.
While the current implementation only deals with one-way binding (only fetching the data from the without the ability to update data on the source) I do plan to show how to implement two-way binding and support comitting and rolling back data in future posts.
Thanks,
Eran

Eran Kampf

Tags: Metadata Driven User Interface, MVC, Patterns, WPF

The Strategic Architect Forum (SAF) conference was hosted at the Microsoft Conference Center in Redmond over three days during November 2007. All the keynotes and informational sessions are available for download at the SAF recorded sessions site.

Be sure to check out the Duet: An Enterprise S+S Offering From Microsoft And SAP session”:

Duet: An Enterprise S+S Offering From Microsoft And SAP
Chris Keyser (Microsoft)
Burra Gopal (Microsoft)
Adi Kavaler (SAP)

[Video] . [Slides] . [Transcript]

Other topics include:

Tags: Duet, Mendocino, SAP

MSDN has released a new article which shows how to implement a Word-like UI object model architecture that supports automation etc. like Office does.
The object model demonstrated is based on the model-view-controller (MVC) design pattern.

All Microsoft Office applications are built on top of an object model that supports automation. Any developer can use the 0bject model to drive the application UI and add, edit, and delete content, just as a real user interacts with the application. The rich object model, together with automation support, makes Office applications truly extensible and pluggable. Anyone can write a powerful add-in within a very short time in order to extend the behavior of Microsoft Word according to their own need. As good object-oriented (OO) developers, we develop our applications with rich architecture and with a reasonably good object model following the Model-View-Controller (MVC) design pattern.

However, until recently, very few applications have been developed that offer automation similar to Microsoft Office applications. As a result, we cannot extend our applications the same way we can extend and customize Office applications using the .NET Framework and Microsoft Visual Basic for Applications (VBA). This article shows you a way to implement a Microsoft Word-like object model for your own .NET application. We will be following the Model-View-Controller design pattern and also .NET Framework events and delegates heavily. The object model we will develop here will add infinite extensibility to our application. It will give us the opportunity to add plug-in and scripting capability to our applications as designed, without writing additional code for them. The plug-in and scripting feature will have the same power as the core application. The design will also produce a very clean code base that truly decouples business logic from the UI logic. Best of all, we will be able to write code to drive the UI and thus create test scripts that not only test business logic but also test the UI behaviors.

Read Implement a Microsoft Word-like Object Model for Your .NET Framework Application on MSDN.

Here is the presentation from a UML lecture I had to give at the university.
Enjoy

UML.pdf (451.16 KB)

Tags: UML

Scott Hanselman has posted a very interesting blog entry about organizing the code for software projects.

A particular interest in the post is the use of NTFS Junction points.
It could be usefull for sharing version information (take versioning out of AssemblyInfo.cs and put it in a shared file using junctions).

My own projects are organized in a rather hectic way which currently makes the process of building and releasing much more complex than it should be.

Note to self : Organize Sharp3D’s code

Well,

I have been using TDD for Sharp3D’s development process for few days now and it already helped me find several bugs.

At first I thought having to write all the tests code will just be a waste of time I could spend on adding features to the library but after some actuall work I noticed that working on the tests isn’t time cosuming as I thought and it helps me find some bugs I wouldn’t find otherwise and deliver a better product.

Having said that, I really have to say that the testing and refactoring capabilities in VS2005 are awsom. I only wish they would add performance testing capabilities (So I can define “test methods“ that would run and be time measured).

Oh and btw, notice the new Stopwatch class used to accuratly meaure times (for peformance etc…)