I’ve been dealing a lot with Amazon’s AWS platform lately. Mostly doing offline data processing using Hadoop but the latest load balancing features finally opened the door for frontend applications to take advantage of Amazon’s cloud computing platform – making it easier for developers to make application more cost efficient an scalable.

Keeping in mind that there are a lot of applications out there who can benefit from moving to the cloud (including my own) I’ve made a list of tasks\considerations to make when preparing for such a move:

Step One: Move Static Content to S3

The first and easiest step is to move all your static content – images, CSS, JavaScript files, etc. – to Amazon S3. Let Amazon worry about storage, backups and availability for you.

Things to consider:

• GZIP content. S3 does not support serving GZIPed content natively so you’ll have to upload both GZIPed and plaintext version of each file and figure out in code which one to use according to the headers sent by the user’s browser.
  The following post describes how its done: Using Amazon S3 as a CDN
• Amazon’s S3 service is “eventually consistent” which means that files uploaded to S3 may not be immediately available to read.
• Use separate sub-domains for content.

Once your content is on S3 you can also use CloudFront, Amazon’s CDN (Content Delivery Network), to serve the files and improve your application’s performance.

Step Two:  Move Web Servers and Backend Servers to EC2

Move your web server code and backend services – database, memcached, etc. – to run on Amazon EC2 instances.

Consider using Amazon’s availability zones to setup servers in different availability zones. This can help your serve customers at different parts of the worlds better, while making your infrastructure tolerant to the unlikely event of a datacenter failures at Amazon.

The Web Servers

Moving your web servers to EC2 should be fairly simple. You can setup EC2 images that are configured exactly the same way your current web servers are.

If you require a queuing service as part of your architecture, consider switching to Amazon’s SQS to make administration easier.

The Database

Moving your database to EC2 is probably the hardest part of the move to AWS. If you plan on keeping your database (as opposed to migrating to a cloud solution like SimpleDB) you should use EBS (Elastic Block Storage) so that your storage is persists independently from the life of your EC2 instance.

Backup. Figure out how to take scheduled snapshots of your EBS and store them on S3.

Consider replication and sharding. If you’re using availability zone you should consider sharding your data. For example, store data European accounts data in Europe only. You should also consider replication between the different availability zones to ensure keep your site available even when one of the datacenter is unavailable.

Related Links:

• Running MySQL on EC2 with EBS
• EC2 SQL Server backup strategies and tactics
• Create a MySQL-EBS Database Setup

Step Three: Scale. Take advantage of the cloud services

Now that your application is entirely running on Amazon’s platform it’s time to take the full advantage of the platform and make it scale.

Setup Monitoring to keep up with what’s going on on your system. Amazon provides a service called CloudWatch that allows you to monitor your machines and applications.

Based on the monitoring metrics you should start using Amazon’s auto-scaling and load balancing capabilities to be able to consume and release computing resources according to demand.

At this point you should also investigate reducing your dependency on relational databases (RDMS) as much as possible (as its the most complex, and hardest to scale, component in the system) and try to move as much functionality as possible to S3 and SimpleDB.
S3 is suitable for storing large objects while SimpleDB is ideal for small stubs of data.

Important notes:

• Amazon’s load balancer doesn’t support SSL. This can be a showstopper for some applications…
• Simple DB has a max row size limitation. If your data exceeds that limit you should consider using SimpleDB as a metadata store that references the full data stored on S3.

Related Links:

• Load Balancing, Auto Scaling and CloudWatch resources on the AWS blog.

Tags: Amazon, AWS

The Amazon Elastic Compute Cloud (Amazon EC2) allows customers build secure, fault-tolerant applications that can scale up and down with demand, at low cost. One of the core features for achieving this kind of efficiency and fault-tolerant is the ability to acquire and release computing resources in a matter of minutes according to demand.

While Amazon’s EC2 has been great so far in allowing companies to run large, computational-heavy distributed tasks on the background, it has been really lacking on allowing companies to run online services in a manner that’s efficient and reliable.

The reason for this lack of support is basically because there was no way to do load balancing on EC2 machines allowing a service to efficiently scale across multiple EC2 instances.

With the launch of Amazon CloudWatch, Auto Scaling and Amazon Elastic Load Balancing, Amazon is effectively making EC2 a viable host for online services, not just background jobs. Using these services you can not only scale a service across machines for reliability, you can also automatically add and remove computing resources as demand increases\decreases driving your operation costs down.

You can find more information at the detail pages for Amazon CloudWatch, Auto Scaling and Elastic Load Balancing and on the AWS developer weblog. Also, check out Werner Vogel’s blog post for some background on how Amazon is horizontally scaling its services.

Tags: Amazon, AMZN, AWS, EC2, fault-tolerant applications, load balancing

(Originally posted on Cloud Avenue)

Jeff Barr from Amazon Web Service reports that Amazon’s Simple Storage Service (S3) is now storing more than 29 billion, an increase of 7 billion from the previous quarter:

As one of the S3 engineers told me last week, that’s over 4 objects for every person now on Earth!

Our customers are keeping S3 pretty busy too. To give you an example of what this means in practice, the peak S3 usage for October 1st was over 70,000 storage, retrieval, and deletion requests per second.

Amazon is also lowering the prices on S3 storage, with a new four-tier pricing plan that takes effect on Nov. 1st.  Customers storing more than 500 terabytes will get a rate of 12 cents per gigabyte.

With such a huge amount of data, low prices, and abundance of success stories, it really seems like Amazon has got a revolutionary service on its hands…

Tags: Amazon, AWS, Cloud Computing, S3, Simple Storage Service

Cloud Computing is the latest, hottest new buzzword in today’s information technology world. However, and much like other buzzwords such as Web x.0, it seems to be losing whatever meaning it once had as an increasing number of companies, not wanting to miss out on the latest hype, are starting to use it for their product’s PR campaigns….

Read the complete post at Cloud Avenue.

Tags: Amazon, Cloud Avenue, Cloud Computing, Oracle

Amazon’s shares jumped by more than 9% following Citigroup analysts Mark Mahaney predicting its Kindle to become the “iPod of books”:

In a report to clients, analyst Mark Mahaney said Amazon could be on track to sell as many as 380,000 units of the Kindle this year. This would match the number of sales for the iPod digital music player in its first year on the market, leading the analyst to predict that the Kindle “is becoming the iPod of the book world.”

Mahaney predicted that the Kindle will likely top the lists of holiday “gadget gifts” this year. He warned that his projections do not account for a possible launch of an updated version of the device.

Personally, I want one! I just can’t wait for Amazon to release a Wifi version that’ll work world wide and will save me delivery costs and time when ordering my books.

I’m exactly the kind of person the Kindle appeals to – a techie who reads lots of books (mostly tech too) via Amazon.
I don’t think its a very large niche…  at least, not as large as the iPod’s target market…

I find it ridiculous comparing Kindle success to the iPod simply because of one people fact: People don’t read anymore!

Apparently, I’m not the only one holding this opinion:

Today he had a wide range of observations on the industry, including the Amazon Kindle book reader, which he said would go nowhere largely because Americans have stopped reading.

“It doesn’t matter how good or bad the product is, the fact is that people don’t read anymore,” he said. “Forty percent of the people in the U.S. read one book or less last year. The whole conception is flawed at the top because people don’t read anymore.”

By the way, the US is ranked 18 in literacy rate…  Just a fact worth mentioning…

Tags: Amazon, AMZN

Via TechCrunch:

When famous computer scientist Jim Gray went missing a few days ago, the coast guard launched a large scale search that found absolutely nothing. On Thursday, they gave up.

Then Amazon stepped in. They arranged for a satellite sweep of the area and stored the images on their S3 storage service. They then created a task on their Mechanical Turk service to allow volunteers to scan the images to look for the boat. It’s a tough task – the boat would only be about six pixels in size in an image, and there was a lot of cloud cover obscuring large parts of the area scanned. But volunteers are pouring in to help out.

If you’d like to help, go to this task on Mechanical Turk. You’ll be asked to view five satellite images and note any that should be looked at more closely.

This initiative took the web by storm. “Jim Gray” is already ranked 3rd on Technorati’s search terms.
Just an indication of how powerful the net could be with today’s technology…

I’m off to look at some photos of the water outside the San Francisco Bay…

Technorati : Amazon, Jim Gray, Web 2.0

Tags: Amazon, AMZN, Jim Gray