New Features for Amazon EC2 – Now You Can Truly Scale Applications

Technology May 18th, 2009

a-m-lbThe Amazon Elastic Compute Cloud (Amazon EC2) allows customers build secure, fault-tolerant applications that can scale up and down with demand, at low cost. One of the core features for achieving this kind of efficiency and fault-tolerant is the ability to acquire and release computing resources in a matter of minutes according to demand.

While Amazon’s EC2 has been great so far in allowing companies to run large, computational-heavy distributed tasks on the background, it has been really lacking on allowing companies to run online services in a manner that’s efficient and reliable.

The reason for this lack of support is basically because there was no way to do load balancing on EC2 machines allowing a service to efficiently scale across multiple EC2 instances.

With the launch of Amazon CloudWatch, Auto Scaling and Amazon Elastic Load Balancing, Amazon is effectively making EC2 a viable host for online services, not just background jobs. Using these services you can not only scale a service across machines for reliability, you can also automatically add and remove computing resources as demand increases\decreases driving your operation costs down.

You can find more information at the detail pages for Amazon CloudWatch, Auto Scaling and Elastic Load Balancing and on the AWS developer weblog. Also, check out Werner Vogel’s blog post for some background on how Amazon is horizontally scaling its services.

Tags: , , , ,

Introduction to MapReduce for .NET Developers

.NET, Programming May 6th, 2009

The basic model for MapReduce derives from the map and reduce concept in functional languages like Lisp.
In Lisp, a map takes as input a function and a sequence of values and applies the function to each value in the sequence.
A reduce takes as input a sequence of elements and combines all the elements using a binary operation (for example, it can use “+” to sum all the elements in the sequence).

MapReduce, inspired by these concepts, was developed as a method for writing processing algorithms for large amounts of raw data. The amount of data is so large that it can’t be stored on a single machine and must be distributed across many machines in order to be processed in a reasonable time.
In systems with such data distribution, the traditional central processing algorithms are useless as just getting the data to the centralized CPU running the algorithm implies huge network costs and months (!) spent on transferring data from the distributed machines.
Therefore, processing such massive scales of distributed data implies the need for parallel computing allowing us to run the required computation “close” to where the data is located.
MapReduce is an abstraction that allows engineers to write such processing algorithms in a way that is easy to parallelize while hiding the complexities of parallelization, data distribution, fault tolerance etc.

This value proposition for MapReduce is outlined in a Google research paper on the topic:

MapReduce is a programming model and an associated implementation for processing and generating large data sets. Users specify a map function that processes a key/value pair to generate a set of intermediate key/value pairs, and a reduce function that merges all intermediate values associated with the same intermediate key. Many real world tasks are expressible in this model, as shown in the paper.

Programs written in this functional style are automatically parallelized and executed on a large cluster of commodity machines. The run-time system takes care of the details of partitioning the input data, scheduling the program’s execution across a set of machines, handling machine failures, and managing the required inter-machine communication. This allows programmers without any experience with parallel and distributed systems to easily utilize the resources of a large distributed system.

Our implementation of MapReduce runs on a large cluster of commodity machines and is highly scalable: a typical MapReduce computation processes many terabytes of data on thousands of machines. Programmers find the system easy to use: hundreds of MapReduce programs have been implemented and upwards of one thousand MapReduce jobs are executed on Google’s clusters every day.

The MapReduce Programming Model

As explained earlier, the purpose of MapReduce is to abstract parallel algorithms into a map and reduce functions that can then be executed on a large  scale distributed system.
In order to understand this concept better lets look at a concrete map reduce example – consider the problem of counting the number of occurrences of each word in a large collection of documents:

map(String key, String value):
// key: document name
// value: document contents
for each word w in value:
  EmitIntermediate(w, "1"); 

reduce(String key, Iterator values):
// key: a word
// values: a list of counts
int result = 0;
for each v in values:
  result += ParseInt(v);
Emit(AsString(result));

The map function goes over the document text and emits each word with an associated value of “1”.

The reduce functions sums together all the values for each word producing the number of occurrences for that word as a result.

First we go through the mapping phase where we go over the input data and create intermediate values as follows:

  • Records from the data source (lines out of files, rows of a database, etc.) are fed into the map function as <key,value> pairs.For example: <filename, file content>
  • The map function produces one or more intermediate values along with an output key from the input

After the mapping phase is over, we go through the reduce phase to process the intermediate values:

  • After the map phase is over, all the intermediate values for a given output key are combined together into a list and fed to the reduce function.
  • The reduce function combines those intermediate values into one or more final values for that same output key

Notice that both the map and the reduce functions run on independent set of input data. Each run of the map function process its own data source and each run of the reduce function processes the values of a different intermediate key.

Therefore both phases can be parallelized with the only bottleneck being the fact that the map phase has to finish for the reduce phase to start.

The underlying system running these method is in takes care of:

  • Initialize a set of workers that can run tasks – map or reduce functions.
  • Take the input data (in our case, lots of document filenames) and send them to the workers to map
  • Streamline values emitted by map function to the worker (or workers) doing the reduce. Note that we don’t have to wait for a certain map run to finish going over the entire file in order to start sending its emitted values to the reducer, so that the system can prepare the data for the reducer while the map function is running
    (In Hadoop – send the map values to the reducer node and andle grouping by key).
  • Handle errors – support a reliable, fault tolerant process as workers may fail, network can crush preventing workers from communicating results, etc.
  • Provides status and monitoring tools.

A Naive Implementation in C#

Lets see how we can build naive MapReduce implementation in C#.

First, we define a generic class to manage our Map-Reduce process:

public class NaiveMapReduceProgram<K1, V1, K2, V2, V3>

The generic types are used the following way:

  • (K1, V1) – key-value types for the input data
  • (K2, V2) – key value types for the intermediate results (results of our Map function)
  • V3 – The type of the result for the entire Map-Reduce process

Next, we’ll define the delegates of our Map and Reduce functions:

public delegate IEnumerable<KeyValuePair<K2, V2>>   MapFunction(K1 key, V1 value);
public delegate IEnumerable<V3>                     ReduceFunction(K2 key, IEnumerable<V2> values);
private MapFunction _map;
private ReduceFunction _reduce;
public NaiveMapReduceProgram(MapFunction mapFunction, ReduceFunction reduceFunction)
{
    _map = mapFunction;
    _reduce = reduceFunction;
}

(Yes, I realize I could use .NET’s Func<T1,T2,TResult> instead but that would just result in horribly long ugly code…)

Now for the actual program execution. The execution flow is as follows: We take the input values, pass them through the map function to get intermediate values, we group those values by key and pass them to the reduce function to get result values.

So first, lets look at the mapping step:

private IEnumerable<KeyValuePair<K2, V2>> Map(IEnumerable<KeyValuePair<K1, V1>> input)
{
    var q = from pair in input
            from mapped in _map(pair.Key, pair.Value)
            select mapped;

    return q;
}

Now after we got the mapped intermediate values we want to reduce them. The Reduce function expects a key and all its mapped values as input so to do that efficiently we want to group the intermediate values by key first and then call the Reduce function for each key.

The output of this process is a V3 value for each of the intermediate K2 keys:

private IEnumerable<KeyValuePair<K2, V3>> Reduce(IEnumerable<KeyValuePair<K2, V2>> intermediateValues)
{
    // First, group intermediate values by key
    var groups = from pair in intermediateValues
                 group pair.Value by pair.Key into g
                 select g;

    // Reduce on each group
    var reduced = from g in groups
                  let k2 = g.Key
                  from reducedValue in _reduce(k2, g)
                  select new KeyValuePair<K2, V3>(k2, reducedValue);

    return reduced;
}

Now that we have the steps code the execution itself is simply defined as Reduce(Map(input)) :

public IEnumerable<KeyValuePair<K2, V3>> Execute(IEnumerable<KeyValuePair<K1, V1>> input)
{
    return Reduce(Map(input));
}

The full source code and tests can be downloaded from here:

Map-Reduce Word Counting Sample – Revisited

Lets go back to the word-counting pseudo code and write it in C#.

The following Map function gets a key and a text value and emits a <word, 1> key-pair for each word in the text:

public IList<KeyValuePair<string, int>> MapFromMem(string key, string value)
{
    List<KeyValuePair<string, int>> result = new List<KeyValuePair<string, int>>();
    foreach (var word in value.Split(' '))
    {
        result.Add(new KeyValuePair<string, int>(word, 1));
    }
    return result;
}

Having calculated a <word, 1> key-pair for each input source, we can group the results by the word and then our Reduce function can sum the values (which are 1 in this case) for each word:

public IEnumerable<int> Reduce(string key, IEnumerable<int> values)
{
    int sum = 0;
    foreach (int value in values)
    {
        sum += value;
    }

    return new int[1] { sum };
}

Our program code looks like this:

MapReduceProgram<string, string, string, int, int> master = new MapReduceProgram<string, string, string, int, int>(MapFromMem, Reduce);
var result = master.Execute(inputData).ToDictionary(key => key.Key, v => v.Value);

The result dictionary contains a <word, number-of-occurrences> pairs.

Other Examples

Distributed LINQ Queries. One of POCs I’m working on using the above naive, LINQ-based implementation, is running a distributed LINQ query. Imagine you have a system where raw data is distributed across several SQL Servers. We can have our map function run a LINQ-to-SQL query on multiple DataContexts in parallel (the value input for the map function – V1 – can be a DataContext) and then reduce it to a single result set. This is probably a naive\simplified implementation of what the guys at Microsoft’s Dryad team are doing.

Count URL Visits. Consider you have several web servers and you want to produce the amount of visits for each page on your site. You can produce pretty much the same way the word-counting example works. The map function parses a log file and produce a <URL, 1> intermediate value. The reduce function then sums the values for each URL and emits <URL, number of visits>

Distributed Grep. You can run a grep search on a large amount of files by having the map function emits a line if it matches a given pattern. The reduce function in this case is just an identity function that copies the supplied intermediate data to the output.

Map-Reduce in the Real World

The real complexity and sophistication in MapReduce is in the underlying system takes care of running and managing the execution of MapReduce jobs. Real world MapReduce implementations, like Google’s system, Hadoop or Dryad have to go beyond the naive implementation shown here and take care of things like resource monitoring, reliability and fault tolerance (for example, handle cases where nodes running map\reduce jobs crush, or go offline due to network problems).

The following resources are worth checking out:

Tags: , , , , ,

A Visit to Maraboo, Restaurants as Tribes in the Digital Age

Uncategorized April 30th, 2009

3486004640_0982dd6589Its a rare occasion that I go out of a restaurant (and I go to many) with an absolute feeling of “WOW!”… This Monday was one of these occasions.

On the past couple of week we’ve been working on a new, food related venture (more details on that in the very near future).  Apparently, we raised enough twitter noise to get noticed, and thanks to Maya we’ve been contacted by Channel 10 and invited to visit Maraboo as part of the promotions for the season finale of the Israeli version of Icon Chef featuring the Maraboo team vs. the Hudson Brasseri – you should tune in on Channel10 this Thursday (April 30, 2009) at 21:00.

At the restaurant we met the owners, Haim Amshalom and Chef Yoav Bar
who showed us around the place and introduced us to each and every dish we got. We had a great time…

image_279B6FB5 IMG_3603_653831A4
* On the left Haim Amshalom and on the right Chef Yoav Bar

The fact that were at Maraboo as bloggers, that Maraboo has its own group on Facebook and that its owner is on Twitter made me think about the opportunities the web has to offer restaurants.

The Web is About Openness… Can Restaurants be Open?

The Web is about openness, its about empowering online communities and individuals to create, remix, share and make things themselves.
Restaurants are essentially small communities, or tribes of people with a common interest (in our case – food) a leader (the Chef) and a platform to communicate (the restaurant). They can use the web as a platform to develop a personal relationship with its customers and the facility itself (the restaurant) as a social hub.

Restaurants, like any other business, can use Twitter or a Facebook group etc. to keep in touch with their diners but what’s more interesting is that they can actually use it to become a platform, or a hub, for its tribe members (diners) to interact and their common interests (food).

The Menu as a Wiki. Ok, the Chef builds the menu and we wouldn’t want to change that… that’s the whole point of the restaurant. But what if he could collaborate or include the diners into that process?

Maybe people think the Calamari dish is lacking? or maybe they just hate Calamari…  wouldn’t you want to know?

It can put recipes online and invite diners to make suggestions. Some users might go through the trouble of preparing the dish at home. They might make all sort of adaptations and report back, discuss them online with their fellow diners (For example, a lot of people tried to make the same wings sauce as Dixie’s famous sauce and they were discussing it online). Maybe some of these adaptations are actually worth putting in the menu?

In fact, what if browsing a restaurant’s menu was more like browsing Amazon?
You could see which dishes people like more, what they think of it. You could see what your friends like (or not) or filter dishes by “spec” (like calories intake?).

Restaurants can become the stars. The popularity of kitchen-based reality shows is soaring (heck, we’re here as a promotion for such a show) because there’s a public interests in restaurants, in chefs, in food making.
Chefs could blog about their taste, inspirations, trends they see, cooking practices and tutorials and become local stars (take Gary Vaynerchuk’s for example)

Restaurant as a community meeting place. People in online communities like to meet. Just see all the Tweetup events where communities of people who know each other from talking on Twitter gather to talk offline. As its community members all share a passion for food, a restaurant can organize bakeups- where people could learn about cooking\cook. Maybe even have a community member as Chef for a night…

Final Thoughts and.. Food Photos!

As Seth Godin puts it, managing a tribe is a whole different concept than the old way of just managing a brand using traditional PR:

Tribe management is a whole different way of looking at the world.

It starts with permission, the understanding that the real asset most organizations can build isn’t an amorphous brand but is in fact the privilege of delivering anticipated, personal and relevant messages to people who want to get them.

It adds to that the fact that what people really want is the ability to connect to each other, not to companies. So the permission is used to build a tribe, to build people who want to hear from the company because it helps them connect, it helps them find each other, it gives them a story to tell and something to talk about.

Anyway, I can’t end this post without some photos of some of the amazing dishes we had at Maraboo:

3485103939_bd56f03f3a 3485052589_965193bcc2 3485054547_81f848a2d4 3485163245_980e6d8da4

3485923748_e60d2e4378 3299_97101179007_646439007_2503321_1740312_n

3485007257_3f098b6303 3485932262_9671e20964

3485930412_6f5734e35c 3485946606_f0b0e0d459

And finally, the entire dinning team :)

IMG_3800.JPG

Tags: ,

Weekend Inspirations – Big Data Visualizations, Innovative Man-Machine Interactions

Uncategorized April 25th, 2009

A bunch of interesting/inspiring topics for the weekend…

Designing “Big Data”

Jeff Veen from Small Batch Inc. gave a 20-minutes talk at the Web2.0 Expo at San Francisco.
During the talk he focuses on some classic examples for information visualization (John Snow pump, Minard‘s map, the tube map, and so on), the challenge of making data more accessible and understandable vs. just “decorating” it and the emerging challenge in Web 2.0 to empower users to find and create their own stories using the data.

And on the same topic of dealing with “big data”, check out the AlloSphere. A system developed by scientists and artists at the University of California for exploring huge sets of data as a visual and audible 3D world:

Cool New Computer Interaction Technology

I saw this amazing UI concept watching the TED podcast:

The concept as computers as tiny blocks that can react to each other is just amazing and the possibilities here, especially in education, are endless…

Kiva Robots Invade the Warehouse

If a bunch of tiny computers interacting can change personal computing here’s what it can do in the enterprise – optimize warehouses supply chain:

The servers work in real-time, receiving orders, immediately dispatching robots to bring the required pods to the worker fulfilling the order, and then returning the pods to their storage locations. The robots receive their orders wirelessly, while using cameras to read navigational barcode stickers on the warehouse floor.

Have a great weekend!

Tags: ,

Developing a Robust Data Driven UI Using WPF – An Overdue Summary (and full source code)

.NET, Programming, WPF April 15th, 2009

I wrote the stocky application more than a year ago as a research project aimed at proving that using WPF we can separate presentation metadata (XAML) from program logic. The goal was to provide the Duet team at SAP with a document reference sample for using M-V-VM to achieve this separation.

I started documenting the proof-of-concept in a series of posts but unfortunately after leaving SAP my interests (and work) shifted away from WPF and I didn’t find the time to finish the series.

I’ve received numerous requests to release the source code but I couldn’t do so because it was part of a larger infrastructure code I wrote at SAP which basically ads a lot of noise to the sample (an d probably ads legal issues for me sharing it).
Anyway, I took some time off this afternoon to re-write the sample independently so that I could share it:

It can be found on my SkyDrive

This, I guess is the long overdue ending for the series:

  • Introduction – introduces the concept of M-V-VM and the reasoning behind it.
  • The DataModel – describes how to write the Model part of our application.
  • Stock DataModel Sample – provides a conrete implementation of a Stock model and its view.. 

However, If you’re interested in M-V-VM in WPF, there are numerous topics worth mentioning that I didn’t get to cover and are definitely worth checking out:

Unit Testing

As I said in the introduction post, one of the most important benefits of seperating the logic code from the presentation (XAML) is that its straightforward to unit test. In fact, my next post following the Stock DataModel Sample was going to be about unit testing – specifically, how to test the DataModel its provider which, because of the use of threading, is a bit tricky.

This post is actually 99% done in the comments of the unit test code that’s in DefaultStockQuoteProviderTest.cs in the  provided source code. So do yourself a favor and go over the code. It’s not long and very well documented…

Using Lambda Expression for DataBinding

Data-binding is pretty much at the heart of the M-V-VM concept and it makes us write Value Converters which is pretty tedious and annoying.
Wouldn’t it be great if we could replace writing lots of IValueConverter classes like this:

<TextBlock Foreground="{Binding Change, Converter={StaticResource StockForegroundConverter}}" … />

[ValueConversion(typeof(double), typeof(Brush))]
public class StockChangeToBrushConverter : IValueConverter
{
    public object Convert(object value, Type targetType, object parameter, CultureInfo culture)
    {
        double change = (double)value;
        if (change == 0) return Brushes.Black;
        return (change < 0) ? Brushes.DarkRed : Brushes.Green;
    }

    public object ConvertBack(object value, Type targetType, object parameter, CultureInfo culture)
    {
        return double.NaN;
    }
}

To just the following XAML statement that embeds the conversion logic:

<TextBlock Foreground="{Binding Change,

Converter={ change=> if (change == 0) return Brushes.Black; return (change < 0) ? Brushes.DarkRed : Brushes.Green; }}" … />

M. Orçun Topdağı wrote an excellent series on using Lambda Expressions for data-binding in WPF to achieve just that:

Reference Applications and Guidance

I haven’t seen a lot of sample WPF LOB reference applications out there but here are some interesting links for further learning:

Tags: , , , ,

Microsoft Can Clone Twitter?!

Technology April 5th, 2009

In a response to Microsoft watcher Todd Bishop’s post saying  Microsoft should buy Twitter, Mary Jo-Foley wrote Microsoft Shouldn’t Buy Twitter saying:

…But I’d argue Microsoft could simply do a Twitter clone — the same way that it has built its own Facebook-notification-like news stream into Windows Live — and reap similar results. In fact, the Softies are hinting they’ve already been experimenting with adding Twitter-like functionality to its business software (possibly via SharePoint). I’d bet the Xbox and maybe the Pink/Danger mobile teams have been looking at doing their own Twitter-like services too.

Seriously?! Microsoft could simply clone Twitter?!

Sure, Microsoft certainly has some brilliant tech folks that can surely implement or clone anything. It also has the resources to do that.
And yet, its mostly following the pack with its online offering pretty stagnant. Last time it tried anything like that was Live Home with its Facebook\FriendFeed like functionality and that’s pretty much dead
It can certainly clone Twitter, probably even do a better job at it than the original, but will anyone bother using it?

Twitter has a fast growing, huge and vibrant community. Its also a well known brand name that’s getting a huge amount of media attention right now.
It survived its own fail whales, upgrade owls and all sorts of other nasty service downtimes simply because no one else can beat that…
You can clone technology but you can’t clone a community and brand strength…

Tags: , ,

Google’s New Behavioral Ad Targeting Should be Excellent for All

Business March 12th, 2009

The discussion over Google’s latest move into behavioral ad targeting is all over TechMeme. Basically what this means is that Google will start selling users (or more specifically, clusters of users, like football fans for example) in addition to words:

Today we are launching “interest-based” advertising as a beta test on our partner sites and on YouTube. These ads will associate categories of interest – say sports, gardening, cars, pets – with your browser, based on the types of sites you visit and the pages you view. We may then use those interest categories to show you more relevant text and display ads. (…)
So if you visit an online sports store, you may later be shown ads on other websites offering you a discount on running shoes during that store’s upcoming sale.

goog-add-interest

Google’ is not the first to embrace behavioral targeting as other companies (including Google’s own DoubleClick) were already offering advertisers these features, but it is certainly the biggest.
Behavioral targeting on Google massive network of sites (just think of anyone who’s using Google Analytics, which is pretty much everyone) is certainly a game changing event to the online advertising industry.

The responses online (and ones I usually get talking to people about the topic as we’re doing stuff in this area at Nuconomy) range from simple skepticism of behavioral targeting effectiveness (like Google’s own Matt Cutts…) to Big Brother concerns.

To the latter group, concerned with privacy and worried about its actions online being tracked I have to say – NEWS FLASH! you don’t have privacy on the internet… everything you do is already being tracked, analyzed and can be linked back to you even if stored anonymously (ever heard of Thelma Arnold?), the only question is what do you get out of it?

I hope that with behavioral targeted ads I will see less irrelevant ads offering me low mortgages and seduction courses and more ads geared towards tech gear, maybe even making online ads actually useful (can’t remember the last time I clicked an ad. with time I just learned to automatically ignore these ad banners…). But I think change is not going to hit just the advertising market…

Google’s move can (hopefully) marks the beginning of a personal internet era where sites will customize their entire layout and content based on the preferences, or habits of the browsing users.
My favorite news site will know I care about tech and biz. and never read sports news and will modify its homepage layout based on these preferences, the recommendation widget on my blog will show post recommendation based on the viewers’ recent browsing history rather than based on traffic, and so on…

Behavioral targeting opens the door on a conceptual change, not just in advertising…

Related Resources:

Tags: , , ,

Playing with the Windows 7 Fish

Humor, Technology February 7th, 2009

I just finished installing Windows 7 Beta on my home machine to find a fish swimming on my desktop:

Windows 7 Beta Default Desktop

But not just any fish, its a Siamese fighting fish, also knows a “betta fish” (or just “betta”).
A subtle Microsoft joke? Hope it does a better job selectively breeding this one (and get rid of some mutations… err… SKUs) …

Tags: , , , ,

Give Up Control, Think Distributed – DLD 2009 Summary

Business, Events February 6th, 2009

DLD09_New_RealitiesI  was very fortunate to get invited to the 2009 DLD Conference as a participant (Thanks to Yossi Vardi!). This was my first time at DLD and I can definitely say it has been the most amazing conference I have been to so far.

As defined by Steffi Czerny "DLD is interdisciplinary, creating interfaces and connecting people from the most different worlds." the conference’s “New Realities” theme encourages participants to discuss and formulates perspectives on markets following an eventful year, setting the agenda for 2009.

Or as Yossi Vardi defines it:

I told people that of my generation no one will understand instant messaging. And I was told my generation was already dead. But our generation has experiences the new generation does not have: real handshakes and real warmth of the hand. This still has to  move into social networks. And this is what we provide here: This old style feeling.

We invite you for 4 things:

  • to get food for thought
  • to make new friends
  • to play
  • to laugh

Think Distributed

So what is the new reality we’re living in? the repeating theme in most panels I’ve been to was the same:
We’re going through a fundamental change from centralized control (over content, decision making, …) to a distributed structure.

This change becomes very clear when you go over the following panels (and most other panels too):

All these panels talk about a new reality where, in order to survive, you have to give up on centralized control and go distributed.
On the 100 Million Uniques and New Media Models we hear about new distributed media networks that distribute information rather than creating and controlling it. On cloud computing we learn about technologies allowing companies to lose control over infrastructure for increased agility and lower costs, and on On Leadership we have the story of Best Buy that transformed its culture from “communicating at employees” to empowering their employees using social media tools.

Distributed Media

DLD09-NewMediaModelsPanelOn New Media Models we’ve heard from Jeff Jarvis who says that the future of media distributed – aggregating content from a lot of independent sources rather than controlling the creation of content in-house.
On his latest book, “What Would Google Do?” (which I got signed at the event. Thanks Jeff! :))  discusses the importance of building a platform for content, handing over control to users to build and distribute content, making your distributed network larger.

Traditional media companies think centralized. They create and control content and then spend a fortune on advertising in order to try and attract us, the consumers, to their content. This worked pretty well for big online media companies, like Yahoo for example, so far.
However, consumer today are relying more and more on social, self organization tools, to collect, sort, filter and rate content. Tools like Digg, Twitter, delicious and Facebook replace the functions that content portals used to provide.

In this new reality media companies need to stop thinking of themselves as an end – a closed content site (or portal) users come to in order to find content – and start distributing themselves.
Google distributes itself. It has its widgets, ads, maps and videos embedded in millions of web pages that it doesn’t own and it is making its profit of this vast distribution network.
Another example for distributed content is Glam. On 100 Million Uniques, Samir Arora, CEO of Glam described his company the following way:

“Glam is a distributed network that recognizes the fact that people go to many sites as opposed to a few, and that is the fundamental change that is happening today. So instead of bringing people to one place, we find out where people go and are there.”

“with every day that was passing, fueled by Google, its easier to find more sites, as opposed to one portal.

Instead of producing women targeted content and competing with all the other women content producers out there, Glam finds the best women content producers out there (some are lone bloggers and other can be bigger media companies) and invites them to join its network. It then sells ads on those sites and shares it revenue with them. It also aggregates the best content of its network to Glam.com, where it sells ads at a higher rate and shares its revenues too.
Glam helps sites on its network by sharing technology and content, and delivering traffic (and advertisers) to its member sites.

Distributed Leadership

DLD09-OnLeadership Changing from centralized control to a distributed structure is not limited to the media and advertising world.
On the On Leadership panel we see how Best Buy made a strategic decision to distributing control to its employees allowing them to self organize and interact using social Web 2.0 tools.
By changing the company culture  “Less control from the top, more taking responsibility” the role of the CEO changes from  to defining a common goal, not way to get there, and relying the companies human capital to do what it takes to get there.
So some control over what’s going on is lost, but according to Best Buy CEO Brad Anderson, “this stuff builds energy. if people have self accountability and can do something that they helped created and love they got more energy. what a customer would get when he gets into that store is that he could tell whether that store has got energy and engaged employees serving him or not”.

So if there’s one thing I have to take from my  time at DLD 2009 its this: give up control, think distributed.

More on DLD:

Recommended session videos:

Tags: ,

No Hope for Traditional Media Companies?

Business, Events February 4th, 2009

DLD09-NewMediaModelsPanel The Internet is a media platform based on content from traditional media companies (Times Magazine, etc.) as well as content produced by its users – blogs, forums and other social communication platforms.

This vast new world of content is taking the lead from traditional media. Newspapers, Magazines and TV Channels are all loosing their audience in favor of the internet. Those traditional media companies who do develop strong internet presence are still having troubles as income from online advertising does not cover the decrease in income from traditional advertising and subscription fees.

The New Media Models (video) panel, on last week’s DLD conference, tries to deal with the question of business models that media companies can use to profit on the web.

When asked about what kind of new media outlet they’d start today, non of the panel’s participants would start a print newspaper or a magazine.

“I wouldn’t start a newspaper. There might be a place for a magazine to sit on my coffee table, but not for a NewsWeek or a BusinessWeek or a daily newspaper I think its absurd… I understand why the do it, revenues on print are so great vs. online but it has to go way” said Michael Arrington. According to Arrington we’re going to be consuming our news and other content online or on digital devices and once a certain threshold is reached it’ll no longer be profitable to print. “It doesn’t make any sense for news to be on paper because of its just the cost structure” he summarizes.

Jeff Jarvis said he would start a distributed content network, like Glam. According to Jeff not owning and controlling the data is what allowed Glam to literally explode to more than 110 Million unique users in 3 years. While content owners have to spend a fortune advertising to bring people in, distributed networks just go to where the people are. People today reach content via rating sites like DIGG or Twitter messages and no longer require central content portals to collect, sort, filter and rate content them. Jeff says that media companies need to start asking “How can we build platforms on which others succeed?”, they have to become platforms for content distributed around the web (like YouTube for example) rather than producing and controlling their own content.

On that same note, Arrington added that traditional media sites, who’s cost structure includes all kinds of expenses that do not produce content – like programmers, office space – can’t compete with an army of bloggers who blog from their home using their laptop and free software.

Carolyn McCall, who is the CEO of Guardian Media Group (representing the “traditional” media companies) confirmed the fact that income from the printing business is going down while internet income, even for a huge company like the Guardian Group, do not compensate for these losses. However, she also mentions that 25 million users use the Guardian’s web and that its income is 300 million dollars a year and that both numbers continue growing.

The panelist could not explain why advertising budget that are taken from the print versions of the media do not move in their entirety to the online advertising world, and they could also not find a definite business model to run media company online (even Jeff’s example for a distributed network – Glam – is still not profitable and taking VC money) but they all agree that print is phasing out and that there’s a need for a new model for online media and journalism.

Tags: , , , , , , ,