Yahoo Releases Its Own Hadoop Distribution

Cloud Computing, Software Industry June 11th, 2009

hadoopYahoo! is releasing its own distribution of Hadoop:

Hadoop is a distributed file system and parallel execution environment that enables its users to process massive amounts of data.
In response to frequent requests from the Hadoop community, Yahoo! is opening up its investment in Hadoop quality engineering to benefit the larger ecosystem and to increase the pace of innovation around open and collaborative research and development.
The Yahoo! Distribution of Hadoop has been tested and deployed at Yahoo! on the largest Hadoop clusters in the world.

Hadoop is free Java software framework born out of an open-source implementation of Google’s published computing infrastructure and fostered within the Apache Software Foundation.
Yahoo! has been the primary developer and contributor to Apache’s Hadoop.
In 2006, Hadoop founder Doug Cutting joined Yahoo, which provided a dedicated team and resources, to lead the project of developing the open-source software and turn Hadoop into a system that ran at web scale. Today, Yahoo! is running the largest Hadoop cluster in the world, which includes more than 25,000 servers and provides the framework for many Yahoo properties including Yahoo Search, Yahoo Mail, and several content and ad services.

Yahoo says its opening up the source code to Hadoop to “benefit the larger ecosystem increase the pace of innovation around open and collaborative research and development.”.
As Nigel Daley, Quality and Release Engineering Manager at Yahoo! Grid Technologies, summarizes:

Hadoop is helping us solve key science and research problems in hours or days instead of months. It provides us a platform to solve extreme problems requiring massive amounts of data processing. It underpins major revenue-generating systems. Opening our distribution enables a faster pace of innovation for the entire Hadoop ecosystem and broadens the use — and ultimately the quality — of this key platform across the industry.

Tags: , , , , , , ,

Google’s New Behavioral Ad Targeting Should be Excellent for All

Software Business March 12th, 2009

The discussion over Google’s latest move into behavioral ad targeting is all over TechMeme. Basically what this means is that Google will start selling users (or more specifically, clusters of users, like football fans for example) in addition to words:

Today we are launching “interest-based” advertising as a beta test on our partner sites and on YouTube. These ads will associate categories of interest – say sports, gardening, cars, pets – with your browser, based on the types of sites you visit and the pages you view. We may then use those interest categories to show you more relevant text and display ads. (…)
So if you visit an online sports store, you may later be shown ads on other websites offering you a discount on running shoes during that store’s upcoming sale.

goog-add-interest

Google’ is not the first to embrace behavioral targeting as other companies (including Google’s own DoubleClick) were already offering advertisers these features, but it is certainly the biggest.
Behavioral targeting on Google massive network of sites (just think of anyone who’s using Google Analytics, which is pretty much everyone) is certainly a game changing event to the online advertising industry.

The responses online (and ones I usually get talking to people about the topic as we’re doing stuff in this area at Nuconomy) range from simple skepticism of behavioral targeting effectiveness (like Google’s own Matt Cutts…) to Big Brother concerns.

To the latter group, concerned with privacy and worried about its actions online being tracked I have to say – NEWS FLASH! you don’t have privacy on the internet… everything you do is already being tracked, analyzed and can be linked back to you even if stored anonymously (ever heard of Thelma Arnold?), the only question is what do you get out of it?

I hope that with behavioral targeted ads I will see less irrelevant ads offering me low mortgages and seduction courses and more ads geared towards tech gear, maybe even making online ads actually useful (can’t remember the last time I clicked an ad. with time I just learned to automatically ignore these ad banners…). But I think change is not going to hit just the advertising market…

Google’s move can (hopefully) marks the beginning of a personal internet era where sites will customize their entire layout and content based on the preferences, or habits of the browsing users.
My favorite news site will know I care about tech and biz. and never read sports news and will modify its homepage layout based on these preferences, the recommendation widget on my blog will show post recommendation based on the viewers’ recent browsing history rather than based on traffic, and so on…

Behavioral targeting opens the door on a conceptual change, not just in advertising…

Related Resources:

Tags: , , ,

Microsoft Updates Its Windows Live Services

Cloud Computing, Software Industry November 14th, 2008

(Cross posted from CloudAve)

Microsoft announced today its rollout plans for the 3rd wave of Windows Live services.

The goal of this latest release wave, according to company officials, is to simplify the use of the offered services and unify the user’s entire online experience into the Windows Live interface.
Microsoft is planning to rollout the new services, currently in beta, to the public within the 1-2 months timeframe.

Windows Live Goes Social

One of Microsoft’s main emphasis with the current wave of services is on social networking between users using its services.

Microsoft finally figured out that its Live Messenger with about 268 million users worldwide, is by far the most popular instant messaging software in the world, is actually a social networks. With the new release, your Live Messenger contacts are now your Friends and you can see aggregated information about their activities on the net.

Very much like Plaxo, FriendFeed etc. Microsoft allows users to bring into their profile content they create in all sorts of services on the web (Live Services, Flickr, LinkedIn, blogs and RSS feeds, …) and share it with their friends and colleagues.
When users add photos, write reviews, and update their profiles directly on Live.com, that content will be put into their activity stream as well.
This activity stream is exposed in all sort of ways throughout Microsoft’s services interface.

For example, Microsoft’s new Live Home portal shows the latest events in your social network. When emailing a friend or chatting on Messenger you’re also able to interact with that friend’s activity stream and more…

Not just for private consumers…

I’ve been told that all these new service updates will not skip Windows Live Domains used by universities and organizations to create a personalized version of Microsoft’s services.
If that really the case, having all these new social capabilities as part of its domain offering can be amazing for collaboration and communication inside the organization.
While Google doesn’t seem to care about its Google Applications for Your Domains customers its good to see that Microsoft is going forward with Live Domains.
This latest update may just be the final straw I need to make the switch to Live Domains…

Where’s Live Mesh?!

It will be really interested to see where Live Mesh comes into the picture in regards to all of these Live services.
Live Mesh should be the glue bridging between Microsoft’s online services and its offline applications and devices (S+S) allowing users to sync all their content- contacts, photos, events, favorites, etc. – across devices and services.
Unfortunately, there’s no clear answer for that…

During the launch we’ve only heard about Live Sync allowing users to sync photos across computers. Some sources say its an incarnation of FolderShare and in any case it doesn’t seem to be based on Live Mesh technology.
With Live Mesh being one of Microsoft’s core platform offering its really hard to understand why we need to have Live Sync too…

Other notes…

  • All the services are released simultaneously in all countries and in 48 (!) languages.
  • Windows Live Skydrive size limit has changed from 5GB to 25GB
  • Windows Live Hotmail looks and feels a lot better to use.
  • I’ve uploaded all the screenshots of the new services to my SkyDrive:

Tags: , , , , , ,

Google Chrome is a Failure… Surprised?

Software Industry October 21st, 2008

Last month Google released its Chrome browser to the public and as usual it made a lot of PR noise (and a comic book) and got lots of favorable reviews

Google released its Chrome browser to the world about a month ago. It made a lot of PR noise, a comic book, and squeezed some outrageously favorable remarks from critics.

“When combined with Gears, which allows for offline access, Chrome is nothing less than a full on desktop operating system that will compete head on with Windows. “

Michael Arrington in “Meet Chrome, Google’s Windows Killer” (TechCrunch)

A month later, is Google Chrome even close to the the expectations set on its launch?
Read about it Cloud Avenue

Update October 21th, 2008:

Be sure to read the discussion on the comments section at CloudAve and Krishnan’s response post

Related stories:

Tags: , , ,

Why Coming Up With an iPhone Killer Will Be Tough…

Software Industry October 5th, 2008

2586687783_bc3cafd4db[1] Nokia recently announced its new 5800 Xpress Music, or Tube, as an “iPhone Killer”. Reviewers, however, disagree, mainly on hardware spec grounds – no touchscreen, S60 software issues.

What most reviewers seem to ignore is that even if Nokia (or Google, Sony, LG etc.) do come up with a device that matches the iPhone’s technical abilities (and I’m sure they have the required resources to do so) they’re still going to have a tough time competing with it.

Why? They way they sell and support their devices is very different than Apple.

When you buy an iPhone you have one company that control’s your entire experience with the device. From purchasing to ongoing operations and services its all ran by Apple that controls the entire product experience etc.
Apple bundles its iPhone with a set of services (AppStore, iTunes, Music Store…) that are tightly integrated with the device allowing easy consumption by customers which eventually is what’s driving Apple’s business.

Nokia, Sony, LG etc., on the other hand, sell the device through partnerships with different communication companies such as Orange Partner, which in turn, works with local sub-diaries. They rely on these partners for supporting their device and providing services to customers which often results in poor (sometime broken) customer experience.

Speaking at an Israeli GarageGeeks meeting, Steve Glagow, VP of Marketing Operations at Orange Partner VP mentioned that due to the fact they have to work with many regional and local partners, it takes more than 3 clicks and several screens for a user to install an application. Although he said their application installation rate is high (percentage of users who reach the application marketplace and end up installing an application) its a misleading measurement as most users will give up and never even reach the marketplace – only users determined to install an application will go through that effort.

Bundling your device with services is a winning strategy, not just for cellphones. Microsoft’s Xbox Live! platform is another great example on how bundling a device with online services helps increasing its value and perception.

Killing the iPhone will require more than just cool new hardware…

Related readings:

Tags: , , , , , , , , ,

On Google’s Chrome and The Future of Cloud Computing

Software Industry September 4th, 2008

I’ve received a lot of good feedback for my initial post about Chrome comparing its “Microsoft Killer” hype to the GTalk (now mostly defunct) launch a few years ago.

However, I’ve also heard from several folks who disagree (like Asaf for example).
The main point of the Chrome supporters is that advanced AJAX execution capabilities, coupled with Google Gears offlining capabilities and bundled in a browser provides a platform for next-gen web cloud applications that can seamlessly work online and offline.

The thing is, that no matter how much you optimize your browser’s rendering and JavaScript capabilities, underneath it all our entire web technology still based on old (even ancient in Internet terms) standards set almost a decade ago by W3C – a now defunct organization that is failing to keep up with the rate of technological changes.
Can you really believe that the future is in technology standards set by W3C and updated once, maybe twice a decade?!

It’s W3C’s inability to keep up that is driving companies to develop their own proprietary standards to lead today’s technological trends – Adobe with Flex and Air and Microsoft with Silverlight.

And maybe we’re going towards a browser-less future were we have AIR\Silverlight cloud enabled applications running on our OS?

The point is, taking Apple’s rendering code (apparently, not even the latest build) and putting it inside a featureless window while adding some optimized JavaScript VM is far from being innovative and light years away from the revolutionary expectations that we could hear about before and during the Chrome launch. Adding a few more horses to pull your carriage around doesn’t turn it into a car…

But hey, we’ve had the same story with a boring featureless (sorry, plain and simple) chat program a few years ago ;)

Tags: , , ,

Is Google Replacing My OS Again?!

Software Industry September 2nd, 2008

It seems like Google is making comic books these days. Its also announcing a new browser – Chrome – which we haven’t and know nothing about except the fact that it’s (obviously, an antitrust waiting to happen?) bundled with Google Gears. This of course, does not prevent Michael Arrington from TechCrunch of making outrageous claims calling it a “Windows Killer”:

When combined with Gears, which allows for offline access, Chrome is nothing less than a full on desktop operating system that will compete head on with Windows.

I’ve heard about Google’s Window-Killing abilities before….  where was that?  ah! right! scrolling all the way back to August 2005 where a new Google App is just about to revolutionize the way we communicate, the world, and the very fabric of the universe:

keith.teare: “It’s almost as if Google is implementing the features Microsoft has announced for Longhorn – Sidebar; voice calls inside IM, RSS integration – but doing it ahead of Microsoft, by about 12 months.

A Google layer between Users and the OS.

Rendering the OS a commodity

GTalk… remember GTalk? Google killed Vista with GTalk and will now kill Microsoft entirety thanks to coupling Google Gears to a JavaScript engine (not just any engine… an optimized one!)

And on a more serious note – Google’s business is all about AdSense.
It’s entire products line revolves around AdSense and anything that doesn’t help its business grow – dies. Even Android, under all the open-source and Apple hating PR, it about serving ads to mobiles.
Having said that, I wonder where Chrome fits in…

Tags: , , , , , , , , ,

Scaling Web Application – Recommended Readings

Software Architecture August 13th, 2008

Designing for scale is one of the greatest challenges when building when building web applications for the Internet. The huge scale of the Internet and the amount of potentials users requires applications to be able to handle huge amounts of data and traffic.

Today’s Internet applications has to be design with large scale in mind:

  • It has to be able to accommodate increased usage
  • It has to be able to accommodate increased data volumes.
  • It has to be maintainable

While the need seems obvious, implementing a working solution seems is not trivial, and so we see a lot of new companies that fail to handle the load (Cuil, Twitter, ….)

Joining Nuconomy’s ranks recently has opened me to the world of web scalability. And so, I’ve had to do quite a lot of reading the past couple of weeks.

I’ve compiled a list of the best resources I came across:

  • The following presentation by Cal Henderson provides a detailed overview of common patterns and approaches when building application for high availability and scale (you might also want to check out his book his book) :

SlideShare Link
(removed their player embed as it was throwing internal exceptions. So much for SlideShare’s QA…)

We describe our experience building a fault-tolerant data-base using the Paxos consensus algorithm.
Despite the existing literature in the field, building such a database proved to be non-trivial. We describe
selected algorithmic and engineering problems encountered, and the solutions we found for them. Our
measurements indicate that we have built a competitive system.

We used the Paxos algorithm (“Paxos”) as the base for a framework that implements a fault-tolerant
log. We then relied on that framework to build a fault-tolerant database. Despite the existing literature on
the subject, building a production system turned out to be a non-trivial task for a variety of reasons:
While Paxos can be described with a page of pseudo-code, our complete implementation contains several
thousand lines of C++ code. The blow-up is not due simply to the fact that we used C++ instead
of pseudo notation, nor because our code style may have been verbose. Converting the algorithm into
a practical, production-ready system involved implementing many features and optimizations – some
published in the literature and some not.

Got some more interesting scalability resources to share?  feel free to leave a comment…

Tags: , , , ,

Sergey Is Leaving Google For Microsoft (Not THAT Sergey…)

Software Industry June 30th, 2008

Dare has written a post that claims there’s an exodus from Google to Microsoft. The post is driven by his own observations and a post entitled Back to Microsoft from Sergey Solanik detailing his departure to Microsoft.
Sergey’s post contains some very interesting observations:

So why did I leave?

There are many things about Google that are not great, and merit improvement. There are plenty of silly politics, underperformance, inefficiencies and ineffectiveness, and things that are plain stupid. I will not write about these things here because they are immaterial. I did not leave because of them. No company has achieved the status of the perfect workplace, and no one ever will.

I left because Microsoft turned out to be the right place for me.

First, I love multiple aspects of the software development process. I like engineering, but I love the business aspects no less. I can’t write code for the sake of the technology alone – I need to know that the code is useful for others, and the only way to measure the usefulness is by the amount of money that the people are willing to part with to have access to my work.

Sorry open source fanatics, your world is not for me!

Google software business is divided between producing the “eye candy” – web properties that are designed to amuse and attract people – and the infrastructure required to support them.

And some observations of Google’s culture (bolding was done by me):

On the other hand, I was using Google software – a lot of it – in the last year, and slick as it is, there’s just too much of it that is regularly broken. It seems like every week 10% of all the features are broken in one or the other browser. And it’s a different 10% every week – the old bugs are getting fixed, the new ones introduced. This across Blogger, Gmail, Google Docs, Maps, and more.

This is probably fine for free software, but I always laugh when people tell me that Google Docs is viable competition to Microsoft Office. If it is, that is only true for the occasional users who would not buy Office anyway. Google as an organization is not geared – culturally – to delivering enterprise class reliability to its user applications.

The culture part is very important here – you can spend more time fixing bugs, you can introduce processes to improve things, but it is very, very hard to change the culture. And the culture at Google values “coolness” tremendously, and the quality of service not as much. At least in the places where I worked.

Since I’ve been an infrastructure person for most of my life, I value reliability far, far more than “coolness”, so I could never really learn to love the technical work I was doing at Google.

Dare also quotes Svetlin Nakov that also have some interesting things to say about the Google culture:

Google interview were not professional. It was like Olympiad in Informatics. Google asked me only about algorithms and data structures, nothing about software technologies and software engineering. It was obvious that they do not care that I had 12 years software engineering experience. They just ignored this. The only think Google wants to know about their candidates are their algorithms and analytical thinking skills. Nothing about technology, nothing about engineering.

Google employ everybody as junior developer, ignoring the existing experience. It is nice to work in Google if it is your first job, really nice, but if you have 12 years of experience with lots of languages, technologies and platforms, at lots of senior positions, you should expect higher position in Google, right?

This just demonstrates another cultural problem – Google doesn’t hire the right people for the job.
Granted, young, enthusiastic developers, with string academic background (and probably several degrees) can do some cool innovative stuff. These are exactly the type of guys you would want in your R&D department.
But it also the type that tends to loose interest when the research phase ends and the projects has goes to scaling and maintenance phases where you have to deal with stuff like support, maintenance (Google doesn’t even provide a roadmap for its products), localization, scalability, …

The bottom line is, as Dare concluded, is that Google isn’t a small startup anymore but it still thinks and acts like it is – in its hiring policies, internal processes and culture.
When measuring it up against other software giants it simply seems to lack…

As Fortune sums it up:

Think about that. Google recently made headlines by bidding almost $5 billion in a government auction of wireless spectrum, even though the company had no plan for using it. Some of its more peculiar products include Google Sky, Google Mars, and Google Ride Finder. It has become a significant investor in alternative-energy projects. Yes, alternative energy. And its founders fret that its risk-taking days are over? Then again, Google’s biggest risk may be recreating the magic it enjoyed as a startup- that intangible quality that makes Silicon Valley tick. Paul Buchheit, the former Google engineer who is on to his second startup now, recalls what he loved about Google’s early days. “I was always so excited at Google, because I didn’t know what would happen next,” he says. “Then I knew what would happen next.” Predictability is a virtue in the world of big business. It’s just not particularly Googley.

Maybe some of us in the industry were writing off Microsoft and crowning Google a little bit too soon…

Technorati Tags: ,,,

Tags: , , ,

Evil is Subjective

Software Industry May 19th, 2008

When asked about Microsoft last week at the GarageGeeks event, Sergey Brin, Google’s co-founder, answered that “Microsoft has a monopoly on being the bad guy”.
Well, I guess Evil is subjective as we’re seeing more and more news items like this from Google:

Google kills Anonymous AdSense account

How Scientology funded the anti-Scientology movement

By Cade Metz in San Francisco More by this author

Published Wednesday 14th May 2008 18:52 GMT


Exclusive Google has murdered the AdSense account run by one of the web’s most influential anti-Scientology sites.

Yesterday, the search giant cut off all ads served to Enturbulation, a fledgling site dedicated to promoting activism against the Church of Scientology and all its related organizations. This could have something do with the nature of the ads Google was serving. Many of the Google-driven ads funding the anti-Scientology site were paid for by the Church of Scientology.

Tags: , ,