Cloud Networking – We are not in Kansas anymore !

A funny thing happened on our way from virtualization to the cloud world ! The domain of cloud application developers and network started to converge. Converge might not be the right word – because network builders continue to build robust networks and that has not been antiquated; application developers will continue to develop interesting applications. But what has happened is a common layer and a convergence of interests between cloud application builders and network builders is happening.

Few points to note:

• I said and meant Cloud Application Builders – I do believe that there are many roles in the application domain and the role of an Application Builder is slightly different from Application Developer . I think I need a blog dedicate to this, for now the difference is in the focus – Cloud Application Builders will focus more on the infrastructure of applications – including scalability, availability, …
• Also, cloud application builders literally have to worry about the essential cloud monikers

… more at my blog at vmworld.

A Simple Minded Cloud Reference Architecture

I have been working on a simple Cloud RA – of course based on discussions and insights from many inside and outside my organization. Here it is:

Cloud Reference Architecture

I will write more as I get time. In the meantime, let me know what you think …

Cheers

<k/>

A Management Consultant’s View of Cloud Computing or Why McKinsey shouldn’t leave it’s day job !

Context:

There have been lots of discussions on McKinsey’s “Clearing the air on Cloud Computing”[1]. So I decided to take a close look at it.

In short, paraphrasing a colleague and a good friend, Chris Wiborg, “There is a reason McKinsey is in Management/Business Consulting and not Technology Consulting !”

Summary:

• The folks at McKinsey got the drivers and strategic value right, but totally missed the technology and are dead wrong on the tactics.
• First of all, while complaining that there are 22 definitions, the McKinsey folks added one more ! Looks like they are not satisfied with the other 22 !
• They didnot understand the significance of private clouds and focused narrowly on EC2 as THE alternative. Again, like the Berkeley study, very anemic cost models.
• Their recommendation to CIOs is dead wrong. I wish these folks stayed at the management consulting level (which they are very good at) and not try to make sense of something they are still learning … ;o)
  • Reading assignment : The RESERVOIR architecture document. Read, chew on it (not literally, of course), debate and then pontify !

Clint (a.k.a The Good):

• Excellent analysis – over-hyped, with potential for unrealistic expectations, significant hurdles for big enterprises, …
• They are right on the dot on the tremendous promise of cloud computing (P.4) viz: Faster Time To market, Lower upfront costs, Easier to scale out and most importantly creation of new value chains.
  • Good work folks …
• They are right on the adoption – Clouds are most attractive to start-ups and small businesses

Lee Van Cleef (a.k.a. The Bad):

• MkKinsey’s first observation was that there is no one definition of cloud computing. They found 22 definitions.
  • So what do they do ? They added a 23rd one ! As if that would solve all the definition problems ;o(
  • First of all there is no need for THE definition of Cloud Computing. Multitude of definition means that different constituents find value in different ways … which is a good thing. And a 23rd definition is not going to make all the 22 go away and make everybody fall in line !
  • My humble suggestion is to leave the definitions alone -Let folks, who are on the ground, define it the way they see value and utility
• Another very important distinction is the public clouds vs. private clouds. McKinsey totally ignored the private cloud space. This has been symptomatic – even the high minds in UCB missed this.
  • In the eyes of McKinsey, it is either virtualization inside one’s data center or Amazon’s EC3. Such a narrow view – total lack of understanding of the domain. Sorry

Eli Wallach (a.k.a. The Ugly):

• In McKinsey’s view, “Rather than create unrealizable expectations, CIOs should focus on virtualization”. Yikes, as ScoobeDoo would say it.
  • A recommendation only a management consultant (with no practical experience) can give
• “Aggressive virtualization” as they call it is the wrong strategy
  • What happens is that if CIOs follow this narrow-minded totally impractical strategy, the respective organizations will spend resources (narrowly focused on virtualization) and will miss the bigger benefits of Cloud Computing.
  • After all the effort in virtualization (like ROI calculations, budget negotiations, evaluation, software and hardware purchase, and so forth) they will have to start Cloud Computing all over again, yet another cycle of  ROI calculations, budget negotiations,… an approach that would bring tears of joy in the eyes of a management consultant ;o)

• My recommendation to CIOs would be -
  • Embrace Cloud Computing Infrastructure as the overall architecture with virtualization being one aspect.  If a CIO is initiating new infrastructure work, it should be Cloud Computing not virtualization.
    
  • They will not only be able to realize the economies of virtualization but also leverage the opportunity to create new applications based on the Cloud paradigm.
    

References:

[1] http://uptimeinstitute.org/content/view/353/319
[2] http://www.decisionstats.com/2009/04/mckinsey-attacks-cloud-computing-having-no-sense/
[3] http://bits.blogs.nytimes.com/2009/04/15/when-cloud-computing-doesnt-make-sense/

Personal Note:

After writing the blog, for some reason I got the urge to listen to the sound tracks of the title music for “For A Few Dollars More” and “The Good, The Bad and the Ugly”. I bought the mp3 tracks from AMZ and so finally they came out ahead ! They have found ways of monetizing even cloud blogging !

Six essential traits of an Enterprise Cloud Infrastructure or how to define a Cloud without defining it

Context

There are many ways of defining and talking about a cloud infrastructure. Dave Malcolm has a set of characteristics in his blog. And Mike has more insighst in his blog.

The view from an Enterprise (so called private clouds) and the capability to extend to clouds provided by service providers (so called Cloudbursting)  is a little different from the pure play internet-based clouds (so called public clouds)  Let me take this view and define what an Enterprise Cloud Infrastructure backbone would be, without actually defining it !




Essential Traits

• Elasticity – This, of course, is given. A cloud essentially is an elastic infrastructure
• Multi-tenancy – Naturally, somebody should be using the extra capacity. This functionality is usually at the SP (Cloud Service Provider) side. An enterprise can also have multi-tenancy when it offers cloud capability to it’s organizations – whether they are business units or functional units (like HR and finance)
• Abstraction- The level of functionality from IaaS (Infrastructure As A Service)  to PaaS (Platform As A Service) to SaaS (Software As A Service)
• Federation – The span – whether it is inside one datacenter or across multiple datacenters of an enterprise or extends to an SP
• Extension of Security Context/Scope – If we assume spanning across Service Providers with compatible infrastructure and with existing relationships, then we are extending the security context – a pragmatic approach
• Spanning Ownership domains – The enterprise has ownership if it is inside it’s firewalls and the ownsersip boundary is crossed to a cloud owned by an SP

CloudBursting

Let us define Cloud Bursting in context of the six traits. It is very easy – Elastic, multi-tenant at the SP side, extending the security domain and a different ownership domain.

Supporting Traits

Careful observers would note that this is missing two traits (mentioned below) that is near and dear to the likes of Amazon and Azure. It is deliberate that I did not include them in the essential traits. While Enterprises can (and will) use public clouds for many projects, for the near future public clouds will not be a *native* part of an Enterprise Cloud Infrastructure backbone …

• Raw Internet – I do not think “raw” Internet is an essential trait of Enterprise Clouds. Meaning, the communication between the Enterprise and the SP would be via some form of overlay (or leased line or similar mechanisms) that has security, QoS, policy and other characteristics
• Pay-per-usage – I do not think an Enterprise Cloud Infrastructure backbone would be pay-per-use.

Notes:

1. This blog is from my presentation at the OMG SATCCI (“Strategies & Technologies for Cloud Computing Interoperability”) BTW, there are a few good presentations from the workshop.

OAuth Header Hash

My first attempt draft-sankar-oauth-headerhash-00a

Social Networking – The next ERP ?

We can probably find enough evidence to argue this point.  An interesting waypoint is the project ESME

<exec_summary>

• Enterprise Social Messaging Experiment (ESME) is an Open Source tool designed by Siemens IT Solutions and Services together with SAP Community specialists.
• One client to the ESME system is,… yep you guessed it – ABAP, which is the programming language for SAP. So literally the ESME is an extension to ERP !
• And as SAP matures it’s cloud products and (inevitably) move into SaaS/Cloud models, an ERP-based social media interface which leverages the multi-tenant capabilities (thus deriving the social graph across enterprises)  is not far behind !
• ESME is a “self organizing communication group” which is interesting, as this is the only way collaboration/communication can scale. They need to acquire context as well as intelligently derive connectivity inferences
• “It serves to identify company employees with particular knowledge or expertise, and networks these experts together so that they can exchange information”
• And it is an open source Apache project . I might contribute … may be an *OpenSocial* interface, which the project lacks now. Also need to see how they are organizing the knowledge graph and social graph.

</exec_summary>

<for-those-who-are-more-technically-oriented>
ESME is written in *Scala* – a programming language which combines the object oriented world and the functional programming world. It also has primitives from Erlang, a very scalable language system – see my blog for a quick review

</for-those-who-are-more-technically-oriented>

Cheers & happy Pi day (3/14)
<k/>

Cloud Interoperability of the 5th Kind

In the last couple of days there have been a few good posts on Cloud Interoperability from Reuven, Greg and Stu. I also had written a blog about standard cloud couple of months ago. To understand the cloud interoperability, we do not have to go that far – from the clouds I mean – just taking a cue from the world of UFOs would be enough !

Cloud Interoperability of the 1st Kind => Clouds exist and can be observed. This is the state we are in

Cloud Interoperability of the 2nd Kind => Cloud services can talk to each other

Cloud Interoperability of the 3rd Kind => Clouds can observe each other and some form of intelligent communication can happen

Cloud Interoperability of the 4th Kind => Cloud are abducted by each other – just VM movement from one cloud to another

Cloud Interoperability of the 5th Kind => This is where “joint, bilateral contact events produced through the conscious, voluntary and proactive … cooperative communication” happens. In essence, the clouds can work together

With this enlightenment, we can examine the POV of the cloud luminaries …

• Reuven says “Cloud Interoperability is to make it easier to use multiple cloud providers who share a common set of application interfaces as well as a consensus on the terminology / taxonomies that describe them” – Yes aliens and earthlings can talk the same language and understand each other, but we cannot change the fundamental nature of aliens and earthlings. So a common API layer does not work here and it should not.We do not want to convert aliens to earthlings or vice-versa. They both have their place in the universe and we should respect that. So Common API is a non-starter.
• Reuven defines “Cloud Portability” as mobility irrespective of OS/technology platform and format. Portability is Cloud interoperability of the 4th kind – mobility between homogeneous format and OS/technology platforms. We are not looking for antidote against platform lock-in, but are looking for cloud provider/vendor lock-in and btw, there is nothing like a cloud lock in. Unfortunately Reuven, in his blog, has a little confusion on this aspect. One has a choice of multiple OS platforms and one can always choose to write apps in one platform, if one wishes so. There are valid reasons for choosing one or more platforms. Windows folks will write in Azure, Google folks would like to work with Python and rest of the folks will use EC2. A cloud substrate should not mitigate the OS platform difference.
• Stu first talks about protocol interoperability only a network engineer can love, Yummy Stu ;o)
• His startling observation, that Microsoft has a clue, based on the .NET services architecture, itself is startling ! It is just interfaces between services. I do not think that is interoperability in any sense. OK, I will give it the Cloud Interoperability of the 2nd kind, if a cloud service on another OS/technology platform can talk to Azure services, and they can (I think)
• I liked Stu’s deployment flexibility and that is one of the key components for interoperable clouds – not translation, semantic or not, not common APIs but just deployment flexibility between cloud providers, on the same OS/technology platform.
• Stu has some interesting predictions – yep, no new run time (read my lips – those who remember that)
• As I have said earlier, we have to look at this from the four planes viz: policy, control, management & data. For Cloud Interoperability of the fifth kind, we need :
  • A declarative policy plane that can be interpreted in a common way but implemented in proprietary ways as the platform sees fit
  • A control plane that can be dynamic with rich attributes that are semantically equivalent across clouds (I do not care if they talk the same words so long as I know how to map them and do equivalence)
  • A management plane – again built on semantically equivalent attributes so that I can make (collective and individual) inferences as and when needed
  • And A data path which is the OS/technology of my choice and I do not want the cloud to change my data path characteristics.
  • And A mature metering plane [Update - March 15,09 : James Urquhart convinced me that this is a fundamental requirement!]
    

Cheers

<k/>

Marc Andreessen with Charlie Ross – Innovation, mobility, Social Media & Viral platforms

A very informative interview – Charlie asked interesting questions and Marc has equally insightful answers & discussions.

Video and full text at http://seekingalpha.com/article/121915-marc-andreessen-on-charlie-rose-internet-and-new-media-companies

For the attention challenged my bullet notes:

1. Future of news papers

   • Two words – kill it ! Stop Printing newspapers !
   • Fundamental structural change happening in the newspaper business. It is happening in all branches of the media industry but the newspaper is at the front
   • Investors have seen thru the transition. But the industry is still trying to survive.
   • An interesting analogy : Chronic pain vs acute pain – How many years of chronic pain vs one year of acute pain for transition ;o)
   • Acute pain will be acute but inevitable. But need to build for future.
   • Wrote a blog New York Times deathwatch ! “What is with you & NY Times ?” Charlie asks. Blog post no longer there

2. Social Media Industry

   • Facebook: Facebook 175 million users, half of them use it every day; many use it 50 times a day. On its way to 500 million users !  Mark is on the board of directors. 135 million active users  equates to 6th most populous country in the world !
     They are taking a more organic growth model if they had taken the normal advertising, they would make been over a billion dollar in ad revenue. Facebook has tremendous potential for example could monetize the home page just a question of how they choose to extract the value. They want to build a long term business, eyes way on the horizon and big vision (to connect everybody on the planet (what about beyond?))
   • Ning: Ning has crossed 20 million users adding 2 million users a month. There are a million social networks on Ning !
   • YouTube, Facebook et al – under-monetized assets
   • Twitter as a real-time electronic nervous system – says you could twitter when a plane lands in water. May be people did, but I wouldn’t be twittering if my plane crash landed on water ;o)
     Story of twitter – Evan Williams had a podcasting company ; raised ~3.5 million; didn’t succeed and returned all (Evan made up the difference!) Twitter was a side business at that time, it took off. So they changed focus, closed the podcast operation and focused on Twitter
   • Social networking is here to stay and it’s potential is just beginning. Marc is big on “viral” applications
   • The Obama campaign employed the social networking approach and philosophy as the engine for fund raising, volunteer coordination
   • Viacom suing YouTube wrong strategy – They should be using it to distribute their videos ! Every time there is a Viacom video in YouTube, there should be a buy button! Distribution channel that bring traffic to their properties !
     Napster – 20 million people showed up. If music industry had a buy button they would have been successful. When people line up, find a way to monetize it

3. Innovation

   • More opportunity than ever before – Cascading effect – every new layer of technology makes another layer of innovation possible and that keeps rolling
     There is an interesting discussion of Intel’s transformation from a memory chip maker to a microprocessor maker in around 1985; was not an obvious bet to make, but they had to do it to escape the overhang of Japanese memory makers who were crushing Intel.
   • Innovation Cycle: Silicon graphics was out of business due to Intel’s microprocessor and that freed up engineers to work on nVidia/ATI which in-turn is posing challenges to Intel in video and graphics business

4. Mobility, iPhone & the new landscape

   • Usually people talk about a new idea for long time, finally the technology comes together and the thing takes off – internet in ‘95 is an example, mobile is in that stage now
   • iPhone is a template every other vendor will copy. For first time iPhone real os, sdk and an application delivery infrastructure – 1st time all of these over a fast network
   • iPhone itself is fantastic – beams from the future as Marc characterizes it – and inspired a lot more creative thinking around it
   • He mentioned an investment of his Qik [http://qik.com/], where any phone can be the source of live streaming video to any device or other phone; will be very effective as phones with HiDef videos capability in 2 years

5. The Magic Business

   • Bill Joy once said : some products have the “it works” feature !
   • There were more than thirty-five search ventures before Google; but Google search really worked in terms of the core technology plus they unlocked the ad business model.
   • Marc characterizes this as the “Magic Business” which happens once in 10 years or so – Cisco was a magic business, intel was one, so was Microsoft and even Amazon. With Magic Business, one goes for scale and size. People had written AMZ off in 2002, but Bezos had the fortitude and foresight to stick with the long vision

6. New form factor

   • Marc believes Kindle is the new form factor along with iPhone and netbook; each with a different but effective purpose.
   • Kindle is the web-pad, a 7” form factor, the next opportunistic screen size which people will for video, telephony and conferencing.
   • Most probably the next new product from Apple would be this 7” e-Book, conference, web appliance !

7. New VC Firm with a slightly different focus

   • Marc is starting a new VC fund with Ben Horowitz. They have invested their own money in the last 3 years in 36 companies
     They focus on smaller companies – 100K-200K; may be 500K to million. Marc is of the opinion that a whole generation of startups do not need very much money (“very much” defined as  200K – 1.5 million)
   • His new VC firm’s name – Andreessen Horowitz ; can be a law firm or a vc firm! Abbreviates to A to Z and will get listed first in yellow pages – could be a good name for a tow truck business as well!

8. Impact of the recent economy related challenges

   • During the 2001 recession, we were the nose of the dog , this time we are the tail. Companies on valley do not generally run on debit financing and so affected the least. But the big recession will impact salesSilicon Valley will be the tragic beneficiary from damage in other industries – like banking et al
   • <KS>
     • I thought the discussion on new types of banks was a little asymptotic but the concept of new way of just-in-time credit scoring and credit provisioning by Bill me Later is interesting.
     • On a tangential discussion, Marc was referred to “Good Banks, bad banks and ugly Assets” and ideas by Paul Romer
   • </KS>
   • Innovation will continue tons of innovation will be bottled up in the next 5 years. Companies like Google, YouTube and Facebook developed thru the last bust. Look for return in 7-10 years from today’s funding.

Updates

1. Good Comments. Thanks.

Twitter Tips

I am collecting best practices, insights and just blogs with attitude on the usage of Twitter. I think this would help twitter newbies. So far I have :

• Twitter Etiquette: Five Dos and Don’ts
• Are you using Facebook, Linkedin and Twitter differently?
• Top 10 Twitter Tips for Beginners

Cheers

<k/>

A Berkeley View Of Cloud Computing : An Analysis – the good, the bad and the ugly

I read thru the technical report from UC Berkeley, Above the Clouds: A Berkeley View of Cloud Computing with interest. My analysis:

<summary>

• As an undergrad work on cloud computing, the paper gets an A+. But as a position paper from eminent academics, I can only give a C-. Granted it correctly identifies many of the trends and obstacles. But that material is widely available !
• With a title “A Berkeley view of cloud computing” the report misses the point. “A Berkeley observation…” is more like it – view requires original thinking and interpolation, which the report lacks.

</summary>

<the_good>

• The authors got some of the essentials of Cloud Computing right viz: infinite capacity, no up-front commitment and pay as you go.
• The three classes viz: amazon , Microsoft and the Google model is interesting. But there are more in-between.
• They have some good points on the cost advantage of power et al and leveraging that aspect by building datacenters at the appropriate locations.
• The new application models viz. analytics, parallel batch processing, compute-intensive desktop applications and so forth are excellent observations.
• They have done some good work in characterizing elasticity. Pages 10 and 11 are good read – the models are very simplistic, though.
• They also have done a good job in showing the economies of scale that can be achieved by a cloud computing infrastructure.
• I like their assertion that “there are no fundamental obstacle to make cloud-computing environments secure as well as compliant to data and processing rules. Declarative policy and enforcement thereof is my answer.
• They have correctly identified scalable storage as one of the bottlenecks. The BigTable(Google), Dymo(AMZ) and Cassandra(facebook) all are solutions for the challenge.

</the_good>

<the_bad>

• But, they got the model wrong ! The essentials of Utility Computing is the consumption model not the payment model. No doubt the pay-as-you-go model is attractive to startups, but the payment model is the second order effect. For enterprises and other organizations, the value proposition is the elasticity and the just-in-time availability of resources.  Even for startups the pay as you go is attractive but elasticity is much more important.
• Argument about increase in performance and resultant cost reduction. This just Moore’s law and it is achievable within IT environments as well as a cloud computing space. I think computers are on a 5 year amortization schedule and depreciation. And a refresh can be done – with associated efficiency whether they are a cloud provider or an IT shop.
• I think the major disconnect in the paper is the basic definition of a cloud as public. The artificial separation of public/private clouds and the focus on payment were the two areas where their definition has gone awry. Cloud is an architectural artifact and a business model of computing. But clouds are clouds – internal or external, public or private. The internal vs. external is only a spatial artifact – which side of the firewall. Not worth a demarcation when we talk about the domain of cloud computing.  Which side of the internet (firewall) does the cloud infrastructure lie, should not be the criteria. By their definition, they have disenfranchised the whole set of clouds inside organizations. The internal-external cloud integration across data, management, policy and compute planes is an important topic which this model conveniently skips. Also as I mentioned earlier, utility is the consumption not a payment model. A big organization can have a cloud computing infrastructure and it’s business units can leverage the elasticity – no need for a credit card, a charge back model will do.

</the_bad>

<the_ugly>

1. I really didn’t get the “statistical multiplexing” they mention a few times. What exactly is this and what is the relevance ? Just a buzz word to jazz up the paper ?
2. I literally got lost in their characterization of DDoS attack and the cost models there of on P.15. Really convoluted and it does not change for traditional vs. cloud. They found a break-even point for DDoS attack based on very slim assumptions.
3. I do not think the data transfer bottleneck, as described in the paper (P.16), is an issue. Who is going to transfer 10TB of data routinely for cloud processing ? Looks like a force fit for some calculations done by someone.
4. The report has no useful tables or equations. Equations 1 and 2 (which are the same, btw) are not right – in thesense that the datacenter cost includes the utilization and I do not think we need to accommodate for it additionally.
5. I am sorry to say all the cost models and the equations look forced and very unnatural. Even the assertion of 1/5 to 1/7 cost advantage of a datacenter is at best questionable.No value what so ever – sorry folks

</the_ugly>

Updates:

1. Good comments. Thanks folks.
2. James Urquhart has an excellent blog on the subject. Thanks James. He is more generous than me ;o)
3. [Feb 19,2009] Blog from GoGrid – good analysis