Big Data State Of The Union

An informative study by TCS on the current state of Big Data “The Emerging Big Returns on Big Data”

.

Of course, you should download and read the whole report. Some interesting highlights:

• There’s a polarity in spending on Big Data, with a minority of companies
  spending massive amounts and a larger number spending very little
• The business functions expecting the greatest ROI on Big Data are not the ones
  you may think – while Sales & Marketing have initiatives, finance & logistics are betting on big data for efficiences & insights
• The biggest challenges to getting business value from Big Data are as much
  cultural as they are technological
• Nearly half the data (49%) is unstructured or semi-structured, while 51% is
  structured. The heavy use of unstructured data is remarkable given that
  just a few years ago it was nearly zero in most companies – Enterprises have gone multi-structured !
• Monitoring how customers use their products to detect product and design
  flaws is seen as a critical application for Big Data

Cheers & Happy Reading …

An ode to the Easter Eggs, Ecstasies & Agonies of a GoogleIO Ticket

Chronicles of my failed attempt at procuring a GioogleIO Ticket … The Google Wallet ate my GogleIO 2013 Ticket !

It was the night before GoogleIO … Excitement was in the air … Tweets were in order …

The order of the day was to find all Easter Eggs in the page …

I clicked and clicked and clicked … and got thru all the easter Eggs …

And I slept …

It was early AM when I woke up … still 15 min before the GoogleIO stores open …

The wait was agonizing, but all for a good cause, so I thought …

I was there when the GoogleIO Ticket store opened …

I was not disappointed when my first try failed after 6 minutes …

And my optimism payed off when it eventually found me a precious little ticket …

I reviewed the purchase … and gave it to Google Wallet … little did I know that …

But the screen stayed there and the time ticked down ….

By now the verdict was clear – The Google Wallet is going to eat my lucky GoogleIO Ticket ….

And It did …..

And soon after the registration ended …. The cold hand of fate …

Can I find a kind soul at Google to help me or should I wait for GoogleIo 2014 ? ….

Google Doodle Celebrating Douglas Adams

Interesting interactions on various parts of the image

I always was a fan of “A Hitchhiker’s Guide To the Galaxy”

Thank You Sophia Foster-Dimino, Corrie Scalisi, Kevin Laughlin, Manuel Clément, and Leon Hong - You made my day …

Bayesian Analytics Processing Pipeline

Here is a diagram from my Bayesian talk at pydata, Would appreciate feedback.

Big Data – Technologies, Platforms & Products

I came across a good diagram depicting the Big Data Eco System companies. I wanted to overlay the products as well as the Data management/Data Science pipeline viz Collect-Store-…. Here is my diagram. Let me know how I can improve it and what it is missing.

The Sign of the 9ers – 3 Lessons from the revival of the 49ers franchise

It is always interesting & informative to understand & learn from how great teams are formed – corporate or sports. Daniel Brown’s Article “How Jed York orchestrated 49ers rebuilding” has excellent three insights:

1. Recognize when a team is faltering & Take bold steps

   

   • On Dec 26,2010 after a bad defeat, Jed York reached a breaking point. He vowed to revive the team- fired the coach and started a set of systemic steps –  from hiring a new coach, “expanded the players’ lounge, built an expansive outdoor weightlifting facility and knocked down walls to give meeting rooms more square footage, upgraded the cafeteria … spending lavishly, demanding excellence and changing the culture of the entire organization”.

2. Have a bold vision & Lead from the top

   • Players stay with the 49ers because they know they’re in a first-class organization. That all starts at the top. If the top is not leading the charge, then you’re going to get mediocrity.The quality of ownership means a ton in pro football -Harris Burton

3. Show the passion, Hire the best & Keep both that way

   • What Jed shows — and what his uncle showed — is a passion for excellence

   • Jed hired Jim Harbaugh as the head coach. It was not easy – he outmaneuvered Stanford (which wanted to keep him), the University of Michigan (Harbaugh’s alma mater), the Denver Broncos and Miami Dolphins to reel in coaching’s biggest prize
   • The key for the 49ers was a six-hour meeting in which York and new general manager Trent Baalke laid out their plan for reinvigorating the franchise.
     

     Jed talked about his vision and that sealed the deal – Jim Harbaugh, on the day he was hired

   • The one thing that Jim Harbaugh has that Bill Walsh has is the ability to motivate … the guys to bring them to their peak potential -Brent Jones

Next Sunday, at the SuperBowl XLVII the 49ers face the Ravens – My best wishes and I predict a great match with the 49ers winning the trophy …

Trivia: Am sure sharp eyes would have caught the title “The Sign of the 9ers” as a tribute to Sir Arthur Conan Doyle’s “Sign Of The Four”

5 Steps to Pragmatic Data …er… Big Data

It is 2013 & Big Data is big news … Time to revisit my older (Nov’11) blog “Top 10 Steps to A Pragmatic Big Data Pipeline” … Some things have changed but many have remained the same …

5.  Chuck the hype, embrace the concept …

This seems to the first obvious step for organizations. From Ed Dumbill (“Big data” is an imprecise term...) to TechCrunch (“Perhaps it’s about the actual functionality of apps vs. the data“) agree with the concept, but the terms and marketing hypes have hit the proverbial roof. The point is, there are many ponies this pile & there is tremendous business value (so long as one is willing to discount the hype and think Big Data = All Data) …

I really like Mike Gualtieri’s very insightful definition of Big Data as

… the frontier of a firm’s ability to store, process, and access (SPA) all the data it needs to operate effectively, make decisions, reduce risks, and serve customers

4. Don’t implement a Technology, implement THE Big Data pipeline

Think of Big Data in multiple dimensions than a point technology & evolve the pipeline focussing on all the aspects of the stages

The technologies, the skill sets and the tools are evolving, so are the business requirements.

Chris Taylor addresses this very clearly (“Big Data must not be an elephant riding a bicycle“) – viz. One has to address the entire spectrum to get value …

Simply applying distributed storage and processing (like Hadoop) to extremely large data sets is like putting an elephant on a bicycle .. it just doesn’t make business sense — Chris Taylor

3. Think Hybrid – Big Data Apps, Appliances & Infrastructure

I had addressed this one in my earlier blog(“Big Data Borgs, Rise of the Big Data Machines & Revenge of the Fallen Algorithms“)

The morale of the story : Think out-of-the box & inside-the-box.

Match the impedence of the use cases with appropriate technologies

2. Tell your stories, leveraging smart data, based on crisp business use cases & requirements

Evolve the systems incrementally focussing on the business values that determine the stories to tell, the inferences to derive, the feature sets to influence & the recommendations to make

Augment, not replace the current BI systems

Notice the comma (I am NOT saying “Augment not, Replace”!)

“Replace Teradata with Hadoop” is not a valid use case, given the current state of the technologies. In fact, integration with BI is an interesting challenge for Big Data …

No doubt Hadoop & NOSQL can add a lot of value, but make the case for co-existence leveraging currently installed technologies & skill set. Products like Hive also minimizes barrier to entry for folks who are familiar with SQL

From a business perspective Patrick Keddy of Iron Mountain has a few excellent suggestions on managing Big Data: 

Big data informs and enhances judgement and intuition, it should not replace them

Opt for progress over perfection

View the data in context

1. Apply the art of Data Science & Smart Data, paying attention to touch points

This still remains my #1. Data Science is the key differentiator resulting in new insights, new products, order of magnitude performance, new customer base et al – “a cohesive narrative from the numbers & statistics”

“Data science is about trying to create a process that allows you to create new ways of thinking about problems that are novel, or you are trying to use data to create or make something.” says D.J.Patil

Smart Data = Big Data + context + inference + declaratively interactive visualization

• Smart Data is (inference) model driven & declaratively interactive
• For example,
  • The information like Wikipedia is big data; the in-memory representation Watson referred to is smart data
  • Device logs from 1000 good mobile handsets and 1000 not-so-good phones is big data;  a gam or glm over the log data after running through several stages of MapReduce is smart data, because it could give you an insight as to what factors or combination of factors make a good phone a bad phone

Focus not only on the Vs (ie Volume,Velocity,Variability & variety) but also on the Cs (ie. Connectedness & Context)

• Contextualization – includes integration with BI systems as well as data mashup with structured and unstructured data. An interesting example is how the 2012 US Presidential Election was won – All The President’s Data Scientists and All The President’s DevOps.

The two main Big Data challenges in 2013 would be:

1st : Data integration across silos to get the comprehensive view &

2nd : Matching the real-time velocity of business viz. CEP, sense & respond et al.

 For example, I have already seen folking looking outside Hadoop for CEP and near-realtime response

“.. 85% of respondents say the issue is not about the volume of data but the ability to analyze and act on data in real time” says Ryan Hollenbeck quoting a 2012 Cap Gemini study (Italics mine)