Data Science with Spark on the Databricks Cloud – Training at SparkSummit (East)

DataSci-03-P24We had a good Data Science training session in Sheraton, Times Square, NY; second day of SparkSummit (East). It was my privilege to co-author and lead the Data Science track, along with Reza, Paco, Andy, Hossein, TD,Joseph and Xiangrui. I have shared the slideset at Slideshare as well as at the Databricks site.

This was the second time I was involved with a training fully based off of the Databricks cloud and it worked out very well ! The Databricks cloud was very robust and resilient. Unfortunately we had problems with the wireless at the Sheraton Hotel !DataSci-03-P27
The training was a mixture of hands-on and lecture.We sterted out with a dataset of 30 records and then moved onto the titanic dataset (900) to the movielens medium (1,000,000) and finally with the RecSyschallenge dataset (33,000,000!). What a progression in a day !

You can see the details in the slides. Ping me if you have any questions.

DataSci-03-P28Data wrangling over the RecSysChallenge 2015 data captures the essence of the Databricks cloud. I will quickly cover the RecSys Challenge dataset as an illustration.

The training data consists of 33,003,944 clicks and 1,150,753 buys. Our mission, if we choose to accept is to predict the session-items bought from a test dataset of 8,251,791 clicks.

A quick data exploration workflowdbc-01:



All at scale, in an elastic cloud, seamlessly moving between dev, model, stage and prod ! The magic of Databricks Cloud !

BTW, we also explored the State Of the Union Speeches from Washington, Lincoln, FDR, Clinton, Bush & Obama. The graphs below show a succinct view of the mood of the nation at each periods …


And finally after 100 slides later …!


The Art of NFL Ranking, the ELO Algorithm and FiveThirtyEight

In this blog, I will focus on the NFL Ranking based on the ELO algorithm that Nate Silver’s FiveThirtyeight uses. The guys at 538 have done a good job.The ELO and NFL ranking was part of my workshop at the Global Big Data Conference this Sunday. The full presentation is in slideshare

ELO – the algorithm made famous by Facebook & depicted in the movie Social Network


 Basic ELO


The k-Factor is the main leverage point to customize the algorithm for different domains.

  • For example Chess has no notion of a season; Soccer,Football & Basket ball are dependent on seasons – teams change during different seasons
  • Chess has no score to consider except WIn,Lose or Draw; but ball games have scores that need to be accommodated
  • For Chess k=10; for soccer it varies from 20 to 60; 20 for friendly matches to 60 for World Cup Finals
  • As we will see later, NFL adjusts k with the Margin Of Victory Multiplier
  • NFL also adjusts k to weigh recent games more heavily, w/ exponential decay
  • There are also mechanisms for weighing playoffs higher than regular season games (We will see this in Basketball)

538’s take on ELO



NFL 2014 Predicts & Results

The R program ELO-538.R is in Github

2014 Ranking Table





To Do

  1. Exponential decay with more weight for recent games – later in the season
  2. Calculate the rankings from 1940 to present, draw graphs like this from 538

Augmented Cognitive Intelligence

Have been working on this architecture for a couple of years. The idea is to build an AI machine that augments the human capabilities. I know IBM has Watson; Google, FB all have their own versions that address different domains.

The diagram below is more for my understanding and to clarify the thinking. I will write more as I get time. Hope you all find it useful.


Business Users Shouldn’t touch Hadoop even with a 99-foot pole !

Yep, I know, it is 10 foot pole; and the origin is from “10-foot poles that river boatmen used to pole their boats with”[1]

Back to the main feature, I was reading an piece by Andrew J Burst at GigaOM that “Hadoop needs a better front-end for business users”

Yikes. This is terrible … I would argue, no, make that insist, that business users be kept as far away as possible from Hadoop (& similar frameworks)

Allow me to elaborate …

  • Business users do need highly interactive analytic dashboards with knobs & dials into our deep machine learning models and sliders onto our AI machines, No doubt.
  • We don’t want to abandon our beloved business users with static-rigid-newtonian-deterministic artifacts; we want them to have living, (fire) breathing intelligent-inferential-predictive-models

  • But that control & interactivity is into a business analytics beast that has multiple layers, not directly onto a Hadoop or hadoop-like system.

Also separating the “what” form the “how” by a declarative interface is very important

You see, analytics has at least four layers viz. Infrastructure, Intelligence, Inference & Interface


  • Hadoop is Infrastructure, Spark is Infrastructure – The “How”
  • Machine Learning algorithms are Intelligence – Again lots of “How”
  • Models are Inference – the “What” Plus some “How”
  • Dashboard is the Interface (usually) – definitely the “How”
  • Interface can be recommendations, financial predictions, ad forecasts or even actual devices that interface to predictive models

  • And business needs knobs & dials at the Inference & Interface layers
    • The Infrastructure then appropriately fires frameworks Hadoop or Spark or Java or iPython …
  • Digging deeper, Hadoop itself has three layers – none of them operable by a business user, but real work horses

    • HDFS – the distributed File System

    • MapReduce – the distributed data parallel computation engine

    • HBase – the NOSQL data store

Back to Andrew’s points, Hadoop (and it’s cousins) should remain as a tool for the Chefs; but diners do need to express their choices and have the ability to “tweak” the seasonings, portions or even the amount of cooking; a declarative interface (which tells what but not how) comes from the domain specific menus catered by the restaurants which focus on respective culinary styles or even a fusion !

Now I am getting Hungry ! On my way to downstairs (am at the Hilton – NY Fashion District) to my favorite Chipotle – who in fact gives me the declarative freedom, without getting into their kitchen and the need to handle the saucepans ;o) It is better that way because I am terrible with cooking and spice measures – I can tell less salt but not the amount !


[2] Interface from

Building a Data Organization that works and works with business

One thing that caught my attention on Netflix’s Neil Hunt’s interview with Gigaom was this :

A Data Organization that works & works with business

Well said. That explains Netflix’s data Science in a nutshell that all Data Scientists should emulate !

From a Chief Data Scientist’s perspective, I really like their way of looking at Data Science viz:

  • The folks who do data Science for the whole business
  • The folks who build algorithms & 
  • The folks who do data engineering

In fact I had a blog on this specialization of Data Science skills

Netflix is putting more weight on actual behavior ! Interesting, we are also seeking similar effects ie differentiate between falling asleep on a couch vs. actually watching a TV show !  It is hard inference … Netflix has the blocker, I have nothing ;o(

Binge watching … interesting … We are actually working on algorithms to figure that out and change the ad mix. I plan to talk more at the TM Forum Digital Disruption Panel on December 9th !

bdtc-py-18-P76Finally, the fact that importance of Netflix recommendation engine is underrated is so true. In many ways the recommendation algorithms and engines are core to many systems.

In fact, we have a reverse recommendation strategy ! We recommend users to ads !

The Best Of the Worst – AntiPatterns & Antidotes in Big Data

Recently I was a panelist at the Big Data Tech Con discussing the Best of the Worst Practices in Big Data. We had an interesting and lively discussion with pragmatic,probing questions from the audience. I will post the video, if they publish it.

Here are my top 5 (Actually Bakers 5 ;o) ) – Anti Patterns and of course, the Antidotes …

1. Data Swamp

swamp-01Typical case of “ungoverned data stores addressing a limited data science audience“.

The scene reads like so :

The company proudly has crossed the chasm to the big data world with a new shiny Hadoop infrastructure. Now every one starts putting their data into this “lake”. After a few months, the disks are full; Hadoop is replicating 3 copies; even some bytes are falling off the floor from the wires – but no one has any clue on what data is in there, the consistency and the semantic coherence.

Larry at IT starts the data import process every week sometime in Friday night – they don’t want to load the DW with a Hadoop feed during customer hours. Sometimes Larry forgets and so you have no clue if the customer data is updated every week or every 14 days; there could be duplicates or missing record sets …

Gartner has an interesting piece about Data Lakes turning to Data Swamps – The Data Lake Fallacy: All Water and Little Substance … A must read …

The antidote is Data Curation. It needn’t be a very process heavy – have a consistent schema & publish them in a Wiki page. Of course, if you are part of a big organization (say retail) and have petabytes of data, naturally it calls for a streamlined process.

Data quality, data lineage, “descriptive metadata and an underlying mechanism to maintain it”, all are part of the data curation layer …

Semantic consistency across diverse multi-structured multi-temporal transactions require a level of data curation and discipline

2. Technology Stampede & Vendor Pollution

  • You start with a few machines, install Apache Hadoop and start loading data. Marketing fols catchup on your big data project and now you have a few apps running. A few projects come onboard and you have a decent sized cluster
  • A famous cloudy Hadoop vendor approaches the management and before you know it, the company has an enterprise license and you are installing the vendor’s Hadoop Manager & MPP database named after a GM car.
  • Naturally the management expects the organization to morph into a data-driven, buzz-compliant organization with this transition. And, naturally a Hadoop infrastructure alone is not going to suddenly morph the organization to a big data analytics poster child … vendor assertions aside …
  • And behold, the Pivotal guys approach another management layer and inevitably an enterprise license deal follows … their engineers come in and revamp the architecture, data formats, flows,…
    • Now you have Apache, the cloudy Hadoop vendor and this Pivotal thing – all Hadoop, all MPP databases, but of course, subtle differences & proprietary formats prevent you from doing anything substantial across the products …
      • In fact even though their offices are few miles apart in US-101, their products look like they are developed by people in different planets !
  • While all is going on, the web store front analytics folks have installed Cassandra & are doing interesting transformations in the NOSQL layer …
  • Of course the brick and mortar side of the business use HBase; now there is no easy way to combine inferences from the web & store-walk-ins
  • One or two applications (may be more) are using MongoDB and they also draw data from the HDFS store
  • And, the recent water cooler conversations indicate that another analytics vendor from Silicon Valley is having top level discussions with the CIO (who by now is frustrated with all the vendors and technology layers) … You might as well order another set of machines and brace for the next vendor wave …

The antidote is called Architecture & a Roadmap (I mean real architecture & roadmap based on business requirements)

Understand that no vendor has the silver bullet and that all their pitches are at best 30% true … One will need products from different vendors, but choose a few (and reject the rest) wisely ! 

3. Big Data to Nowhere

bridge-to-nowhereUnfortunately this is a very common scenario – IT sees an opportunity and starts building the infrastructure, sometimes massive, and puts petabytes of data in the Big Data Hub or lake or pool or … But no relevant business facing apps. A conversation goes like this …

Business : I heard that we have a big data infrastructure, cool. When can I show a demo to our customers ? 

IT : We have petabytes of data and I can show the Hadoop admin console. We even have the Spark UI !

Business : … (unprintable)

The Antidote is very simple. Build the full stack ie bits to business … do the data management ie collect-store-transform as well as a few apps that span the model-reason-predict-infer

Build incremental Decision Data Science & Product Data Science layers, as appropriate … for example the following conversation is a lot better …

Business : I heard that we have a big data infrastructure, cool. When can I show a demo to our customers ? 

IT : Actually we don’t have all the data. But from the transaction logs and customer data, we can infer that Males between 34 -36 buy a lot of stuff from us between 11:00 PM & 2:00 AM !

Business : That is interesting … Show me a graph. BTW, do you know what is the revenue is and the profit margin from these buys ?

IT : Graph is no problem. We have a shiny app with the dynamic model over the web logs.

IT: With the data we have, we only know that they comprise ~30% of our volume by transaction. But we do not have the order data in our Hadoop yet. We can … let me send out a budget request …

Now, business got an app, not fully production ready, still useful. IT has proved the value of the platform and can ask for more $ !

4. A Data too far

This is more of a technology challenge. When disparate systems start writing to a common store/file system, the resulting formats and elements would be different. You might get a few .gz files, a few .csv files and of course, parquet files. Some will have IDs, some names, some aggregated by week, some aggregated by day and others pure transactional. The challenge is that we have the data, but there is no easy way to combine them for interesting inferences …

5. Technology Maze

This is the flip side of the data format challenge (#4 above) This one is because of the technology stampede and the resulting incompatible systems where data is locked in.

Antidote : Integrated workflows.

Came across an interesting article by Jay Kreps. Couple of quotes:

” .. being able to apply arbitrary amounts of computational power to this data using frameworks like Hadoop let us build really cool products. But adopting these systems was also painful. Each system had to be populated with data, and the value it could provide was really only possible if that data was fresh and accurate.”

“…The current state of this kind of data flow is a scary thing. It is often built out of csv file dumps, rsync, and duct tape, and it’s in no way ready to scale with the big distributed systems that are increasingly available for companies to adopt.”

“..The most creative things that happen with data are less about sophisticated algorithms and vast computation (though those are nice) than it is about putting together different pieces of data that were previously locked up in different silos.”

The solution : A pub-sub infrastructure (Kafka) + “surrounding code that made up this realtime data pipeline—code that handled the structure of data, copied data from system to system, monitored data flows and ensured the correctness and integrity of data that was delivered, or derived analytics on these streams

5. Where is the Tofu ?

This is succinctly articulated by Xavier Amatriain when he talked about the Netflix Prize:

It is very simple to produce “reasonable” recommendations and extremely difficult to improve them to become “great”. But there is a huge difference in business value between reasonable Data Set and great …

The Antidote : The insights and the algorithms should be relevant and scalable … There is a huge gap between Model-Reason and Deploy …

What says thou ? Pl add comments about the Anti Patterns that you have observed and lived thru!


[Jan 3, 15] Interesting blog in Forbes “Four Common Mistakes That Can Make For A Toxic Data Lake” says :

  1. Your big data strategy ends at the data lake

  2. The data in your data lake is abstruse

  3. The data in your data lake is entity-unaware

  4. The data in your data lake is not auditable