What would you want AI to do, if it could do whatever you wanted it to do ?


P.S: Copy of my blog in Linkedin

Note : I am capturing interesting updates at the end of the blog.

AI-06-01

Exponential Advances:

An interesting article in Nature points out that exponential advanced in technological growth can result is a very alternate world very soon.

IBM X Prize:

 

AI-07

And the IBM AI X Prize is offering a chance to showcase powerful ideas that tackle challenges.

Got me thinking … What do would we want our machines/AI to do ?

I am interested in your thoughts. Pl comment on what you would like AI to do.

Earlier I had written about us not wanting our machines to be like us; understand us – may be, help us – definitely, but imitate us – absolutely not  …

So what does that mean ?

  • Driving cars ? – Definitely
  • Image recognition, translation and similar tasks ? – Absolutely
  • Write like Shakespeare just by feeding all the plays to a neural network like the LSTM ? – Definitely not !

I see folks training deep Learning systems by feeding them Shakespeare plays and see what the AI can write. Good exercise, but is that something we would get an X Prize for ? Of course, that is putting the cart before the horse !

We don’t write just by memorizing the dictionary and Elements of Style !!

  • We write because we have a story to tell.
  • The story comes before writing;
  • Experience & imagination comes before a story …
  • A good story requires both the narrative power as well as a powerful content with it’s own anti-climax, and of course the hanging chads ;o)
  • Which the current AI systems do not possess …
  • Already we have robots (Google Atlas) that can walk like a human – leaving aside the the goofy gait – which, of course, is mainly a mechanical/balance problem than an AI challenge
  • Robots can drive way better than a human
  • They translate a lot better than humans can (Of course language semantics is a lot more mechanical than storytelling)
  • Robots and AI machines do all kinds of stuff (Even though Mercedes Assembly plant found that they cannot handle the versatile customization!)

Is there anything remaining for an AI prize One wonders …

In the article “How Google’s impressive new robot demo will fuel your nightmares” , at 2:09, the human (very rudely) pushes the robot to the ground and the robot gets up on it’s own ! That proves that we have solved the mechanical aspects of human anatomy w.r.t movements & balance.

[Update 3/17/16] Looks like Google is pushing Boston Dynamics out of the fold !

AI-08

 

But a meta question remains.

  • Would the robot be upset at the human ?
  • Would it know the difference – if it was pushed to keep it away from harm’s way (say a falling object) vs. out of spite ?
  • And, if we later hug the robot (as the author suggests we do) would it feel better ?
  • Will it forget the insult ?

So there is something to be done after all !

Impart into our AI – the capability to imagine, the ability to understand what life is;  feel sadness & joy; understand what it is to struggle through a loss,…

This is important – for example, if we want robots to act as companions for the sick, the elderly and the disabled, may be even the occasional lonely, the desolate and for that matter even the joyous!

If the AI cannot comprehend sadness, how can it offer condolences as a companion ? Wouldn’t understanding our state of mind help it to help us better? 

AI-09

 In many ways, by helping AI to understand us, the ultimate utility might not be whether AI really comprehends us or not, but whether we get to understand us better, in the process !! And that might be the best outcome out of all of these innovations.

As H2O-ai Chief SriSatish points out,

Over the past 100 years, we’ve been training humans to be as punctual and predictable as machines; … we’re so used to being machines at work—AI frees us up to be humans again ! – Well said SriSatish

With these points in mind, it is interesting to speculate what the AI X-Prize TED talks would look like in 2017; in 2018. And what better way to predict the future than to invent it ? I am planning on working on one or two submissions …

And what says thee ?

[Update 3/12/16] Very interesting article in GoGaneGuru about AlphaGo’s 3rd win.

AI-11

  • AlphaGo’s strength was simply remarkable and it was hard not to feel Lee’s pain
  • Having answered many questions about AlphaGo’s strengths and weakness, and exhausting every reasonable possibility of reversing the game, Lee was tired and defeated. He resigned after 176 moves.
  • It’s time for broader discussion about how human society will adapt to this emerging technology !!

And Jason Millar @guardian agrees.

AI-12

Maybe all is not lost after all, WSJ says … !

AI-13.png

[Update 3/9/16] Rolling Stone has a 2-part report – Inside the Artificial Intelligence Revolution. They end the report with a very ominous statement.

AI-10

 

[Update 3/4/16] Baidu Chief Scientist Andrew Ng has insightful observations

  • “What I see today is that computer-driven cars are a fundamentally different thing than human-driven cars and we should not treat them the same”- so true !

[Update 3/6/16] An interesting post from Tom Devenport about Cognitive Computing.

  • Good insights into what Cognitive Computing is, as a combination of Intelligence(Algorithms), Inference(Knowledge) and Interface (Visualization, Recommendation, Prediction,…)
  • IMHO, Cognitive Computing is more than Analytics over unstructured data, it also has touches of AI in there.
  • Reason being, Cognitive Computing understands humans – whether it is about buying patterns or the way different bodies reacts to drugs or the various forms of diseases or even the way humans work and interact
  • And that knowledge is the difference between Analytics and Cognitive Computing !

I like Cognitive Computing as an important part of AI, probably that is where most of the applications are … again understanding humans rather than being humans !

Reference & Thanks:

My thanks to the following links from which I created the collage:

  1. http://www.nature.com/news/a-world-where-everyone-has-a-robot-why-2040-could-blow-your-mind-1.19431
  2. http://lifehacker.com/the-three-most-common-types-of-dumb-mistakes-we-all-mak-1760826426
  3. https://e27.co/know-artificial-intelligence-business-20160223/
  4. http://spectrum.ieee.org/tech-talk/computing/software/digital-baby-project-aims-for-computers-to-see-like-humans
  5. http://www.techrepublic.com/article/10-artificial-intelligence-researchers-to-follow-on-twitter/
  6. http://www.lifehack.org/309644/book-lovers-alert-8-the-most-spectacular-libraries-the-world
  7. http://www.headlines-news.com/2016/02/18/890838/can-ai-fix-the-world-ibm-ted-and-x-prize-will-give-you-5-million-to-prove-it
  8. http://www.lifehack.org/366158/10-truly-amazing-places-you-should-visit-india?ref=tp&n=1
  9. http://www.lifehack.org/articles/lifestyle/20-most-magnificent-places-read-books.html

The Art of NFL Ranking, the ELO Algorithm and FiveThirtyEight


In this blog, I will focus on the NFL Ranking based on the ELO algorithm that Nate Silver’s FiveThirtyeight uses. The guys at 538 have done a good job.The ELO and NFL ranking was part of my workshop at the Global Big Data Conference this Sunday. The full presentation is in slideshare


ELO – the algorithm made famous by Facebook & depicted in the movie Social Network

gbdc-r-04-P30


 Basic ELO

gbdc-r-05-P20

The k-Factor is the main leverage point to customize the algorithm for different domains.

  • For example Chess has no notion of a season; Soccer,Football & Basket ball are dependent on seasons – teams change during different seasons
  • Chess has no score to consider except WIn,Lose or Draw; but ball games have scores that need to be accommodated
  • For Chess k=10; for soccer it varies from 20 to 60; 20 for friendly matches to 60 for World Cup Finals
  • As we will see later, NFL adjusts k with the Margin Of Victory Multiplier
  • NFL also adjusts k to weigh recent games more heavily, w/ exponential decay
  • There are also mechanisms for weighing playoffs higher than regular season games (We will see this in Basketball)

538’s take on ELO

gbdc-r-05-P21

gbdc-r-05-P22


NFL 2014 Predicts & Results

The R program ELO-538.R is in Github

2014 Ranking Table

gbdc-r-05-P27

gbdc-r-05-P29

gbdc-r-05-P31

gbdc-r-05-P32


To Do

  1. Exponential decay with more weight for recent games – later in the season
  2. Calculate the rankings from 1940 to present, draw graphs like this from 538

Business Users Shouldn’t touch Hadoop even with a 99-foot pole !


Yep, I know, it is 10 foot pole; and the origin is from “10-foot poles that river boatmen used to pole their boats with”[1]

Back to the main feature, I was reading an piece by Andrew J Burst at GigaOM that “Hadoop needs a better front-end for business users”

Yikes. This is terrible … I would argue, no, make that insist, that business users be kept as far away as possible from Hadoop (& similar frameworks)

Allow me to elaborate …

  • Business users do need highly interactive analytic dashboards with knobs & dials into our deep machine learning models and sliders onto our AI machines, No doubt.
  • We don’t want to abandon our beloved business users with static-rigid-newtonian-deterministic artifacts; we want them to have living, (fire) breathing intelligent-inferential-predictive-models

  • But that control & interactivity is into a business analytics beast that has multiple layers, not directly onto a Hadoop or hadoop-like system.

Also separating the “what” form the “how” by a declarative interface is very important

You see, analytics has at least four layers viz. Infrastructure, Intelligence, Inference & Interface

I4

  • Hadoop is Infrastructure, Spark is Infrastructure – The “How”
  • Machine Learning algorithms are Intelligence – Again lots of “How”
  • Models are Inference – the “What” Plus some “How”
  • Dashboard is the Interface (usually) – definitely the “How”
  • Interface can be recommendations, financial predictions, ad forecasts or even actual devices that interface to predictive models

  • And business needs knobs & dials at the Inference & Interface layers
    • The Infrastructure then appropriately fires frameworks Hadoop or Spark or Java or iPython …
  • Digging deeper, Hadoop itself has three layers – none of them operable by a business user, but real work horses

    • HDFS – the distributed File System

    • MapReduce – the distributed data parallel computation engine

    • HBase – the NOSQL data store

Back to Andrew’s points, Hadoop (and it’s cousins) should remain as a tool for the Chefs; but diners do need to express their choices and have the ability to “tweak” the seasonings, portions or even the amount of cooking; a declarative interface (which tells what but not how) comes from the domain specific menus catered by the restaurants which focus on respective culinary styles or even a fusion !

Now I am getting Hungry ! On my way to downstairs (am at the Hilton – NY Fashion District) to my favorite Chipotle – who in fact gives me the declarative freedom, without getting into their kitchen and the need to handle the saucepans ;o) It is better that way because I am terrible with cooking and spice measures – I can tell less salt but not the amount !


[1] http://en.wiktionary.org/wiki/not_touch_something_with_a_ten_foot_pole

[2] Interface from http://img1.mxstatic.com/wallpapers/1bb91493c637d7c5ed6e1cefbef87ec1_large.jpeg

Building a Data Organization that works and works with business


One thing that caught my attention on Netflix’s Neil Hunt’s interview with Gigaom was this :

A Data Organization that works & works with business

Well said. That explains Netflix’s data Science in a nutshell that all Data Scientists should emulate !

From a Chief Data Scientist’s perspective, I really like their way of looking at Data Science viz:

  • The folks who do data Science for the whole business
  • The folks who build algorithms & 
  • The folks who do data engineering

In fact I had a blog on this specialization of Data Science skills

Netflix is putting more weight on actual behavior ! Interesting, we are also seeking similar effects ie differentiate between falling asleep on a couch vs. actually watching a TV show !  It is hard inference … Netflix has the blocker, I have nothing ;o(

Binge watching … interesting … We are actually working on algorithms to figure that out and change the ad mix. I plan to talk more at the TM Forum Digital Disruption Panel on December 9th !

bdtc-py-18-P76Finally, the fact that importance of Netflix recommendation engine is underrated is so true. In many ways the recommendation algorithms and engines are core to many systems.

In fact, we have a reverse recommendation strategy ! We recommend users to ads !

The Sense & Sensibility of a Data Scientist DevOps


The other day I was pondering the subject of a Data Scientist & model deployment at scale as we are developing our data science layers consisting of Hadoop, HBase & Apache Spark. Interestingly earlier today I came across two artifacts – a talk by Cloudera’s @josh_wills and a presentation by (again) Cloudera’s Ian Buss.

The talks made a lot of sense independently, but add a lot more insight – collectively !  The context, of course, is the exposition of the curious case of data scientists as devops. The data products need an evolving data science layer …

It is well worth your time to follow the links above and listen to Josh as well as go thru Ian’s slides. Let me highlight some of the points that I was able to internalize …

Let me start with one picture that “rules them all” & summarizes the  synergy. The “Shift In Perspective” from Josh & the Spark slide from Ian

JW-01

The concept of Data Scientist devops is very relevant. It extends the curious case of the Data Scientist profession to the next level.

Data products live & breath in the wild, they cannot be developed and maintained with a static set of the data. Developing an R model and then throwing it over the wall for a developer to translate won’t work.  Secondly, we need models that can learn & evolve in their parameter space.

JW-02

 I agree with the current wisdom that Apache Spark is a good framework that spans the reason,model & deploy stages of data. 

Other interesting insights from Josh’s talk.

Finally,

The virtues of being really smart is massively overrated; the virtues of being able to learn faster is massively underrated

Well said Josh.
P.S: Couldn’t find the video of Ian’s talk at the Data Science London meetup. Should be an interesting talk to watch …