Oz Analytics

Data Mining in 1,000 Words

The Atlantic has published a useful short article describing what data mining is. "Everything You Wanted to Know About Data Mining but Were Afraid to Ask". Worth a read and forwarding to anyone in the general management community who needs to know this stuff.

If you are looking for some deeper background on data mining then there is a link within the article to a paper by Usama Fayyad, Gregory Piatetsky-Shapiro and Padhraic Smyth. This is a more meaty introduction to the topic but the authors illustrate the concepts with a particulalry clear (and practical) example. Published in 1996, a lot, and I mean a lot has happened in the intervening years. But when you rightly worry about the current hype around Big Data, this article reminds you that under the hyperbole the business drivers underpinning mining are decades old. Dealing with data is still one of the major challenges of business. Get used to it.

ps. If you clicked on a link within The Atlantic article and got Sesame Street, this is the wrong link. Although maybe this is a more effective way of communicating to your boss after all ... ?

The Data Journalism Process

As a part of the the 60 day information management refresher course from the CORTEX, I recently blogged about The Guardian newspaper's outstanding work leveraging data. I have done a bit of cutting and pasting and reconstructed the process referred to by Simon Rogers, Editor at The Guardian, in his talk on September 21st, 2011. The talk is available on my blog here.

Click on the image to display a larger version.

Yes, there is a lot of manual work, but it is interesting that The Guardian sees so much value being generated that they are prepared to invest the time and money. I recommend bookmarking their Data Store section.

Day 3 Week 6 of the CORTEX MBAnalytics

Continuing week 6 - part of the 60 day information management refresher course from the CORTEX.

The CORTEX

Master of Business Analytics

Week 6, Day 3

Theme: Analytics In The Real World

Week 6, Wednesday

This week's theme is application - not in the sense of a computer program - rather in the sense of 'use it or lose it'. This week we look at how analytics is being applied in the real world.

WATCH    Today's task:  Rich reward for data miners  

10 minutes

'Rich reward for data miners', by Glenn Mulcaster, The Sydney Morning Herald, September 26, 2011

Key takeout: Why should an organisation pay for a small team of analysts when it can run a competition that attract some of the greatest analytic minds in the world. This is not a one-off example. Netflix and others have generated serious returns from other competitions.

As always, comments, questions and ideas are all welcome. 

The MBAnalytics is just one part of the Business Intelligence CORTEX - a community of Business Intelligence, Data Warehousing, Analytics and Project Management professionals in Australia, New Zealand and Asia. Membership is free and the forums are very active with over 20,000 posts offering a uniquely Australasian viewpoint.

Day 2 Week 6 of the CORTEX MBAnalytics

Continuing week 6 - part of the 60 day information management refresher course from the CORTEX.

The CORTEX

Master of Business Analytics

Week 6, Day 2

Theme: Analytics In The Real World

Week 6, Tuesday

This week's theme is application - not in the sense of a computer program - rather in the sense of 'use it or lose it'. This week we look at how analytics is being applied in the real world.

WATCH    Today's task:  Data-Driven Journalism at the Guardian    Streamed

17 minutes

'Data-Driven Journalism at the Guardian', presentation at the Strata New York Summit 2011, Simon Rogers, Editor at The Guardian, September 21st, 2011

Key takeout: The Guardian has reinvented itself as one of the leading users of data analysis while keeping their website free to all. This talk gives you a good overview of how data is driving some of their journalism and how their readers engage with the newspaper.

As always, comments, questions and ideas are all welcome. 

The MBAnalytics is just one part of the Business Intelligence CORTEX - a community of Business Intelligence, Data Warehousing, Analytics and Project Management professionals in Australia, New Zealand and Asia. Membership is free and the forums are very active with over 20,000 posts offering a uniquely Australasian viewpoint.

Day 3 Week 2 of the CORTEX MBAnalytics

Welcome back to our second week - part of the 60 day information management refresher course from the CORTEX.

The CORTEX

Master of Business Analytics

Week 2, Day 3

Theme: Big Data Practitioners

Week 2, Wednesday

Continuing on from our exploration of big data we look at how this is being used at Facebook.

READ       Today's task:  Facebook Engineering Scaling Analytics Solutions   120Mb

45 minutes (includes 15 minutes of Q&A)

Facebook Inc., 'Facebook Engineering Scaling Analytics Solutions Tech

Talk Facebook', Alex Himel, Engineering Manager at Facebook, March, 2011.

Key takeout: The architecture and features behind Facebook’s new realtime

analytics solutions scale to handle many millions of events across millions of

sites per day. This was achieved through the methodical elimination of road-

blocks to realtime analysis. The scale is impressive but the message is a very

familiar one: Content must be engaging and the analytics must be actionable

to have any value.

Note: The audio for the first 4 minutes of the presentation is not great.

It gets better after that.

You can also access the video stream on Facebook here.

WATCH    For extra credit:  Facebook High Scalability    200KB

10 minutes

'Facebook's New Realtime Analytics System: HBase to Process 20 Billion Events

Per Day', by Todd Hoff author of the 'High Scalability' blog, March 2011.

Key takeout: The use of social plugins in Facebook has driven an explosion of

content and connections that resonate with users. Analytics has enabled the

company to make sense - and money - out of all of this data. Todd explains

how by providing a great deal of information beyond the Facebook presentation

by Alex Himel.

I recommend becoming a regular reader of the High Scalability blog. It is an

excellent source of insight.

 Comments,questions and ideas are all welcome. 

The MBAnalytics is just one part of the Business Intelligence CORTEX - a community of Business Intelligence, Data Warehousing, Analytics and Project Management professionals in Australia, New Zealand and Asia. Membership is free and the forums are very active with over 20,000 posts offering a uniquely Australasian viewpoint.

Day 2 Week 1 of the CORTEX MBAnalytics

Part of the 60 day information management refresher course from the CORTEX.

The CORTEX

Master of Business Analytics

Week 1, Day 2

Week 1, Tuesday

 WATCH   Case Study:  Helping Retailers Understand Their Customers    40Mb

12 minutes

Keynote Presentation, 'Big Data + Mobile: Helping Retailers Understand

Who Their Customers Are', by Charlie Kim, Next Jump Inc, speaking at the

O'Reilly Where2.0 Conference, Santa Clara, CA, April 2011

Online here.

READ      For extra credit:  Managing the Data Flood    200KB

15 minutes

MIT Sloan Management Review, 'No Magic to Managing The Data Flood',

An Interview with Rick Echevarria by Michael Hopkins, March 2010.

Online here.

 Comments,questions and ideas are all welcome. 

The MBAnalytics is just one part of the Business Intelligence CORTEX - a community of Business Intelligence, Data Warehousing, Analytics and Project Management professionals in Australia, New Zealand and Asia. Membership is free and the forums are very active with over 20,000 posts offering a uniquely Australasian viewpoint.

Crowdsourced Insight

I have just been reading David Cameron's blog where he talks about an interesting hack from India. You can read the story here but in short it is about Arun Ganesh, a student at the National Institute of Design in Bangalore, who was frustrated that the poor quality of the bus timetable for Chennai. With over 5,000 bus routes this was a real world problem for Chennai's 4 million residents. How did Arun solve this? He created his own map, crowd-sourced data from his fellow bus passengers and then published the new improved map along with an accompanying app BusRoutes.in for interactive route planning. The maps use OpenStreetMap - a free editable map of the whole world. In 3 days he had crowd-sourced enough data to kick-start everything. An impressive solution to an everyday data problem.

I know that corporate crowdsourcing for R&D is nothing new and prediction markets are gaining some traction. Take a look also at InnoCentive for some very serious idea outsourcing of industrial problems and Brand Tag for insight into what people really think of your brand.

I wonder if it is possible to apply similar ingenuity for information management problems within the corporation? What problems am I talking about? For me today in my role they include:

• Creating a single business glossary across a conglomerate of businesses
• Building a knowledge portal to collect information useful to management report writers and users
• Identifying solutions to complex analytic problems such as:
  • predicting propensities to substitute physical products with digital ones
  • measures of copyright protected revenues.

My initial thoughts are to invest in the following capabilities:

• Business Glossary - set-up an intranet portal (probably using SharePoint because that's what we use internally) to publish glossary term definitions and link these to discussion forums where anyone who is interested can discuss, provide feedback and mediate agreement on standard terms and definitions
• Knowledge Portal - another portal linked directly into our reporting platforms (SAP, Oracle, TM1, etc.) so that our:
  • Decision makers have access to use and can contribute to wiki's, FAQ's, known issues and documentation (tutorials, reporting guidelines, etc.) about our information assets and related business processes.
  • Project Staff and Analysts have access to the scope, structure and lineage of data from the report semantic layer back through the data marts and warehouses and into the source systems where data is created.
• Analytic Solutions - you guessed it: YAP (yet-another-portal) that lets people publish data sets that have either been purchased from 3rd parties (such as market research) or contain discovered insight that may be useful to others.

My hope is that these investments will deliver not just better support to our decision makers but also better engagement with the information they use. Give them a sense that they not only own their management reporting but also feel empowered to improve the information asset without relying on IT experts.

This may not be as innovative as people like Arun have achieved, but in the my corporate world these are sufficient wins. Maybe we will even create more super users in the organisation? Sounds good to me.

Looking again out into the rest of the world and taking a lateral step from directly involving your users, kaggle is a site that aims to facilitate better predictions by providing a platform for data mining, forecasting and bioinformatics competitions. As a side note, it is interesting to see that 'kagglers' favourite software is R, with SAS now a fairly distant 3rd in popularity. SPSS is even further away with only a quarter of the users that R has.

If you want to learn more about open innovation I suggest you start at Open Innovators website.

The Data Deluge

The Economist this week has an excellent 16 page special feature on 'The Data Deluge'. Well worth reading.

Mining Social Media

 The finest example of social media mining that I have found to-date is the superb site Please Robe Me.

In case you can't guess, it mines the twitter stream in realtime to republish anyone who is telling the world that they are not at home. Who would want to know this? Take a guess.

 So all you Twitter, Foursquare, Brightkite, Google Buzz, etc. users - keep that good info coming!

Check out my new site I'm-bored-and-broke-so-who-can-I-rob?.com. We feature full integration with Google Maps and their latest realtime street stealview technology^TM© - so you can watch your place being robbed as it happens via our iPhone app (a bargain at $599)!

Head First Data Analysis

Continuing the book theme established in my last blog, here is another book that looks very interesting. I say 'looks' because I haven't purchased it yet. The book is Head First Data Analysis: A Learner's Guide to Big Numbers, Statistics, and Good Decisions by Michael Milton. It is very different from the last book I recommended.

The main differences are that it is aimed at the beginner, is highly visual, and at under 500 pages is significantly shorter.

Extensive examples make up the majority of the book and use is made of Excel, OpenOffice, and statistical computing software package R.

What is really interesting is that R is a major feature in this book. It doesn't even rate a mention in the Handbook of Statistical Analysis and Data Mining Applications.

The publisher's recommend that you first know how to use basic spreadsheet formulas before tackling the book. After reading this book they also recommend Head First Statistics in the same series. I haven't yet looked into this title.

Author Michael Milton's approach in the book is very refreshing and engaging as well as being very practical. It is a radical departure from the dry texts of most (all?) university courses on the subject. Michael has degrees in philosophy and religious ethics and this may help to explain his innovative approach.

Below are a couple of screen shots of the book to give you an idea of how topics are presented. Click on the images to open a larger version.

Using the Google book preview on publisher O'Reilly's website, you can get a real flavour of the book. I'm impressed by what I see and I have put it on my list of books to buy soon.

You can find the book here on Amazon for about AUD 35. You can also buy it from O'Reilly as a printed book (but it is much more expensive than Amazon) or as an ebook.