Oz Analytics

Data Mining in 1,000 Words

The Atlantic has published a useful short article describing what data mining is. "Everything You Wanted to Know About Data Mining but Were Afraid to Ask". Worth a read and forwarding to anyone in the general management community who needs to know this stuff.

If you are looking for some deeper background on data mining then there is a link within the article to a paper by Usama Fayyad, Gregory Piatetsky-Shapiro and Padhraic Smyth. This is a more meaty introduction to the topic but the authors illustrate the concepts with a particulalry clear (and practical) example. Published in 1996, a lot, and I mean a lot has happened in the intervening years. But when you rightly worry about the current hype around Big Data, this article reminds you that under the hyperbole the business drivers underpinning mining are decades old. Dealing with data is still one of the major challenges of business. Get used to it.

ps. If you clicked on a link within The Atlantic article and got Sesame Street, this is the wrong link. Although maybe this is a more effective way of communicating to your boss after all ... ?

Day 1 Week 7 of the CORTEX MBAnalytics

Welcome to week 7 - part of the 60 day information management refresher course from the CORTEX.

The CORTEX

Master of Business Analytics

Week 7, Day 1

Theme: The S Word

Week 7, Monday

This week's theme is statistics - yes, the dreaded S word. This is what is at the heart of the mechanics of analysis. So assuming that you have a reasonable grounding in basic stats, this week we will look at some of the current issues raised when stats is applied in the real world.

WATCH       Today's task:  Dance of the P Values  

10 minutes

YouTube video by Emeritus Professor Geoff Cumming, Statistical Cognition Laboratory, School of Psychological Science at La Trobe University, and author of 'The New Statistics' book published in 2012 by Psychology Press and Routledge, and the free software package 'ESCI' (“ESS-key”, Exploratory Software for Confidence Intervals) running under Microsoft excel.

Key takeout: This video's message is to highlight how commonly accepted techniques can mislead. To me it is the analysts responsibility to choose the most appropriate measures and not just to frame outcomes as favourably as possible. In the short-term you may gain, but beyond that you will lose every time.

As always, comments, questions and ideas are all welcome.

The MBAnalytics is just one part of the Business Intelligence CORTEX - a community of Business Intelligence, Data Warehousing, Analytics and Project Management professionals in Australia, New Zealand and Asia. Membership is free and the forums are very active with over 20,000 posts offering a uniquely Australasian viewpoint.

Day 4 Week 6 of the CORTEX MBAnalytics

Continuing week 6 - part of the 60 day information management refresher course from the CORTEX.

The CORTEX

Master of Business Analytics

Week 6, Day 4

Theme: Analytics In The Real World

Week 6, Thursday

This week's theme is application - not in the sense of a computer program - rather in the sense of 'use it or lose it'. This week we look at how analytics is being applied in the real world.

READ       Today's task Part 1:   Data Mining At The ATO 

20 minutes

'Data Mining in the ATO', presentation by Warwick Graco, Director Operational Analytics, Office of the Chief Knowledge Officer, ATO at a 2007 UTS Faculty of Information Technology  (FIT) Research Seminar

Today's task Part 2:  ATO No Longer Taxing 

10 minutes

'No Longer Taxing, Using SAS to make it easier to interact with the Australian Taxation Office', SAS Case Study, 2007

Key takeout: The Australian Tax Office (ATO) is by most measures a pretty serious user of analytics. So it is interesting to see how both traditional big-vendor tools like SAS and open source Rattle and R are co-existing. Perhaps it's 'horses for courses'? Unfortunately, little is publicly known about how the SAS and R communities interact at the ATO? Perhaps a reader of this blog knows more?

 
As always, comments, questions and ideas are all welcome. 

The MBAnalytics is just one part of the Business Intelligence CORTEX - a community of Business Intelligence, Data Warehousing, Analytics and Project Management professionals in Australia, New Zealand and Asia. Membership is free and the forums are very active with over 20,000 posts offering a uniquely Australasian viewpoint.

The Data Journalism Process

As a part of the the 60 day information management refresher course from the CORTEX, I recently blogged about The Guardian newspaper's outstanding work leveraging data. I have done a bit of cutting and pasting and reconstructed the process referred to by Simon Rogers, Editor at The Guardian, in his talk on September 21st, 2011. The talk is available on my blog here.

Click on the image to display a larger version.

Yes, there is a lot of manual work, but it is interesting that The Guardian sees so much value being generated that they are prepared to invest the time and money. I recommend bookmarking their Data Store section.

Day 3 Week 6 of the CORTEX MBAnalytics

Continuing week 6 - part of the 60 day information management refresher course from the CORTEX.

The CORTEX

Master of Business Analytics

Week 6, Day 3

Theme: Analytics In The Real World

Week 6, Wednesday

This week's theme is application - not in the sense of a computer program - rather in the sense of 'use it or lose it'. This week we look at how analytics is being applied in the real world.

WATCH    Today's task:  Rich reward for data miners  

10 minutes

'Rich reward for data miners', by Glenn Mulcaster, The Sydney Morning Herald, September 26, 2011

Key takeout: Why should an organisation pay for a small team of analysts when it can run a competition that attract some of the greatest analytic minds in the world. This is not a one-off example. Netflix and others have generated serious returns from other competitions.

As always, comments, questions and ideas are all welcome. 

The MBAnalytics is just one part of the Business Intelligence CORTEX - a community of Business Intelligence, Data Warehousing, Analytics and Project Management professionals in Australia, New Zealand and Asia. Membership is free and the forums are very active with over 20,000 posts offering a uniquely Australasian viewpoint.

Day 2 Week 6 of the CORTEX MBAnalytics

Continuing week 6 - part of the 60 day information management refresher course from the CORTEX.

The CORTEX

Master of Business Analytics

Week 6, Day 2

Theme: Analytics In The Real World

Week 6, Tuesday

This week's theme is application - not in the sense of a computer program - rather in the sense of 'use it or lose it'. This week we look at how analytics is being applied in the real world.

WATCH    Today's task:  Data-Driven Journalism at the Guardian    Streamed

17 minutes

'Data-Driven Journalism at the Guardian', presentation at the Strata New York Summit 2011, Simon Rogers, Editor at The Guardian, September 21st, 2011

Key takeout: The Guardian has reinvented itself as one of the leading users of data analysis while keeping their website free to all. This talk gives you a good overview of how data is driving some of their journalism and how their readers engage with the newspaper.

As always, comments, questions and ideas are all welcome. 

The MBAnalytics is just one part of the Business Intelligence CORTEX - a community of Business Intelligence, Data Warehousing, Analytics and Project Management professionals in Australia, New Zealand and Asia. Membership is free and the forums are very active with over 20,000 posts offering a uniquely Australasian viewpoint.

Day 1 Week 6 of the CORTEX MBAnalytics

Welcome to the start of week 6 - part of the 60 day information management refresher course from the CORTEX.

The CORTEX

Master of Business Analytics

Week 6, Day 1

Theme: Analytics In The Real World

Week 6, Monday

This week's theme is application - not in the sense of a computer program - rather in the sense of 'use it or lose it'. This week we look at how analytics is being applied in the real world.

WATCH    Today's task:  HSBC    Streamed

11 minutes

Untitled set of 3 videos on YouTube,posted by Responsetek. This is an interview with Justin Kenny, HSBC eChannel Programme Director

Key takeout: I've started the week with these videos because they give you an insight into the world of large corporate use of analytics. It is not as 'sexy' as the examples the large BI vendors are always presenting but it is all the more realistic for that. This is the type of work that the vast majority of organisations are focussed on. 

As always, comments, questions and ideas are all welcome. 

The MBAnalytics is just one part of the Business Intelligence CORTEX - a community of Business Intelligence, Data Warehousing, Analytics and Project Management professionals in Australia, New Zealand and Asia. Membership is free and the forums are very active with over 20,000 posts offering a uniquely Australasian viewpoint.

Day 5 Week 5 of the CORTEX MBAnalytics

Concluding week 5 - part of the 60 day information management refresher course from the CORTEX.

The CORTEX

Master of Business Analytics

Week 5, Day 5

Theme: Analytic Errors

Week 5, Friday

This week's theme is mistakes. Today we look into how easy it is to start out with an incorrect assumption. The example here is also very entertaining.

WATCH    Today's task:  Why do crack dealers still live with their moms?     Streamed

20 minutes

"Why do crack dealers still live with their moms?", TED Talk in February 2004 by Steven Levitt Professor of Economics at the University of Chicago and co-author of the 2005 book Freakonomics.

Key takeout: Populist - definitely. Entertaining - absolutely. In this talk Levitt examines the links between real-world events, and finds many instances where the data simply doesn't back up popular belief. How often do you re-examine your basic assumptions? Doing so when building or evaluating your hypotheses should be a mandatory part of any analysts checklist. Now go out and read the blog!

As always, comments, questions and ideas are all welcome. 

The MBAnalytics is just one part of the Business Intelligence CORTEX - a community of Business Intelligence, Data Warehousing, Analytics and Project Management professionals in Australia, New Zealand and Asia. Membership is free and the forums are very active with over 20,000 posts offering a uniquely Australasian viewpoint.

Day 4 Week 5 of the CORTEX MBAnalytics

Continuing week 5 - part of the 60 day information management refresher course from the CORTEX.

The CORTEX

Master of Business Analytics

Week 5, Day 4

Theme: Analytic Errors

Week 5, Thursday

This week's theme is mistakes. Today we look into how people make decisions and you have a choice. 

WATCH    Choice 1:  Think Twice: Michael Mauboussin at CEP 2011     Streamed

30 minutes

"Think Twice:" Michael Mauboussin at CEP's 2011 Conference, YouTube video of Michael Mauboussin talking at the Center for Effective Philanthropy's 2011 conference.

READ       Choice 2:   Think Twice A cautionary tale for investors     70 KB

5 minutes

'Michael Mauboussin's 'Think Twice': A cautionary tale for investors' article by Vishesh Kumar, on the AOL DailyFinance site, August 2009. Available on AOL here.

READ       Choice 3:  Think Twice: Harnessing the Power of Counterintuition   To buy

??? hours. How fast can you read 204 pages?

'Think Twice: Harnessing the Power of Counterintuition' by Michael Mauboussin Chief Investment Strategist at Legg Mason Capital Management and adjunct professor of finance at Columbia Business School, hardcover book published by Harvard Business School Press, October, 2009, 204 pages. Available to purchase on Amazon.

Key takeout: Fascinating insight into how we make decisions - especially insightful on the subject of mistakes and experts. Choice 1, the video, is a great way to be introduced to Mauboussin's ideas. Choice 2 is the one to take if you really can't spend half an hour watching choice 1. Choice 3 is something to keep in mind. Now go out and read the book!

As always, comments, questions and ideas are all welcome. 

The MBAnalytics is just one part of the Business Intelligence CORTEX - a community of Business Intelligence, Data Warehousing, Analytics and Project Management professionals in Australia, New Zealand and Asia. Membership is free and the forums are very active with over 20,000 posts offering a uniquely Australasian viewpoint.

Day 3 Week 5 of the CORTEX MBAnalytics

Continuing week 5 - part of the 60 day information management refresher course from the CORTEX.

The CORTEX

Master of Business Analytics

Week 5, Day 3

Theme: Analytic Errors

Week 5, Wednesday

This week's theme is mistakes. Today we look at the most common types of analytic mistakes. 

READ       Today's task:    Top 10 analytic mistakes     70 KB (also online below)

5 minutes

'Top 10 analytic mistakes', blog entry by James Taylor CEO of Decision Management Solutions and co-author of "Smart (Enough) Systems". Also available online here.

Disclaimer: James is a fellow blogger on the Smart Data Collective.

Key takeout: Short and insightful. Imagine how good his book Smart (Enough) Systems is. Now go out and read it!

As always, comments, questions and ideas are all welcome. 

The MBAnalytics is just one part of the Business Intelligence CORTEX - a community of Business Intelligence, Data Warehousing, Analytics and Project Management professionals in Australia, New Zealand and Asia. Membership is free and the forums are very active with over 20,000 posts offering a uniquely Australasian viewpoint.