Oz Analytics

Day 2 Week 7 of the CORTEX MBAnalytics

Continuing week 7 - part of the 60 day information management refresher course from the CORTEX.

The CORTEX

Master of Business Analytics

Week 7, Day 2

Theme: The S Word

Week 7, Tuesday

This week's theme is statistics - yes, the dreaded S word. This is what is at the heart of the mechanics of analysis. So assuming that you have a reasonable grounding in basic stats, this week we will look at some of the current issues raised when stats is applied in the real world.

READ       Today's task:    Why statistics rule 

15 minutes

"Why Statistics Rule", article by Stathi Paxinos, The Age newspaper, June 29th, 2004.

Key takeout: Stats matter and you cannot be an effective analyst without a solid grounding in the subject. Most will be familiar with the book Moneyball by Michael Lewis - now an apparently very good film starring Brad Pitt "the man who made statistics sexy". Moneyball is the story of US baseball team Oakland A's general manager Billy Beane's successful attempt to put together a baseball club on a budget by employing computer-generated analysis to draft his players. This article shows how fundamental stats have become to sports - in this case in Australian AFL. Remember - this article is from 7 years ago, so it's more historical record than news now. Why should we care? Because the AFL is just one example among many industries where stats is an essential tool of management. In a way this is the justification for the whole of the CORTEX and this refresher course.

As always, comments, questions and ideas are all welcome.

The MBAnalytics is just one part of the Business Intelligence CORTEX - a community of Business Intelligence, Data Warehousing, Analytics and Project Management professionals in Australia, New Zealand and Asia. Membership is free and the forums are very active with over 25,000 posts offering a uniquely Australasian viewpoint.

Day 1 Week 7 of the CORTEX MBAnalytics

Welcome to week 7 - part of the 60 day information management refresher course from the CORTEX.

The CORTEX

Master of Business Analytics

Week 7, Day 1

Theme: The S Word

Week 7, Monday

This week's theme is statistics - yes, the dreaded S word. This is what is at the heart of the mechanics of analysis. So assuming that you have a reasonable grounding in basic stats, this week we will look at some of the current issues raised when stats is applied in the real world.

WATCH       Today's task:  Dance of the P Values  

10 minutes

YouTube video by Emeritus Professor Geoff Cumming, Statistical Cognition Laboratory, School of Psychological Science at La Trobe University, and author of 'The New Statistics' book published in 2012 by Psychology Press and Routledge, and the free software package 'ESCI' (“ESS-key”, Exploratory Software for Confidence Intervals) running under Microsoft excel.

Key takeout: This video's message is to highlight how commonly accepted techniques can mislead. To me it is the analysts responsibility to choose the most appropriate measures and not just to frame outcomes as favourably as possible. In the short-term you may gain, but beyond that you will lose every time.

As always, comments, questions and ideas are all welcome.

The MBAnalytics is just one part of the Business Intelligence CORTEX - a community of Business Intelligence, Data Warehousing, Analytics and Project Management professionals in Australia, New Zealand and Asia. Membership is free and the forums are very active with over 20,000 posts offering a uniquely Australasian viewpoint.

Day 3 Week 6 of the CORTEX MBAnalytics

Continuing week 6 - part of the 60 day information management refresher course from the CORTEX.

The CORTEX

Master of Business Analytics

Week 6, Day 3

Theme: Analytics In The Real World

Week 6, Wednesday

This week's theme is application - not in the sense of a computer program - rather in the sense of 'use it or lose it'. This week we look at how analytics is being applied in the real world.

WATCH    Today's task:  Rich reward for data miners  

10 minutes

'Rich reward for data miners', by Glenn Mulcaster, The Sydney Morning Herald, September 26, 2011

Key takeout: Why should an organisation pay for a small team of analysts when it can run a competition that attract some of the greatest analytic minds in the world. This is not a one-off example. Netflix and others have generated serious returns from other competitions.

As always, comments, questions and ideas are all welcome. 

The MBAnalytics is just one part of the Business Intelligence CORTEX - a community of Business Intelligence, Data Warehousing, Analytics and Project Management professionals in Australia, New Zealand and Asia. Membership is free and the forums are very active with over 20,000 posts offering a uniquely Australasian viewpoint.

Day 5 Week 5 of the CORTEX MBAnalytics

Concluding week 5 - part of the 60 day information management refresher course from the CORTEX.

The CORTEX

Master of Business Analytics

Week 5, Day 5

Theme: Analytic Errors

Week 5, Friday

This week's theme is mistakes. Today we look into how easy it is to start out with an incorrect assumption. The example here is also very entertaining.

WATCH    Today's task:  Why do crack dealers still live with their moms?     Streamed

20 minutes

"Why do crack dealers still live with their moms?", TED Talk in February 2004 by Steven Levitt Professor of Economics at the University of Chicago and co-author of the 2005 book Freakonomics.

Key takeout: Populist - definitely. Entertaining - absolutely. In this talk Levitt examines the links between real-world events, and finds many instances where the data simply doesn't back up popular belief. How often do you re-examine your basic assumptions? Doing so when building or evaluating your hypotheses should be a mandatory part of any analysts checklist. Now go out and read the blog!

As always, comments, questions and ideas are all welcome. 

The MBAnalytics is just one part of the Business Intelligence CORTEX - a community of Business Intelligence, Data Warehousing, Analytics and Project Management professionals in Australia, New Zealand and Asia. Membership is free and the forums are very active with over 20,000 posts offering a uniquely Australasian viewpoint.

Day 4 Week 5 of the CORTEX MBAnalytics

Continuing week 5 - part of the 60 day information management refresher course from the CORTEX.

The CORTEX

Master of Business Analytics

Week 5, Day 4

Theme: Analytic Errors

Week 5, Thursday

This week's theme is mistakes. Today we look into how people make decisions and you have a choice. 

WATCH    Choice 1:  Think Twice: Michael Mauboussin at CEP 2011     Streamed

30 minutes

"Think Twice:" Michael Mauboussin at CEP's 2011 Conference, YouTube video of Michael Mauboussin talking at the Center for Effective Philanthropy's 2011 conference.

READ       Choice 2:   Think Twice A cautionary tale for investors     70 KB

5 minutes

'Michael Mauboussin's 'Think Twice': A cautionary tale for investors' article by Vishesh Kumar, on the AOL DailyFinance site, August 2009. Available on AOL here.

READ       Choice 3:  Think Twice: Harnessing the Power of Counterintuition   To buy

??? hours. How fast can you read 204 pages?

'Think Twice: Harnessing the Power of Counterintuition' by Michael Mauboussin Chief Investment Strategist at Legg Mason Capital Management and adjunct professor of finance at Columbia Business School, hardcover book published by Harvard Business School Press, October, 2009, 204 pages. Available to purchase on Amazon.

Key takeout: Fascinating insight into how we make decisions - especially insightful on the subject of mistakes and experts. Choice 1, the video, is a great way to be introduced to Mauboussin's ideas. Choice 2 is the one to take if you really can't spend half an hour watching choice 1. Choice 3 is something to keep in mind. Now go out and read the book!

As always, comments, questions and ideas are all welcome. 

The MBAnalytics is just one part of the Business Intelligence CORTEX - a community of Business Intelligence, Data Warehousing, Analytics and Project Management professionals in Australia, New Zealand and Asia. Membership is free and the forums are very active with over 20,000 posts offering a uniquely Australasian viewpoint.

Day 2 Week 5 of the CORTEX MBAnalytics

Continuing week 5 - part of the 60 day information management refresher course from the CORTEX.

The CORTEX

Master of Business Analytics

Week 5, Day 2

Theme: Analytic Errors

Week 5, Tuesday

This week's theme is mistakes. Today we look at the common types of analytic bias and some pretty spectacular ensuing falures. 

READ       Today's task:   A Tradecraft Primer     400 Kb (also online below)

5 minutes (to read only the introduction)

'A Tradecraft Primer: Structured Analytic Techniques for Improving Intelligence Analysis', Introduction pp 1 - 3 prepared by the US Government, March 2009. Also available online here.

Key takeout: Biases are many in number and rich in variety. Of course, the single page (3) included as reading today is just the tip of the iceberg - but I have found that many practicing analysts are unaware of many of the biases on this page. This is a serious issue that is often overlooked when corporations are building training programs for analysts. Too often they focus exclusively on software tools and what I call data orientation - introducing analysts to the organisation's data, it's meaning and known issues/limitations. If as an analyst you are ignorant of the causes of bias and the techniques that can overcome them, then I question how good you can ever be.

Oh, and the rest of the primer is well worth a read.

As always, comments, questions and ideas are all welcome. 

The MBAnalytics is just one part of the Business Intelligence CORTEX - a community of Business Intelligence, Data Warehousing, Analytics and Project Management professionals in Australia, New Zealand and Asia. Membership is free and the forums are very active with over 20,000 posts offering a uniquely Australasian viewpoint.

Day 4 Week 4 of the CORTEX MBAnalytics

Continuing week 4 - part of the 60 day information management refresher course from the CORTEX.

The CORTEX

Master of Business Analytics

Week 4, Day 4

Theme: Data Visualisation

Week 4, Thursday

This week's theme is data visualisation, or perhaps more accurately, transforming data into insight. Today we have something light and easy to digest - to make up for the challenge of Magicink yesterday. 

WATCH    Today's task:  Statistical Inspiration    (streamed below)

20 minutes

'Hans Rosling shows the best stats you've ever seen' by Hans Rosling professor of global health at Sweden's Karolinska Institute and co-founder of Médecins sans Frontièrs, speaking in 2006. Available online here.

Key takeout: Rosling has developed software that can visualize virtually any data set. His talk demonstrates the powerful stories that can be created from data. Combine this with an overwhelming enthusiasm for the subject you are talking about and you have something akin to an unstoppable force of nature. It's this sort of passion that can change the world. Can you harness a little bit of this to advance analytics in your organisation? Oh, and 2 final points: 

• the Rosling software is a free download here.
• The above original presentation has been expanded and featured on the BBC here.

As always, comments, questions and ideas are all welcome. 

The MBAnalytics is just one part of the Business Intelligence CORTEX - a community of Business Intelligence, Data Warehousing, Analytics and Project Management professionals in Australia, New Zealand and Asia. Membership is free and the forums are very active with over 20,000 posts offering a uniquely Australasian viewpoint.

Head First Data Analysis

Continuing the book theme established in my last blog, here is another book that looks very interesting. I say 'looks' because I haven't purchased it yet. The book is Head First Data Analysis: A Learner's Guide to Big Numbers, Statistics, and Good Decisions by Michael Milton. It is very different from the last book I recommended.

The main differences are that it is aimed at the beginner, is highly visual, and at under 500 pages is significantly shorter.

Extensive examples make up the majority of the book and use is made of Excel, OpenOffice, and statistical computing software package R.

What is really interesting is that R is a major feature in this book. It doesn't even rate a mention in the Handbook of Statistical Analysis and Data Mining Applications.

The publisher's recommend that you first know how to use basic spreadsheet formulas before tackling the book. After reading this book they also recommend Head First Statistics in the same series. I haven't yet looked into this title.

Author Michael Milton's approach in the book is very refreshing and engaging as well as being very practical. It is a radical departure from the dry texts of most (all?) university courses on the subject. Michael has degrees in philosophy and religious ethics and this may help to explain his innovative approach.

Below are a couple of screen shots of the book to give you an idea of how topics are presented. Click on the images to open a larger version.

Using the Google book preview on publisher O'Reilly's website, you can get a real flavour of the book. I'm impressed by what I see and I have put it on my list of books to buy soon.

You can find the book here on Amazon for about AUD 35. You can also buy it from O'Reilly as a printed book (but it is much more expensive than Amazon) or as an ebook.

Statistical Analysis and Data Mining

I don't keep many actual books next to my desk these days. I have found that my hard drive has become my main knowledge repository. For those interested, everything I receive online (email, documents, spreadsheets, video, research papers, etc.) is feed into my knowledgebase using Devonthink. It's unix (Mac) software.

A rare exception to this is a a new book that has really impressed me: Handbook of Statistical Analysis and Data Mining Applications by Robert Nisbet, John Elder IV, and Gary Miner.  Available on Amazon for about AUD80.

Why has this 800+ page book squeezed its way onto my crowded desk? It's useful to a part-time data miner whose post-graduate maths and stats courses are in the dim and distant 1990s. I have found it useful in a number of ways:

• Reference Guide. Section II is a lexicon of the algorithms used in structured and unstructured (i.e. text) data mining.
• Problem Solving. Section III is a substantial how-to guide of data mining in practise. The 13 tutorials cover a wide range of problems and industries/fields.
• Mentoring. Section I is a great primer for people new to the field. I would use it to help any analyst who joins one of my teams.

I haven't yet made use of Section IV of the book (Measuring True Complexity, the "right model for the right use", Top Mistakes, and the Future of Analytics) but I know it's something I should get to.

The book is a practical guide for how to use SAS-Enterprise Miner and STATISTICA Data Miner. There is also a section on SPSS Clementine and sprinkled throughout the book are STATISTICA's C&RT, CHAID, MARSpline, and other data mining and graphical analytic tools. It's a pity that R is not included but you can't have everything.

Here's a link to the table of contents.

I don't need it every week, but when I do I'm really glad I have it to hand.

A Brief Analysis Of 100 Australian BI Projects

Yesterday I gave a talk at the Mastering Business Intelligence with SAP 2009 conference. The objective was to talk about the core reasons for 'doing' BI/IM/DWH and to identify best practises. I did this by looking at a bunch of projects that I have been involved with over the last several years. The key features of the projects are:

• Lessons from approximately 100 BI projects
• Projects were active in the period 2003 - 2009
• Tools used:
• • Analytic Tools:
  • • 50% in environments with SAP
    • 30% SAS
    • 20% with Cognos
    • 20% Microsoft
    • 20% Other
  • Underlying databases:
  • • 60% Oracle DB
    • 20% DB2
    • 20% SQLServer/MySQL
    • 10% Teradata
• 70% Australian, 30% Asian
• Mainly in Banking, Insurance and Leisure industries.

The most common benefits of the projects were the development of the following capabilities:

• Better understand current performance (issues)
• Identify efficiency opportunities (insight)
• Reliably predict future performance
• Make better decisions because they are based on correct data and insight (trusted)
• Reduce risks (less surprises)
• Enhance agility (in handling new requirements)
• Promote a shared understanding (same language)
• Reduce internal conflicts (SSOT)

Best practises observed were:

• Data Quality champions were active in both IT and BI Departments.
• Domain experts were drawn from across the enterprise
• Data owners existed for every data item and measure in the BI platform
• Data Council members are empowered to make decisions: 1. All business data and their definitions - including reference data; 2. Enterprise performance measures and their calculation basis; 3. solutions (business processes and IT); 4. Monthly meetings to solve DQ issues

Finally, some of the lessons to be drawn are that to develop these capabilities you need to keep in mind:

• BI investments often spread over several years. It is not usual for stakeholders to fund projects for this period without seeing significant returns on their investments in the first 12 months.
• The actual solution(s) is usually only known when well into the BI project (once detailed business data requirements are agreed).
• 30 – 50% of your investment in BI will be spent on meeting additional requirements that were not included in your original business case.

And perhaps most importantly: The more successful you are, the more new requirements you will have to meet - with the same original budget.

I think that the talk went OK as I had a few questions from the floor but only the audience can give you honest feedback. If you are interested, here is a slideshow of the presentation:

Download 20091020Presentation