Oz Analytics

Do you know what is bigger than Big Data?

From episode 10 of my Naked Analyst Channel on YouTube.

I think I do - and it is the ‘appification’ of analytics. What I mean by this is the reduction of a complex analytic activity such as market segmentation, down to a single button on your computer interface. Very much like the Apps on your smartphone, tablet or increasingly your desktop.

That’s what it looks like but the impacts are more profound. That’s because it makes it possible for analytics to be successfully done by people who may not understand how it works, but do understand the ‘why’ and ‘when’ they need to do it.

For example, a marketer in a company can access more sophisticated views of their campaigns without the need of a specialist analyst. Appification extends the range of analytic things that a non-specialist can do.

This appification is made possible because of three things that have emerged in recent years:

1. The rapid increase in the number and sophistication of APIs (Application Programming Interface).
2. The rise of open source analytic platforms like R (http://www.r-project.org). These platforms have created vast libraries of algorithms freely available to anyone who knows how to use the platform.
3. The shear number of people and organisations involved in creating open source analytic platforms like R.

The last enabler needs a little further explanation. R is a free software programming language and software environment for statistical computing and graphics. It contains thousands of packages (10,000?) specializing in topics like econometrics, data mining, spatial analysis, and bio-informatics. Nobody knows how many R users there are, but a reliable estimate (see http://spatial.ly/2013/06/r_activity/) puts it in the millions. Many thousands have helped R develop over the years.  I think that this sort of large-scale self-organising open source effort is beginning to teach the world how to use analytic algorithms.

The above is all supposition, but I can back this up with evidence. Here are 4 examples of algorithm markets - or at least they exhibit varying degrees of ‘algorithm marketness’.

• Dataxu - senses and reacts in real time to changing consumer behavior. Openness is at the technical core of DataXu. The DataXu platform is a flexible technology stack with open APIs that make it easy to integrate and extend functionality.
• algorithms.io - delivers machine learning for streaming data. The Algorithms.io cloud platform makes it easy to use machine learning algorithms to classify streaming data from connected devices. Turn the Internet of Things into the Internet of Action
• Snapanalytx - aim to provide predictive analytics for all and make them more accessible and affordable.
•  Algorithmia - are building a community around state-of-the-art algorithm development, where users can create, share, and build, on other algorithms, and then instantly make them available as a web service.

In conclusion, there is still one issue needing resolution before algorithm markets take-off: How will the world’s business people get data into and out of these algorithm apps? I’m not sure yet, but I think the answer will be more apps. Apps that themselves appify the transformation and loading of data into and out of algorithm apps.

Confused? Well don’t be, like most new and shiny things, what we are talking about is just the next generation of ETL - something business intelligence people like myself have been building for the last 20 or more years.

What’s old is new again? Maybe, but from my perspective the future looks exciting for analytics.

My YouTube channel contains tips from a 25 year veteran of the analytic profession.

Changing Tack

My 2 regular readers (thanks mum and dad!) may have noticed that the frequency of my posts has dropped dramatically since 2012. There was a reason for this - beyond my strongly held opinion that those who have nothing sensible to say should simply shut-up. That reason was/is that I have been starting a new venture and the 'stealth mode' has lasted a lot longer than I expected.

In 2012 I suffered an information crisis and I suspected that I was not alone. The crisis arrived because I was being increasingly overwhelmed with information and most of the information swamping me was not on the web. Instead it was in the form of documents and email. I pride myself as being pretty well organised with effective email filters in place and an organised corpus of professional and personal knowledge approaching 1,000,000 objects. Mostly documents of one sort or another.

I was quietly proud of my knowledge management skills but after pride cometh the fall - and in 2012 that's what happened. Let me explain how it happened and I hope the story will resonate with you.

 As a serial entrepreneur and corporate analyst, I have a broad range of professional interests. These range from all aspects of analytics and information management, and broaden out into general management and IT. I am also a bit of a pack rat and if I come across some information that looks useful I store it. For those interested, my tools of choice are Mac OSX, HoudahSpot, DevonAgent and Devonthink.

So over the last 20 years I have built up a very useful corpus of knowledge. By 2011 however I was becoming increasingly frustrated by my inability to effectively use this corpus in my daily work. Each day I need to deal with the usually pile of emails and read/respond to/create multiple documents. This presented the following problems:

1. Identifying and dealing with those emails and documents that I had to.
2. Ignoring the rest.
3. Locating past knowledge/information that would help me to deal effectively with the day's emails and documents.

So the problems were pretty much at the core of what I do.

If only there was a way for me to more efficiently decide whether-or-not a long email or document was worth me reading. Then, when I needed to make a decision or give advice, how could I tap into my corpus of knowledge?

These are the problems solved by my new venture Six Degrees Of Data.

Six Degrees Of Data is building Apps and web services that we hope turn the web on its head. Our first product will launch later this year. It is called Bridger and is an App and Web Service that intelligently evaluates, summarizes and links your documents to trusted sources on the web.

Bridger is a research assistant for busy people suffering from information overload.

We are trying to release the value currently locked inside documents because despite the phenomenal growth of the internet, 80% of the new information created each year is still unstructured – in other words it is text, graphics and video contained in things like documents, emails, Facebook shares, web pages and tweets, etc. This unstructured information is poorly integrated with the world wide web. Even if they are published to a web site, they have remained isolated islands of information – until now.

Within the last five years, the size of the average web page has more than tripled, and the number of external objects has more than doubled. The average web page is now 1 MB and growing at over 20% each year. Today the average web page has about 10 links and 600 words. So it is reasonable to say that less than 2% of the content is hyperlinked to something else.

Your documents often contain only a single link (or less) and many more words than a web page.

If you are lucky, there is a single bridge (hyperlink) from the web to your document. Think of your documents as information that is connected to the web superhighway only via an off-ramp. You can find published documents easily enough because published documents are indexed by the likes of Google and Bing. But what you can’t do is start from your document and get easily back onto the web in an intelligent way.

That’s why we see the current web-document world as a one-way off-ramp.

Bridger builds you an internet on-ramp for all of your documents – whether they are published on the web or not. This is how we can release the untapped value of the 80%.

How do we do this? Well, it’s pretty complicated technically but the good news is that you really don’t have to know about this if you don’t want to. All you need to understand is that Bridger takes your isolated documents, emails, etc. and automatically links them to over 50 billion trusted additional sources of information on the world wide web.

Drop a document on to the Bridger app and it will instantly tell you what the document is about and give you concise profiles of all the relevant people, organizations, places, technologies and more.

Your document is magically transformed into a fully integrated web body of knowledge.

And Bridger is the first of a stream of new products we re planning. Future areas that we will concentrate on are enriching our personal corpus' of knowledge, collaboration and sharing.

So my posts may remain sporadic but I hope to share more about Bridger and our journey as Six Degrees Of Data gets up and running over the coming months.

In the meantime, if you have a passion for the Semantic Web, Linked Open Data, URI's, sparkle and tuples, then give me a call.

 

Day 2 Week 7 of the CORTEX MBAnalytics

Continuing week 7 - part of the 60 day information management refresher course from the CORTEX.

The CORTEX

Master of Business Analytics

Week 7, Day 2

Theme: The S Word

Week 7, Tuesday

This week's theme is statistics - yes, the dreaded S word. This is what is at the heart of the mechanics of analysis. So assuming that you have a reasonable grounding in basic stats, this week we will look at some of the current issues raised when stats is applied in the real world.

READ       Today's task:    Why statistics rule 

15 minutes

"Why Statistics Rule", article by Stathi Paxinos, The Age newspaper, June 29th, 2004.

Key takeout: Stats matter and you cannot be an effective analyst without a solid grounding in the subject. Most will be familiar with the book Moneyball by Michael Lewis - now an apparently very good film starring Brad Pitt "the man who made statistics sexy". Moneyball is the story of US baseball team Oakland A's general manager Billy Beane's successful attempt to put together a baseball club on a budget by employing computer-generated analysis to draft his players. This article shows how fundamental stats have become to sports - in this case in Australian AFL. Remember - this article is from 7 years ago, so it's more historical record than news now. Why should we care? Because the AFL is just one example among many industries where stats is an essential tool of management. In a way this is the justification for the whole of the CORTEX and this refresher course.

As always, comments, questions and ideas are all welcome.

The MBAnalytics is just one part of the Business Intelligence CORTEX - a community of Business Intelligence, Data Warehousing, Analytics and Project Management professionals in Australia, New Zealand and Asia. Membership is free and the forums are very active with over 25,000 posts offering a uniquely Australasian viewpoint.

Day 1 Week 7 of the CORTEX MBAnalytics

Welcome to week 7 - part of the 60 day information management refresher course from the CORTEX.

The CORTEX

Master of Business Analytics

Week 7, Day 1

Theme: The S Word

Week 7, Monday

This week's theme is statistics - yes, the dreaded S word. This is what is at the heart of the mechanics of analysis. So assuming that you have a reasonable grounding in basic stats, this week we will look at some of the current issues raised when stats is applied in the real world.

WATCH       Today's task:  Dance of the P Values  

10 minutes

YouTube video by Emeritus Professor Geoff Cumming, Statistical Cognition Laboratory, School of Psychological Science at La Trobe University, and author of 'The New Statistics' book published in 2012 by Psychology Press and Routledge, and the free software package 'ESCI' (“ESS-key”, Exploratory Software for Confidence Intervals) running under Microsoft excel.

Key takeout: This video's message is to highlight how commonly accepted techniques can mislead. To me it is the analysts responsibility to choose the most appropriate measures and not just to frame outcomes as favourably as possible. In the short-term you may gain, but beyond that you will lose every time.

As always, comments, questions and ideas are all welcome.

The MBAnalytics is just one part of the Business Intelligence CORTEX - a community of Business Intelligence, Data Warehousing, Analytics and Project Management professionals in Australia, New Zealand and Asia. Membership is free and the forums are very active with over 20,000 posts offering a uniquely Australasian viewpoint.

Day 3 Week 6 of the CORTEX MBAnalytics

Continuing week 6 - part of the 60 day information management refresher course from the CORTEX.

The CORTEX

Master of Business Analytics

Week 6, Day 3

Theme: Analytics In The Real World

Week 6, Wednesday

This week's theme is application - not in the sense of a computer program - rather in the sense of 'use it or lose it'. This week we look at how analytics is being applied in the real world.

WATCH    Today's task:  Rich reward for data miners  

10 minutes

'Rich reward for data miners', by Glenn Mulcaster, The Sydney Morning Herald, September 26, 2011

Key takeout: Why should an organisation pay for a small team of analysts when it can run a competition that attract some of the greatest analytic minds in the world. This is not a one-off example. Netflix and others have generated serious returns from other competitions.

As always, comments, questions and ideas are all welcome. 

The MBAnalytics is just one part of the Business Intelligence CORTEX - a community of Business Intelligence, Data Warehousing, Analytics and Project Management professionals in Australia, New Zealand and Asia. Membership is free and the forums are very active with over 20,000 posts offering a uniquely Australasian viewpoint.

Day 5 Week 5 of the CORTEX MBAnalytics

Concluding week 5 - part of the 60 day information management refresher course from the CORTEX.

The CORTEX

Master of Business Analytics

Week 5, Day 5

Theme: Analytic Errors

Week 5, Friday

This week's theme is mistakes. Today we look into how easy it is to start out with an incorrect assumption. The example here is also very entertaining.

WATCH    Today's task:  Why do crack dealers still live with their moms?     Streamed

20 minutes

"Why do crack dealers still live with their moms?", TED Talk in February 2004 by Steven Levitt Professor of Economics at the University of Chicago and co-author of the 2005 book Freakonomics.

Key takeout: Populist - definitely. Entertaining - absolutely. In this talk Levitt examines the links between real-world events, and finds many instances where the data simply doesn't back up popular belief. How often do you re-examine your basic assumptions? Doing so when building or evaluating your hypotheses should be a mandatory part of any analysts checklist. Now go out and read the blog!

As always, comments, questions and ideas are all welcome. 

The MBAnalytics is just one part of the Business Intelligence CORTEX - a community of Business Intelligence, Data Warehousing, Analytics and Project Management professionals in Australia, New Zealand and Asia. Membership is free and the forums are very active with over 20,000 posts offering a uniquely Australasian viewpoint.

Day 4 Week 5 of the CORTEX MBAnalytics

Continuing week 5 - part of the 60 day information management refresher course from the CORTEX.

The CORTEX

Master of Business Analytics

Week 5, Day 4

Theme: Analytic Errors

Week 5, Thursday

This week's theme is mistakes. Today we look into how people make decisions and you have a choice. 

WATCH    Choice 1:  Think Twice: Michael Mauboussin at CEP 2011     Streamed

30 minutes

"Think Twice:" Michael Mauboussin at CEP's 2011 Conference, YouTube video of Michael Mauboussin talking at the Center for Effective Philanthropy's 2011 conference.

READ       Choice 2:   Think Twice A cautionary tale for investors     70 KB

5 minutes

'Michael Mauboussin's 'Think Twice': A cautionary tale for investors' article by Vishesh Kumar, on the AOL DailyFinance site, August 2009. Available on AOL here.

READ       Choice 3:  Think Twice: Harnessing the Power of Counterintuition   To buy

??? hours. How fast can you read 204 pages?

'Think Twice: Harnessing the Power of Counterintuition' by Michael Mauboussin Chief Investment Strategist at Legg Mason Capital Management and adjunct professor of finance at Columbia Business School, hardcover book published by Harvard Business School Press, October, 2009, 204 pages. Available to purchase on Amazon.

Key takeout: Fascinating insight into how we make decisions - especially insightful on the subject of mistakes and experts. Choice 1, the video, is a great way to be introduced to Mauboussin's ideas. Choice 2 is the one to take if you really can't spend half an hour watching choice 1. Choice 3 is something to keep in mind. Now go out and read the book!

As always, comments, questions and ideas are all welcome. 

The MBAnalytics is just one part of the Business Intelligence CORTEX - a community of Business Intelligence, Data Warehousing, Analytics and Project Management professionals in Australia, New Zealand and Asia. Membership is free and the forums are very active with over 20,000 posts offering a uniquely Australasian viewpoint.

Day 2 Week 5 of the CORTEX MBAnalytics

Continuing week 5 - part of the 60 day information management refresher course from the CORTEX.

The CORTEX

Master of Business Analytics

Week 5, Day 2

Theme: Analytic Errors

Week 5, Tuesday

This week's theme is mistakes. Today we look at the common types of analytic bias and some pretty spectacular ensuing falures. 

READ       Today's task:   A Tradecraft Primer     400 Kb (also online below)

5 minutes (to read only the introduction)

'A Tradecraft Primer: Structured Analytic Techniques for Improving Intelligence Analysis', Introduction pp 1 - 3 prepared by the US Government, March 2009. Also available online here.

Key takeout: Biases are many in number and rich in variety. Of course, the single page (3) included as reading today is just the tip of the iceberg - but I have found that many practicing analysts are unaware of many of the biases on this page. This is a serious issue that is often overlooked when corporations are building training programs for analysts. Too often they focus exclusively on software tools and what I call data orientation - introducing analysts to the organisation's data, it's meaning and known issues/limitations. If as an analyst you are ignorant of the causes of bias and the techniques that can overcome them, then I question how good you can ever be.

Oh, and the rest of the primer is well worth a read.

As always, comments, questions and ideas are all welcome. 

The MBAnalytics is just one part of the Business Intelligence CORTEX - a community of Business Intelligence, Data Warehousing, Analytics and Project Management professionals in Australia, New Zealand and Asia. Membership is free and the forums are very active with over 20,000 posts offering a uniquely Australasian viewpoint.

Day 4 Week 4 of the CORTEX MBAnalytics

Continuing week 4 - part of the 60 day information management refresher course from the CORTEX.

The CORTEX

Master of Business Analytics

Week 4, Day 4

Theme: Data Visualisation

Week 4, Thursday

This week's theme is data visualisation, or perhaps more accurately, transforming data into insight. Today we have something light and easy to digest - to make up for the challenge of Magicink yesterday. 

WATCH    Today's task:  Statistical Inspiration    (streamed below)

20 minutes

'Hans Rosling shows the best stats you've ever seen' by Hans Rosling professor of global health at Sweden's Karolinska Institute and co-founder of Médecins sans Frontièrs, speaking in 2006. Available online here.

Key takeout: Rosling has developed software that can visualize virtually any data set. His talk demonstrates the powerful stories that can be created from data. Combine this with an overwhelming enthusiasm for the subject you are talking about and you have something akin to an unstoppable force of nature. It's this sort of passion that can change the world. Can you harness a little bit of this to advance analytics in your organisation? Oh, and 2 final points: 

• the Rosling software is a free download here.
• The above original presentation has been expanded and featured on the BBC here.

As always, comments, questions and ideas are all welcome. 

The MBAnalytics is just one part of the Business Intelligence CORTEX - a community of Business Intelligence, Data Warehousing, Analytics and Project Management professionals in Australia, New Zealand and Asia. Membership is free and the forums are very active with over 20,000 posts offering a uniquely Australasian viewpoint.

Head First Data Analysis

Continuing the book theme established in my last blog, here is another book that looks very interesting. I say 'looks' because I haven't purchased it yet. The book is Head First Data Analysis: A Learner's Guide to Big Numbers, Statistics, and Good Decisions by Michael Milton. It is very different from the last book I recommended.

The main differences are that it is aimed at the beginner, is highly visual, and at under 500 pages is significantly shorter.

Extensive examples make up the majority of the book and use is made of Excel, OpenOffice, and statistical computing software package R.

What is really interesting is that R is a major feature in this book. It doesn't even rate a mention in the Handbook of Statistical Analysis and Data Mining Applications.

The publisher's recommend that you first know how to use basic spreadsheet formulas before tackling the book. After reading this book they also recommend Head First Statistics in the same series. I haven't yet looked into this title.

Author Michael Milton's approach in the book is very refreshing and engaging as well as being very practical. It is a radical departure from the dry texts of most (all?) university courses on the subject. Michael has degrees in philosophy and religious ethics and this may help to explain his innovative approach.

Below are a couple of screen shots of the book to give you an idea of how topics are presented. Click on the images to open a larger version.

Using the Google book preview on publisher O'Reilly's website, you can get a real flavour of the book. I'm impressed by what I see and I have put it on my list of books to buy soon.

You can find the book here on Amazon for about AUD 35. You can also buy it from O'Reilly as a printed book (but it is much more expensive than Amazon) or as an ebook.