Oz Analytics

Do you know what is bigger than Big Data?

From episode 10 of my Naked Analyst Channel on YouTube.

I think I do - and it is the ‘appification’ of analytics. What I mean by this is the reduction of a complex analytic activity such as market segmentation, down to a single button on your computer interface. Very much like the Apps on your smartphone, tablet or increasingly your desktop.

That’s what it looks like but the impacts are more profound. That’s because it makes it possible for analytics to be successfully done by people who may not understand how it works, but do understand the ‘why’ and ‘when’ they need to do it.

For example, a marketer in a company can access more sophisticated views of their campaigns without the need of a specialist analyst. Appification extends the range of analytic things that a non-specialist can do.

This appification is made possible because of three things that have emerged in recent years:

1. The rapid increase in the number and sophistication of APIs (Application Programming Interface).
2. The rise of open source analytic platforms like R (http://www.r-project.org). These platforms have created vast libraries of algorithms freely available to anyone who knows how to use the platform.
3. The shear number of people and organisations involved in creating open source analytic platforms like R.

The last enabler needs a little further explanation. R is a free software programming language and software environment for statistical computing and graphics. It contains thousands of packages (10,000?) specializing in topics like econometrics, data mining, spatial analysis, and bio-informatics. Nobody knows how many R users there are, but a reliable estimate (see http://spatial.ly/2013/06/r_activity/) puts it in the millions. Many thousands have helped R develop over the years.  I think that this sort of large-scale self-organising open source effort is beginning to teach the world how to use analytic algorithms.

The above is all supposition, but I can back this up with evidence. Here are 4 examples of algorithm markets - or at least they exhibit varying degrees of ‘algorithm marketness’.

• Dataxu - senses and reacts in real time to changing consumer behavior. Openness is at the technical core of DataXu. The DataXu platform is a flexible technology stack with open APIs that make it easy to integrate and extend functionality.
• algorithms.io - delivers machine learning for streaming data. The Algorithms.io cloud platform makes it easy to use machine learning algorithms to classify streaming data from connected devices. Turn the Internet of Things into the Internet of Action
• Snapanalytx - aim to provide predictive analytics for all and make them more accessible and affordable.
•  Algorithmia - are building a community around state-of-the-art algorithm development, where users can create, share, and build, on other algorithms, and then instantly make them available as a web service.

In conclusion, there is still one issue needing resolution before algorithm markets take-off: How will the world’s business people get data into and out of these algorithm apps? I’m not sure yet, but I think the answer will be more apps. Apps that themselves appify the transformation and loading of data into and out of algorithm apps.

Confused? Well don’t be, like most new and shiny things, what we are talking about is just the next generation of ETL - something business intelligence people like myself have been building for the last 20 or more years.

What’s old is new again? Maybe, but from my perspective the future looks exciting for analytics.

My YouTube channel contains tips from a 25 year veteran of the analytic profession.

How to Find Your First Job as an Analyst

Continuing on my YouTube Channel covering everything you wanted to know about being an analyst, business analyst, data scientist, and any other data-using role in business today.

Here is part 2 of my tips on how to land a job in analytics:

 

Part 1 was published a few days ago here and you can subscribe to the channel here as well.

As mentioned before, I've also put together a related playlist on YouTube of a few videos that include those made by others on similar topics that I have found on the net. 30 minutes of watching this whole list should give you some insight into how you can go about finding that first job. Good luck!

The Naked Analyst

I've just started a YouTube Channel covering everything you wanted to know about being an analyst, business analyst, data scientist, and any other data-using role in business today.

I'm doing this because I want to give back a little to the analytic community that has given me such a great career. In short, each video will give you (hopefully useful) tips from a 25 year veteran of the analytic profession. My hope is that people will find it a useful way to spend a couple of minutes and that it will compliment this blog.

I may live to regret the name 'The Naked Analyst' but hopefully most viewers will notice that I am, in fact, not broadcasting naked and will move on to more fertile pastures! I chose the name hoping that this will indicate my 'unaligned' status. Unaligned in that I am talking about analytics without any particular bias or influence from the myriad of vendors and consulting companies.

So no hidden agendas and no marketing push to buy the lastest tool from a vendor promising to solve all of the problems in your organisation. Not that this ever happens ...

Here are the first 2 videos. The first is a quick introduction to the channel:

 

This second video is Part 1 of some tips for people who would like to get their first job as an analyst:

 

I've also put together a related playlist on YouTube of a few videos that include those made by others on similar topics that I have found on the net. 30 minutes of watching this whole list should give you some insight into how you can go about finding that first job. Good luck!

Data Mining in 1,000 Words

The Atlantic has published a useful short article describing what data mining is. "Everything You Wanted to Know About Data Mining but Were Afraid to Ask". Worth a read and forwarding to anyone in the general management community who needs to know this stuff.

If you are looking for some deeper background on data mining then there is a link within the article to a paper by Usama Fayyad, Gregory Piatetsky-Shapiro and Padhraic Smyth. This is a more meaty introduction to the topic but the authors illustrate the concepts with a particulalry clear (and practical) example. Published in 1996, a lot, and I mean a lot has happened in the intervening years. But when you rightly worry about the current hype around Big Data, this article reminds you that under the hyperbole the business drivers underpinning mining are decades old. Dealing with data is still one of the major challenges of business. Get used to it.

ps. If you clicked on a link within The Atlantic article and got Sesame Street, this is the wrong link. Although maybe this is a more effective way of communicating to your boss after all ... ?

The Data Journalism Process

As a part of the the 60 day information management refresher course from the CORTEX, I recently blogged about The Guardian newspaper's outstanding work leveraging data. I have done a bit of cutting and pasting and reconstructed the process referred to by Simon Rogers, Editor at The Guardian, in his talk on September 21st, 2011. The talk is available on my blog here.

Click on the image to display a larger version.

Yes, there is a lot of manual work, but it is interesting that The Guardian sees so much value being generated that they are prepared to invest the time and money. I recommend bookmarking their Data Store section.

Day 3 Week 6 of the CORTEX MBAnalytics

Continuing week 6 - part of the 60 day information management refresher course from the CORTEX.

The CORTEX

Master of Business Analytics

Week 6, Day 3

Theme: Analytics In The Real World

Week 6, Wednesday

This week's theme is application - not in the sense of a computer program - rather in the sense of 'use it or lose it'. This week we look at how analytics is being applied in the real world.

WATCH    Today's task:  Rich reward for data miners  

10 minutes

'Rich reward for data miners', by Glenn Mulcaster, The Sydney Morning Herald, September 26, 2011

Key takeout: Why should an organisation pay for a small team of analysts when it can run a competition that attract some of the greatest analytic minds in the world. This is not a one-off example. Netflix and others have generated serious returns from other competitions.

As always, comments, questions and ideas are all welcome. 

The MBAnalytics is just one part of the Business Intelligence CORTEX - a community of Business Intelligence, Data Warehousing, Analytics and Project Management professionals in Australia, New Zealand and Asia. Membership is free and the forums are very active with over 20,000 posts offering a uniquely Australasian viewpoint.

Day 2 Week 6 of the CORTEX MBAnalytics

Continuing week 6 - part of the 60 day information management refresher course from the CORTEX.

The CORTEX

Master of Business Analytics

Week 6, Day 2

Theme: Analytics In The Real World

Week 6, Tuesday

This week's theme is application - not in the sense of a computer program - rather in the sense of 'use it or lose it'. This week we look at how analytics is being applied in the real world.

WATCH    Today's task:  Data-Driven Journalism at the Guardian    Streamed

17 minutes

'Data-Driven Journalism at the Guardian', presentation at the Strata New York Summit 2011, Simon Rogers, Editor at The Guardian, September 21st, 2011

Key takeout: The Guardian has reinvented itself as one of the leading users of data analysis while keeping their website free to all. This talk gives you a good overview of how data is driving some of their journalism and how their readers engage with the newspaper.

As always, comments, questions and ideas are all welcome. 

The MBAnalytics is just one part of the Business Intelligence CORTEX - a community of Business Intelligence, Data Warehousing, Analytics and Project Management professionals in Australia, New Zealand and Asia. Membership is free and the forums are very active with over 20,000 posts offering a uniquely Australasian viewpoint.

Day 3 Week 2 of the CORTEX MBAnalytics

Welcome back to our second week - part of the 60 day information management refresher course from the CORTEX.

The CORTEX

Master of Business Analytics

Week 2, Day 3

Theme: Big Data Practitioners

Week 2, Wednesday

Continuing on from our exploration of big data we look at how this is being used at Facebook.

READ       Today's task:  Facebook Engineering Scaling Analytics Solutions   120Mb

45 minutes (includes 15 minutes of Q&A)

Facebook Inc., 'Facebook Engineering Scaling Analytics Solutions Tech

Talk Facebook', Alex Himel, Engineering Manager at Facebook, March, 2011.

Key takeout: The architecture and features behind Facebook’s new realtime

analytics solutions scale to handle many millions of events across millions of

sites per day. This was achieved through the methodical elimination of road-

blocks to realtime analysis. The scale is impressive but the message is a very

familiar one: Content must be engaging and the analytics must be actionable

to have any value.

Note: The audio for the first 4 minutes of the presentation is not great.

It gets better after that.

You can also access the video stream on Facebook here.

WATCH    For extra credit:  Facebook High Scalability    200KB

10 minutes

'Facebook's New Realtime Analytics System: HBase to Process 20 Billion Events

Per Day', by Todd Hoff author of the 'High Scalability' blog, March 2011.

Key takeout: The use of social plugins in Facebook has driven an explosion of

content and connections that resonate with users. Analytics has enabled the

company to make sense - and money - out of all of this data. Todd explains

how by providing a great deal of information beyond the Facebook presentation

by Alex Himel.

I recommend becoming a regular reader of the High Scalability blog. It is an

excellent source of insight.

 Comments,questions and ideas are all welcome. 

The MBAnalytics is just one part of the Business Intelligence CORTEX - a community of Business Intelligence, Data Warehousing, Analytics and Project Management professionals in Australia, New Zealand and Asia. Membership is free and the forums are very active with over 20,000 posts offering a uniquely Australasian viewpoint.

Day 2 Week 1 of the CORTEX MBAnalytics

Part of the 60 day information management refresher course from the CORTEX.

The CORTEX

Master of Business Analytics

Week 1, Day 2

Week 1, Tuesday

 WATCH   Case Study:  Helping Retailers Understand Their Customers    40Mb

12 minutes

Keynote Presentation, 'Big Data + Mobile: Helping Retailers Understand

Who Their Customers Are', by Charlie Kim, Next Jump Inc, speaking at the

O'Reilly Where2.0 Conference, Santa Clara, CA, April 2011

Online here.

READ      For extra credit:  Managing the Data Flood    200KB

15 minutes

MIT Sloan Management Review, 'No Magic to Managing The Data Flood',

An Interview with Rick Echevarria by Michael Hopkins, March 2010.

Online here.

 Comments,questions and ideas are all welcome. 

The MBAnalytics is just one part of the Business Intelligence CORTEX - a community of Business Intelligence, Data Warehousing, Analytics and Project Management professionals in Australia, New Zealand and Asia. Membership is free and the forums are very active with over 20,000 posts offering a uniquely Australasian viewpoint.

Crowdsourced Insight

I have just been reading David Cameron's blog where he talks about an interesting hack from India. You can read the story here but in short it is about Arun Ganesh, a student at the National Institute of Design in Bangalore, who was frustrated that the poor quality of the bus timetable for Chennai. With over 5,000 bus routes this was a real world problem for Chennai's 4 million residents. How did Arun solve this? He created his own map, crowd-sourced data from his fellow bus passengers and then published the new improved map along with an accompanying app BusRoutes.in for interactive route planning. The maps use OpenStreetMap - a free editable map of the whole world. In 3 days he had crowd-sourced enough data to kick-start everything. An impressive solution to an everyday data problem.

I know that corporate crowdsourcing for R&D is nothing new and prediction markets are gaining some traction. Take a look also at InnoCentive for some very serious idea outsourcing of industrial problems and Brand Tag for insight into what people really think of your brand.

I wonder if it is possible to apply similar ingenuity for information management problems within the corporation? What problems am I talking about? For me today in my role they include:

• Creating a single business glossary across a conglomerate of businesses
• Building a knowledge portal to collect information useful to management report writers and users
• Identifying solutions to complex analytic problems such as:
  • predicting propensities to substitute physical products with digital ones
  • measures of copyright protected revenues.

My initial thoughts are to invest in the following capabilities:

• Business Glossary - set-up an intranet portal (probably using SharePoint because that's what we use internally) to publish glossary term definitions and link these to discussion forums where anyone who is interested can discuss, provide feedback and mediate agreement on standard terms and definitions
• Knowledge Portal - another portal linked directly into our reporting platforms (SAP, Oracle, TM1, etc.) so that our:
  • Decision makers have access to use and can contribute to wiki's, FAQ's, known issues and documentation (tutorials, reporting guidelines, etc.) about our information assets and related business processes.
  • Project Staff and Analysts have access to the scope, structure and lineage of data from the report semantic layer back through the data marts and warehouses and into the source systems where data is created.
• Analytic Solutions - you guessed it: YAP (yet-another-portal) that lets people publish data sets that have either been purchased from 3rd parties (such as market research) or contain discovered insight that may be useful to others.

My hope is that these investments will deliver not just better support to our decision makers but also better engagement with the information they use. Give them a sense that they not only own their management reporting but also feel empowered to improve the information asset without relying on IT experts.

This may not be as innovative as people like Arun have achieved, but in the my corporate world these are sufficient wins. Maybe we will even create more super users in the organisation? Sounds good to me.

Looking again out into the rest of the world and taking a lateral step from directly involving your users, kaggle is a site that aims to facilitate better predictions by providing a platform for data mining, forecasting and bioinformatics competitions. As a side note, it is interesting to see that 'kagglers' favourite software is R, with SAS now a fairly distant 3rd in popularity. SPSS is even further away with only a quarter of the users that R has.

If you want to learn more about open innovation I suggest you start at Open Innovators website.