Oz Analytics

The Data Journalism Process

As a part of the the 60 day information management refresher course from the CORTEX, I recently blogged about The Guardian newspaper's outstanding work leveraging data. I have done a bit of cutting and pasting and reconstructed the process referred to by Simon Rogers, Editor at The Guardian, in his talk on September 21st, 2011. The talk is available on my blog here.

Click on the image to display a larger version.

Yes, there is a lot of manual work, but it is interesting that The Guardian sees so much value being generated that they are prepared to invest the time and money. I recommend bookmarking their Data Store section.

Day 2 Week 6 of the CORTEX MBAnalytics

Continuing week 6 - part of the 60 day information management refresher course from the CORTEX.

The CORTEX

Master of Business Analytics

Week 6, Day 2

Theme: Analytics In The Real World

Week 6, Tuesday

This week's theme is application - not in the sense of a computer program - rather in the sense of 'use it or lose it'. This week we look at how analytics is being applied in the real world.

WATCH    Today's task:  Data-Driven Journalism at the Guardian    Streamed

17 minutes

'Data-Driven Journalism at the Guardian', presentation at the Strata New York Summit 2011, Simon Rogers, Editor at The Guardian, September 21st, 2011

Key takeout: The Guardian has reinvented itself as one of the leading users of data analysis while keeping their website free to all. This talk gives you a good overview of how data is driving some of their journalism and how their readers engage with the newspaper.

As always, comments, questions and ideas are all welcome. 

The MBAnalytics is just one part of the Business Intelligence CORTEX - a community of Business Intelligence, Data Warehousing, Analytics and Project Management professionals in Australia, New Zealand and Asia. Membership is free and the forums are very active with over 20,000 posts offering a uniquely Australasian viewpoint.

Crowdsourced Insight

I have just been reading David Cameron's blog where he talks about an interesting hack from India. You can read the story here but in short it is about Arun Ganesh, a student at the National Institute of Design in Bangalore, who was frustrated that the poor quality of the bus timetable for Chennai. With over 5,000 bus routes this was a real world problem for Chennai's 4 million residents. How did Arun solve this? He created his own map, crowd-sourced data from his fellow bus passengers and then published the new improved map along with an accompanying app BusRoutes.in for interactive route planning. The maps use OpenStreetMap - a free editable map of the whole world. In 3 days he had crowd-sourced enough data to kick-start everything. An impressive solution to an everyday data problem.

I know that corporate crowdsourcing for R&D is nothing new and prediction markets are gaining some traction. Take a look also at InnoCentive for some very serious idea outsourcing of industrial problems and Brand Tag for insight into what people really think of your brand.

I wonder if it is possible to apply similar ingenuity for information management problems within the corporation? What problems am I talking about? For me today in my role they include:

• Creating a single business glossary across a conglomerate of businesses
• Building a knowledge portal to collect information useful to management report writers and users
• Identifying solutions to complex analytic problems such as:
  • predicting propensities to substitute physical products with digital ones
  • measures of copyright protected revenues.

My initial thoughts are to invest in the following capabilities:

• Business Glossary - set-up an intranet portal (probably using SharePoint because that's what we use internally) to publish glossary term definitions and link these to discussion forums where anyone who is interested can discuss, provide feedback and mediate agreement on standard terms and definitions
• Knowledge Portal - another portal linked directly into our reporting platforms (SAP, Oracle, TM1, etc.) so that our:
  • Decision makers have access to use and can contribute to wiki's, FAQ's, known issues and documentation (tutorials, reporting guidelines, etc.) about our information assets and related business processes.
  • Project Staff and Analysts have access to the scope, structure and lineage of data from the report semantic layer back through the data marts and warehouses and into the source systems where data is created.
• Analytic Solutions - you guessed it: YAP (yet-another-portal) that lets people publish data sets that have either been purchased from 3rd parties (such as market research) or contain discovered insight that may be useful to others.

My hope is that these investments will deliver not just better support to our decision makers but also better engagement with the information they use. Give them a sense that they not only own their management reporting but also feel empowered to improve the information asset without relying on IT experts.

This may not be as innovative as people like Arun have achieved, but in the my corporate world these are sufficient wins. Maybe we will even create more super users in the organisation? Sounds good to me.

Looking again out into the rest of the world and taking a lateral step from directly involving your users, kaggle is a site that aims to facilitate better predictions by providing a platform for data mining, forecasting and bioinformatics competitions. As a side note, it is interesting to see that 'kagglers' favourite software is R, with SAS now a fairly distant 3rd in popularity. SPSS is even further away with only a quarter of the users that R has.

If you want to learn more about open innovation I suggest you start at Open Innovators website.

Just Tell Me What I'm Doing

In my last post I took us all in the local analytic community to task for a shared lack of ambition. Having burnt some bridges there, I thought that I'd finish the job and have a go at putting a torch to the global community at the same time. Forget about moving back to France - I've picked out a nice remote location where the 'net can't find me¹.

So enough about my future. What's my beef this time?

It's our inability to agree on a common language to hold our collective body of knowledge. What do I mean? Well let's start at the top. How do you describe your profession to others? Are you a business intelligence expert? An analytics expert? Data Warehousing? Database Marketing? Performance Manager?

I bet that if I asked 10 of us in similar functions, that we would not agree on a single description. I'm not even sure that we would all recognise that we share the same profession.

You get the picture.

Why does this matter?

This inability to agree on the most basic common language presents a serious barrier to gaining a wider appreciation of what we do. How can we expect the business community to understand us when we confuse then with a mish-mash of words seemingly designed to confuse and obfuscate?

And this problem is exacerbated as we look deeper into the language we use to describe the key concepts of our profession. Have you ever tried to create a set of standard definitions for the foundation concepts of our field of knowledge? Well good luck. I have over the last couple of years but it is not something you can do easily. There is no single source of truth for the [insert here whatever term you like to describe what you do] body of knowledge. I'm going to call this analytics.

There is no single place on the web. No professional group to codify most of the key concepts we use. Wikipedia is a good start - take a look at business intelligence:

"Business Intelligence (BI) refers to skills, processes, technologies, applications and practices used to support decision making."

I can live with that. But what about plain English definitions and examples of:

• Business Analysis Models
• Business Analytics
• Business Data Model
• Conformed dimensions
• Customer Data Integration
• Data Integration
• Data Lineage
• Data Management
• Data Mining
• Data Profiling
• Data Stewards

and the list goes on. Now I'm not advocating that we need to agree on a single definition for everything, but a definition that we can all easily access, edit and make use of would be a great start.

I don't claim to have all the answers, but I am going to try and create a solution. I will start by making a modest effort to publish definitions on this blog by assembling the best information I can glean from the web, research papers, books and presentations. I will try to acknowledge the original sources wherever I can. I will naturally be drawing deeply on the work of many past and present practitioners in the Analytic field – not least Ralph Kimball, Bill Inmon, Howard Dresner, Thomas Davenport and the rest of the BI Community. I will stand on the shoulders of giants and I'm hoping that they won't complain too much.

Into this mix I will blend my own experiences and learnings. My hope is that we will begin to have a practical set of definitions that over time will contribute to the core of our body of knowledge.

In addition to blogging individual definitions, I will collect them together on the CORTEX.

Let the arguments and the plagiarism accusations begin!

.

¹ Campbell Island - a lovely place I visited in 2008. Population 0.

Forrester On BI Mashups

This is the second time in as many weeks that I am writing about a research report that has crossed my desk. This may be a trend - or a sign that I am relying on other people's opinions when I should be drawing my own conclusions. Only time will tell.

Forrester is a research organisation that I have long admired as they have a track record of producing some real insights into the business uses of information technologies. A July 2009 report entitled "Mighty Mashups: Do-It-Yourself Business Intelligence For The New Economy" by James Kobielus continues this tradition.

This report gives you a good overview of the why and how of business intelligence mashups. In essence they can give an organisation a greater capability to self-service their BI needs. This is something I believe is the key to unlocking greater value from your BI investments.

Mashups wont replace pre-canned reporting or well thought out analysis by statistical experts, but they have their place, and if you want to learn more then I recommend starting by reading this report.

Will The Cloud Democratise BI?

I read with interest the announcement that Canadian BI vendor startup Indicee has successfully completed a second round of venture capital funding. In brief, Indicee sells a cloud solution where you upload your data and it works out automatically the relationships - freeing you to concentrate on answering your business questions. Here's a video from Indicee summarising their service:

Indicee Movie

I don't know the terms of the financing agreement but the US$6 million raised is another sign that BI in the cloud is an area that we should all keep a track of.

Why The Cloud?

Here are four reasons:

1. Indicee was founded by Mark Cunningham and Fred Tummonds both originally with Crystal Services - makers of Crystal Reports. Mark was a developer of the original Crystal Reports product - possibly the most ubiquitous BI tool outside of Excel in the market today.

2. Their early seed funding included all of the original founders of Crystal Services.

3. This second round adds Granite Ventures who have also previously invested in companies such as:

• Biz360 - Hosted Analytical Applications
• C-Decisions (dba DecisionView Software) - Analytic Software
• Crowd Science - Online Marketing Research
• Lucid Imagination, Inc. - Open Source Search
• Oversight Systems - Analytic Software
• Purisma - Customer Data Management SaaS, since sold to Dun & Bradstreet
• Siebel Systems - Analytic Software, since sold to Oracle
• Skytide - Digital Media Performance Management Software
• Vignette - Content Management Software, now a public company.

4. Their pricing model. It's cheap and offers small and medium sized enterprises (SMEs) some real analytic grunt without the hassles of installing and running the software themselves:

Indicee is certainly not alone and there are other startups taking similar but distinctly unique services to market. For example, take a look at CloudSwitch, RightNow Technologies, RightScale or Good Data. Good Data offers real OLAP capabilities and runs on Amazon Web Services. It is also backed by an impressive list of people including Marc Andreessen, Tim O’Reilly, Esther Dyson and John Landry.

Many of these startups are building their services on top of the EC2 infrastructure provided by Amazon's web services.

New Choices

As we approach the end of the naughties the BI market is offering some interesting and fundamentally new choices to us:

• BI in the cloud
• SaaS
• Open source.

... and we can now choose to buy each of these services from all the big vendors (at big cost) as well as a raft of new entrants at substantially lower cost.

With prices like those from Indicee and others, we could be seeing a massive increase in the number of organisations making active use of analytics.

My Predictions?

It is still early days for SaaS and full cloud solutions - especially for larger organisations with mission critical BI. For the big boys SaaS and cloud BI will remain a niche option for a number of years. In 2010, of these 3 alternatives, open source will have the biggest impact by far.

SMEs will perhaps be the biggest beneficiaries of these market developments in 2010. As new adopters of BI they are more able to make use of the cloud - especially if the entry cost is less than AU$100 a month to launch your first analytic service.

Stay tuned.

Decision Mashups

Last week I presented my thoughts on mashups at the Data Warehouse 2.0 Asia Pacific Summit. I talked mainly about why mashups can be a useful item in your toolbox of analytic solutions and looked at some best practices.

Why Are Mashups Popular?

• They are less technical (no programming skills needed)
• Data is easier to get (XML)
• Free data tools (Google, MSFT, Intel, etc.)
• Bandwidth is available
• Web 2.0 (Internet as an interactive platform)

What Are The Benefits of Mashups?

They deliver a greater variety of:

• Data to decision makers without having to (formally) invest in data integration and warehousing
• Presentation options so that the data is better understood (greater insight).

Can enhance trust:

• If you can access data from another internal data source/warehouse that is already established
• If the external data is from government, regulatory or other trusted source.

Helps people think about data and logic (information centric)

Increases the potential size (and value) of the user community (many ‘programmers’)

Can deliver extra value to customers and partners (analytic suites)

Enhances your analytic community (encourages cooperation)

Encourages innovation (unintended uses, knowledge worker can react quickly to the business)

Changes the type of innovations (pull by the BI function becomes push from the business)

Can offer better data quality (integrity, completeness and freshness)

Saves time (faster development: days not months, ROI, lowers barrier to entry)

Low infrastructure costs (individual computers as servers)

Some Potential Issues

You are using data sets that haven’t been conformed to the same standard as your static data stores (DWH, ODS, etc.)

Users will see the quality of data (little ETL invested, master data not used)

These data sets may not be:

• Always available
• Consistent over time
• Refreshed consistently

Access security

Expectations management (“why can’t I see all my spreadsheets?”)

Application performance may suffer (inefficient queries, network traffic spikes)

Tactical developments may overtake the strategy

If done in a self-service environment:

• Standards
• Maintenance (support, documentation, ownership)
• Scalability 
• Availability

may all suffer (lose IT SDLC benefits)

Resource misallocation (budgets?) Who governs use? (multiple SSOT)

Here is a (13MB) pdf of the presentation: Decision Mashups: Strategies for Combining Structured and Unstructured Data for Better Decision Making. Please note that I have removed a couple of case studies to reduce the size of the file.

Mining The Gartner Symposium

Gartner's biggest annual get-together (Gartner Symposium/ITxpo 2009) is on in Orlando and although I generally don't think too much about it, it does (claim to) bring together over 1,600 CIO's and I was wondering what knowledge I could mine by looking at what people are tweeting, blogging and in other ways posting online.

Here's what I have found. I include some interesting snippets and also a sprinkling of 'stating the obvious'. Gartner remains a fascinating mix of deep insight and banality. I can't resist it, but you can judge the value yourself:

@lgreski: #gartnersym keynote: 75% of leaders surveyed say bad data constrains business. Too much info managed in spreadsheets.

@andreagoulet: Avg team size for big brand twitter team? 4. @jetblue :12 @wholefoods :11 @ford :6 @starbucks :2 Most common 1-2 people "on duty" #ims09

“Just as your business counterparts use fact-based business intelligence data to make decisions regarding investments and resource prioritization to achieve their goals, IT needs to gather and analyze relevant data to run the business of IT,” said the Gartner analysts. By Mike Hill, Gartner

The most blogged news of the conference comes from Gartner Analysts Carl Claunch and David Cearley’s Top 10 Strategic Technologies for 2010:

1. Cloud Computing
2. Advanced Analytics
3. Client Computing
4. IT for Green
5. Reshaping the Data Center
6. Social Computing
7. Security – Activity Monitoring
8. Flash Memory
9. Virtualization for Availability
10. Mobile Applications

Most made me yawn, but these stand-out (from a Gartner post):

#1: Cloud Computing. Cloud computing is a style of computing that characterizes a model in which providers deliver a variety of IT-enabled capabilities to consumers. Cloud-based services can be exploited in a variety of ways to develop an application or a solution. Using cloud resources does not eliminate the costs of IT solutions, but does re-arrange some and reduce others. In addition, consuming cloud services enterprises will increasingly act as cloud providers and deliver application, information or business process services to customers and business partners.

#2 Advanced Analytics. Optimization and simulation is using analytical tools and models to maximize business process and decision effectiveness by examining alternative outcomes and scenarios, before, during and after process implementation and execution. This can be viewed as a third step in supporting operational business decisions. Fixed rules and prepared policies gave way to more informed decisions powered by the right information delivered at the right time, whether through customer relationship management (CRM) or enterprise resource planning (ERP) or other applications. The new step is to provide simulation, prediction, optimization and other analytics, not simply information, to empower even more decision flexibility at the time and place of every business process action. The new step looks into the future, predicting what can or will happen.

#6 Social Computing. Workers do not want two distinct environments to support their work – one for their own work products (whether personal or group) and another for accessing “external” information. Enterprises must focus both on use of social software and social media in the enterprise and participation and integration with externally facing enterprise-sponsored and public communities. Do not ignore the role of the social profile to bring communities together.

#7 Security – Activity Monitoring. Traditionally, security has focused on putting up a perimeter fence to keep others out, but it has evolved to monitoring activities and identifying patterns that would have been missed before. Information security professionals face the challenge of detecting malicious activity in a constant stream of discrete events that are usually associated with an authorized user and are generated from multiple network, system and application sources. At the same time, security departments are facing increasing demands for ever-greater log analysis and reporting to support audit requirements. A variety of complimentary (and sometimes overlapping) monitoring and analysis tools help enterprises better detect and investigate suspicious activity – often with real-time alerting or transaction intervention. By understanding the strengths and weaknesses of these tools, enterprises can better understand how to use them to defend the enterprise and meet audit requirements.

From a presentation by Gartner Senior Analyst Eric Knipp on “Citizen Developers: When End Users Surpass the IT Department in Delivering Business Applications”:

"as much as 25 percent of business applications will be built by “citizen developers” – users outside of the scope of formal enterprise IT – by 2014."

“Future citizen-developed applications will leverage the IT investments beneath the surface,” he said. “Mashups will enable IT to focus on deeper architectural concerns, while end users will focus on wiring together services into business processes and workflows. Furthermore, citizen development introduces the opportunity for end users to address projects that IT has never had time to get to — the vast expanse of departmental and situational projects found beneath the surface.”

A recent report published by Gartner entitled Introducing Pattern-Based Strategy states: “We live in a world of patterns — competitive advantage and survival are about recognizing and acting on patterns before others. Previous strategic planning models were strained to seek changes, predict outcomes, adequately guide decisions and assist leaders in defining governance models that embrace change. Through a focus on pattern-based strategy organizations will move from being ‘reactive’ to being ‘proactive.’” The research note goes on to say, “The environment emerging from the recession demands an increased focus on detecting leading indicators of change, and on identifying and quantifying risk emerging from new patterns, rather than obsessing over lagging indicators of performance. In this way, we move from a world of ’sense and respond’ to one focused on ’seek and act.’”

"Introducing Pattern-Based Strategy," Gartner, 7 August 2009.

A cut-down version of the Gartner roadshow comes to Sydney 17-19 November 2009.

Mastering Business Intelligence with SAP 2009

Looks like I'm speaking at the Mastering Business Intelligence with SAP 2009 Conference in Sydney this October. Let me know if any of my 2 readers of this blog are going to be there so we can meet.

Here's a copy of the conference brochure:

Mastering SAP Business Intelligence with SAP 2009

The organizers are saying that demand is excellent (for the conference not my presentation). I went last year and the buzz around the event was very good. I'm looking forward to it.

Data Mashups Using Excel and Gemini

Here's a demo from Microsoft that shows the Gemini plug-in to the new version of Excel. It shows:

• how fast Excel will work with 20 million rows of data
• combining data from a warehouse with other 'private' data sets 
• out-of-the-box data mining in Excel 
• a 2 minute dashboard. 

All-in-all a pretty impressive demo that makes me wonder if this is going to make end user adoption of business intelligence suites like BOBJ or Cognos much more difficult. Is this the future of power user BI?

Here is the demo:

Download this demo