Oz Analytics

The Death of BIG Business intelligence

In the second decade of the 21st century the BI platforms from the SIMO vendors (SAP, IBM, Microsoft and Oracle) are dinosaurs.

A provocative statement - but bear with me.

Over the past few years I've blogged about a lot on the emerging trends of information management: real-time, BIG data, agile BI, actionable performance management and the like. The one thing that they have in common is that they are all big. Either big in terms of expensive or big in terms of the changes that they require within organisations.

I've tried to relate many of these trends to practical actions that (for those of us without large budgets) we can still extract value from. The context of these actions is that I have often been advising medium and large commercial businesses with existing business intelligence operations. So what I have experienced may not necessarily apply to government or small organisations.

Frequently my advice has involved technologies from one or more of the BI platforms from the SIMO vendors. SIMO and cheap are not words that go together. It's not a criticism of the platforms. They are what they are but I argue that their time has past because there are now a number of alternative approaches available that leverage newer low cost products and methods.

I have been increasingly looking to the benefits of open standards and web-based solutions to create low cost platforms without further investment in the SIMO vendor platforms. In the past I have used open source to extend SIMO platforms so that I didn't have to pay for the 'enterprise version' or 'additional module' that the SIMO vendors always seem to suggest when I want my platform to do something new.

An example of this was to extend our web-based dashboards to include sparklines¹. This type of graphic was not supported by the SIMO vendor product at the time but was easily extended using open source. It also saved the company the considerable expense of buying the enterprise dashboard for $250,000 + 20% annual support charge.

Now I have questioned the idea of using the SIMO platform at all. In the above scenario I today have invested in something like Tableau which although very definitely not based on open source code, is significantly cheaper to purchase and more adaptable to a variety of data sources and presentation types.

At this point, I will digress into more technical considerations. So if you become glassy-eyed at the mention of java, then it's best if you skip over the next couple of paragraphs and start reading about the example.

Open standards-based initiatives make our world of data so much more open than it once was. Everyone has heard of Big Data, but there are also dozens of open initiatives such as Mondrian (or more correctly: the Pentaho Analysis Services Community Edition) - an Online Analytical Processing (OLAP) server that enables business users to analyze large quantities of data in real-time.

Why do we care? Mondrian gives us an open standard way to access OLAP using Java on the client side. I'm not referring to XML/A the 'open' standard that Microsoft has been promoting for some years. XML/A is not truly open. Mondrian and Saiku should provide a true open driver capability to those of us building best-of-breed platforms. In time, I hope that market pressures will convince the SIMO vendors to build their own drivers for Mondrian.

A lot of brainpower has been invested over recent years improving database technologies so that they better support business intelligence activities. Combine this with the computing power now available on our desktops and in our data centres and clouds and I think you will find that best-of-breed technology stacks where you decide each layer will often do the job more sustainably and cheaper than an equivalent SIMO investment.

An Example

I can best explain the business logic behind my ideas by taking an example such as business MDM (master data management).

Old School

You purchase an MDM solution from the SIMO vendor you already have in-house. Let's take IBM in this case. So you will need to purchase the Websphere Business Glossary and probably the MetaData Server and Workbench. What's this going to cost? I'm guessing $1 million or more for an enterprise license once you factor in the specialist technical skills you will need to integrate it into your organisation.

IBM's Websphere Platform (courtesy IBM)

Now Websphere is a nice platform and the integration between DataStage, QualityStage, Information Analyzer and Business Glossary is a joy to have as a business manager of this stuff. I've just never had the business case to buy the entire platform. This is true of the majority of IBM platforms implemented across Asia-Pacific today. IBM is no exception and this is true of the other SIMO platforms. Budgets, capabilities and maturity levels almost always mean that only parts of a SIMO platform are implemented in an organisation. What a missed opportunity - but there is no point in crying over spilt milk. If the business case to implement entire platforms existed, then the SIMO vendors would be the first to let you know.

Microsoft is the exception to this rule, having a much cheaper licensing model. Unfortunately their MDM capability is very much a 'you build it' one, so the TCO (total cost of ownership) is still significant.

The challenge today is being smarter and more effective than past attempts. 

New School

Means that you leverage your existing investments - often those in web solutions.

For example, an MDM on-the-cheap solution would be to have your content management (CMS) platform hold critical business metadata. SharePoint is frequently the CMS of choice for large traditional corporations. Even if not already implemented, Using SharePoint to build a custom workflow and collaboration solution is pretty straightforward. So with a modest effort, the CMS can support an intranet-based community of business owners to create and maintain business metadata.

Making use of this business metadata is also possible. Continuing with the MDM scenario above, and assuming you use BusinessObjects, Cognos or similar, then you can export the business metadata easily to universes/cubes so that it is available to report writers, users and decision makers. This will drive the use of standard definitions of data and measures and make enterprise performance management easier.

We are also reaching the point where the presentation layer products (reports, dashboards and scorecards) from the SIMO vendors are now intelligently integrating back into - you guessed it - the CMS's like SharePoint. So production and consumption of business metadata is done on the most widespread platform available within the organisation.

This takes BI and IM out of the analytic ghetto and integrates it seamlessly into the corporate intranet - and beyond.

Cost? A conservative $100,000.

 Conclusions

What's the overall result? You get 50% or more of the benefits at 15% or less of the cost. Ongoing maintenance is also less. Most crucial of all is that the cheaper option is also the one most likely to be funded by an organisation. Why? Simple: it's CHEAPER.

This blog started with the statement that the SIMO vendors were dinosaurs. This is not quite true, but as my example above illustrates, it is now time to integrate your existing SIMO investments into your corporate intranet. Not by buying more SIMO products. Instead use more open technologies to integrate them into the intranet. Treat the SIMO-based data and report repositories as services to be consumed via a web front-end that does not require those extra SIMO products.

I thought about calling this approach 'Poorman's BI' and 'Lagging-edge BI' but I will stick with another term that I have used in the past: 'Slow BI'. It is slow in a good way because I have seen this approach allow an organisation to build a level of support for BI that is solid, broadbased and delivering benefits that the organisation can measure and acknowledge.

Slow BI provides an enduring organisational and technological platform that focuses limited resources on delivering practical business capabilities. These become part of the operational business and decision making processes among people that are only dimly aware of business intelligence and information management. This is an important step towards the elimination of analytic ghettos and the creation of an information workforce.

Slow BI over the long term exceeds the benefits of the SIMO platforms as they frequently struggle to achieve widespread adoption outside of the analytic ghettos. Perhaps slow-and-steady beats the big-bang?

 

 ¹ Sadly Microsoft has recently tried to patent sparklines - despite the wide acknowledgement that Edward Tufte invented and defined sparkline as a term to describe "small, high resolution graphics embedded in a context of words, numbers, images".

dataFUD

 

Fear, Uncertainty and Doubting your data is a feature of organisations who are failing to effectively manage their information resources. As a result, it is one of the main issues to address when building an information management strategy for a dataFUD organisation. How do you know if your organisation is failing? Well, if you're reading this blog then there is a good chance you already know, but just in case, here are some things to think about. Do you or others in your organisation spend a lot of time:

• Performing work incorrectly or making a wrong decision?
• Re-doing work because it was previously performed incorrectly?
• Recovering from the impact of making a wrong decision?
• Taking unnecessary time to investigate the integrity of the data before using it?
• Performing calculations or reformatting the data before it can be used?
• Hunting for additional information in order to use the data?
• Losing customers because it caused work to be performed incorrectly?
• Causing unrecoverable damage?
• Missing business opportunities?
• Miscommunication within the business or with end customers and other information stakeholders?

If any of these ring true, then there is a good chance that you need to manage information better. How will this help? I've identified 7 main benefits that an organisation can realise by managing information better. They are:

1. Improved Data Quality – Poor information management leads to poor data quality and subsequently poorer business decisions. Early indicators of the business costs of non-quality data (listed in the bullet list above) point to potentially foregoing revenue as high as 10-20% of the revenue or total budget of the organisation.
2. Elimination of Duplicate Work – Removing redundancies, reducing the need for rework and therefore reducing operational costs by repairing data quality issues that create the need for manual reconciliation and avoid futile marketing and sales expenditure. On its own, by halving the costs related to poor data quality, a 5-10% saving on operating expenses can potentially be achieved.
3. Unified Vision – or Single Source Of the Truth. Organisations that break the barriers between their silos of operations can unlock value by obtaining unified views of business-critical data like customers and products, as well as apply existing information in new value adding ways such as opportunity identification and cross selling opportunities. This will improves the ability to coordinate activities between lines of business.
4. Broader insights and Better Decisions – into data across products and business units. Decision makers gain a deeper understanding of operations and an accurate picture of past and present performance as well as predictions of future performance that shape decisions.
5. Support More Complex Operations – The ability to serve numerous users with varying requirements at a lower cost. Users include employees from different areas within the organisation, external suppliers, partners and any other party that may interact with the company.
6. Continuous workflow – Continuous workflow with timely information available. This stops people having to wait for information that they require to perform their work and make decisions. Delay can cause lost opportunities or added costs. A well designed information management capability will reduce the lead time an organisation requires to go to market with something new.
7. Compliance – Aids compliance to legislation such as Sarbanes-Oxley as knowing where to find the information required will decrease the risk of non-compliance and allow for greater responsibility.

These benefits are pretty fundamental to the competitiveness and sustainability of business in the 21st century. Just remember that although information management may often be talked about in technological terms, the true key to improving performance is found by up-skilling staff so that they are able to take advantage of your technology investments. There is a reason that almost all management and information experts are talking about competing on analytics. My favourite response? A fool with a tool is still a fool - it doesn't matter how good the data quality.

 

As always with my blogs, I may be repeating ideas from one of the many books I have read, websites I visited or methods I use. I stand on the shoulders of giants and acknowledge their original thoughts when I can. I haven't checked if I am directly quoting from them, but the Mike2.0 community has been an inspiration in recent years so you may well find their ideas mixed with mine. As always, any mistakes or controversies are my responsibility alone.

How Many Analysts Does It Take To Screw In A Lightbulb?

An old joke but a question I face each time I am asked gather a team to create a sustainable analytic capability in an organisation. Well I've finally collected together my thoughts and I believe I have a model that can be used to plan the resource requirements for a range of analytic objectives.

A couple of things to be aware of if you want to use this:

• It is based on my work with large commercial organisations - those with 1,000 employees or more (sometimes 10's of thousands more).
• I almost always use specialist BI toolsets such as Business Objects, Cognos, SAS, Informatica, Infosphere, etc.
• My experience has been that the model works equally well for business as usual (BAU), project activities, or a combination of both.

It may be that this model is less applicable outside of the 'large commercial organisation' domain - but I don't yet know why it wouldn't be. I have used this model across a range of industries and have talked with many people working in many types of non-commercial environments. Nothing they have said has indicated that they have fundamentally different analytic needs from my own. I am interested if anyone reading this has a different opinion.

Anyway, enough prevaricating, here is the resource model:

 

 (Click to open a larger version in a new window)

 So the minimum team required for an organisation wanting to have a modest analytic capability is 6 people: A business sponsor and a team of 5:

• Business Subject Matter Expert who understands what the business requires analytics for - what will work, and what wont. This person is also an analytics evangelist (but usually not a 'tools' or techniques expert when they start in the role) who will work with the business as a trusted advisor. This is the person that understands the real business requirement for analytics.
• Data Steward who is responsible for data conversion and ongoing quality. This is the person that understands how to deliver business data that is 'fit for purpose'.
• Data Business Analyst who is able to understand business requirements and create new and original (innovative) solutions to the analytic requirements of the business. This is the person that understands the 'nature of data' and can apply this knowledge using diverse tools and techniques like UML, BPR, SQL to give structure and process to the solution.
• Analytics Manager and Technology Solution Architect who manages the analytics team and is the main designer of the technology solution that will be used to deliver analytic products to the business. This is the person that understands how to deliver analytic services.
• Report Writer, help desk and trainer who gets business people to use the analytic solution and then helps support them and further improve the 'analytic products'.

If you are really, really lucky, then either the Data Business Analyst or the Report Writer also have the skills to use structured analytic techniques. But 9 times out of 10, this isn't the case. Small team such as the one I describe, are effective at data integration and starting the business in using reports and dashboards for insight. Structured analytic techniques will be important a few years down the track as the organisation's capabilities grow.

The table below gives you an idea of how these roles become more specialised as the team grows:

 

 (Click to open a larger version in a new window)

So these are the roles and skills that I believe are required if you are to be successful. They do reflect my personal biases for:

• A strong business focus and led approach where IT's main role is providing the infrastructure platform
• Delivering a self-service analytic capability into the business wherever possible (as opposed to building a large team that provides analytics to the business)

Like the BICC advocated by the Gartners and Forresters of this world, I'm sure that there are a number of models that work. The above is one that has stood the test of time for me.

BI Personas

Boris Evelson of Forrester has recently published a matrix of 'BI Personas' on his blog. As he says, this is his first attempt - but the concept struck an immediate chord with me. The matrix is trying to present on a single page a view of the many different dimensions of BI creation and use that exist today.

His blog contains the original matrix of 5 types of personas across 7 dimensions. Here is my attempt at producing a version 2.0:

(Click on image to see a larger version of this summary view) 

This matrix has added 2 new roles (Data Entry and splitting Developers into 2 types) and a new dimension (Reliance on Business SME). I've also changed some of the details in the matrix so I'm not sure what Boris will think of all this. From my perspective, the changes better reflect practices that I have seen in large corporations around the world:

• I am a strong advocate of making business subject matter expertise a critical dimension in any BI capability. It delivers a huge analytic punch that contributes directly to the amount of insight (and thus value) a business gains.
• BI and Data Integration developers rarely span both skill sets although if money is tight I will go with hiring a data integrator everytime. In this case, the BI Application Developer role is often wrapped into the Power Report Writer role as they are the ones who best understand the very specific requirements of vendor-specific constructs such as Universes and Frameworks (see persona definitions below).

Medium sized businesses will undoubtedly see a collapse in the number of roles (rows) they have. Differing levels of IM maturity will have a similar effect on the number of dimensions (columns).

Here is a more detailed matrix that includes persona descriptions and other explanations (and attributions):

(Click on image to see a larger version of this detailed view)

I have also tried to define each of the personas:

• Executives - Business people who consume highly structured reports or dashboards
• Analytic Experts - Business experts in structured analytic techniques and related toolsets
• Power Report Users - Business people who consume reports or dashboards frequently and who are highly skilled in the specific BI tool used
• Average, Regular Users - Business people who consume reports or dashboards frequently (typically each week)
• Casual Users - Business people who consume reports or dashboards infrequently (typically less than weekly)
• Data Entry People - Business people working in Call Centres, etc. that use transactional applications to create and modify data
• BI Application Developers - Experts in Presentation Layer toolsets that support data access and report writing. Examples include Business Object Universes and Frameworks in Cognos
• Data Integration Developers - Experts in ETL and data quality toolsets and methods. Examples include DataStage, Informatica, SAS and SQL

I hope this helps people communicate the complexities of modern BI activities. If you have comments (good and bad) and ideas to improve the matrix then please contact me.

Please note that Boris is the originator of the model so he gets the credit. Any errors in the above I proudly own alone.

Just Tell Me What I'm Doing

In my last post I took us all in the local analytic community to task for a shared lack of ambition. Having burnt some bridges there, I thought that I'd finish the job and have a go at putting a torch to the global community at the same time. Forget about moving back to France - I've picked out a nice remote location where the 'net can't find me¹.

So enough about my future. What's my beef this time?

It's our inability to agree on a common language to hold our collective body of knowledge. What do I mean? Well let's start at the top. How do you describe your profession to others? Are you a business intelligence expert? An analytics expert? Data Warehousing? Database Marketing? Performance Manager?

I bet that if I asked 10 of us in similar functions, that we would not agree on a single description. I'm not even sure that we would all recognise that we share the same profession.

You get the picture.

Why does this matter?

This inability to agree on the most basic common language presents a serious barrier to gaining a wider appreciation of what we do. How can we expect the business community to understand us when we confuse then with a mish-mash of words seemingly designed to confuse and obfuscate?

And this problem is exacerbated as we look deeper into the language we use to describe the key concepts of our profession. Have you ever tried to create a set of standard definitions for the foundation concepts of our field of knowledge? Well good luck. I have over the last couple of years but it is not something you can do easily. There is no single source of truth for the [insert here whatever term you like to describe what you do] body of knowledge. I'm going to call this analytics.

There is no single place on the web. No professional group to codify most of the key concepts we use. Wikipedia is a good start - take a look at business intelligence:

"Business Intelligence (BI) refers to skills, processes, technologies, applications and practices used to support decision making."

I can live with that. But what about plain English definitions and examples of:

• Business Analysis Models
• Business Analytics
• Business Data Model
• Conformed dimensions
• Customer Data Integration
• Data Integration
• Data Lineage
• Data Management
• Data Mining
• Data Profiling
• Data Stewards

and the list goes on. Now I'm not advocating that we need to agree on a single definition for everything, but a definition that we can all easily access, edit and make use of would be a great start.

I don't claim to have all the answers, but I am going to try and create a solution. I will start by making a modest effort to publish definitions on this blog by assembling the best information I can glean from the web, research papers, books and presentations. I will try to acknowledge the original sources wherever I can. I will naturally be drawing deeply on the work of many past and present practitioners in the Analytic field – not least Ralph Kimball, Bill Inmon, Howard Dresner, Thomas Davenport and the rest of the BI Community. I will stand on the shoulders of giants and I'm hoping that they won't complain too much.

Into this mix I will blend my own experiences and learnings. My hope is that we will begin to have a practical set of definitions that over time will contribute to the core of our body of knowledge.

In addition to blogging individual definitions, I will collect them together on the CORTEX.

Let the arguments and the plagiarism accusations begin!

.

¹ Campbell Island - a lovely place I visited in 2008. Population 0.

Slow BI and the BIG Method Part 3

Continuing on from my recent posts introducing the BIG Method of Slow BI, here are the major steps and also the amount of effort I would usually expects each step to take (expressed as a percent of the total budget):

So under Slow BI, you expend a minimum of 65% of your budget before you start the strategic development. If you were to do this under a more traditional SDLC, you would probably lose political support for your program, quickly followed by loss of budget. And you will not have had the time to deliver anything close to justifying the money you have spent up to that point in time. On the other hand, if you launch into rapid prototyping you will know so little about the underlying data, any prototype you show the business is unlikely to be successful and/or feasible.

How can you afford to spend this 65%? Well, you can still implement and deliver value to your stakeholders prior to this but you do it as a rapid response. In a sense, you mask the main activities of your team (data analysis) with a series of tactical, quick wins that address the low hanging fruit. You identify a portfolio of deliverables that are:

• quick to achieve
• specific to a small group of stakeholders
• a solution has clear support from the benefiting stakeholder(s)
• do not require an enterprise platform
• solve a problem that is causing business pain
• if at all possible, result in a concrete benefit - i.e. dollars saved through removal of a cost item, especially staff cuts. Mitigation of an identified risk is also good (especially an audit finding/compliance risk)
• have a commitment from a stakeholder to quickly realise the benefit(s). 

What sort of projects are these in my experience? Usually they involve one or more of the following:

• Replacement of a manual  (time consuming and/or unreliable) process. Usually involving excel or Access
• Involve a single department
• Take less than 3 months from start to benefit realisation
• Have permission from IT Architecture and Risk Management.

What are the main activities and deliverables of step 03?

03 Rapid Response

03.1 Business Problem Prioritisation

03.1.1 Risk/Reward Matrix

03.1.2 Rapid Response Priorities

03.2 Solution Design

03.2.1 Impact Statement

03.2.2 Business Benefit Statement

03.2.3 Business Commitment Statement

03.3 Rapid Response Program Schedule

03.9 Solution Build and Implement.

Next post: Planning, Analysis and Design.

Previous posts in this series:

• Slow Business Intelligence
• Slow BI and the BIG Method Part 1
• Slow BI and the BIG Method Part 2

Slow BI and the BIG Method Part 2

Continuing on from my last post introducing the BIG Method of Slow BI, here are the major steps and also the amount of effort I would usually expects each step to take (expressed as a percent of the total budget):

    Step Expected % Effort

    01 Initiate 5

    02 Definition 10

    03 Rapid Response 10 - 20

    04 Plan 10 - 15

    05 Analysis and Design 30 - 40

    06 Develop 20 - 30

    07 Implement 10 - 20

    08 Benefit Realisation and Review 5

So under Slow BI, you expend a minimum of 65% of your budget before you start the strategic development. You can still implement and deliver value to your stakeholders prior to this. This will be the subject of my next post.

What are the main activities and deliverables of steps 01 and 02?

01 Initiate

01.1 Business Case

01.1.1 Project Initiation Document

01.2 Corporate Performance Management Vision

01.3 Information Management Vision

01.9 Definition Stage Gate

02 Definition

02.1 Governance

02.1.1 Business Governance

02.1.2 Technical Governance

02.2 Change Management and Communication Strategy

02.3 Analytic Strategy

02.4 Information Management Strategy

02.5 Detailed Business Requirements

02.5.1 Data Definitions/Owners/Profiles

02.5.2 Data Prioritisation

02.5.3 Performance Measure Definitions

02.5.4 Business Process Matrix

02.6 Performance Management Framework

02.6.1 Corporate Performance KPIs

02.6.2 BI Performance KPIs

02.6.3 Stage Gate Success Criteria

02.7 Information Management Framework

02.7.1 Information Management Architecture

02.7.2 Information Management and Enterprise Architecture

02.8 Knowledge Repository Solution

02.9 Rapid Response and Plan Stage Gate.

Next post: Step 03 Rapid Response.

Previous posts in this series:

• Slow Business Intelligence
• Slow BI and the BIG Method Part 1

Slow BI and the BIG Method Part 1

A while ago, I blogged about what I call Slow BI. With this I am advocating that we take time to analyse data and arrive at better decisions. Better, quicker numbers are only worth something if they are to make better decisions - not just any decision.

In my next few posts I want to present to you a Slow BI methodology for the development of business intelligence capabilities for medium and large organisations. I call this the BIG Method, after the name of a company I run - The Business Intelligence Group.

BIG, and the people involved with us, have been the driving force behind the development of a number of enterprise data warehouses and analytic platforms in Australia, Europe and the US. These experiences have led to the slow evolution of a method that I have now successfully used a couple of times and I want to share the method and see if other's find any value.

Why the BIG Method? It arose as a solution to the conflicting requirements of many platform development projects: quick delivery and data mastery. Your stakeholders want and need fast results and if they don’t get it, you wont be given the time to invest sufficiently in collecting, understanding, transforming and storing data. If you don't master your data, then any analysis based on it has a good chance of being wrong or poorly interpreted. Both can lead to poor decisions and the failure of the BI platform.

Slow BI is a method to achieve both of these apparently conflicting needs.

The key features that make the BIG Method uniquely suited to BI are:

• Rapid tactical responses
• Data centric
• Analysis intensive
• Enterprise focus
• Portfolio project management.

The first 3 features are also why the BIG Method differs from most other SDLCs. Of course, the BIG Method owes a lot to other SDLCs - especially in respect to waterfall and rapid prototyping. It also incorporates aspects of iterative development techniques.

So what is the BIG Method?

Over the next few posts, I will talk about the stages, key activities and deliverables of the method. If anyone can contribute with ideas and criticism, then please do. I don't have all the answers, but the BIG Method has worked for me.