Oz Analytics

The Death of BIG Business intelligence

In the second decade of the 21st century the BI platforms from the SIMO vendors (SAP, IBM, Microsoft and Oracle) are dinosaurs.

A provocative statement - but bear with me.

Over the past few years I've blogged about a lot on the emerging trends of information management: real-time, BIG data, agile BI, actionable performance management and the like. The one thing that they have in common is that they are all big. Either big in terms of expensive or big in terms of the changes that they require within organisations.

I've tried to relate many of these trends to practical actions that (for those of us without large budgets) we can still extract value from. The context of these actions is that I have often been advising medium and large commercial businesses with existing business intelligence operations. So what I have experienced may not necessarily apply to government or small organisations.

Frequently my advice has involved technologies from one or more of the BI platforms from the SIMO vendors. SIMO and cheap are not words that go together. It's not a criticism of the platforms. They are what they are but I argue that their time has past because there are now a number of alternative approaches available that leverage newer low cost products and methods.

I have been increasingly looking to the benefits of open standards and web-based solutions to create low cost platforms without further investment in the SIMO vendor platforms. In the past I have used open source to extend SIMO platforms so that I didn't have to pay for the 'enterprise version' or 'additional module' that the SIMO vendors always seem to suggest when I want my platform to do something new.

An example of this was to extend our web-based dashboards to include sparklines¹. This type of graphic was not supported by the SIMO vendor product at the time but was easily extended using open source. It also saved the company the considerable expense of buying the enterprise dashboard for $250,000 + 20% annual support charge.

Now I have questioned the idea of using the SIMO platform at all. In the above scenario I today have invested in something like Tableau which although very definitely not based on open source code, is significantly cheaper to purchase and more adaptable to a variety of data sources and presentation types.

At this point, I will digress into more technical considerations. So if you become glassy-eyed at the mention of java, then it's best if you skip over the next couple of paragraphs and start reading about the example.

Open standards-based initiatives make our world of data so much more open than it once was. Everyone has heard of Big Data, but there are also dozens of open initiatives such as Mondrian (or more correctly: the Pentaho Analysis Services Community Edition) - an Online Analytical Processing (OLAP) server that enables business users to analyze large quantities of data in real-time.

Why do we care? Mondrian gives us an open standard way to access OLAP using Java on the client side. I'm not referring to XML/A the 'open' standard that Microsoft has been promoting for some years. XML/A is not truly open. Mondrian and Saiku should provide a true open driver capability to those of us building best-of-breed platforms. In time, I hope that market pressures will convince the SIMO vendors to build their own drivers for Mondrian.

A lot of brainpower has been invested over recent years improving database technologies so that they better support business intelligence activities. Combine this with the computing power now available on our desktops and in our data centres and clouds and I think you will find that best-of-breed technology stacks where you decide each layer will often do the job more sustainably and cheaper than an equivalent SIMO investment.

An Example

I can best explain the business logic behind my ideas by taking an example such as business MDM (master data management).

Old School

You purchase an MDM solution from the SIMO vendor you already have in-house. Let's take IBM in this case. So you will need to purchase the Websphere Business Glossary and probably the MetaData Server and Workbench. What's this going to cost? I'm guessing $1 million or more for an enterprise license once you factor in the specialist technical skills you will need to integrate it into your organisation.

IBM's Websphere Platform (courtesy IBM)

Now Websphere is a nice platform and the integration between DataStage, QualityStage, Information Analyzer and Business Glossary is a joy to have as a business manager of this stuff. I've just never had the business case to buy the entire platform. This is true of the majority of IBM platforms implemented across Asia-Pacific today. IBM is no exception and this is true of the other SIMO platforms. Budgets, capabilities and maturity levels almost always mean that only parts of a SIMO platform are implemented in an organisation. What a missed opportunity - but there is no point in crying over spilt milk. If the business case to implement entire platforms existed, then the SIMO vendors would be the first to let you know.

Microsoft is the exception to this rule, having a much cheaper licensing model. Unfortunately their MDM capability is very much a 'you build it' one, so the TCO (total cost of ownership) is still significant.

The challenge today is being smarter and more effective than past attempts. 

New School

Means that you leverage your existing investments - often those in web solutions.

For example, an MDM on-the-cheap solution would be to have your content management (CMS) platform hold critical business metadata. SharePoint is frequently the CMS of choice for large traditional corporations. Even if not already implemented, Using SharePoint to build a custom workflow and collaboration solution is pretty straightforward. So with a modest effort, the CMS can support an intranet-based community of business owners to create and maintain business metadata.

Making use of this business metadata is also possible. Continuing with the MDM scenario above, and assuming you use BusinessObjects, Cognos or similar, then you can export the business metadata easily to universes/cubes so that it is available to report writers, users and decision makers. This will drive the use of standard definitions of data and measures and make enterprise performance management easier.

We are also reaching the point where the presentation layer products (reports, dashboards and scorecards) from the SIMO vendors are now intelligently integrating back into - you guessed it - the CMS's like SharePoint. So production and consumption of business metadata is done on the most widespread platform available within the organisation.

This takes BI and IM out of the analytic ghetto and integrates it seamlessly into the corporate intranet - and beyond.

Cost? A conservative $100,000.

 Conclusions

What's the overall result? You get 50% or more of the benefits at 15% or less of the cost. Ongoing maintenance is also less. Most crucial of all is that the cheaper option is also the one most likely to be funded by an organisation. Why? Simple: it's CHEAPER.

This blog started with the statement that the SIMO vendors were dinosaurs. This is not quite true, but as my example above illustrates, it is now time to integrate your existing SIMO investments into your corporate intranet. Not by buying more SIMO products. Instead use more open technologies to integrate them into the intranet. Treat the SIMO-based data and report repositories as services to be consumed via a web front-end that does not require those extra SIMO products.

I thought about calling this approach 'Poorman's BI' and 'Lagging-edge BI' but I will stick with another term that I have used in the past: 'Slow BI'. It is slow in a good way because I have seen this approach allow an organisation to build a level of support for BI that is solid, broadbased and delivering benefits that the organisation can measure and acknowledge.

Slow BI provides an enduring organisational and technological platform that focuses limited resources on delivering practical business capabilities. These become part of the operational business and decision making processes among people that are only dimly aware of business intelligence and information management. This is an important step towards the elimination of analytic ghettos and the creation of an information workforce.

Slow BI over the long term exceeds the benefits of the SIMO platforms as they frequently struggle to achieve widespread adoption outside of the analytic ghettos. Perhaps slow-and-steady beats the big-bang?

 

 ¹ Sadly Microsoft has recently tried to patent sparklines - despite the wide acknowledgement that Edward Tufte invented and defined sparkline as a term to describe "small, high resolution graphics embedded in a context of words, numbers, images".

Taser Your Data Warehouse

 Earlier this week I was having yet-another-end-of-year-lunch¹, this time with a bunch of friends who are all data warehouse developers. Now we could have spent the time enjoying the food and wine and talking about what we are all going to do over the holidays.

I say could because of course we didn't. Instead we talked about when a data warehouse is not a data warehouse. Go figure.

If you are still reading this, then you too must be a data warehouse developer so I will dispense with any attempt to include non-experts in the conversation. If this isn't you, just skip to the end of the article and then go to have lunch with friends who are happy to talk about the real world!

We all know that data warehousing is a term that is widely used to describe a database - any database - that somebody (dare I say a vendor?) wants you to think is somehow endowed with mystical powers that normal databases don't have.

To me, a data warehouse exists if the following three conditions are met:

1. A data warehouse is a repository of an organisation's electronically stored data.
2. It is designed to facilitate reporting and analysis.
3. Is optimised for fast data retrieval (as opposed to update) across large data sets.

There is often confusion between a data warehouse and databases used in operational systems. Here is how I think of the differences.

Operational databases are optimised to preserve data integrity while also handling the insertion of transactions into normalised databases. Operational system designers generally follow the Codd rules²of database normalisation in order to ensure data integrity.

Fully normalized database designs (that is, those satisfying the five (of the 12 in total) Codd rules related to normalisation, result in transaction data being stored in many different tables. Relational databases are efficient at managing the relationships between these tables and, in order to improve performance, older data are usually regularly purged from operational systems.

Data warehouses are optimised for data retrieval - by placing all relevant information together in a single table - and analysis - by holding historical information that shows you how the data has changed over time.

So how does this work in practise? Even more importantly, how does this relate to my recent lunch?

Well, somebody mentioned that they had read about a new cloud-based data warehouse built by Taser (yes, that fun-loving bunch that zap criminals for a living). The data warehouse captures tamper-proof video recordings of police 'incidents' and stores them for use as evidence in subsequent criminal trials. 

I argued that this isn't a data warehouse because it wasn't designed to facilitate analysis, reporting or even fast retrieval. Others argued that it supported analysis by providing the raw video footage of incidents in the same way that other warehouses provide data to business intelligence applications. Their logic was that the warehouse doesn't do the analysis but it is the analytic software that delivers the insight.

I still don't buy it and we (amicably) agreed to disagree.

On reflection, perhaps it comes down to our different worlds. I'm only interested in data warehouses because of the opportunities they give me to drive greater insight and better decisions in the business.

I think my data warehouse friends have the more modest aim of giving the business the data that makes insight possible - but it's up to the business to make smart use of this data. Isn't this just like a scientist inventing a cure and the drug company making it available to people? Sounds like a big enough challenge to me.

But hold on, doesn't that make me a drug pusher?

---

Since the lunch I have tracked-down an article about this and you can read how Taser Builds Cisco-Based Data Warehouse. In case the article disappears, you can also access it here.

¹ That's what happens to you when you stop doing actual business intelligence work and become a manager. It's a tough life.

² Ted Codd was the IBM scientist who invented the relational model for database management. Interestingly, Codd also invented OLAP back in 1993 when he published an article in Computerworld. The article was then retracted by the publisher when they found out that Codd wrote it while working as a consultant for Arbor Software, the inventors of Essbase (which later merged with Hyperion and is now Oracle).

Will The Cloud Democratise BI?

I read with interest the announcement that Canadian BI vendor startup Indicee has successfully completed a second round of venture capital funding. In brief, Indicee sells a cloud solution where you upload your data and it works out automatically the relationships - freeing you to concentrate on answering your business questions. Here's a video from Indicee summarising their service:

Indicee Movie

I don't know the terms of the financing agreement but the US$6 million raised is another sign that BI in the cloud is an area that we should all keep a track of.

Why The Cloud?

Here are four reasons:

1. Indicee was founded by Mark Cunningham and Fred Tummonds both originally with Crystal Services - makers of Crystal Reports. Mark was a developer of the original Crystal Reports product - possibly the most ubiquitous BI tool outside of Excel in the market today.

2. Their early seed funding included all of the original founders of Crystal Services.

3. This second round adds Granite Ventures who have also previously invested in companies such as:

• Biz360 - Hosted Analytical Applications
• C-Decisions (dba DecisionView Software) - Analytic Software
• Crowd Science - Online Marketing Research
• Lucid Imagination, Inc. - Open Source Search
• Oversight Systems - Analytic Software
• Purisma - Customer Data Management SaaS, since sold to Dun & Bradstreet
• Siebel Systems - Analytic Software, since sold to Oracle
• Skytide - Digital Media Performance Management Software
• Vignette - Content Management Software, now a public company.

4. Their pricing model. It's cheap and offers small and medium sized enterprises (SMEs) some real analytic grunt without the hassles of installing and running the software themselves:

Indicee is certainly not alone and there are other startups taking similar but distinctly unique services to market. For example, take a look at CloudSwitch, RightNow Technologies, RightScale or Good Data. Good Data offers real OLAP capabilities and runs on Amazon Web Services. It is also backed by an impressive list of people including Marc Andreessen, Tim O’Reilly, Esther Dyson and John Landry.

Many of these startups are building their services on top of the EC2 infrastructure provided by Amazon's web services.

New Choices

As we approach the end of the naughties the BI market is offering some interesting and fundamentally new choices to us:

• BI in the cloud
• SaaS
• Open source.

... and we can now choose to buy each of these services from all the big vendors (at big cost) as well as a raft of new entrants at substantially lower cost.

With prices like those from Indicee and others, we could be seeing a massive increase in the number of organisations making active use of analytics.

My Predictions?

It is still early days for SaaS and full cloud solutions - especially for larger organisations with mission critical BI. For the big boys SaaS and cloud BI will remain a niche option for a number of years. In 2010, of these 3 alternatives, open source will have the biggest impact by far.

SMEs will perhaps be the biggest beneficiaries of these market developments in 2010. As new adopters of BI they are more able to make use of the cloud - especially if the entry cost is less than AU$100 a month to launch your first analytic service.

Stay tuned.

Data Modelling Book

I just read a post on a CORTEX blog about a book that sounds interesting for data miners. Many thanks to Arthur for the post. Here are the details:

The Multidimensional Data Modeling Toolkit
By John Paredes, OLAP World Press (March 1, 2009), 330 pages, US$47 from Amazon, ISBN: 978-0981775302

Table of Contents

Part I: Preparing for the Journey

• Oveview
• What is OLAP?
• What is the Oracle OLAP DML?
• Pit Stop: Working with Analytic Workspaces

Part II: Mastering Dimensions

• Creating Dimensions
• Describing Dimension Members with Labels
• Categorizing Dimension Members with Attributes
• Organizing Dimension Members into Hierarchies
• Unnatural, Ragged, and Other Hierarchy Variations

Part III: Designing Multidimensional Data

• Storing Multidimensional Data in the Analytic Workspace
• Calculating Data On-the-Fly
• Filtering, Relationships, and Combinations
• Working with Sets
• Aggregations

Part IV: Graphical User Interfaces for OLAP

• Exploring Metadata
• The Star Schema and Beyond
• Selectors

Part V: OLAP DML Programming

• Writing DML Programs
• Programs for Getting Data In and Out of an Analytic Workspace
• Tune Up: Working with Analytic Workspaces

Part VI: Analytical Alchemy

• Design Principles for Creating Multidimensional Information
• Migration Analysis: Showing Changes in Customer Choices
• The Relationship between OLAP, Statistics, and Data Mining

Available on Amazon or directly from the author as an ebook. The book has been written from an Oracle 10g perspective but from what I can gather online most of the content is relevant for any RDBMS.

10 Signs That You Need Analytics

I live in a world where the majority of my clients have Management Information Systems (MIS) - but not Analytics. The difference? Well, MIS is (my definition):

The people, processes and technologies used by managers of an organisation to answer their questions. Management information systems differ from regular information systems because they are for analysing other information systems used in operating the organisation.

Analytics is a specialised form (or subset of) MIS that I define as:

How an organisation uses modeling (involving extensive computation) to arrive at an optimal or realistic decision based on existing data.

In other words, MIS and Analytics are both intended to support decision making and the difference between the two is that Analytics makes use of more advanced techniques to interpret data.

The above are my practical definitions that I use when creating an analytic capability in an organisation. I can already hear the shouts and objections from my former Gartner analyst colleagues and I leave further academic debate to them. These definitions work for me and maybe you if you want.

I also see that statistical analysis and data mining are closely related - but often those activities do not involve modeling to the same degree. As to Business Intelligence I see it as an umbrella term for analytics and technologies such as reporting, OLAP, data mining, business performance management, decision performance management, benchmarking, scorecarding, dashboarding, text mining, and predictive analytics.

Now that we have that out of the way, here are the top 10 signs that you need to improve your organisation's analytic capability:

1. You have to wait longer than a day for either IT or your business intelligence department to make/change a report for you.
2. Across the organisation there are more than 100 requests pending for reporting /dashboard /scorecard changes waiting for a specialist to deliver them.  
3. When you attend meetings, there are multiple numbers being quoted for the same thing - and you don't know which of them is correct. 
4. When you talk about fundamental things like transaction, account, balance or available stock - and you discover that the person you are talking to is using the same words but means something different to what you mean. 
5. You can't get an instantly understanding when glancing at a report/dashboard/scorecard and what it is telling you. 
6. The commentary is larger than the automatically generated report. 
7. The report is not generated automatically but is a handcrafted labour of love by either yourself or one of your staff, or you spend hours trying to locate the right data and then have to consolidate it manually into Excel. 
8. It takes longer than 5 minutes to view a new report. 
9. You can't access the report when and where you most need it. 
10. There are hundreds of reports available to you but you don't trust them and you spend time trying to manually validate key numbers.  

If these are the signs, then what are the solutions? That, dear readers, I leave to another blog entry ...