Oz Analytics

Shiny New Toys

Just published on my YouTube channel:

 

The Naked Analyst Episode 5

This video contains some advice on how to address Big Data opportunities. Believing vendor and pundit hype is not always healthy. Test their assertions and make use of your own analytic brain - it's good practice!

Think also about the impact you can have on your colleagues when you build a solution using your current toolset. It may not be perfect, but if you can deliver a 70% solution for $0 then that's impressive.

But don't get too confident - under promise and over deliver is always better than the other way around. Start with a small pilot or test to prove viability if you can.

And just to make life difficult - there are a large number of solutions that are possible only with the new Big Data tools. Again, thinking for yourself and knowing who to trust is the best strategy of all.

Tips from a 25 year veteran of the analytic profession.

The Difference Between Analytics and Big Data

Just published on my YouTube channel:

The Difference Between Analytics and Big Data (The Naked analyst Episode 4)

 

The Death of BIG Business intelligence

In the second decade of the 21st century the BI platforms from the SIMO vendors (SAP, IBM, Microsoft and Oracle) are dinosaurs.

A provocative statement - but bear with me.

Over the past few years I've blogged about a lot on the emerging trends of information management: real-time, BIG data, agile BI, actionable performance management and the like. The one thing that they have in common is that they are all big. Either big in terms of expensive or big in terms of the changes that they require within organisations.

I've tried to relate many of these trends to practical actions that (for those of us without large budgets) we can still extract value from. The context of these actions is that I have often been advising medium and large commercial businesses with existing business intelligence operations. So what I have experienced may not necessarily apply to government or small organisations.

Frequently my advice has involved technologies from one or more of the BI platforms from the SIMO vendors. SIMO and cheap are not words that go together. It's not a criticism of the platforms. They are what they are but I argue that their time has past because there are now a number of alternative approaches available that leverage newer low cost products and methods.

I have been increasingly looking to the benefits of open standards and web-based solutions to create low cost platforms without further investment in the SIMO vendor platforms. In the past I have used open source to extend SIMO platforms so that I didn't have to pay for the 'enterprise version' or 'additional module' that the SIMO vendors always seem to suggest when I want my platform to do something new.

An example of this was to extend our web-based dashboards to include sparklines¹. This type of graphic was not supported by the SIMO vendor product at the time but was easily extended using open source. It also saved the company the considerable expense of buying the enterprise dashboard for $250,000 + 20% annual support charge.

Now I have questioned the idea of using the SIMO platform at all. In the above scenario I today have invested in something like Tableau which although very definitely not based on open source code, is significantly cheaper to purchase and more adaptable to a variety of data sources and presentation types.

At this point, I will digress into more technical considerations. So if you become glassy-eyed at the mention of java, then it's best if you skip over the next couple of paragraphs and start reading about the example.

Open standards-based initiatives make our world of data so much more open than it once was. Everyone has heard of Big Data, but there are also dozens of open initiatives such as Mondrian (or more correctly: the Pentaho Analysis Services Community Edition) - an Online Analytical Processing (OLAP) server that enables business users to analyze large quantities of data in real-time.

Why do we care? Mondrian gives us an open standard way to access OLAP using Java on the client side. I'm not referring to XML/A the 'open' standard that Microsoft has been promoting for some years. XML/A is not truly open. Mondrian and Saiku should provide a true open driver capability to those of us building best-of-breed platforms. In time, I hope that market pressures will convince the SIMO vendors to build their own drivers for Mondrian.

A lot of brainpower has been invested over recent years improving database technologies so that they better support business intelligence activities. Combine this with the computing power now available on our desktops and in our data centres and clouds and I think you will find that best-of-breed technology stacks where you decide each layer will often do the job more sustainably and cheaper than an equivalent SIMO investment.

An Example

I can best explain the business logic behind my ideas by taking an example such as business MDM (master data management).

Old School

You purchase an MDM solution from the SIMO vendor you already have in-house. Let's take IBM in this case. So you will need to purchase the Websphere Business Glossary and probably the MetaData Server and Workbench. What's this going to cost? I'm guessing $1 million or more for an enterprise license once you factor in the specialist technical skills you will need to integrate it into your organisation.

IBM's Websphere Platform (courtesy IBM)

Now Websphere is a nice platform and the integration between DataStage, QualityStage, Information Analyzer and Business Glossary is a joy to have as a business manager of this stuff. I've just never had the business case to buy the entire platform. This is true of the majority of IBM platforms implemented across Asia-Pacific today. IBM is no exception and this is true of the other SIMO platforms. Budgets, capabilities and maturity levels almost always mean that only parts of a SIMO platform are implemented in an organisation. What a missed opportunity - but there is no point in crying over spilt milk. If the business case to implement entire platforms existed, then the SIMO vendors would be the first to let you know.

Microsoft is the exception to this rule, having a much cheaper licensing model. Unfortunately their MDM capability is very much a 'you build it' one, so the TCO (total cost of ownership) is still significant.

The challenge today is being smarter and more effective than past attempts. 

New School

Means that you leverage your existing investments - often those in web solutions.

For example, an MDM on-the-cheap solution would be to have your content management (CMS) platform hold critical business metadata. SharePoint is frequently the CMS of choice for large traditional corporations. Even if not already implemented, Using SharePoint to build a custom workflow and collaboration solution is pretty straightforward. So with a modest effort, the CMS can support an intranet-based community of business owners to create and maintain business metadata.

Making use of this business metadata is also possible. Continuing with the MDM scenario above, and assuming you use BusinessObjects, Cognos or similar, then you can export the business metadata easily to universes/cubes so that it is available to report writers, users and decision makers. This will drive the use of standard definitions of data and measures and make enterprise performance management easier.

We are also reaching the point where the presentation layer products (reports, dashboards and scorecards) from the SIMO vendors are now intelligently integrating back into - you guessed it - the CMS's like SharePoint. So production and consumption of business metadata is done on the most widespread platform available within the organisation.

This takes BI and IM out of the analytic ghetto and integrates it seamlessly into the corporate intranet - and beyond.

Cost? A conservative $100,000.

 Conclusions

What's the overall result? You get 50% or more of the benefits at 15% or less of the cost. Ongoing maintenance is also less. Most crucial of all is that the cheaper option is also the one most likely to be funded by an organisation. Why? Simple: it's CHEAPER.

This blog started with the statement that the SIMO vendors were dinosaurs. This is not quite true, but as my example above illustrates, it is now time to integrate your existing SIMO investments into your corporate intranet. Not by buying more SIMO products. Instead use more open technologies to integrate them into the intranet. Treat the SIMO-based data and report repositories as services to be consumed via a web front-end that does not require those extra SIMO products.

I thought about calling this approach 'Poorman's BI' and 'Lagging-edge BI' but I will stick with another term that I have used in the past: 'Slow BI'. It is slow in a good way because I have seen this approach allow an organisation to build a level of support for BI that is solid, broadbased and delivering benefits that the organisation can measure and acknowledge.

Slow BI provides an enduring organisational and technological platform that focuses limited resources on delivering practical business capabilities. These become part of the operational business and decision making processes among people that are only dimly aware of business intelligence and information management. This is an important step towards the elimination of analytic ghettos and the creation of an information workforce.

Slow BI over the long term exceeds the benefits of the SIMO platforms as they frequently struggle to achieve widespread adoption outside of the analytic ghettos. Perhaps slow-and-steady beats the big-bang?

 

 ¹ Sadly Microsoft has recently tried to patent sparklines - despite the wide acknowledgement that Edward Tufte invented and defined sparkline as a term to describe "small, high resolution graphics embedded in a context of words, numbers, images".

Another One Bites The Dust

"We intend to nurture that culture of innovation at Sybase
and to do that we have to leave them alone"

SAP Australia managing director Tim Ebbeck

So Sybase is now part of the SAP empire.

The deal is being portrayed in the media as a mobile play, with Sybase bringing complex financial event processing, analytics and a mobile platform. SAP brings in-memory technology. As expected, SAP has been quick to emphasize that SAP remains committed to other leading database vendors.

Now I expect PR spin to feature in these announcements, but how can they play down Sybase's crown jewels - the Sybase IQ and RAP database platforms? And what happened to SAP's decade long down playing of Sybase as an analytic platform?

I guess the addition of SAP's in-memory technology into the arguement was the straw that broke the camel's back for me. I'd like to know if there is anyone not employed by SAP that thinks their in-memory technology is currently more than a niche capability at best.

In my mind, the end result of this acquisition is that the leading columnar DBMS player now becomes a decision to buy SAP. I think that this may lose them a few customers. Perhaps lost to the smaller columnar DBMS players like ParAccel and Vertica. How long they remain independent is anyone's guess. Personally, I think that in some strange way, this deal will make me take a closer look at the open source options. PostgreSQL, Ingres, or more niche technologies like One$DB and Hypertable.

In SAP's defence, I think that they are getting an excellent technology capability in Sybase. And it's not like SAP and Sybase are strangers as they have existing partnerships in mobile services (Sybase Mobile Workflow for SAP Business Suite and Sybase Mobile Sales for SAP CRM) and SAP Business One - to name just three.

Also, when you look across our region, there are additional potential benefits. For example, Sybase is doing well in China where smartphones are one of the hottest growth sectors. In Japan, the news is all about SAP attacking Oracle, and in India the move is not being seen as so important given Sybase's small user base there.

On balance, the deal is good for SAP's product line and Sybase's shareholders. The jury is out for us users.

Big Data

Several month's ago a short video appeared on YouTube with an interview of LinkedIn's Chief Scientist DJ Patil. In it he discusses how 'Big Data' impacts the practise of analytics. I've only just got around to posting about it but I am doing so now because he has some insights that I agree with and would like to share as they are still relevant. 

Big data is today most often associated with the internet superstars like Google, eBay and Amazon. There are 3 other areas with lower profiles where big data is important: intelligence (spooks, the military, etc.), scientific and academic research, and the financial markets.

Big data's future is much bigger than this because more and more areas of human activity are going to be faced with vast data sets. When you hear people talking about the growth of knowledge and statements like 'if this data were printed then the stack would grow faster than NASA’s fastest rocket', you have to remember that there is a good chance that each page of new data is adding to someone's analytic data set.

I'm not quoting the guy verbatim but here's what I heard and my takeouts to his comments:

• Open source 'big data ready' technologies like Hadoop (see my earlier blog or here) have come into their own now. Look to people with these skills over those only with SQL if you are facing big data challenges.
• We have reached a tipping point in the use of open source for commercial solutions to big data problems.
• If you want good analysts then the best place is to look is in occupations where people will already have the practical skills in manipulating big data sets: scientific fields like meteorology, oceanography and the like. I agree but this is not the only place as in my experience I also need analysts that relate well to business decision makers - i.e. those people that make commercial decisions based on the analytics. This is perhaps less important in pure tech plays like LinkedIn.
• Open source will transform the practise of analytics in the next 3 - 5 years. I think it will take longer than this to really impact the more traditional industries. I'm not happy about this but I am realistic about the difficulty in convincing business leaders that open source is a superior solution to proprietary ones. The money behind the big vendors will keep them going for a number of years yet.

One potential qualifier to DJ Patil's perspective is that although he has a very impressive big data background as a mathematician, US Department of Defence analyst ('Threat Anticipation'), and former eBay Director of Strategy and Analytics, his current employer is LinkedIn.

The core of LinkedIn's big data is structured and fairly static: profiles of people. So I'm not sure how similar their big data challenges would be to, say, those faced with processing, understanding and predicting large streams of real time data from financial markets or very large sensor arrays. On the other hand, the growth of LinkedIn communities and their related activities must generate large amounts of semi-structured data.

I also have no idea what LinkedIn's own analytic goals are beyond what DJ mentions on his own profile where he says his analytics drives product features like:

• "People You May Know"
• "Who Viewed My Profile"
• "Groups You Might Like"

Maybe somebody reading this blog knows more?

The video is on YouTube and I embed it here for convenience:

Or you can download 'DJ Patil on How Big Data Impacts Analytics' directly from this blog.

Review Of The BI Survey 8

The BI Survey is created each year by analyst Nigel Pendse of the Business Application Research Center (BARC) headquartered in Germany. BARC was founded in 1994 as a spin-off from the University of Würzburg’s Chair for Information Science.

The BI Survey is based on the world's largest independent survey of Business Intelligence and Performance Management users. For this year's report over 2600 respondents (1894 users, 256 consultants and 472 vendors) from around the world took part. I have been lucky enough to have been given a review copy and on reading it I have had two reactions:

1. This is a BIG report with over 550 pages, 350 figures and is a 34 MB pdf file.
2. It lives up to the author's claims to be comprehensive, insightful and practical.

This report is so comprehensive that it is difficult to give you a fair summary of its content. Here are a couple of details to give you an idea.

The products covered are: MicroStrategy, Infor PM OLAP, QlikTech QlikView, Board, Microsoft Analysis Services, Cognos Reporting, BusinessObjects, SAP BI/BW, WebFOCUS, Cubeware Cockpit, Cognos TM1 Server, arcplan, Microsoft Reporting Services, Panorama NovaView, Bissantz, Hyperion Essbase, Crystal Reports, Cognos Analysis, Targit, Oracle BIEE/BISEO, Actuate Platform, Microsoft Excel PivotTables, MIK, LogiXML, Oracle Discoverer, SAS, Palo, Hyperion Interactive Reporting, Microsoft PPS, ProClarity or BSM, Other Business Objects product, CyberQuery, Other Cognos product, Oracle OLAP Option, Hyperion Planning, SAP BPC (OutlookSoft), Hyperion Financial Management.

There are of course many hundreds of other tools out there that are not included. This report targets the top 30 - 40 globally. Take a look at the CORTEX forum on local vendors for examples.

Here are the major sections of the report:

1. Introduction¹
2. The sample
3. Products included
4. Age Profiles
5. The Business Benefits Index
6. The purchase cycle
7. The BI ownership experience
8. Vendor effectiveness
9. Implementation
10. Timescales
11. What goes wrong?
12. Applications
13. Web BI
14. Server platforms
15. Client/server combos
16. Source databases
17. Data Volumes
18. Performance at the speed of thought?
19. The customers’ verdict dashboards
20. Appendix: Survey questionnaire

Each section contains extensive and useful (i.e. insightful) analysis from the survey.

The report is not cheap at US$5,000 but if I was in the market to buy a BI platform then the cost would not be a barrier. After all what are the alternatives? I don't know of any, although options often taken are:

1. Employ a consultant - directly or through a firm like Accenture, Deloittes, etc. You would never get someone with recent experience of all 35+ products included in the BI Survey. In my defence (as a working consultant), I believe that the BI Survey (facts and insight) and a consultant (local knowledge and experience) can be a winning combination.
2. Subscribe to Gartner or Forrester to get access to their research. They certainly publish some good stuff but it can be a little light on detail unless you pay a lot of dollars for the detailed industry analysis. Either way, it will cost a lot more than the 5K for the BI Survey. This is not really a fair comparison as they are addressing different needs. The BI Survey is focussed on an investment decision. Gartner and Forrester both offer a stream of research that covers many more business needs².
3. Do the research yourself and speak to the vendors. You will need to do this anyway as part of your normal purchasing process. Trusting vendors exclusively to advise you is a brave strategy.

 It is a little easier to say why the BI Survey report could be useful to anyone making BI related investments as the report gives you:

• Detailed evaluations (and sometimes pricing) of all the major BI tools.
• Deep perspective on how the tools are being used by your peers and across different industries.
• Customer Verdicts that provide a quick snapshot of how a particular product was rated in the last edition of The BI Survey. 23 products are assessed across a set of 26 normalized root KPIs (see a sample below). This is a new feature and I hope to see it retained in subsequent surveys.

 

So you can tell I like the report. Are there any ways to improve it? Well yes, of course there are. Here are my suggestions for the 2009 BI Survey:

• Include more on open source and cloud (SaaS) based solutions. The authors of the report note that they found it "very hard to find users of these tools, even with the help of the vendors (Actuate, JasperSoft, Jedox and Pentaho)" I am surprised. The authors go on the observe that "It may be that open source BI components are mainly embedded in other applications, so the end users may not even be aware that they are using them." Elsewhere in the report is included an open source database BI league table. I look forward to more detail being included in the next annual report.
• Price guidance is a current gap. The report does include pages on a Cost of Ownership Index and an analysis of the types of licenses purchased but more information would be useful. Anything that can provide more hard details on the total cost of ownership of each tool would be a valuable additional feature. Difficult for BARC to do perhaps. As mentioned, there are insights given into license fees - I just want more!
• Drop the practise of using vendors to to inform their customers of the survey as this could lead to skewed results if the vendor decided only to inform customers they knew were happy. The temptation for vendors to not inform unhappy customers is too great.
• Increase SME participation. Small, medium and large companies are all well represented and I would like to have more insight into how smaller and medium sized enterprises differ from the big end of town.

The 2009 Survey is now open and if you are a BI user and want to complete the survey then you can go online here. I encourage you to do it as it should only take about 15-20 minutes to complete.

¹Includes a good 20 page executive summary.

²BARC also publishes the The BI Verdict (the new name for The OLAP Report).

Quick Guide To Choosing An MPP Database

I have just come across an interesting blog by Charlie Wardell in the States. If you are looking for a short discussion on the relative merits of MPP (Massively Parallel Processing) database products from Teradata, Netezza and Vertica then read "...And This Database Is Just Right".

He has also recently looked into the popularity of Google searches on Teradata, Netezza, Vertica, ParAccel, and Greenplum here. If you're about to place a bet on which vendor will increase their market share then this is a good place to start.

The Size Of The Local Market

I have always wondered how big the Australian BI market was. Even when I was a Research Director of BI and IM at Gartner the market research didn't easily reveal data down at the Australian level. So I was pleased to read about some apparently 'hard' data:

Gartner's prediction for the business intelligence [market] for 2009 through 2012 says that business units, rather than IT, will control at least 40 percent of the total budget for BI. The problem here, Gartner wrote, is that "business users have lost confidence in the ability of [IT] to deliver the information they need to make decisions." 

The report continues to note that the market for business intelligence (BI) platform software in Australia is forecast to reach A$174.8 million in 2009, up 16.8 percent from A$149.6 million in 2008.

This little bit of intelligence is care of local vendor eXant - the local reseller of the reporting and dashboarding business tool eXPorter. This looks like an interesting tool but I don't know what their pricing is like.

I know enough about Gartner's research methods to add a note of caution to their figures. The true size may be very different from this estimate - but at least they try.

Now all I need is my 1% market share ...

Top 10 Cloud Computing Vendors

I came across an interesting article discussing and analysing cloud computing vendors. I was interested to see that the top 10 vendors are:

1. Amazon Web Services
2. Google
3. IBM
4. Microsoft
5. Salesforce.com
6. Oracle
7. EMC and VMware
8. HP
9. SAP
10. Dell

Further down the list are other companies making a contribution to analytics. Take a look at the likes of:

• Cloud9Analytics (15) - for Salesforce.com users. They provide more detailed data and reports on sales activities with the goal of improving overall sales performance
• Cloudera (27) - runs on Apache Hadoop, a Java-based framework supporting data-intensive applications
• Cordys (32) - software to develop, manage and monitor business processes, including: Business Operations Platform (BOP), which is a suite that includes Business Process Management (BPM) and Business Activity Monitoring (BAM)
• Quantiva (50) - on-demand behavior analytics software for retailers, 
• NetSuite (61) - accounting, BI, CRM, e-commerce and ERP.
• CloudScale (77) - early stage startup promising to bring “real-time analytics to everyone" 
• 10Gen (81) - commercial support for the open source MongoDB database. MongoDB is "suited for a high volume, low value data like a Web site’s data store"

Another Wave Of Vendor Consolidation?

We all woke up today to the news that IBM is to buy SPSS. IBM is acquiring analytics-software maker SPSS in an all-cash transaction at US$50 per share for a deal valued at about US$1.2 billion.

With this news I think that it is worth taking a look at IBM's recent moves in analytics as they now include:

• IBM on Tuesday unveiled a business analytics system—a bundle of preconfigured and optimized hardware and software—designed to cut deployment times.
• They are rolling out a global network of IBM Analytics Solution centres. Already announced are Berlin and Tokyo, with further centres planned for London, New York City, Beijing and Washington, D.C. Nothing for our region unfortunately.
• Back in April of this year IBM also purchased Exeros, a privately-held company based in Santa Clara, California. Exeros makes data mining software.
• Also in April IBM unveiled a new services unit focused on business analytics and optimization stating the need for businesses to use data to make better predictions. This is IBM's first new services business since 2002. Take a look at Larry Dignan's blog for insight into IBM's thinking.

So IBM is making some sizable moves and it will be interesting to see if the new analytics business unit can make real inroads with customers. I think that these are moves in the right direction for IBM - but I'm not yet convinced that their new business analytics unit will deliver the end-to-end analytic platform customers want. I think that we will have to first see much greater coordination between the technology components (DataStage, DB2, Cognos, SPSS, etc.) and the new business analytics unit.

As a side note: IBM analytics is also tweeting and worth following here.

In the meantime the financial markets are reacting with Actuate Software, Informatica, Microstrategy, Netezza, and Teradata all being identified as potential acquisition targets by the likes of Hewlett-Packard, IBM, Oracle, and SAP AG.

Make sure your seat belts are tightened - change could be accelerating in our world.