Oz Analytics

Day 1 Week 3 of the CORTEX MBAnalytics

Congratulations for reaching week 3 - part of the 60 day information management refresher course from the CORTEX.

The CORTEX

Master of Business Analytics

Week 3, Day 1

Theme: Communication & Collaboration

Week 3, Monday

This week's themes are communication, and if you are listening and willing, a resulting collaboration. 

WATCH    Today's task:   Google's Effects on Memory     80Mb

5 minutes. You can also stream it here.

PBS NewsHour 'Google's Effects on Memory' Psychologist Betsy Sparrow

interviewed by Hari Sreenivasan in 2011

Key takeout: The ability to retrieve information from the net has already

allowed us to move more of the 'burden' of memory from our brains and this

is reducing our ability to recall information without the net. It hadn't occurred

to me before reading this research but this is an important step towards the

singularity. 

READ       For extra credit:  Cognitive Consequences of Having Information at Our Fingertips 

  300KB, 15 minutes

'Google Effects on Memory Cognitive Consequences of Having Information at

Our Fingertips', by Betsy Sparrow, Jenny Liu, and Daniel M. Wegner,

Science Express report, July 2011.

Key takeout: How does this impact the design of our collaboration, analytic and

business intelligence portals, etc.?

 

As always, comments, questions and ideas are all welcome. 

The MBAnalytics is just one part of the Business Intelligence CORTEX - a community of Business Intelligence, Data Warehousing, Analytics and Project Management professionals in Australia, New Zealand and Asia. Membership is free and the forums are very active with over 20,000 posts offering a uniquely Australasian viewpoint.

Head First Data Analysis

Continuing the book theme established in my last blog, here is another book that looks very interesting. I say 'looks' because I haven't purchased it yet. The book is Head First Data Analysis: A Learner's Guide to Big Numbers, Statistics, and Good Decisions by Michael Milton. It is very different from the last book I recommended.

The main differences are that it is aimed at the beginner, is highly visual, and at under 500 pages is significantly shorter.

Extensive examples make up the majority of the book and use is made of Excel, OpenOffice, and statistical computing software package R.

What is really interesting is that R is a major feature in this book. It doesn't even rate a mention in the Handbook of Statistical Analysis and Data Mining Applications.

The publisher's recommend that you first know how to use basic spreadsheet formulas before tackling the book. After reading this book they also recommend Head First Statistics in the same series. I haven't yet looked into this title.

Author Michael Milton's approach in the book is very refreshing and engaging as well as being very practical. It is a radical departure from the dry texts of most (all?) university courses on the subject. Michael has degrees in philosophy and religious ethics and this may help to explain his innovative approach.

Below are a couple of screen shots of the book to give you an idea of how topics are presented. Click on the images to open a larger version.

Using the Google book preview on publisher O'Reilly's website, you can get a real flavour of the book. I'm impressed by what I see and I have put it on my list of books to buy soon.

You can find the book here on Amazon for about AUD 35. You can also buy it from O'Reilly as a printed book (but it is much more expensive than Amazon) or as an ebook.

Quick Guide To Choosing An MPP Database

I have just come across an interesting blog by Charlie Wardell in the States. If you are looking for a short discussion on the relative merits of MPP (Massively Parallel Processing) database products from Teradata, Netezza and Vertica then read "...And This Database Is Just Right".

He has also recently looked into the popularity of Google searches on Teradata, Netezza, Vertica, ParAccel, and Greenplum here. If you're about to place a bet on which vendor will increase their market share then this is a good place to start.

Local Intelligence Beta Testers Wanted

As most of my readers know by now, I and a small group of like-minded people are building an online community for business intelligence, information management, data warehousing, analytics and performance management enthusiasts in Australia and New Zealand. Called the CORTEX for short.

We launched officially this year and membership is approaching 700 people. It's great to know that so many other people like the concept. Everyone is welcome and membership is free. OK, that's the end of the PR bull.

We are trying to regularly add new features and in the spirit of Google Labs we have just launched a one-click search function that scours the CORTEX for news and views on specific leading ANZ organisations. Currently we have the following industries covered:

• Construction and Property
• Financial Services
• Government, Health and Education
• Manufacturing and Distribution
• Publishing and Media
• Retailers
• Rural, Minining, Oil and Gas
• Telecommunications
• Tourism and Hospitality
• Transportation
• Utilities

I know that we have missed several organisations and I ask anyone who is active in these (or other) industries to:

• Beta-test the search function
• Suggest other organisations we should include
• Add news to the CORTEX local industry channels if you know of something that we are missing.

All comments and criticisms are welcome!

Top 10 Cloud Computing Vendors

I came across an interesting article discussing and analysing cloud computing vendors. I was interested to see that the top 10 vendors are:

1. Amazon Web Services
2. Google
3. IBM
4. Microsoft
5. Salesforce.com
6. Oracle
7. EMC and VMware
8. HP
9. SAP
10. Dell

Further down the list are other companies making a contribution to analytics. Take a look at the likes of:

• Cloud9Analytics (15) - for Salesforce.com users. They provide more detailed data and reports on sales activities with the goal of improving overall sales performance
• Cloudera (27) - runs on Apache Hadoop, a Java-based framework supporting data-intensive applications
• Cordys (32) - software to develop, manage and monitor business processes, including: Business Operations Platform (BOP), which is a suite that includes Business Process Management (BPM) and Business Activity Monitoring (BAM)
• Quantiva (50) - on-demand behavior analytics software for retailers, 
• NetSuite (61) - accounting, BI, CRM, e-commerce and ERP.
• CloudScale (77) - early stage startup promising to bring “real-time analytics to everyone" 
• 10Gen (81) - commercial support for the open source MongoDB database. MongoDB is "suited for a high volume, low value data like a Web site’s data store"

Is Information Alive?

It's a new week and I thought we'd start it with a discussion to get our brains working after the weekend.

I have just read an interesting article in New Scientist by Dr Susan Blackmore, a writer and psychologist based in the UK. The article is called 'Evolution's third replicator: Genes, memes, and now what?' Essentially the article is proposing that there are 3 evolutionary processes now on Earth.

I must be dense because I was only aware of one: Darwinian evolution and the passing of genes from parent to child. Well I'm wrong. There are two others: 'natural' and 'artificial' memes. A definition of each may help:

• genes, the copying of characteristics through DNA
• "natural" human memes, such as spoken words, habits, fashions, art and religions
• "artificial" memes, such as websites and high-tech goods

I was only vaguely aware of what a meme (rhymes with 'cream') was. For the record (and a layman's definition), a meme is the "basic building block of our minds and culture" and you can think of memes as cultural analogues to genes, in that they self-replicate and evolve following a process similar to natural selection.

The term was coined by Richard Dawkins in the mid 1970's and Dr Blackmore is a leading researcher in the meme field. The concept of memes are still very controversial and are not accepted by the majority of social scientists. I am also sceptical but I haven't completely dismissed the meme. My main problem is that it seems too simple a concept to explain the richness of human culture and the transmission of ideas. 

So if you take a leap of faith (I can find no generally accepted empirical evidence) there is now a new evolution happening as electronically processed binary information is copied and changed automatically by computers. To quote from the article:

"Or think of Google. It copies information, selects what it needs and puts the selections together in new variations - that's all three [copying, varying and selection]. The temptation is to think that since we designed search engines and other technologies for our own use they must remain subservient to us. But if a new replicator is involved we must think again. Search results go not only to screens for people to look at, but to other programs, commercial applications and even viruses - that's machines copying information to other machines without the intervention of a human brain."

Sounds very similar to the concept of The Singularity - the idea that artificial intelligence will suddenly (and in the near future) outsmart humans. Take a look at a recent article in the New York Times for a good overview.

So is information (on the web) alive?

Anyway, it's all very thought provoking so I will leave you with a quote from Dr Blackmore that echos the recent posts of another CORTEX blogger (Navigating the Information Management maze) on the information explosion. Dr Blackmore:

"Last year Google announced that the web had passed the trillion mark, with more than 1,000,000,000,000 unique URLs. Many countries now have nearly as many computers as people, and if you count phones and other connected gadgets they far outnumber people. Even if we all spent all day reading this stuff it would expand faster than we could keep up.

Billions of years ago, free-living bacteria are thought to have become incorporated into living cells as energy-providing mitochondria. Both sides benefited from the deal. Perhaps the same is happening to us now. The growing web of machines we let loose needs us to run the power stations, build the factories that make the computers, and repair things when they go wrong - and will do for some time yet. In return we get entertainment, tedious tasks done for us, facts at the click of a mouse and as much communication as we can ask for. It's a deal we are not likely to turn down."

Top 6 BI Social Network Organisations

Just published is a new survey of social networking by Wetpaint and Altimeter Group. The result is called the ENGAGEMENTdb. The survey is designed to answer the question: 

What companies are on the social web and how well are they doing it?

ENGAGEMENTdb ranks the world's most valuable brands. these are the brands identified in the separate survey by BusinessWeek / Interbrand called "Best Global Brands 2008".  ENGAGEMENTdb goes on to rank how they leverage social media to interact with customers. It is a great piece of work.

Success varies greatly by industry as shown by the picture on the left.

I thought it would be interesting to mine the survey and take a look at how the business intelligence vendors/ participants performed:

Position	Organisation	# of Channels	Score	Place in Top 100
1	Google	11	105	4
2	Microsoft	10	103	5
3	SAP	10	86	9
4	Oracle	10	73	14
5	HP	8	50	24
6	IBM	9	46	28

 

I also include two other organisations: Reuters because of their extensive textual data mining activities like factiva; and Accenture due to their sizable global BI practice. If they were included in the above list, then they would rank as follows:

Position	Organisation	# of Channels	Score	Place in Top 100
3	Thomson Reuters	11	101	6
4	Accenture	10	76	13

 

Overall, the IT firms have greatly varied performances as shown by the figure to the left. All of the business intelligence vendors perform in the top quartile of the top 100.

The ENGAGEMENTdb has a lot to say about all of these organisations. I took a closer look at SAPs Social Profile and (with a little extra digging) you learn that:

Its core social presence is the SCN (SAP Community Network ) that has 1.7 million registered users and has within it four distinct communities:

• SDN - SAP Developer Network for SAP NetWeaver. Contains technical articles, white papers, and how-to guides, as well as some 200 moderated forums, expert blogs, software downloads, extensive eLearning materials, and a Wiki that supports open communication.
• SBX - Business Process Expert Community. Contains articles, white papers, how-to guides, forums, blogs, and Wiki, as well as virtual events and eLearning. You'll also find up-to-date strategic information on business processes and the software solutions that support them.
• BusinessObjects - SAP Business Objects Community. Contains forums and articles that reinforce the value of open and heterogeneous Business Objects solutions, leveraging technologies from SAP, Microsoft, IBM, Oracle, and more. You'll also have access to software downloads for Crystal Reports and Xcelcius.
• SAP EcoHub - SAP Application Partner Solutions. A knowledgebase.

Other communities are operated by SAP and they include the SAP Community Network 2.0 (beta).

SAP also is active through a number of independent communities such as BOB (BusinessObjects Board http://www.forumtopics.com/busobj/index.php) with 42,000 registered users.

Apart from the shear size of the network, the quality of the communities activities impress. Perhaps this is because of the following:

• The network is open to anyone who wants to post, blog or comment. SCN has over 5,000 bloggers now. Contributions to the SCN are encouraged by rewarded by points for participation that contribute to your reputation.
• Employees are encouraged to participate. 1,500 of the 5,000 SCN bloggers are SAP employees.
• New channels are embraced as they become popular. SAP uses Twitter for example.
• Actively reaching out to customers. Twitter.com/saplistens http://www.Twitter.com/saplistens is a twitter account whose purpose is to get feedback from customers.

Checkout the ENGAGEMENTdb website or read the whole report here: 

Download ENGAGEMENTdb_Report_2009

I'm Afraid I Can't Let You Do That Dave

When you've got a hammer the whole world is a nail - so goes the saying.

So I was wondering if this is the case at Google as they continue to grapple with a serious brain drain of their best staff.

The latest reports are that Google has developed an algorithm to predict which of their staff are most likely to quit. Details are very difficult to come by but here are the features of the algorithm that I have read about in various reports:

• it mines employee reviews, promotion and salary histories
• attempts to identify employees who feel 'underused' 
• inputs also include information from surveys and peer reviews

Google's algorithm helps the company "get inside people's heads even before they know they might leave," said Laszlo Bock, who runs human resources for the company. This data crunching supplements more traditional measures like employee training and leadership meetings to evaluate talent.

Is Google using this to categorise employees into other performance groups? Is this another take on the old six sigma (forced) ranking of the bottom 10 percent of employees?

You could easily argue that almost 5 years on from Google's IPO, this drain is normal and expected from a highly successful company. Way back in April 2007 it was reported that:

"Hundreds of the 2,300 Googlers hired before the Internet juggernaut's initial public offering in August 2004 are hitting their fourth anniversary. When they do, they'll be free to cash in the final portions of their pre-IPO options, collectively worth an estimated $2.6 billion before taxes."
San Jose Mercury News

Presumably the employees leaving were all wealthy from their stock grants and options as under Google's vesting rules, employees typically earn the right to cash in 25 percent of their options when they hit their first anniversary, then the rest in monthly chunks over the next three years. So August 2008 was the month many had earned full cash in rights and could leave with no penalty. Add to that the feeling that your personal contribution is diminishing as the company grows and matures. The desire the seek greener pastures in other companies or exciting start-ups is hardly unexpected.

All credit to Google for being proactive however. If you can retain even a handful from leaving then it must be a worthwhile effort. It should be no surprise that Google is a trend setter here. How many companies do you know with a Chief Culture Officer and the 10 Golden Rules for knowledge workers:

1. Hire by committee. Virtually every person who interviews at Google talks to at least half-a-dozen interviewers, drawn from both management and potential colleagues. Everyone's opinion counts, making the hiring process more fair and pushing standards higher. Yes, it takes longer, but we think it's worth it. If you hire great people and involve them intensively in the hiring process, you'll get more great people. We started building this positive feedback loop when the company was founded, and it has had a huge payoff.
2. Cater to their every need. As Drucker says, the goal is to "strip away everything that gets in their way." We provide a standard package of fringe benefits, but on top of that are first-class dining facilities, gyms, laundry rooms, massage rooms, haircuts, carwashes, dry cleaning, commuting buses—just about anything a hardworking engineer might want. Let's face it: programmers want to program, they don't want to do their laundry. So we make it easy for them to do both.
3. Pack them in. Almost every project at Google is a team project, and teams have to communicate. The best way to make communication easy is to put team members within a few feet of each other. The result is that virtually everyone at Google shares an office. This way, when a programmer needs to confer with a colleague, there is immediate access: no telephone tag, no e-mail delay, no waiting for a reply. Of course, there are many conference rooms that people can use for detailed discussion so that they don't disturb their office mates. Even the CEO shared an office at Google for several months after he arrived. Sitting next to a knowledgeable employee was an incredibly effective educational experience.
4. Make coordination easy. Because all members of a team are within a few feet of one another, it is relatively easy to coordinate projects. In addition to physical proximity, each Googler e-mails a snippet once a week to his work group describing what he has done in the last week. This gives everyone an easy way to track what everyone else is up to, making it much easier to monitor progress and synchronize work flow.
5. Eat your own dog food. Google workers use the company's tools intensively. The most obvious tool is the Web, with an internal Web page for virtually every project and every task. They are all indexed and available to project participants on an as-needed basis. We also make extensive use of other information-management tools, some of which are eventually rolled out as products. For example, one of the reasons for Gmail's success is that it was beta tested within the company for many months. The use of e-mail is critical within the organization, so Gmail had to be tuned to satisfy the needs of some of our most demanding customers—our knowledge workers.
6. Encourage creativity. Google engineers can spend up to 20 percent of their time on a project of their choice. There is, of course, an approval process and some oversight, but basically we want to allow creative people to be creative. One of our not-so-secret weapons is our ideas mailing list: a companywide suggestion box where people can post ideas ranging from parking procedures to the next killer app. The software allows for everyone to comment on and rate ideas, permitting the best ideas to percolate to the top.
7. Strive to reach consensus. Modern corporate mythology has the unique decision maker as hero. We adhere to the view that the "many are smarter than the few," and solicit a broad base of views before reaching any decision. At Google, the role of the manager is that of an aggregator of viewpoints, not the dictator of decisions. Building a consensus sometimes takes longer, but always produces a more committed team and better decisions
8. Don't be evil. Much has been written about Google's slogan, but we really try to live by it, particularly in the ranks of management. As in every organization, people are passionate about their views. But nobody throws chairs at Google, unlike management practices used at some other well-known technology companies. We foster to create an atmosphere of tolerance and respect, not a company full of yes men.
9. Data drive decisions. At Google, almost every decision is based on quantitative analysis. We've built systems to manage information, not only on the Internet at large, but also internally. We have dozens of analysts who plow through the data, analyze performance metrics and plot trends to keep us as up to date as possible. We have a raft of online "dashboards" for every business we work in that provide up-to-the-minute snapshots of where we are.
10. Communicate effectively. Every Friday we have an all-hands assembly with announcements, introductions and questions and answers. (Oh, yes, and some food and drink.) This allows management to stay in touch with what our knowledge workers are thinking and vice versa. Google has remarkably broad dissemination of information within the organization and remarkably few serious leaks. Contrary to what some might think, we believe it is the first fact that causes the second: a trusted work force is a loyal work force.

Don't Be Evil is the one that often gets the headline, but the next one down is just as radical in my view: Data Drive Decisions. Here is the whole article from Eric Schmidt, CEO of Google.

What a pity that other companies are not turning to analytics for the same reasons.

Media reactions have been mixed. Some outright hostile. I may be biased but I don't think that using quantitative analysis of 'HR' data is a bad thing. If anything it should help to remove the bias of human interpretation when reviewing performance. We call it 'fact based decision making' everywhere else. What's so special about HR? I'm not saying that it is a universal panacea but used in conjunction with other things like encouraging supportive cultures and inspirational management styles.

It also struck me that Google should perhaps be concentrating more on those employees who are most under-used as an end in itself. Making better use of existing people must have a positive impact on the performance of the company.

A roll call of brain drainers is available here if you are interested. As recently as last week Tom Phillips, the head of Search and Analytics was rumoured to have left Google.

Gartner's View That Search Is Not Business Intelligence

I was rummaging through my knowledgebase of BI stuff this morning and came across a curious news report on Gartner from late last year. Here's the bit that caught my eye:

So the jist of the claim is that search is not business intelligence (or ERP). I agree with the ERP bit but I think it is odd to split Search and BI. Sondergaard says "this new trend in business IT should not be confused with 'business intelligence' which refers to the collection, integration and analysis of business information to make decisions and forecasts". Sounds like the same reasons you use search in an organisation.

One prediction I am confident to make is that Search and BI will steadily become synonymous as practical distinctions disappear between structured and unstructured data. This is already happening today as better methodologies and algorithms emerge that have the sophistication to deeply understand unstructured data. You only have to experiment with Google to get how smart the googlesphere is getting.

Even my personal knowledgebase of BI stuff has become a vital part of my professional life. I use Devonthink Pro to manage a huge variety of BI stuff - currently running to over 250,000 objects from MS Office documents to podcasts, pdf's and 100 other formats. Searching it is a breeze and the ranking of results is very good. Good enough to make it one of the core tools I own and use.

Sure beats having a good memory (not that I have that choice!).

So here is my 2 cents on Search and BI:

• Both structured and unstructured data will be vital 'grist in the BI mill' 
• Powerful algorithms will enable realtime mining of unstructured data into BI reports and dashboards 
• Where such realtime processing is not feasible (because of slow response times or data integration bottlenecks) then batch processing of unstructured data to create structured data will happen 
• Search of unstructured data within an organisation will be achieved by using algorithms developed and refined first by public use of search engines like Google.
• Mining public (or industry shared) structured and unstructured data will be as important as mining unstructured data within an organisation
• XBRL will be a major contributor to the structuring of publicly available corporate data in Australia - but as always we will be at least 5 years behind the USA in this.  

Here is the whole Sondergaard article if you are interested. Download 200811_Sondergaard_on_Search

Google and Open Source

Google makes much of its friendliness towards open source. Some may say this is to distract people from noticing their virtual monopoly of search on the net. I'm not so cynical as there seems to be a lot of competitive activity in search these days - that plus the fact that Google continues to innovate and deliver great products and services.

Here is an interesting leaked internal presentation from Google. Take a look at page 8 where their internal 'Data Liberation Front' team is discussed.

Download Googlepresentation

They even have a manifesto that you can read on their Google Code Wiki (where else?!). Trouble is, there's nothing more than the manifesto there. Nothing about what they are actually doing.

Of more use are the tweets from a team member.

On the whole this seems like an interesting initiative as they seek (no pun intended - seek is a job search site in Oz) to liberate more data sets. More power to them.

Fellow blogger John Gordon has done some digging about this team and you can read about it on his blog Gordon's Notes.