Oz Analytics

Google+ and Bridger

A surprising (for me) outcome of our work creating Bridger has been how often I use it in conjunction with my social media interactions.

I guess that I shouldn't be surprised as Bridger is a universal document and web content analyser and we want to have it play nice with as many file types, applications and platforms as is possible. This includes Google, Facebook, LinkedIn and the like.

My use of Google+ has been growing over the last year and I now find myself relying on it to keep me informed in a number of areas - both professionally and privately. It's also our platform of choice to hold Six Degrees Of Data meetings and chats.

So Google+ has ingratiated itself into my work habits and the following short video shows you how Bridger enhances the value of social media by analyzing the documents that your communities share in Google+:

You can also check out our other videos on YouTube and Bridger is now in Public Beta so you can try it out for yourself. It's free to use and there is nothing to install.

Changing Tack

My 2 regular readers (thanks mum and dad!) may have noticed that the frequency of my posts has dropped dramatically since 2012. There was a reason for this - beyond my strongly held opinion that those who have nothing sensible to say should simply shut-up. That reason was/is that I have been starting a new venture and the 'stealth mode' has lasted a lot longer than I expected.

In 2012 I suffered an information crisis and I suspected that I was not alone. The crisis arrived because I was being increasingly overwhelmed with information and most of the information swamping me was not on the web. Instead it was in the form of documents and email. I pride myself as being pretty well organised with effective email filters in place and an organised corpus of professional and personal knowledge approaching 1,000,000 objects. Mostly documents of one sort or another.

I was quietly proud of my knowledge management skills but after pride cometh the fall - and in 2012 that's what happened. Let me explain how it happened and I hope the story will resonate with you.

 As a serial entrepreneur and corporate analyst, I have a broad range of professional interests. These range from all aspects of analytics and information management, and broaden out into general management and IT. I am also a bit of a pack rat and if I come across some information that looks useful I store it. For those interested, my tools of choice are Mac OSX, HoudahSpot, DevonAgent and Devonthink.

So over the last 20 years I have built up a very useful corpus of knowledge. By 2011 however I was becoming increasingly frustrated by my inability to effectively use this corpus in my daily work. Each day I need to deal with the usually pile of emails and read/respond to/create multiple documents. This presented the following problems:

1. Identifying and dealing with those emails and documents that I had to.
2. Ignoring the rest.
3. Locating past knowledge/information that would help me to deal effectively with the day's emails and documents.

So the problems were pretty much at the core of what I do.

If only there was a way for me to more efficiently decide whether-or-not a long email or document was worth me reading. Then, when I needed to make a decision or give advice, how could I tap into my corpus of knowledge?

These are the problems solved by my new venture Six Degrees Of Data.

Six Degrees Of Data is building Apps and web services that we hope turn the web on its head. Our first product will launch later this year. It is called Bridger and is an App and Web Service that intelligently evaluates, summarizes and links your documents to trusted sources on the web.

Bridger is a research assistant for busy people suffering from information overload.

We are trying to release the value currently locked inside documents because despite the phenomenal growth of the internet, 80% of the new information created each year is still unstructured – in other words it is text, graphics and video contained in things like documents, emails, Facebook shares, web pages and tweets, etc. This unstructured information is poorly integrated with the world wide web. Even if they are published to a web site, they have remained isolated islands of information – until now.

Within the last five years, the size of the average web page has more than tripled, and the number of external objects has more than doubled. The average web page is now 1 MB and growing at over 20% each year. Today the average web page has about 10 links and 600 words. So it is reasonable to say that less than 2% of the content is hyperlinked to something else.

Your documents often contain only a single link (or less) and many more words than a web page.

If you are lucky, there is a single bridge (hyperlink) from the web to your document. Think of your documents as information that is connected to the web superhighway only via an off-ramp. You can find published documents easily enough because published documents are indexed by the likes of Google and Bing. But what you can’t do is start from your document and get easily back onto the web in an intelligent way.

That’s why we see the current web-document world as a one-way off-ramp.

Bridger builds you an internet on-ramp for all of your documents – whether they are published on the web or not. This is how we can release the untapped value of the 80%.

How do we do this? Well, it’s pretty complicated technically but the good news is that you really don’t have to know about this if you don’t want to. All you need to understand is that Bridger takes your isolated documents, emails, etc. and automatically links them to over 50 billion trusted additional sources of information on the world wide web.

Drop a document on to the Bridger app and it will instantly tell you what the document is about and give you concise profiles of all the relevant people, organizations, places, technologies and more.

Your document is magically transformed into a fully integrated web body of knowledge.

And Bridger is the first of a stream of new products we re planning. Future areas that we will concentrate on are enriching our personal corpus' of knowledge, collaboration and sharing.

So my posts may remain sporadic but I hope to share more about Bridger and our journey as Six Degrees Of Data gets up and running over the coming months.

In the meantime, if you have a passion for the Semantic Web, Linked Open Data, URI's, sparkle and tuples, then give me a call.

 

Day 1 Week 3 of the CORTEX MBAnalytics

Congratulations for reaching week 3 - part of the 60 day information management refresher course from the CORTEX.

The CORTEX

Master of Business Analytics

Week 3, Day 1

Theme: Communication & Collaboration

Week 3, Monday

This week's themes are communication, and if you are listening and willing, a resulting collaboration. 

WATCH    Today's task:   Google's Effects on Memory     80Mb

5 minutes. You can also stream it here.

PBS NewsHour 'Google's Effects on Memory' Psychologist Betsy Sparrow

interviewed by Hari Sreenivasan in 2011

Key takeout: The ability to retrieve information from the net has already

allowed us to move more of the 'burden' of memory from our brains and this

is reducing our ability to recall information without the net. It hadn't occurred

to me before reading this research but this is an important step towards the

singularity. 

READ       For extra credit:  Cognitive Consequences of Having Information at Our Fingertips 

  300KB, 15 minutes

'Google Effects on Memory Cognitive Consequences of Having Information at

Our Fingertips', by Betsy Sparrow, Jenny Liu, and Daniel M. Wegner,

Science Express report, July 2011.

Key takeout: How does this impact the design of our collaboration, analytic and

business intelligence portals, etc.?

 

As always, comments, questions and ideas are all welcome. 

The MBAnalytics is just one part of the Business Intelligence CORTEX - a community of Business Intelligence, Data Warehousing, Analytics and Project Management professionals in Australia, New Zealand and Asia. Membership is free and the forums are very active with over 20,000 posts offering a uniquely Australasian viewpoint.

Head First Data Analysis

Continuing the book theme established in my last blog, here is another book that looks very interesting. I say 'looks' because I haven't purchased it yet. The book is Head First Data Analysis: A Learner's Guide to Big Numbers, Statistics, and Good Decisions by Michael Milton. It is very different from the last book I recommended.

The main differences are that it is aimed at the beginner, is highly visual, and at under 500 pages is significantly shorter.

Extensive examples make up the majority of the book and use is made of Excel, OpenOffice, and statistical computing software package R.

What is really interesting is that R is a major feature in this book. It doesn't even rate a mention in the Handbook of Statistical Analysis and Data Mining Applications.

The publisher's recommend that you first know how to use basic spreadsheet formulas before tackling the book. After reading this book they also recommend Head First Statistics in the same series. I haven't yet looked into this title.

Author Michael Milton's approach in the book is very refreshing and engaging as well as being very practical. It is a radical departure from the dry texts of most (all?) university courses on the subject. Michael has degrees in philosophy and religious ethics and this may help to explain his innovative approach.

Below are a couple of screen shots of the book to give you an idea of how topics are presented. Click on the images to open a larger version.

Using the Google book preview on publisher O'Reilly's website, you can get a real flavour of the book. I'm impressed by what I see and I have put it on my list of books to buy soon.

You can find the book here on Amazon for about AUD 35. You can also buy it from O'Reilly as a printed book (but it is much more expensive than Amazon) or as an ebook.

Quick Guide To Choosing An MPP Database

I have just come across an interesting blog by Charlie Wardell in the States. If you are looking for a short discussion on the relative merits of MPP (Massively Parallel Processing) database products from Teradata, Netezza and Vertica then read "...And This Database Is Just Right".

He has also recently looked into the popularity of Google searches on Teradata, Netezza, Vertica, ParAccel, and Greenplum here. If you're about to place a bet on which vendor will increase their market share then this is a good place to start.

Local Intelligence Beta Testers Wanted

As most of my readers know by now, I and a small group of like-minded people are building an online community for business intelligence, information management, data warehousing, analytics and performance management enthusiasts in Australia and New Zealand. Called the CORTEX for short.

We launched officially this year and membership is approaching 700 people. It's great to know that so many other people like the concept. Everyone is welcome and membership is free. OK, that's the end of the PR bull.

We are trying to regularly add new features and in the spirit of Google Labs we have just launched a one-click search function that scours the CORTEX for news and views on specific leading ANZ organisations. Currently we have the following industries covered:

• Construction and Property
• Financial Services
• Government, Health and Education
• Manufacturing and Distribution
• Publishing and Media
• Retailers
• Rural, Minining, Oil and Gas
• Telecommunications
• Tourism and Hospitality
• Transportation
• Utilities

I know that we have missed several organisations and I ask anyone who is active in these (or other) industries to:

• Beta-test the search function
• Suggest other organisations we should include
• Add news to the CORTEX local industry channels if you know of something that we are missing.

All comments and criticisms are welcome!

Top 10 Cloud Computing Vendors

I came across an interesting article discussing and analysing cloud computing vendors. I was interested to see that the top 10 vendors are:

1. Amazon Web Services
2. Google
3. IBM
4. Microsoft
5. Salesforce.com
6. Oracle
7. EMC and VMware
8. HP
9. SAP
10. Dell

Further down the list are other companies making a contribution to analytics. Take a look at the likes of:

• Cloud9Analytics (15) - for Salesforce.com users. They provide more detailed data and reports on sales activities with the goal of improving overall sales performance
• Cloudera (27) - runs on Apache Hadoop, a Java-based framework supporting data-intensive applications
• Cordys (32) - software to develop, manage and monitor business processes, including: Business Operations Platform (BOP), which is a suite that includes Business Process Management (BPM) and Business Activity Monitoring (BAM)
• Quantiva (50) - on-demand behavior analytics software for retailers, 
• NetSuite (61) - accounting, BI, CRM, e-commerce and ERP.
• CloudScale (77) - early stage startup promising to bring “real-time analytics to everyone" 
• 10Gen (81) - commercial support for the open source MongoDB database. MongoDB is "suited for a high volume, low value data like a Web site’s data store"

Is Information Alive?

It's a new week and I thought we'd start it with a discussion to get our brains working after the weekend.

I have just read an interesting article in New Scientist by Dr Susan Blackmore, a writer and psychologist based in the UK. The article is called 'Evolution's third replicator: Genes, memes, and now what?' Essentially the article is proposing that there are 3 evolutionary processes now on Earth.

I must be dense because I was only aware of one: Darwinian evolution and the passing of genes from parent to child. Well I'm wrong. There are two others: 'natural' and 'artificial' memes. A definition of each may help:

• genes, the copying of characteristics through DNA
• "natural" human memes, such as spoken words, habits, fashions, art and religions
• "artificial" memes, such as websites and high-tech goods

I was only vaguely aware of what a meme (rhymes with 'cream') was. For the record (and a layman's definition), a meme is the "basic building block of our minds and culture" and you can think of memes as cultural analogues to genes, in that they self-replicate and evolve following a process similar to natural selection.

The term was coined by Richard Dawkins in the mid 1970's and Dr Blackmore is a leading researcher in the meme field. The concept of memes are still very controversial and are not accepted by the majority of social scientists. I am also sceptical but I haven't completely dismissed the meme. My main problem is that it seems too simple a concept to explain the richness of human culture and the transmission of ideas. 

So if you take a leap of faith (I can find no generally accepted empirical evidence) there is now a new evolution happening as electronically processed binary information is copied and changed automatically by computers. To quote from the article:

"Or think of Google. It copies information, selects what it needs and puts the selections together in new variations - that's all three [copying, varying and selection]. The temptation is to think that since we designed search engines and other technologies for our own use they must remain subservient to us. But if a new replicator is involved we must think again. Search results go not only to screens for people to look at, but to other programs, commercial applications and even viruses - that's machines copying information to other machines without the intervention of a human brain."

Sounds very similar to the concept of The Singularity - the idea that artificial intelligence will suddenly (and in the near future) outsmart humans. Take a look at a recent article in the New York Times for a good overview.

So is information (on the web) alive?

Anyway, it's all very thought provoking so I will leave you with a quote from Dr Blackmore that echos the recent posts of another CORTEX blogger (Navigating the Information Management maze) on the information explosion. Dr Blackmore:

"Last year Google announced that the web had passed the trillion mark, with more than 1,000,000,000,000 unique URLs. Many countries now have nearly as many computers as people, and if you count phones and other connected gadgets they far outnumber people. Even if we all spent all day reading this stuff it would expand faster than we could keep up.

Billions of years ago, free-living bacteria are thought to have become incorporated into living cells as energy-providing mitochondria. Both sides benefited from the deal. Perhaps the same is happening to us now. The growing web of machines we let loose needs us to run the power stations, build the factories that make the computers, and repair things when they go wrong - and will do for some time yet. In return we get entertainment, tedious tasks done for us, facts at the click of a mouse and as much communication as we can ask for. It's a deal we are not likely to turn down."

Top 6 BI Social Network Organisations

Just published is a new survey of social networking by Wetpaint and Altimeter Group. The result is called the ENGAGEMENTdb. The survey is designed to answer the question: 

What companies are on the social web and how well are they doing it?

ENGAGEMENTdb ranks the world's most valuable brands. these are the brands identified in the separate survey by BusinessWeek / Interbrand called "Best Global Brands 2008".  ENGAGEMENTdb goes on to rank how they leverage social media to interact with customers. It is a great piece of work.

Success varies greatly by industry as shown by the picture on the left.

I thought it would be interesting to mine the survey and take a look at how the business intelligence vendors/ participants performed:

Position	Organisation	# of Channels	Score	Place in Top 100
1	Google	11	105	4
2	Microsoft	10	103	5
3	SAP	10	86	9
4	Oracle	10	73	14
5	HP	8	50	24
6	IBM	9	46	28

 

I also include two other organisations: Reuters because of their extensive textual data mining activities like factiva; and Accenture due to their sizable global BI practice. If they were included in the above list, then they would rank as follows:

Position	Organisation	# of Channels	Score	Place in Top 100
3	Thomson Reuters	11	101	6
4	Accenture	10	76	13

 

Overall, the IT firms have greatly varied performances as shown by the figure to the left. All of the business intelligence vendors perform in the top quartile of the top 100.

The ENGAGEMENTdb has a lot to say about all of these organisations. I took a closer look at SAPs Social Profile and (with a little extra digging) you learn that:

Its core social presence is the SCN (SAP Community Network ) that has 1.7 million registered users and has within it four distinct communities:

• SDN - SAP Developer Network for SAP NetWeaver. Contains technical articles, white papers, and how-to guides, as well as some 200 moderated forums, expert blogs, software downloads, extensive eLearning materials, and a Wiki that supports open communication.
• SBX - Business Process Expert Community. Contains articles, white papers, how-to guides, forums, blogs, and Wiki, as well as virtual events and eLearning. You'll also find up-to-date strategic information on business processes and the software solutions that support them.
• BusinessObjects - SAP Business Objects Community. Contains forums and articles that reinforce the value of open and heterogeneous Business Objects solutions, leveraging technologies from SAP, Microsoft, IBM, Oracle, and more. You'll also have access to software downloads for Crystal Reports and Xcelcius.
• SAP EcoHub - SAP Application Partner Solutions. A knowledgebase.

Other communities are operated by SAP and they include the SAP Community Network 2.0 (beta).

SAP also is active through a number of independent communities such as BOB (BusinessObjects Board http://www.forumtopics.com/busobj/index.php) with 42,000 registered users.

Apart from the shear size of the network, the quality of the communities activities impress. Perhaps this is because of the following:

• The network is open to anyone who wants to post, blog or comment. SCN has over 5,000 bloggers now. Contributions to the SCN are encouraged by rewarded by points for participation that contribute to your reputation.
• Employees are encouraged to participate. 1,500 of the 5,000 SCN bloggers are SAP employees.
• New channels are embraced as they become popular. SAP uses Twitter for example.
• Actively reaching out to customers. Twitter.com/saplistens http://www.Twitter.com/saplistens is a twitter account whose purpose is to get feedback from customers.

Checkout the ENGAGEMENTdb website or read the whole report here: 

Download ENGAGEMENTdb_Report_2009

I'm Afraid I Can't Let You Do That Dave

When you've got a hammer the whole world is a nail - so goes the saying.

So I was wondering if this is the case at Google as they continue to grapple with a serious brain drain of their best staff.

The latest reports are that Google has developed an algorithm to predict which of their staff are most likely to quit. Details are very difficult to come by but here are the features of the algorithm that I have read about in various reports:

• it mines employee reviews, promotion and salary histories
• attempts to identify employees who feel 'underused' 
• inputs also include information from surveys and peer reviews

Google's algorithm helps the company "get inside people's heads even before they know they might leave," said Laszlo Bock, who runs human resources for the company. This data crunching supplements more traditional measures like employee training and leadership meetings to evaluate talent.

Is Google using this to categorise employees into other performance groups? Is this another take on the old six sigma (forced) ranking of the bottom 10 percent of employees?

You could easily argue that almost 5 years on from Google's IPO, this drain is normal and expected from a highly successful company. Way back in April 2007 it was reported that:

"Hundreds of the 2,300 Googlers hired before the Internet juggernaut's initial public offering in August 2004 are hitting their fourth anniversary. When they do, they'll be free to cash in the final portions of their pre-IPO options, collectively worth an estimated $2.6 billion before taxes."
San Jose Mercury News

Presumably the employees leaving were all wealthy from their stock grants and options as under Google's vesting rules, employees typically earn the right to cash in 25 percent of their options when they hit their first anniversary, then the rest in monthly chunks over the next three years. So August 2008 was the month many had earned full cash in rights and could leave with no penalty. Add to that the feeling that your personal contribution is diminishing as the company grows and matures. The desire the seek greener pastures in other companies or exciting start-ups is hardly unexpected.

All credit to Google for being proactive however. If you can retain even a handful from leaving then it must be a worthwhile effort. It should be no surprise that Google is a trend setter here. How many companies do you know with a Chief Culture Officer and the 10 Golden Rules for knowledge workers:

1. Hire by committee. Virtually every person who interviews at Google talks to at least half-a-dozen interviewers, drawn from both management and potential colleagues. Everyone's opinion counts, making the hiring process more fair and pushing standards higher. Yes, it takes longer, but we think it's worth it. If you hire great people and involve them intensively in the hiring process, you'll get more great people. We started building this positive feedback loop when the company was founded, and it has had a huge payoff.
2. Cater to their every need. As Drucker says, the goal is to "strip away everything that gets in their way." We provide a standard package of fringe benefits, but on top of that are first-class dining facilities, gyms, laundry rooms, massage rooms, haircuts, carwashes, dry cleaning, commuting buses—just about anything a hardworking engineer might want. Let's face it: programmers want to program, they don't want to do their laundry. So we make it easy for them to do both.
3. Pack them in. Almost every project at Google is a team project, and teams have to communicate. The best way to make communication easy is to put team members within a few feet of each other. The result is that virtually everyone at Google shares an office. This way, when a programmer needs to confer with a colleague, there is immediate access: no telephone tag, no e-mail delay, no waiting for a reply. Of course, there are many conference rooms that people can use for detailed discussion so that they don't disturb their office mates. Even the CEO shared an office at Google for several months after he arrived. Sitting next to a knowledgeable employee was an incredibly effective educational experience.
4. Make coordination easy. Because all members of a team are within a few feet of one another, it is relatively easy to coordinate projects. In addition to physical proximity, each Googler e-mails a snippet once a week to his work group describing what he has done in the last week. This gives everyone an easy way to track what everyone else is up to, making it much easier to monitor progress and synchronize work flow.
5. Eat your own dog food. Google workers use the company's tools intensively. The most obvious tool is the Web, with an internal Web page for virtually every project and every task. They are all indexed and available to project participants on an as-needed basis. We also make extensive use of other information-management tools, some of which are eventually rolled out as products. For example, one of the reasons for Gmail's success is that it was beta tested within the company for many months. The use of e-mail is critical within the organization, so Gmail had to be tuned to satisfy the needs of some of our most demanding customers—our knowledge workers.
6. Encourage creativity. Google engineers can spend up to 20 percent of their time on a project of their choice. There is, of course, an approval process and some oversight, but basically we want to allow creative people to be creative. One of our not-so-secret weapons is our ideas mailing list: a companywide suggestion box where people can post ideas ranging from parking procedures to the next killer app. The software allows for everyone to comment on and rate ideas, permitting the best ideas to percolate to the top.
7. Strive to reach consensus. Modern corporate mythology has the unique decision maker as hero. We adhere to the view that the "many are smarter than the few," and solicit a broad base of views before reaching any decision. At Google, the role of the manager is that of an aggregator of viewpoints, not the dictator of decisions. Building a consensus sometimes takes longer, but always produces a more committed team and better decisions
8. Don't be evil. Much has been written about Google's slogan, but we really try to live by it, particularly in the ranks of management. As in every organization, people are passionate about their views. But nobody throws chairs at Google, unlike management practices used at some other well-known technology companies. We foster to create an atmosphere of tolerance and respect, not a company full of yes men.
9. Data drive decisions. At Google, almost every decision is based on quantitative analysis. We've built systems to manage information, not only on the Internet at large, but also internally. We have dozens of analysts who plow through the data, analyze performance metrics and plot trends to keep us as up to date as possible. We have a raft of online "dashboards" for every business we work in that provide up-to-the-minute snapshots of where we are.
10. Communicate effectively. Every Friday we have an all-hands assembly with announcements, introductions and questions and answers. (Oh, yes, and some food and drink.) This allows management to stay in touch with what our knowledge workers are thinking and vice versa. Google has remarkably broad dissemination of information within the organization and remarkably few serious leaks. Contrary to what some might think, we believe it is the first fact that causes the second: a trusted work force is a loyal work force.

Don't Be Evil is the one that often gets the headline, but the next one down is just as radical in my view: Data Drive Decisions. Here is the whole article from Eric Schmidt, CEO of Google.

What a pity that other companies are not turning to analytics for the same reasons.

Media reactions have been mixed. Some outright hostile. I may be biased but I don't think that using quantitative analysis of 'HR' data is a bad thing. If anything it should help to remove the bias of human interpretation when reviewing performance. We call it 'fact based decision making' everywhere else. What's so special about HR? I'm not saying that it is a universal panacea but used in conjunction with other things like encouraging supportive cultures and inspirational management styles.

It also struck me that Google should perhaps be concentrating more on those employees who are most under-used as an end in itself. Making better use of existing people must have a positive impact on the performance of the company.

A roll call of brain drainers is available here if you are interested. As recently as last week Tom Phillips, the head of Search and Analytics was rumoured to have left Google.