Death Of The Relational Database

A recent entry by Tony Bain in his excellent 'Innovations in Data Management' blog caught me a little by surprise. In it he talks about the NoSQL movement - the group of people and organisations that say we can do without the RDBMS's from the likes of Oracle, Microsoft and IBM. This is new to me.

In a nutshell, the argument runs as follows: Massively scalable databases exist that are not relational and they power some of the biggest sites on the internet - Amazon, Google and Facebook to name three.

For those who don't know about NoSQL, take a look at ACM blogger Michael Stonebraker or a nice summary from Computerworld. You can't find anything about NoSQL on wikipedia yet - that's how new it is. Stonebraker constructs a convincing argument in favour of 'the death of the RDBMS':

• For data warehouses, a column store beats a row store by approximately a factor of 50 on typical business intelligence queries. The reason is because column stores read only the columns of interest to the query and not all of them. In addition, compression is more effective in a column store. Since the legacy systems are all row stores, they are vulnerable to competition from the newer column stores.
• For online transaction processing (OLTP), a lightweight main memory DBMS beats a row store by a factor of 50. Leveraging main memory and the fact that no DBMS application will send a message to a human user in the middle of a transaction, allows an OLTP DBMS to run transactions to completion with no resource contention or locking overhead. 
• In XML, where the current major vendors have spent a great deal of energy extending their engines, it is claimed that specialized engines, such as Mark Logic or Tamino, run circles around the major vendors. 

The argument then runs that if you don't want performance like this, then just get an open source RDBMS for free. I am beginning to agree (even the Cortex is MySQL) and if free is too hard for you, the price of SQLServer is good value for most organisations these days.

So what are the reasons for spending big dollars on an RDBMS? Here are the reasons I can think of for a large company:

• Nobody every got sacked for buying Oracle or DB2.
• The IT specialists have built their career and expertise on a specific vendor's product line. What's in it for them to support a change that they see as undermining that?
• Who's going to hire someone with Voldemort or MongoDB experience?? 
• For most applications, the RDBMS can do the job - so what if the company spends $500,000 more on hardware to do it. Don't forget, the hardware guys are also comfortable with running the big RDBMS on 'their' boxes.
• A surprising number of data warehouse developer's lack the skills to really understand the differences pointed out by the NoSQL people. Besides - they're not the one's paying for the infrastructure they use.
• When something goes wrong with the software - who do you call? Will they be around next year/month? 
• "Our RDBMS solution works today - or it will when we upgrade/implement the new module." So why change?

I am happy to concede that most of these reasons are not technical. Politics is very real in larger enterprises and you don't stay long if you ignore this fact. In the past I have been lucky to hire some very good developers because of their frustration with life in a large corporate data shop.

Anyway, the NoSQL revolutionaries got together recently and you can read/view/listen to the presentations on Johan Oskarsson's site (he is a developer for Last.fm in London). The presentation on the Cassandra database by Avinash Lakshman of Facebook has some interesting stats comparing Cassandra and MySQL for example.

It is curious that the NoSQL people don't call their solutions databases. Instead they are a a "highly available key-value store" (Amazon) and a "distributed storage system for managing structured data" (Google). At least MongoDB does describe itself as "a high-performance, open source, schema-free document-oriented database." Not exactly as snappy a label as RDBMS and I wonder why CDBMS (Columnar RDBMS) isn't good enough.

Check out NoSQL - it could be useful to you.

Comments

[James B]

"Nobody every got sacked for buying Oracle or DB2"

Perhaps they should. Megavendors are expensive and often supply a feature set well above requirements. MySQL serves a large number of businesses very well (I have run a moderately complex web game using it for about 5 years with no trouble from the DB engine). SQL Server is vastly cheaper that Oracle and is not that far off it in feature set. For a small - mid size data warehouse both those systems will do just fine, Oracle / DB2 are overkill.

But, back to the point of the article - it doesn't surprise me that an alternative to RDBMS is starting to emerge. Whether this is it is another question, but the RDBMS is getting pretty old now, and was well adapted to an environment where (relatively) small amounts of well structured data existed. Now data volumes are getting larger and less structured, a new approach is required and i'll be interested to see what comes next. Thanks for keeping us updated on developments.

[OzAnalytics]

Good points Tony. For me, and my more modest data needs, the RDBMS does me fine. The main factor in my decision for database selection is usually the licensing cost - at least when it is my own online venture. MySQL has served me well so far - as it has millions of others.

With my other hat on - corporate manager in the big-end of town - the decision is different. In the recent past the primary data warehouses I have had built used Oracle and DB2. This was for many of the reasons I listed in my blog but ultimately I didn't care that much so long as the DB of choice supported the business activities I required of it. All the major DBs did.

My decisions were made easier by the discounting arrangements of the vendors. For their global clients the vendors will negotiate massive (80% or more) discounts when they sign global enterprise agreements. So I 'go with the flow' regarding the DB technology choice. I focus my efforts on delivering analytics that answer business questions.

As a footnote - I have found that the presentation layer is where the biggest performance issues are found in my 'analytic' world. If I need to get interactive data mining or discovery capabilities to end users, then it is the performance of the online report or dashboard that kills my infrastructure - and it is the fragility of the presentation layer's technology not the underlying database that is the root cause.

I suspect that my next BI platform will use one of the newer (and much cheaper) presentation layer products rather than sticking with the mega-vendor product again.

[Tony Bain]

There is a lot of debate going on. Personally I don’t believe in the death of the RDBMS, in fact my position is far from this.

But I do believe we went too far down the “RDBMS for everything mentality” this decade (which largely translated to “Oracle, SQL Server or MySQL for everything”). Instead I think we need a richer set of alternatives for our data management layer. Relational databases clearly are not the best fit for a number of requirements, yet we have been shoehorning them into anything & everything. The “NoSQL” distributed key/value stores have clear benefits for some needs, but RDBM’s are still most suitable for most applications.

However the RDBMS isn’t without fault. There are some significant problems that are compounding as both data volumes & processing response times grow. Now there is a bunch of interesting start ups focused on keeping the essence of the RDBMS but being innovative and fixing some of these problems, which will help to take it through the next decade.