The Future of the NoSQL, SQL, and RDBMS Markets
Matt Asay wrote an interesting article for The Register titled SQL Survives Murder Attempt by Mutant Stepchild where he opines that “NoSQL remains a tiny blip in the overall datastore universe“. And he’s correct. When it comes to the universe of data management deployments, NoSQL usage is a tiny fraction of the overall data management market.
The term NoSQL implies that these emerging data management technologies are fighting the SQL establishment. I would argue that, instead, they are fighting the traditional Relational Database Management System (RDBMS) establishment. The NoSQL movement has evolved out of a loose association of technologies that solve challenges that traditional relational solutions are not designed to solve well. RDBMS software is good at addressing the majority of our data management challenges. However, there are instances where the relational approach simply does not work well. While these situations are a relatively small part of the data management universe, they are nonetheless important. After all, these emerging technologies are meeting a very real market need, and the likelihood is that this market need will grow as the business world shifts towards use cases where these NoSQL solutions shine. So, essentially we have a situation where a bunch of data management technologies are emerging to solve a subset of data management challenges that are not well served by currently available technologies. I expect that some of these NoSQL use cases will evolve into reasonable, if relatively small, segments of the overall data management market.
To further illustrate that the term NoSQL is probably a misnomer, some of these NoSQL technologies have plans to adopt SQL interfaces. How will the NoSQL movement react when some of its products start adopting SQL interfaces? As Alanis Morissette would say, isn’t it ironic!
But anyway, back to the topic at hand. While certain segments of the high tech media are portraying this as a big battle between the incumbent and a challenger, I would instead portray it as the emergence of new technologies to augment the incumbent. The NoSQL solutions are essentially a set of technologies that address use cases that are not well served by existing relational technology. The relational database software market is huge today, and I don’t see this changing in any significant way in the foreseeable future. Despite what some wide-eyed and naive smaller vendors may claim, these emerging technologies are simply not in a position to wholesale unseat the incumbent relational database technology. Instead, they will likely augment relational technology in many IT environments. In some IT environments, where their business is built around NoSQL-friendly use cases, it may actually be the opposite with relational technologies augmenting the more dominant NoSQL technologies. However, as Matt points out in his article, the fact that SQL-based systems have such a low barrier-to-entry will ensure their long-term dominance. Another significant factor in determining how thing will evolve is the huge investment and significant maturity of the ease-of-use, ease-of-maintenance, stability, reliability, and security features that make RDBMS systems enterprise-ready today. And don’t forget that, as emerging technologies play catch-up with this huge investment, the relational vendors will continue to innovate.
In my opinion, the likely outcome here is that there will be a set of separate battles among vendors for each of the individual market segments corresponding to the NoSQL use cases. And the larger vendors will participate in the more lucrative of these market segments, either with organically-developed or acquired products. And, for the most part, the servicing of these use cases will be relatively independent of the larger relational database market. What’s you opinion?
Hi Conor,
For those of us in the more enlightened participants in the alternative datastore space, we’ve adopted NOSQL rather than NoSQL. That is, “Not Only SQL” rather than “No [to] SQL” which is far more sane and likely.
It’s unfortunate that we’ve gotten SQL mixed up in all of this, since the innovation going on in the NOSQL world is really about experimenting with different data models not query languages.
Jim
Jim Webber
August 11, 2011 at 6:58 am
Hi Jim,
Thanks for your comment. Your second sentence conveys what I was trying to articulate, but in a much more elegant manner.
Cheers,
Conor.
Conor O'Mahony
August 23, 2011 at 11:52 pm
There are different application scenarios, and different requirements. RDBMS has its strength but also its weakness. I would love to use RDBMS to solve all my problems, but it doesn’t. For managing and querying big data with RDBMS, the loading performance, the hardware requirements, the license cost for parallel database, and the complexity on optimizing parallel queries is a huge challenge.
I would see the two technologies will eventually merge: integrating DBMS capabilities such as query language and indexing into MapReduce will be the trend.
See our work on this: Hadoop-GIS: https://web.cci.emory.edu/confluence/display/HadoopGIS
wangfsh
August 11, 2011 at 11:42 am
I agree. However, relational technology does solve many problems well, and many more problems “well enough”. It’s certainly going to be interesting watching all of this pan out. I continue to be surprised at the level of attention garnered by the NoSQL movement. All this attention can only help with the adoption levels for the constituent technologies.
Regards,
Conor.
Conor O'Mahony
August 23, 2011 at 11:56 pm
Hi Connor, I wish it was as simple as SQL & RDBMS is good for this and NoSQL is good for that. For me at least, the waters are much muddier than that.
The benefit of SQL & RDBMS is that its general purpose nature has meant it can be applied to a lot of problems, and because of its applicability it is become mainstream to the point every developer on the planet can probably write basic SQL. And it is justified, there aren’t many data problems you can’t through a RDBMS at and solve.
The problem with SQL & RDBMS, well essentially I see two. Firstly, distributed scale is a problem in a small number of cases. This can be solved by losing some of the generic nature of RDBMS and keeping SQL such as with MPP or attempts like Stonebraker’s NewSQL. The other way is to lose RDBMS and SQL altogether to achieve scale with alternative key/value methods such as Cassandra, HBase etc. But these NoSQL databases don’t seem to be the ones gaining the most traction. From my perspective, the most “popular” and fastest growing NoSQL databases tend to be those which aren’t entirely focused on pure scale but instead focus first on the development model, such as Couch and MongoDB. Which brings me to my second issue with SQL & RDBMS.
Without a doubt the way in which we build applications has changed dramatically over the last 20 years. We now see much greater application volumes, much smaller developer teams, shorter development timeframes and faster changing requirements. Much of what the RDBMS has offered developers – such as strong normalization, enforced integrity, strong data definition, documented schemas – have become less relevant to applications and developers. Today I would suspect most applications use a SQL database purely as a application specific dumb datastore. Usually there aren’t multiple applications accessing that database, there aren’t lots of direct data import/exports into other aplications, no third party application reporting, no ad-hoc user queries and the data store is just a repository for a single application to retain data purely for the purpose of making that application function. Even several major ERP applications have fairly generic databases with soft schemas without any form of constraints of referential integrity. This is just handled better, from a development perspective, in the code that populates it.
Now of course the RDBMS can meet this requirement – but the issue is the cost of doing this is higher than what it needs to be. People write code with classes, RDBMS uses SQL. The translation between these two structures, the plumbing code, can be in cases 50% of more of an applications code base (be that it hand-written code or automatic code generated by a modeling tool). Why write queries if you are just retrieving and entire row based on key. Why have a strict data model if you are the only application using it and you maintain integrity in the code? Why should a change in requirements require you to now to go through the process of building a schema change script/process that has to have deployed sync’d with application version. Why have cost based optimization when all the data access paths are 100% known at the time of code compilation?
Now I am still largely undecided on all of this. I get why NoSQL can be appealing. I get how it fits with today’s requirements, what I am unsure about if it is all very short sighted. Applications being built today with NoSQL will themselves grow over time. What may start off today as simple gets/puts within a soft schema’d datastore may overtime gain certain reporting or analytics requirements unexpected when initial development began. What might have taken a simple SQL query to meet such a requirement in RDBMS now might require data being extracted into something else, maybe Hadoop or MPP or maybe just a simple SQL RDBMS – where it can be processed and re-extracted back into the NoSQL store in a processed form. It might make sense if you have huge volumes of data but for the small scale web app, this could be a lot of cost and overhead to summarize data for simple reporting needs.
Of course this is all still evolving. And RDBMS vendors and NoSQL are both on some form of convergence path. We have already started hearing noises about RBDMS looking to offer more NoSQL like interfaces to the underlying data stores as well as the NoSQL looking to offer more SQL like interfaces to their repositories. They will meet up eventually, but by then we will all be talking about something new like stream processing
Thanks Connor for the thought provoking post.
Tony Bain
August 11, 2011 at 8:09 pm
Hi Tony,
Very interesting. I’ve talked to other people who have also talked about the developer-driven aspects of NoSQL adoption, and the role that programming interfaces are playing in this adoption. I’ve also heard people being attracted to the relatively low overhead of these “NoSQL” solutions, but that may change as they mature.
As regards the relatively high cost of relational solutions for transactional systems, there is a very real potential for significant disruption here. I’m not sure that the current pricing levels are sustainable in the market. We have seen significant downward price pressure for analytical systems. Surely this will eventually also happen for transactional systems.
The key will be who can take advantage of this situation, and how they can make it happen. And, of course, the reality is that if someone can come along and disrupt this market, then it will take quite a few years for them to do it. In my experience, significant market disruption does not happen over night, and usually takes some time.
Regards,
Conor.
Conor O'Mahony
August 24, 2011 at 12:14 am
[...] Conor O’Mahony over at IBM wrote a good post on a favorite topic of mine “The Future of the NoSQL, SQL, and RDBMS Markets”. [...]
Log Buffer #233, A Carnival of the Vanities for DBAs | The Pythian Blog
August 12, 2011 at 2:06 am
[...] A recent article offers insight into the current state and future prospects of NoSQL solutions. Conor O’Mahony writes, “Matt Asay wrote an interesting article for The Register titled SQL Survives Murder Attempt by Mutant Stepchild where he opines that ‘NoSQL remains a tiny blip in the overall datastore universe.’ And he’s correct. When it comes to the universe of data management deployments, NoSQL usage is a tiny fraction of the overall data management market. The term NoSQL implies that these emerging data management technologies are fighting the SQL establishment. I would argue that, instead, they are fighting the traditional Relational Database Management System (RDBMS) establishment. The NoSQL movement has evolved out of a loose association of technologies that solve challenges that traditional relational solutions are not designed to solve well.” [...]
NoSQL’s Place on the Data Map | DATAVERSITY
August 12, 2011 at 12:42 pm
Thanks for the comments everyone. Don’t get me wrong, I’m excited by these technologies. I’ve spent quite a lot of time evangelizing native XML storage. And I’m currently very interested in both key-value stores and graph stores, as well as Hadoop of course. I primarily wrote the blog post after multiple conversations/debates with people, where I felt they had unrealistic expectations regarding the revenue potential for NoSQL technologies. These are exciting technologies, and they will serve a segment of our data management market, but I found that some people are looking at the overall database software revenue numbers and making, what I feel are, unreasonable “guess-timates” about the addressable market size for these new technologies. Remember, technical excellence doesn’t necessarily guarantee business success. The best technology doesn’t always win. The best “whole product” (to use Moore’s terminology) stands a better chance of winning, but that is not even guaranteed to win either. There are many factors at play, some of which are related to the technology.
Conor O'Mahony
August 18, 2011 at 11:50 am
I hear a lot of SQL and NoSQL with no mention of Object Databases. Where do you see these technologies fitting in Conor?
German
August 25, 2011 at 1:42 pm
I imagine that Object Database software falls under the NoSQL umbrella. Similar to Native XML Database software, Graph Database software, and so on… there has been a market for Object Database software for some time. Albeit a relatively small market.
Given the relational-object impedance mismatch, in theory you would have thought that the market for Object Database software would have been huge. However, the reality is that Object Database software has never gained widespread adoption. There are many theories as to why that has happened. The adoption of stored procedures, user-defined functions, and triggers by the relational vendors probably had a significant part to play.
It’s amusing to see vendors who have been selling Native XML Database software and Graph Database software for a number of years now trying to “draft” in the “NoSQL slipstream”. Its understandable. After all, they want to seize any opportunity to get attention for their products.
Conor O'Mahony
August 26, 2011 at 11:55 am
[...] recommend you read Conor O'Mahony’s great post “The Future of the NoSQL, SQL, and RDBMS Markets” to get his perspective on the implications of NoSQL on the database [...]
The Syncfusion Company Blog | NoSQL is no fad
September 26, 2011 at 4:12 pm