Philip Howard, Research Director at Bloor Research, recently evaluated the performance, scalability, administration, and cost considerations for the leading integrated systems from IBM and Oracle for OnLine Transaction Processing (OLTP) environments. Here is a summary of his conclusions:
And here is a video with his evaluation. It is packed with practical advice regarding storage capacity, processing capacity, and more.
Last year, I blogged about how IBM DB2 has the Leading x86-based TPC-C Result. Well, IBM has further cemented DB2′s position as the leading database for x86-64 systems with a new record-breaking TPC-C benchmark result. The new benchmark result achieved more than 3 million transactions per minute on an IBM System x 3850 X5. The entire system for this result is housed in a single, space-saving 42U rack. The system runs DB2 9.7 on SUSE Linux. It has four Intel Xeon E7-8870 processors running at 2.40GHz (4 processors/40 cores/80 threads). It should also be noted that the system uses Solid State Drive (SSD) storage for faster database access.
For me, the most interesting aspect of this result is not just the performance; it is the price for that performance. And, of course, price/performance is a key consideration for all systems, but especially for cost-conscious x86-64 purchasing decisions. The new system costs US$0.59 per tpmC. As of today, this is the lowest cost of any system in the Top 10 TPC-C performance results (by the way, the next lowest cost also features DB2). See for yourself at TPC-C – Top Ten Performance Results.
If you look at the TPC-C – Top Ten Price/Performance Results, you will see some results from Oracle that offer better price/performance. However, these Oracle results are for very small benchmark systems; approximately one-tenth the size of the IBM DB2 systems. And they include only Oracle Database 11g Standard Edition One; whereas the IBM results include the full enterprise edition of DB2. Not only do the IBM benchmark system give you more product capability for your money, but you can clearly see that the performance of the IBM systems and the cost per transaction for the IBM systems both scale up very nicely.
IBM System x®3850 X5 (Intel Xeon E7-8870 processors 2.40GHz, 4 processors/40 cores/80 threads) TPC-C result of 3,014,684 tpmC, $.59 USD/tpmC, available 9/22/11, DB2 9.7, SUSE Linux® Enterprise Server 11 (SP1)
Last week, I was in Dallas speaking at an event. In the morning, as I left my hotel room, I picked up the Wall Street Journal which was outside my door. I was surprised to see that Oracle are re-running an old advertisement:
Why was I surprised? Because this advertisement is based on an industry benchmark that shows that Oracle uses 9 times the number of CPU cores to achieve only 3 times the performance of the IBM result. To put it another way, if you look at the per-core throughput, the IBM system is 3 times faster than the Oracle system. Oracle highlight the overall throughput of the system, but if you do some investigating you will see that the Oracle system in question uses 1,728 CPU cores, whereas the IBM system in question uses only 192 CPU cores. Considering that you typically pay for software based upon the number of CPU cores, I know which system I’d prefer to be buying software for 
By the way, if you want to see how big these benchmark systems are, check out this blog post… TPC-C Result in Real World Terms: Big Macs and Walmart. Of course, while the benchmark systems themselves are—for the most part—disconnected from today’s real world situations, that is not to say that they are not useful. They are. They serve a very useful purpose in stress testing the different vendor’s products. And they also demonstrate how efficiently those systems scale out.
This is why I’m surprised that Oracle is persisting with advertising this benchmark result. For fun, let’s create a graph that doesn’t show the overall throughput of the systems. Let’s instead create a graph that shows the throughput per CPU core for these benchmark systems. Some people might consider this to be a good measure of efficiency for the systems. As you can see, when you look at this measure of efficiency, it paints a very different picture (of course, the higher the number, the better).
Results on Transaction Processing Performance Council Web site at www.tpc.org. Results as of 06/08/11.
Oracle SPARC SuperCluster (108 chips, 1728 cores, 13824 threads); 30,249,688 tpmC; $1.01/tpmC; available 6/1/11.
IBM Power 780 cluster (24 chips, 192 cores, 768 threads); 10,366,254 tpmC; $1.38/tpmC; available 10/13/10.
HP Integrity Superdome (64 chips, 128 cores, 256 threads); 4,092,799 tpmC; $2.93/tpmC; available 08/06/07.
The second of the series of DB2 ads that compares IBM DB2 with Oracle Database has been released. Here it is…
You can find out more about these ads, and the details behind the claims at ibm.com/facts.
This is the conclusion of my series of blog posts about the Solitaire Interglobal research, which measures various aspects of database environments. In this post, I’m going to focus on performance and cost in IBM Power Systems environments.
Solitaire examined database performance in 1,430 production environments that use IBM Power Systems. You can see the specific breakdown on the counts of the different types of systems in the full report. Their research includes production systems for credit card processing systems, CRM systems, transaction processing systems, and DSS systems.
Here are the summary performance findings for the credit card, CRM, and transaction-processing systems. They indicate the average number of Transactions Per Second (TPS) for these systems. As you can see, DB2 appears to offer a clear performance advantage over Oracle Database. The full report includes details of the number of TPS for each production system in the analysis.
And here are the summary performance findings for the Decision Support System (DSS) environments, which use an Average Queries per Minute metric.
Solitaire also determined the operational costs for these environments. These are the costs for infrastructure and staffing. It does not include overhead costs like facilities, acquisition, and initial deployment. As you can see, the operational costs for IBM DB2 compare very favorably with Oracle Database, especially when you consider the superior performance of the DB2-based systems.
And when you include overhead costs to determine the overall costs, as you might imagine, DB2 offers even better value.
You can read and download the full Solitaire report at Comparing Real World Database Performance: IBM® DB2® versus Oracle® Database and Microsoft SQL Server®.
It is difficult to get direct performance comparisons between Oracle Exadata and competing products. Last month, The Register published what may be such a comparison in its IBM: Our appliance servers smoke Ellison’s ‘phony baloney’ article. They include an image that compares the performance and price of the IBM Smart Analytics System with a leading competitor. The IBM Smart Analytics System is, of course, an integrated hardware/software system for data warehousing and analytics that is based on DB2. The article covers an IBM Investor Day presentation that was delivered by Steve Mills of IBM, and includes the following explanatory passage:
“We benchmark all the time,” Mills said, and he pulled out some real tests to support his point. “We have a favorite competitor who likes the color red. We like the color blue. This is real workload benchmarking, not some phony baloney made-up thing that goes in an ad. We deliver a system that is fast for what customers run.”
Here is the chart that is included in the article. The Register assert that the competitor in red is Oracle Exadata.
According to IDC’s latest server market share report, released earlier this week, Oracle is languishing in fourth place with 6.6 points of market share (IBM has 30.5 points of market share). When you consider that Oracle/Sun has lost server market share in each of the past seven quarters, you have to imagine that Oracle are desperate to stop the rot. Well, today our friends at Redwood Shores attempted to stem the tide by announcing a new TPC-C benchmark result for a cluster of SPARC systems. However, the benchmark result is far from impressive. Sure, the benchmark system has a huge throughput. However, it is woefully inefficient.
Today’s Oracle benchmark result uses 27 64-core Sun SPARC T3-4 servers to process more than 30M tpmC*. In contrast, IBM’s most recent clustered TPC-C result uses 3 64-core IBM Power 780 servers to process more than 10M tpmC**. There are many ways to look at this. You could claim that Oracle uses nine times the number of CPU cores to achieve only three time the performance. Alternatively, you could claim that each CPU core of the IBM system is able to achieve three times the performance of a CPU core in the Oracle system. Either way, in my opinion, it points to a very inefficient benchmark run. Such inefficiencies are surely a concern for customers who are paying for Oracle Database based upon the number of CPU cores in their systems.
At first glance, the cost efficiency of this new benchmark system from Oracle may appear to be impressive—their system costs 1.01 USD per tpmC. However, if you scratch below the surface, you will find that number is quite deceptive. Oracle do not use the perpetual licenses that you would expect, and Oracle do not use the kinds of support contracts that you would expect. If they did use the licenses and support contracts that are most commonly used, then the system costs would skyrocket, and the relative cost inefficiencies of this system would be plain for all to see. For prior coverage of Oracle’s price/performance tactics, see Sun and Oracle TPC Price/Performance Tactics Revealed.
Also, you should be aware that Oracle have once again resorted to sacrificing data integrity for performance in its benchmark systems. They have turned off page integrity checking—I imagine because, according to the Oracle documentation, it incurs a performance degradation of between 1% and 10%. So, even though it is highly unlikely that you would run a production system without page integrity checking, Oracle has chosen to do just that in the interests of squeezing extra performance out of its system.
Given all this context of misleading cost information and questionable system settings, it was timely to read the following article yesterday… Larry Ellison Hearsay: “We Can’t Be Successful if We Don’t Lie to Customers”
Results on Transaction Processing Performance Council Web site at www.tpc.org. Results as of 12/02/10.
* Oracle SPARC SuperCluster with T3-4 Servers (27 x 64 core) (108 chips, 1728 cores, 13824 threads); 30,249,688 tpmC; $1.01/tpmC; available 6/1/11.
** IBM Power 780 cluster (3 x 64 core) (24 chips, 192 cores, 768 threads); 10,366,254 tpmC; $1.38/tpmC; available 10/13/10
Lately, I’ve been hearing a lot about Oracle Exadata performance. It certainly performs quite well. Although it is not necessarily faster than an IBM system. For instance, I know of a recent customer bake-off where an IBM Power 780 system was 2.4 times faster than a half-rack Exadata system. And when you combined the performance difference with IBM’s price advantage, it made the decision a no-brainer for the customer. Naturally, each customer bake-off has so many variables, as to make it useful only for that customer. The key thing to remember is that you need to keep your vendors honest by performing these competitive bake-offs, and not simply comparing a new system’s performance to an old system.
But anyway, the real reason for this blog post is to remind you that product maturity is important and to remind you of the 15 years of product maturity that have gone into the IBM Smart Analytics System. A level of product maturity that is not yet present in Oracle Exadata. Oracle are relative newcomers when it comes to developing integrated hardware/software systems, or engineered systems as they like to call them. And this shows when it comes to the challenges currently facing Exadata users. In the following presentation, I have assembled some references to independent experiences with Oracle Exadata. In each case, I include information about the source of the information, whether it is a Web page or a session at an Oracle event.
Admittedly, every product has bugs and issues. My intention here is simply to highlight that Oracle Exadata may not necessarily be the “IT nirvana” that Larry Ellison may portray it to be. While we have heard Oracle touting the ease of managament of Oracle Exadata, the reality is that this is a very complex system with many of the issues you might expect with a complex system that is so early in its maturity. And, of course, remember that as your system grows, the system will only be as reliable as the underlying software (ie. Oracle RAC).
[slideshare id=5971616&doc=exadataissues-101129135131-phpapp02]
As of yesterday, IBM DB2 has the best x86-based TPC-C result in the industry. IBM’s x86-based system achieved more than 2.3 million transactions per minute on TPC-C benchmark. The benchmark uses the IBM System x 3850 X5, which uses the latest Intel Xeon 7500 Series processor technology. And importantly, the benchmark system is cost efficient, with a price of $0.64 USD per tpmC.
The x3850 X5 achieved this tpmC result using DB2 9.7 and SUSE Linux Enterprise Server 11 (SP1). The configuration included 1.5TB of memory using the IBM MAX5 for System x and eXFlash SSD storage. MAX5 is an industry-first technology from IBM that decouples the memory from the processor, eliminating the need to buy another server to support memory-intensive workloads. Basically, it allows you to increase the system’s memory and storage, which in turn supports larger, faster databases.
For more details, see the benchmark result.
IBM DB2 recently beat Oracle in the race to 10 million tpmC for the TPC-C benchmark. For fun, some of my engineering colleagues spent time putting this in “real world” terms. They wanted to be able to explain to their friends and family just how big the system is, without using jargon like tpmC.
Remember that the DB2 benchmark system processed 10.6 million new orders per minute. Well, if you extrapolate those numbers, such a system would be able to process the transactions if every man, woman, and child on the planet happened to go to McDonalds three times a day to purchase a Big Mac™. Not only that, but all of those transactions would be recorded in real time in a single database system. Of course, I am not advocating that everyone on the planet go out and exclusively dine in that manner. But if a retailer wants to cater for that eventuality, then we have the system for you
Next, they looked at these numbers from another angle. The TPC-C benchmark attempts to model the transactions of a hypothetical wholesale supplier. It includes information for sales districts, where each district has a sales warehouse. This benchmark system was able to process transactions for almost 1 million different sales distribution warehouses. Well, Walmart—the World’s largest retailer—currently has 8,446 stores. (Now, this is not intended to be a rigorous calculation, it is a simple and quick calculation that makes the questionable assumption that Walmart has one distribution warehouse for each store and ignores the relative size of those warehouses.) If you do the math, then this benchmark system can handle 120 times as many distribution warehouses as would be needed by the world’s largest retailer. Yes, that’s right… more than 100 times the number of the world’s largest.
At this stage, you are probably wondering about the applicability of these benchmark results. After all, would you really ever need a system that handles transactions for everyone on the planet eating three times a day in real time? The answer to that question is that today very, very few organizations need to process anything approaching that volume of transactions (and those that do, are often secretive). However, who knows what tomorrow may bring? After all, the world is becomming more instrumented all the time. Every day, there are new electronic devices that have the ability to create transactions (or simply supply information). There are new electronic health monitoring devices, electronic utility meters, electronic sensors in automobiles, and so on. We are only at the beginning of this era of machine-generated data and Big Data. Just think back 5 or 10 years at the volume of data that your IT department was processing. And now apply that growth level to the next 5 or 10 years. If we follow similar patterns, then such large systems might be here sooner than you think. And it will be good if you can be confident that your systems can scale to meet those needs.