A Closer Examination of Oracle’s “Database Performance” Advertisement
Last week, I was in Dallas speaking at an event. In the morning, as I left my hotel room, I picked up the Wall Street Journal which was outside my door. I was surprised to see that Oracle are re-running an old advertisement:
Why was I surprised? Because this advertisement is based on an industry benchmark that shows that Oracle uses 9 times the number of CPU cores to achieve only 3 times the performance of the IBM result. To put it another way, if you look at the per-core throughput, the IBM system is 3 times faster than the Oracle system. Oracle highlight the overall throughput of the system, but if you do some investigating you will see that the Oracle system in question uses 1,728 CPU cores, whereas the IBM system in question uses only 192 CPU cores. Considering that you typically pay for software based upon the number of CPU cores, I know which system I’d prefer to be buying software for 
By the way, if you want to see how big these benchmark systems are, check out this blog post… TPC-C Result in Real World Terms: Big Macs and Walmart. Of course, while the benchmark systems themselves are—for the most part—disconnected from today’s real world situations, that is not to say that they are not useful. They are. They serve a very useful purpose in stress testing the different vendor’s products. And they also demonstrate how efficiently those systems scale out.
This is why I’m surprised that Oracle is persisting with advertising this benchmark result. For fun, let’s create a graph that doesn’t show the overall throughput of the systems. Let’s instead create a graph that shows the throughput per CPU core for these benchmark systems. Some people might consider this to be a good measure of efficiency for the systems. As you can see, when you look at this measure of efficiency, it paints a very different picture (of course, the higher the number, the better).
Results on Transaction Processing Performance Council Web site at www.tpc.org. Results as of 06/08/11.
Oracle SPARC SuperCluster (108 chips, 1728 cores, 13824 threads); 30,249,688 tpmC; $1.01/tpmC; available 6/1/11.
IBM Power 780 cluster (24 chips, 192 cores, 768 threads); 10,366,254 tpmC; $1.38/tpmC; available 10/13/10.
HP Integrity Superdome (64 chips, 128 cores, 256 threads); 4,092,799 tpmC; $2.93/tpmC; available 08/06/07.
Get over it. Oracle has beat IBM in total performance and even price/performance so regardless of whether IBM’s cores are faster, the outcome is that Oracles solution is faster by 3x and lower cost over 3 years! And now that Oracle just published a result on TPC-H @ 1TB scale, once again, Oracle shows superior system performance!
Phil
June 9, 2011 at 5:59 am
Phil, you’re drinking the Oracle Kool-Aid. Licensing aside, it is cheaper to run 192 CPU cores versus 1,700+ CPU cores. Come on Phil, wake up!
Brian
June 14, 2011 at 7:58 am
Licensing aside? Last time I checked, licensing is what makes the huge difference in TCO. Unfortunately in this comparison above, IBM ran DB2 and Oracle ran Oracle DB and yes, theres different licensing terms used so equalizing the licensing I think is the first step in understanding the real cost differences.
So, yes, I agree, its cheaper to run 192 cores versus 1,700+ cores assuming *equal performance* unless those 1,700+ cores provide 3x the performance of the 192 cores (which they do here) and offer better performance/license if you’re running Oracle DB. Theres no proof from IBM that its cheaper to run 192 CPU cores versus 1,700+ CPU cores.
If you look at the details, the SPARC T3-4 system has a performance of 30,249,688 tpmC and required 1,728 cores so with an Oracle database license multiplier of 0.25, that means you’d need 432 Oracle database licenses and therefore in calculating tpmC/Oracle license you get 70,022.
The IBM Power 780 achieved 10,366,254 tpmC and required 192 cores and if the Power system is running Oracle DB, it has an Oracle multiplier of 1.0 so that’s 192 Oracle DB licenses and therefore tpmC/license is 53,991.
So in comparing the SPARC tpmC/Oracle license of 70,022 vs IBM Power 780 tpmC/license is 53,991 and therefore, SPARC has a 30% performance/license advantage. So in this comparison, 1,700+ SPARC CPU cores is cheaper to run than 192 IBM CPU cores due to the Oracle licensing.
Phil
June 14, 2011 at 12:44 pm
Hi Phil,
Your logic is flawed. See below in my previous reply to your position on June 13, 2011 at 11:23 am. You are using a pricing core factor, but neglecting to normalize the pricing. But again, if you are trying to hone in on pricing as the measure of efficiency, why not use the actual Price/Performance key metric from these benchmark runs? After all, that will be a more accurate measure of what you are trying to do, as it will be the actual Price/Performance for the systems in question here. By taking the throughput of these systems, and analyzing pricing considerations without fully considering all aspects of the cost of those systems, you risk drawing incorrect conclusions. Of course, if we do hone in on the Price/Performance key metric from those TPC-C benchmarks, we will need to account for the fact that the IBM result offers full database licenses, whereas the Oracle result only offers 3-year term licenses. And you will also need to account for the fact that the Oracle result uses an unrealistic support package, whereas the IBM result uses a more conventional support package. These two items, if corrected, will have a huge impact on the Price/Performance of the Oracle result.
Regards,
Conor.
Conor O'Mahony
June 15, 2011 at 7:33 am
I suppose you’re aware that after the three years you have to buy new licenses or stop using the system, right?
Fernando Nunes
June 16, 2011 at 2:20 pm
That’s correct Fernando. The Oracle benchmark system includes a term license that coincidentally lasts only as long as the amount of time necessary for the benchmark Price/Performance metric
Whereas the IBM benchmark system includes full perpetual licenses.
Conor O'Mahony
June 16, 2011 at 2:29 pm
Hi Phil,
I’m glad you brought up price/performance.
For people interested in learning more about price/performance for industry benchmarks like TPC, make sure to check out Sun and Oracle TPC Price/Performance Tactics Revealed. IBM and Oracle use different tactics for developing their Price/Performance metrics. People should make sure to understand those differences when comparing Price/Performance numbers, otherwise they risk making incorrect conclusions.
And if people are interested in reading independent analysis regarding the cost of running these database systems, please see Solitaire: Comparing the Performance and Cost of IBM DB2 and Oracle Database and ITG: DB2 Costs are 36% Lower than Oracle Database.
Another interesting source of real-world cost information, for those considering a switch from Oracle Database to IBM DB2, is Forrester Total Economic Impact: Migrating a Heterogeneous Database Environment to IBM DB2.
Regards,
Conor.
Conor O'Mahony
June 9, 2011 at 6:14 am
People looking at the TPC-C numbers look at 2 things…raw performance and price per transaction. All the other points you are bringing may be valid, but it is noise.
Scale out 128 PureScale nodes. IBM needs to demonstrate a comparitive shared disk cluser to Exadata.
Or can the DPF system be scaled out to run 400,000,000 tpm? Considering the clustering limitations of a RAC system, this would be a very high bar for Oracle to cross.
This is a lot like drag racing. It may not be a practical system to run production (i.e. real life), but it shows the upper limits of high performance.
Norm Wong
June 9, 2011 at 12:49 pm
Hi Norm, I think people who do look at just the perf and price/perf numbers from TPC-C are doing themselves a disservice (maybe what the DBMS vendors would like) but they are also being fooled in my personal opinion. I think the TPC did a good thing at the outset by trying to get away from the “drag racing” analogy by including the price/performance metric. The intent I think was to make it so vendors could not just build crazy configurations to get the highest possible performance without thinking of the real world considerations. A valiant effort by the TPC but in my own personal opinion, I think Oracle has made a mockery of the price/performance metric by creating pricing options that I don’t see real world customers using. Some may call it benchmark special pricing but it’s completely legal because it’s on their price books. So no fault of TPC because they obey the letter of the rule…but as a consumer I think people should be a bit more aware of what really makes up these numbers.
Imagine getting a support contract that doesn’t let you upgrade to the next version of the software or talk to a live person if you have a sev 1 problem? And it just so happens that the only products you can get this support for are the ones they just happen to use in benchmarks…coincidence…hmmm. And if you look at how they priced this support offering…well let’s just ask the readers here…how many Oracle customers do you know of that pay less than 1% for software support? I bet more like around 22% is the norm. Then of course there is the “leased” software vs. perpetual license differences as well.
So it may sound like noise but if I’m buying software and paying by the core and getting real world support, I should know that looking at just those two TPC-C metrics for guidance would be, in my own personal opinion, not a wise decision.
Just my two cents but of course I have my own bias…but I do like to look at all the facts (and not just the flashy highlights).
Chris Eaton
June 9, 2011 at 9:17 pm
Hi Norm. Thanks for the comment.
I agree that Performance and Price/Performance are the key TPC-C metrics. And this is why it is important for people to understand how Oracle arrives at their Price/Performance numbers.
Oracle does not have a public TPC-C benchmark result for Exadata yet (the above referenced benchmark was not on Exadata). And IBM does not have one for pureScale. I’m sure these will follow in time.
Your comments are very insightful. You have cut right to the heart of the matter, and touch upon a couple of important considerations:
– The TPC-C data set is highly partitionable (90% of transactions for a given customer are serviced by the same sales warehouse). Therefore, a shared-nothing partitioning approach (using the Database Partitioning Feature) makes sense for the TPC-C benchmark.
– Shared-nothing partitioning, as implemented by DPF, has a near-linear scale-out efficiency. And, there are almost no bounds to that scale-out. So yes, if IBM so chose, it could set an extremely high bar.
Is it worth the cost/effort/time to do that? That’s a really good question, a question that has been debated on more than one occasion here
Conor O'Mahony
June 9, 2011 at 8:50 pm
Conor, thanks for sharing these resources. I appreciate having more facts around these discussions. The third-party resources are particularly helpful.
Tiffany Winman
June 10, 2011 at 12:11 pm
“Oracle system in question uses 1,728 CPU cores, whereas the IBM system in question uses only 192 CPU cores.” can someone boil this down to core factor or pvu and tell the real story.
me
June 10, 2011 at 10:27 pm
Hi me (whoever you are),
Vendors have different ratings for processors.
IBM rates POWER7 processors at 120 PVU and SPARC T3 at 70 PVU. So, if you consider this to be an indicator of relative processing power, IBM reckons that IBM’s POWER7 processors are 1.7 times better than Oracle’s SPARC T3 processors.
Oracle rates POWER7 processors at a 1.0 core factor and SPARC T3 at 0.25. So you could infer that Oracle reckons that IBM’s POWER7 processors are 4 times better than Oracle’s SPARC T3 processors.
That’s a big relative difference (1.7x versus 4x). Of course, this is because the core factors are heavily influenced by each vendor’s pricing strategies. And they can be subject to change, as you can see from this story from The Register: Oracle slashes software prices on own iron, where they talk about Oracle decreasing the SPARC core factor, increasing the POWER core factor, and increasing the Itanium core factor.
So to answer your question, if you apply core factors here, you get different answers depending on which vendor’s core factor you use. And it probably does not make sense to apply a pricing core factor from one vendor to another vendor’s solution. If we go down this path, then we should hone in on the TPC-C Price/Performance primary metric (which incorproates that vendor’s core factor AND their prices). However, if you do that, then you need to be aware of the tricks that Oracle are playing.
Conor O'Mahony
June 11, 2011 at 9:42 am
Well, I think its quite easy to do a quick calculation on (tpmC) performance/license which I guess is the ultimate factor since licensing costs usually far exceed hardware costs.
SPARC T3-4 system has a performance of 30,249,688 tpmC and required 1,728 cores so with an Oracle multiplier of 0.25, that means 432 Oracle licenses and therefore tpmC performance/Oracle license is 70,022.
The IBM Power 780 achieved 10,366,254 tpmC and required 192 cores and if we assume Power780 would deliver same performance running Oracle DB as DB2, with an Oracle multiplier of 1.0 that’s 192 Oracle licenses and so tpmC/license is 53,991.
And in summary, if you run both systems with Oracle DB, you’d have a 30% performance/license advantage running the SPARC T3 system and at $47,500/license, that’s quite a savings! From what I heard, Oracles focus is selling Oracle on Oracle so they’ve made sure its lower cost to run Oracle DB on Oracle (SPARC) hardware. I am sure if you do the calcs for IBM’s DB2 with PVU, you’ll see that the favor is on IBM hardware. Surprise Surprise.
Phil
June 11, 2011 at 2:27 pm
> Well, I think its quite easy to do a quick calculation on (tpmC)
> performance/license which I guess is the ultimate factor
So, instead of a computing efficiency by physical processor, you are suggesting computing efficiency by virtual database license. Interesting. A couple of things to keep in mind:
1) These “virtual database license numbers” are subject to change. For instance, if vendors change their core factor, then these number could change significantly. For instance Oracle had a core factor of .5 for SPARC T2, now it is .25 for SPARC T3. And POWER was 0.75, and now it is 1.0. So these virtual numbers are susceptible to significant change, which can dramatically change the outcome of this type of analysis.
2) But if we are to proceed with this type of analysis, then we need to keep in mind that pricing core factors are only one aspect of pricing. If you use these core factors, then you also need to take into consideration the relative difference in license costs. A license for DB2 costs $40,500, and that includes the first year of subscription and support. A license of Oracle Database costs $57,950 when you include the first year of maintenance ($47,500 + 22% maintenance). Therefore, if you are to use these pricing core factors, then you also need to account for the fact that an Oracle Database license costs 1.43 as much as a DB2 license.
So, by my calculation, the throughput is:
Oracle: 48,966 tpmC per normalized virtual database license
IBM: 45,070 tpmC per normalized virtual database license
where these numbers are:
– The total tpmC
– Divided by the number of CPU cores
– Multiplied by that vendor’s pricing core factor or PVU
– Normalized for that vendor’s pricing
Of course, we should keep in mind that we are getting away from measures that are based on physical entities like a processor. And we are now measuring based upon enterprise software pricing factors, which by their nature not suitable to comparison given the highly-variable discounting practices that define enterprise software pricing.
> if we assume Power780 would deliver same performance running
> Oracle DB as DB2
I don’t think you can make this assumption. IBM DB2 has been shown in real world environments to be faster than Oracle Database on IBM Power Systems. See Comparing the Performance and Cost of IBM DB2 and Oracle Database. This analysis was independently derived from 1,430 production environments that use IBM Power Systems.
> in summary, if you run both systems with Oracle DB, you’d have
After all, this is the price for the respective systems in the actual benchmarks. But, of course, we would need to factor in that the Oracle price does not include perpetual licenses and that the Oracle price has support costs that are only approximately 1% of the software costs (something that is not realistics for many organizations out there).
> a 30% performance/license advantage running the SPARC T3 system
See above for correction. However, if we want to compare price, shouldn’t we just use the TPC-C Price/Performance metric for these results
Conor O'Mahony
June 13, 2011 at 11:23 am
[...] investigation and math, you can usually come up with a reasonable analysis. For instance, here is A Closer Examination of Oracle’s “Database Performance” Advertisement where I look at tpmC per CPU core, which focuses on the number of transactions per CPU rather than [...]
Smarter Questions for a Smarter Planet » What Really Matters in a TPC-C Benchmark Result?
June 13, 2011 at 3:30 pm
Hello Conor,
I am a regular reader of you blog but well I am not sure if user has time to do all these investigations after seeing the Oracle’s advertisement (go into the details like how many cores IBM DB2 or Oracle using) . He will simply assume that “Okay cool Oracle is better choice compare to IBM DB2.”
My question is ‘When Oracle is publishing their stuff in their manner, why IBM is not publishing their theory, their results blah blah in Wall Street or other Journals” Why IBM is not advertising like Oracle (I am not saying to do false advertising but atleast publish whatever you are saying in your blog).
When customer will see your advertisement “OK IBM is giving much more stuff compare to Oracle” then he will look for more info on IBM DB2, then customer will come to your blog, get more info about DB2 and the difference between the advertisements and I am pretty much sure that then customer will go into the details like core, throughput, per-core throughput etc etc
IBM has to find out why after giving so many awesome products like DB2 9.7 or DB2 pureScale, customers are still talking about Oracle, Oracle Exadata etc. Well in my opinion Oracle is very aggressive in product marketing and sales, it is good to be aggressive and you have to when you know your competitors are big guns like IBM, Microsoft etc. IBM has to do something different (or atleast do whatever Oracle is doing
) to beat Oracle.
Thanks
Vinu
Vinu
June 14, 2011 at 6:10 am
Hi Vinu,
Thanks for your comment. IBM does indeed have an advertising campaign at the moment that runs in both print and online media (see Checkmate! New IBM DB2 Advertisement that Compares with Oracle Database and New IBM DB2 vs. Oracle Database Advertising Campaign). These advertisements are running in the Wall Street Journal, and other prominent publications. And this advertising campaign is being rolled out internationally, with more and more countries running the advertisements as we move forward.
As you can see, this advertising campaign is very much focused on Oracle. In the advertisements, we include the headlines from investigations like these, and the details behind the advertisements are at ibm.com/facts.Our goal with the advertisements is to ensure that people have a more balanced set of information presented to them. You will probably notice that many of the facts that appear on these advertisements have already been presented on this blog.
Regards,
Conor.
Conor O'Mahony
June 14, 2011 at 7:48 am
Hi,
Interesting.
Also interesting about no TPC benchmarks regarding IBM Informix.
JJ
June 15, 2011 at 10:05 am
I have tried to do a marketing analysis on Oracle in this pair of posts (sorry, in spanish):
http://logos.com.uy/tiki-view_blog_post.php?postId=35
http://logos.com.uy/tiki-view_blog_post.php?postId=36
Alfonso Vicente
December 27, 2011 at 12:26 am