Conor O'Mahony's Database Diary

Your source of IBM database software news (DB2, Informix, Hadoop, & more)

Anatomy of an Oracle Marketing Claim

with 10 comments

Yesterday, Oracle announced a new TPC-C benchmark result. They claim:

In this benchmark, the Sun Fire X4800 M2 server equipped with eight Intel® Xeon® E7-8870 processors and 4TB of Samsung’s Green DDR3 memory, is nearly 3x faster than the best published eight-processor result posted by an IBM p570 server equipped with eight Power 6 processors and running DB2. Moreover, Oracle Database 11g running on the Sun Fire X4800 M2 server is nearly 60 percent faster than the best DB2 result running on IBM’s x86 server.

Let’s have a closer look at this claim, starting with the first part: “nearly 3x faster than the best published eight-processor result posted by an IBM p570 server“. Interestingly, Oracle do not lead by comparing their new leading x86 result with IBM’s leading x86 result. Instead they choose to compare their new result to an IBM result from 2007, exploiting the fact that even though this IBM result was on a different platform, it uses the same number of processors. Of course, we all know that the advances in hardware, storage, networking, and software technology over half a decade are simply too great to form any basis for reasonable comparison. Thankfully, most people will see straight through this shallow attempt by Oracle to make themselves look better than they are. I cannot imagine any reasonable person claiming that Oracle’s x86 solutions offer 3x the performance of IBM’s Power Systems solutions, when comparing today’s technology. I’m sure most people will agree that this first comparison is simply meaningless.

Okay, now let’s look at the second claim: “nearly 60 percent faster than the best DB2 result running on IBM’s x86 server“. Oracle now compare their new leading x86 result with IBM’s leading x86 result. However, if you look at the benchmark details, you will see that IBM’s result uses half the number of CPU processors, CPU cores, and CPU threads. If you look at performance per core, the Oracle result achieves 60,046 tpmC per CPU core, while the IBM result achieves 75,367 tpmC per core. While Oracle claims to be 60% faster, if you take into account relevant system size and determine the performance per core, IBM is actually 25% faster than Oracle.

Finally, let’s not forget the price/performance metric from these benchmark results. This new Oracle result achieved US$.98/tpmC, whereas the leading IBM x86 result achieved US$.59/tpmC. That’s correct, when you determine the cost of processing each transaction for these two benchmark results IBM is 39% less expensive than Oracle. (BTW, I haven’t had a chance yet to determine if Oracle Used their Usual TPC Price/Performance Tactics for this benchmark result, as the result details are not yet available to me; but if they have, the IBM system will prove to be even less expensive again than the Oracle system.)

Benchmark results are as of January 17, 2012: Source: Transaction Processing Performance Council (TPC), http://www.tpc.org.
Oracle result: Oracle Sun Fire X4800 M2 server (8 chips/80 cores/160 threads) – 4,803,718 tpmC, US$.98/tpmC, available 06/26/12.
IBM results: IBM System p 570 server (8 chips/16 cores/32 threads) -1,616,162 tpmC, US$3.54 /tpmC, available 11/21/2007. IBM System x3850 X5 (4 chips/40 cores/80 threads) – 3,014,684 tpmC, US$.59/tpmC, available 09/22/11.

About these ads

Written by Conor O'Mahony

January 18, 2012 at 11:01 am

10 Responses

Subscribe to comments with RSS.

  1. “If you look at performance per core, the Oracle result achieves 60,046 tpmC per CPU core, while the IBM result achieves 75,367 tpmC per core”

    And if you look at the Cisco/Oracle result they get 1,053,100 tpmC / 12 cores = 87,758 tpmC/core which is better than both of those results. Granted, it is a different CPU family, but still. Not really any different than comparing a 4 socket point-to-point memory system to an 8 socket NUMA (up to 2 hop memory) which isn’t really “fair” either.

    For fun let’s compare the 10M IBM POWER7 number to Intel: 10,366,254 tpmC / 192 cores = 53,991 tpmC/core

    Doh! All three Intel numbers are better than the 10M tpmC IBM POWER7 system number. But, in all fairness, that really isn’t a good comparison. Big SMP systems won’t have as good of ratio as they bleed some performance for additional capacity.

    A more appropriate number to compare is the 2 socket POWER7 result which yields 1,200,011 tpmC / 8 cores = 150,001 tpmC/core which is an impressive per core number, however, when you look at it at a socket level, it’s only a bit better than the Cisco/Oracle number (13.95% to be exact). Looking at that, I’d say Intel Xeon is an extremely strong competitor to IBM POWER and price/performance wise beats it.

    It will be fun to see how well Intel’s Sandy Bridge and Ivy Bridge compete with IBM POWER.

    Greg Rahn

    January 22, 2012 at 3:00 am

    • Hi Greg. Thanks so much for your comments.

      You say that “Intel Xeon is an extremely strong competitor to IBM POWER and price/performance wise beats it”. The IBM result you reference has a price/performance of $0.69 and the Cisco result has a price/performance of 0.58. However, if you look at the details, you will see that database software license and maintenance is $544,320 for the IBM configuration and is $12,700 for the Cisco/Oracle configuration. Obviously, this price/performance comparison isn’t quite an apples-to-apples comparison. The IBM configuration includes the enterprise-level software, the kind of perpetual license that organizations typically purchase, and the kind of maintenance organizations typically purchase. Whereas the Cisco/Oracle configuration includes Standard One software, an unrealistic 3-year term license, and a highly-restrictive maintenance policy. When you take all of this into account, it significantly shifts the price/performance story strongly in IBM’s favor.

      As regards, whether Intel Xeon is a strong competitor to IBM POWER, again let’s look at the details. Processors are an important driver of overall system performance, but so are memory and storage. The IBM configuration had 32GB of memory and less than 37TB of storage; the Cisco/Oracle configuration had 384GB of memory and more than 62TB of storage. The significantly larger amounts of memory (more than 10x) and storage in the Cisco/Oracle configuration will certainly have contributed significantly to their performance number. When you take this into account, the processor comparison will be more accurate, and again heavily tilt in IBM’s favor.

      Thanks,
      Conor.

      Conor O'Mahony

      January 23, 2012 at 11:06 am

      • That 2 socket POWER7 result has 16 x 32GB = 512GB of RAM, not 32GB total, thus 128GB more memory than the Cisco/Oracle kit which had 384GB. But sure, the Cisco/Oracle result had more storage capacity but the storage space difference is really a red herring – space in itself does not drive performance. Quite the contrast though when it comes to RAM, which has a significant impact on performance.

        Let’s only look at the hardware costs of both (that’s really the interesting bit here anyway):

        IBM: $714,029 + $113,339 + $56,289 + $3,580 = $887,237
        Cisco/Oracle: $42,777 + $1,698 + $704,282 + $70,718 = $819,475

        IBM: $887,237 / 1,200,011 tpmC = $0.73935738922393/tpmC
        Cisco/Oracle: $819,475 / 1,053,100 tpmC = $0.77815497103789/tpmC

        This IBM result edges out the Cisco/Oracle result by a few pennies when looking at hardware costs alone. However, the entire Cisco server is ~2x cheaper than the IBM POWER7 CPUs alone with maintenance ($42,777 + $1,698 vs $67,000 + $23,040 = $44,475 vs $90,040) [if I read the IBM pricing sheet correctly]. Now focusing in on processing power, I think your statement of “heavily tilt[ed] in IBM’s favor” is quite a stretch.

        My point here is that given a two socket Intel Xeon 5600 system and a two socket IBM POWER7 system with the same amounts of RAM and identical storage, I’d wager the price performance number will easily be in Intel’s favor.

        Greg Rahn

        January 24, 2012 at 1:51 am

      • Hi Greg,

        I stand corrected on the memory front. That was very sloppy of me… thanks for catching… more evidence that despite what I think, I really cannot multi-task :-)

        Looking at your numbers below, those two system configurations are interesting:
        – The IBM configuration has $714k+ worth of servers and $56k+ worth of storage,
        – Whereas the Cisco/Oracle configuration has $42k+ worth of servers and $704k+ worth of storage.

        The IBM configuration focused their spending on the servers, whereas the Cisco/Oracle configuration focused their spending on the storage. And the difference is huge. When I get some time, I must dig a little deeper into this to understand the system characteristics…

        Regards,
        Conor.

        Conor O'Mahony

        January 24, 2012 at 11:09 am

      • The IBM configuration focused their spending on the servers, whereas the Cisco/Oracle configuration focused their spending on the storage. And the difference is huge. When I get some time, I must dig a little deeper into this to understand the system characteristics…

        Once you analyze the FDR you will realize that statement is not as true as you would first be led to believe. TPC breaks pricing into “server” and “storage” but if the storage is actually internal it falls into the “server” pricing, if it is external then it falls into the “storage” category. In the IBM case the SSDs were part of the “server” category since they are in the I/O Drawers, whereas the Cisco/Oracle result they used Violin storage which falls into the “external” category.

        Greg Rahn

        January 24, 2012 at 12:24 pm

  2. [...] I read and heard a few seemingly unrelated items this week that led me to the title of this week’s post.    First were the IBM 4Q and 2011 results and our CFO’s presentation and Q&A with financial analysts, next an analysis of these results in context of other technology providers, and finally was Conor O’Mahony’s analysis of recent Oracle benchmarks. [...]

  3. I work in new system sales in Europe and read technology comparisons with interest looking for some ammunition. In practice most opportunities are competitive replacement and the comparatrive differences in performance between IBM and the competition do not make a sufficiently compelling reason for change. On numerous occasions IBM has shown via POTs that our servers and database are superior but the prospects stuck with what they have. Change carries risk and acts against entrenched thinking and vested interests.

    The reality is that with this sort of noise Oracle are telling their customers that they are investing and improving. For most this is enough; the technology they have is alive and not a dead end. I believe that performance comparisons will only carry weight if they appear to be a quantum leap that show Oracle technology to be seriously dated and that we have something inarguably superior. So we can show that while Oracle have improved by 10%, we have improved by 50%.

    But, the practical value of improved performance lies in the ability to build a cost case around reducing software licences. (I think you are making this point?) So keep up the good work.

    Brian Alford

    January 24, 2012 at 5:36 am

  4. When I make purchasing decisions, I look at more than the benchmarking hype that is generated. I can twist any benchmark result to hype it and make it seem like “we are leading the industry.” Surprised nobody went after the TPMC per thread, TPMC per Watt or TPMC per gHZ claims.

    Most of the benchmarks do not provide you with any indication of TCA, TCO, reliability, up-time and so on. And, as far as I know, POWER is one of the few that actually use server virtualization as part of their benchmarking process. In today’s environment, server virtualization should be considered a key component of any benchmark that is submitted.

    Something that I think is a more important and distinguishing difference between x86, SPARC, Itanium, POWER and the z Architecture actually comes from HP’s Odyssey announcement http://h18004.www1.hp.com/products/solutions/mcci/index.html. If you read between the lines, HP is saying that x86, Windows on x86 and Linux on x86 are not mission critical, do not have the RAS required to run a business and cannot meet business required SLA’s in their current state.

    When I read the announcement, I was just shaking my head. Ouch to trusted partners Microsoft, Red Hat and Intel.

    Yar Ydnar

    January 24, 2012 at 3:08 pm

  5. I found some interesting information from IBM at the following websites. Looks like they make a pretty good overall case and have proof points on performance.

    http://www.ibm.com/systems/power/hardware/benchmarks
    http://benchmarkingblog.wordpress.com/

    Wonder what the rest of the vendors have.

    R. Kus

    January 24, 2012 at 3:16 pm

  6. Conor and Greg – you’ve both done a great job probing and clarifying the numbers. It would be great to see each of you, on your respective blogs, now post a new piece that lays out the details you fleshed out clearly. What we have now is a great thread that has to be read serially to get the point. You’ve punctured a few of each other’s claims – in an admirably courteous way – and reinforced others. I’d love to see each of your takes on the net effect of the discussion.

    Merv Adrian

    January 30, 2012 at 8:33 pm


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 71 other followers

%d bloggers like this: