Category Archives: Benchmark

Anatomy of an Oracle Marketing Claim

Posted on by
10

Yesterday, Oracle announced a new TPC-C benchmark result. They claim:

In this benchmark, the Sun Fire X4800 M2 server equipped with eight Intel® Xeon® E7-8870 processors and 4TB of Samsung’s Green DDR3 memory, is nearly 3x faster than the best published eight-processor result posted by an IBM p570 server equipped with eight Power 6 processors and running DB2. Moreover, Oracle Database 11g running on the Sun Fire X4800 M2 server is nearly 60 percent faster than the best DB2 result running on IBM’s x86 server.

Let’s have a closer look at this claim, starting with the first part: “nearly 3x faster than the best published eight-processor result posted by an IBM p570 server“. Interestingly, Oracle do not lead by comparing their new leading x86 result with IBM’s leading x86 result. Instead they choose to compare their new result to an IBM result from 2007, exploiting the fact that even though this IBM result was on a different platform, it uses the same number of processors. Of course, we all know that the advances in hardware, storage, networking, and software technology over half a decade are simply too great to form any basis for reasonable comparison. Thankfully, most people will see straight through this shallow attempt by Oracle to make themselves look better than they are. I cannot imagine any reasonable person claiming that Oracle’s x86 solutions offer 3x the performance of IBM’s Power Systems solutions, when comparing today’s technology. I’m sure most people will agree that this first comparison is simply meaningless.

Okay, now let’s look at the second claim: “nearly 60 percent faster than the best DB2 result running on IBM’s x86 server“. Oracle now compare their new leading x86 result with IBM’s leading x86 result. However, if you look at the benchmark details, you will see that IBM’s result uses half the number of CPU processors, CPU cores, and CPU threads. If you look at performance per core, the Oracle result achieves 60,046 tpmC per CPU core, while the IBM result achieves 75,367 tpmC per core. While Oracle claims to be 60% faster, if you take into account relevant system size and determine the performance per core, IBM is actually 25% faster than Oracle.

Finally, let’s not forget the price/performance metric from these benchmark results. This new Oracle result achieved US$.98/tpmC, whereas the leading IBM x86 result achieved US$.59/tpmC. That’s correct, when you determine the cost of processing each transaction for these two benchmark results IBM is 39% less expensive than Oracle. (BTW, I haven’t had a chance yet to determine if Oracle Used their Usual TPC Price/Performance Tactics for this benchmark result, as the result details are not yet available to me; but if they have, the IBM system will prove to be even less expensive again than the Oracle system.)

Benchmark results are as of January 17, 2012: Source: Transaction Processing Performance Council (TPC), www.tpc.org.
Oracle result: Oracle Sun Fire X4800 M2 server (8 chips/80 cores/160 threads) – 4,803,718 tpmC, US$.98/tpmC, available 06/26/12.
IBM results: IBM System p 570 server (8 chips/16 cores/32 threads) -1,616,162 tpmC, US$3.54 /tpmC, available 11/21/2007. IBM System x3850 X5 (4 chips/40 cores/80 threads) – 3,014,684 tpmC, US$.59/tpmC, available 09/22/11.

Benchmark Results for Informix TimeSeries in Meter Data Management

Posted on by
Reply

AMT-SYBEX are a leading provider of platforms for traditional and smart metering. They created a Meterflow Benchmark to help customers choose the best underpinning infrastructure for their platform, and they worked with IBM to run that benchmark with Informix TimeSeries. I previous blogged about Why Informix Rules for Time Series Data Management. Well, the results of this benchmark further illustrate the benefits of Informix TimeSeries. The following quote is from the resulting AMT-SYBEX case study:

We believe that this represents ground breaking levels of performance which is ten times faster than other published benchmarks in this area.

As you can see, Informix is 10x faster than the leading database software they previously worked with. If you read the Executive Summary, you will also see that IBM Informix enjoys almost linear scalability when going from 10 million meters up to 100 million meters, which is a great testament to the efficiency of operation for Informix TimeSeries.

Industry Benchmark Result for DB2 pureScale: SAP Transaction Banking (TRBK) Benchmark

Posted on by
7

A couple of years ago, IBM introduced the pureScale feature, which provides application cluster transparency (allowing you to create shared-disk database clusters). At the time, IBM had taken their industry-leading clustering architecture from the mainframe, and brought it to Unix environments. IBM subsequently also brought it to Linux environments.

Today, IBM announced its first public industry benchmark result for this cluster technology. IBM achieved a record result for the SAP Transaction Banking (TRBK) Benchmark, processing more than 56 million posting transactions per hour and more than 22 million balanced accounts per hour. The results were achieved using IBM DB2® 9.7 on SUSE Linux® Enterprise Server. The cluster contained five IBM System x 3690 X5 database servers, and used the IBM System Storage® DS8800 disk system. The servers were configured to take over workload in case of a single system failure, thereby supporting high application availability. For more details, see the official certification from SAP.

IBM DB2 Improves Performance Lead for x86-64 Systems with a new Record-Breaking Result

Posted on by
2

Last year, I blogged about how IBM DB2 has the Leading x86-based TPC-C Result. Well, IBM has further cemented DB2′s position as the leading database for x86-64 systems with a new record-breaking TPC-C benchmark result. The new benchmark result achieved more than 3 million transactions per minute on an IBM System x 3850 X5. The entire system for this result is housed in a single, space-saving 42U rack. The system runs DB2 9.7 on SUSE Linux. It has four Intel Xeon E7-8870 processors running at 2.40GHz (4 processors/40 cores/80 threads). It should also be noted that the system uses Solid State Drive (SSD) storage for faster database access.

For me, the most interesting aspect of this result is not just the performance; it is the price for that performance. And, of course, price/performance is a key consideration for all systems, but especially for cost-conscious x86-64 purchasing decisions. The new system costs US$0.59 per tpmC. As of today, this is the lowest cost of any system in the Top 10 TPC-C performance results (by the way, the next lowest cost also features DB2). See for yourself at TPC-C – Top Ten Performance Results.

If you look at the TPC-C – Top Ten Price/Performance Results, you will see some results from Oracle that offer better price/performance. However, these Oracle results are for very small benchmark systems; approximately one-tenth the size of the IBM DB2 systems. And they include only Oracle Database 11g Standard Edition One; whereas the IBM results include the full enterprise edition of DB2. Not only do the IBM benchmark system give you more product capability for your money, but you can clearly see that the performance of the IBM systems and the cost per transaction for the IBM systems both scale up very nicely.

IBM System x®3850 X5 (Intel Xeon E7-8870 processors 2.40GHz, 4 processors/40 cores/80 threads) TPC-C result of 3,014,684 tpmC, $.59 USD/tpmC, available 9/22/11, DB2 9.7, SUSE Linux® Enterprise Server 11 (SP1)

Oracle uses 9x CPUs to Achieve only 3x the Performance for TPC-C Benchmark

Posted on by
5

According to IDC’s latest server market share report, released earlier this week, Oracle is languishing in fourth place with 6.6 points of market share (IBM has 30.5 points of market share). When you consider that Oracle/Sun has lost server market share in each of the past seven quarters, you have to imagine that Oracle are desperate to stop the rot. Well, today our friends at Redwood Shores attempted to stem the tide by announcing a new TPC-C benchmark result for a cluster of SPARC systems. However, the benchmark result is far from impressive. Sure, the benchmark system has a huge throughput. However, it is woefully inefficient.

Today’s Oracle benchmark result uses 27 64-core Sun SPARC T3-4 servers to process more than 30M tpmC*. In contrast, IBM’s most recent clustered TPC-C result uses 3 64-core IBM Power 780 servers to process more than 10M tpmC**. There are many ways to look at this. You could claim that Oracle uses nine times the number of CPU cores to achieve only three time the performance. Alternatively, you could claim that each CPU core of the IBM system is able to achieve three times the performance of a CPU core in the Oracle system. Either way, in my opinion, it points to a very inefficient benchmark run. Such inefficiencies are surely a concern for customers who are paying for Oracle Database based upon the number of CPU cores in their systems.

At first glance, the cost efficiency of this new benchmark system from Oracle may appear to be impressive—their system costs 1.01 USD per tpmC. However, if you scratch below the surface, you will find that number is quite deceptive. Oracle do not use the perpetual licenses that you would expect, and Oracle do not use the kinds of support contracts that you would expect. If they did use the licenses and support contracts that are most commonly used, then the system costs would skyrocket, and the relative cost inefficiencies of this system would be plain for all to see. For prior coverage of Oracle’s price/performance tactics, see Sun and Oracle TPC Price/Performance Tactics Revealed.

Also, you should be aware that Oracle have once again resorted to sacrificing data integrity for performance in its benchmark systems. They have turned off page integrity checking—I imagine because, according to the Oracle documentation, it incurs a performance degradation of between 1% and 10%. So, even though it is highly unlikely that you would run a production system without page integrity checking, Oracle has chosen to do just that in the interests of squeezing extra performance out of its system.

Given all this context of misleading cost information and questionable system settings, it was timely to read the following article yesterday… Larry Ellison Hearsay: “We Can’t Be Successful if We Don’t Lie to Customers”

Results on Transaction Processing Performance Council Web site at www.tpc.org. Results as of 12/02/10.
* Oracle SPARC SuperCluster with T3-4 Servers (27 x 64 core) (108 chips, 1728 cores, 13824 threads); 30,249,688 tpmC; $1.01/tpmC; available 6/1/11.
** IBM Power 780 cluster (3 x 64 core) (24 chips, 192 cores, 768 threads); 10,366,254 tpmC; $1.38/tpmC; available 10/13/10

IBM DB2 has the Leading x86-based TPC-C Result

Posted on by
1

As of yesterday, IBM DB2 has the best x86-based TPC-C result in the industry. IBM’s x86-based system achieved more than 2.3 million transactions per minute on TPC-C benchmark. The benchmark uses the IBM System x 3850 X5, which uses the latest Intel Xeon 7500 Series processor technology. And importantly, the benchmark system is cost efficient, with a price of $0.64 USD per tpmC.

The x3850 X5 achieved this tpmC result using DB2 9.7 and SUSE Linux Enterprise Server 11 (SP1). The configuration included 1.5TB of memory using the IBM MAX5 for System x and eXFlash SSD storage. MAX5 is an industry-first technology from IBM that decouples the memory from the processor, eliminating the need to buy another server to support memory-intensive workloads. Basically, it allows you to increase the system’s memory and storage, which in turn supports larger, faster databases.

For more details, see the benchmark result.

IBM Again Shatters World Record on Two-Tier SAP SD Benchmark

Posted on by
Reply

IBM today announced a new two-tier SAP Sales and Distribution (SD) benchmark result that crushes the Oracle/Sun results. The new benchmark result uses a 256-core Power 795 system with DB2 to support 126,063 SAP SD benchmark users. This is more than three times the number of users than Oracle’s largest system (a 256-core Sun SPARC Enterprise M9000 running Oracle Database). The IBM system was also able to handle more than three times the number of SAP SD users than Oracle’s result from September of this year that runs four clustered 32-core Sun Fire X4470 servers.

This new result is also the first to break the 500,000 SAPS level on a single system with more than 688,000 SAPS. The SAP Application Performance Standard (SAPS) is a measure of system throughput for business deliverables like customer sales orders or invoices. This result follows hot on the heels of last month’s result from IBM, which used a 128-core Power 795 system running DB2 to support 70,032 users. If you look at the effect of doubling the number of CPU cores from one benchmark to the next, you can see that the number of users supported is almost doubled. This near linear scaling is very reassuring for organizations whose SAP environments are growing significantly.

This benchmark result indicates that IBM can support your largest SAP systems, and that IBM can support the growth of those systems in a very efficient manner. But don’t forget that IBM Power Systems have tremendous virtualization capabilities. In fact, the latest optional PowerVM virtualization software allows customers to run more than 1,000 virtual servers on a single physical system. This means that you could also use a system like one in this benchmark to consolidate your various existing SAP environments onto one physical system, vastly simplifying your infrastructure and reducing costs in data center floor space, energy, management resources, and database and web application software.

TPC-C Result in Real World Terms: Big Macs and Walmart

Posted on by
2

IBM DB2 recently beat Oracle in the race to 10 million tpmC for the TPC-C benchmark. For fun, some of my engineering colleagues spent time putting this in “real world” terms. They wanted to be able to explain to their friends and family just how big the system is, without using jargon like tpmC.

Remember that the DB2 benchmark system processed 10.6 million new orders per minute. Well, if you extrapolate those numbers, such a system would be able to process the transactions if every man, woman, and child on the planet happened to go to McDonalds three times a day to purchase a Big Mac™. Not only that, but all of those transactions would be recorded in real time in a single database system. Of course, I am not advocating that everyone on the planet go out and exclusively dine in that manner. But if a retailer wants to cater for that eventuality, then we have the system for you :-)

Next, they looked at these numbers from another angle. The TPC-C benchmark attempts to model the transactions of a hypothetical wholesale supplier. It includes information for sales districts, where each district has a sales warehouse. This benchmark system was able to process transactions for almost 1 million different sales distribution warehouses. Well, Walmart—the World’s largest retailer—currently has 8,446 stores. (Now, this is not intended to be a rigorous calculation, it is a simple and quick calculation that makes the questionable assumption that Walmart has one distribution warehouse for each store and ignores the relative size of those warehouses.) If you do the math, then this benchmark system can handle 120 times as many distribution warehouses as would be needed by the world’s largest retailer. Yes, that’s right… more than 100 times the number of the world’s largest.

At this stage, you are probably wondering about the applicability of these benchmark results. After all, would you really ever need a system that handles transactions for everyone on the planet eating three times a day in real time? The answer to that question is that today very, very few organizations need to process anything approaching that volume of transactions (and those that do, are often secretive). However, who knows what tomorrow may bring? After all, the world is becomming more instrumented all the time. Every day, there are new electronic devices that have the ability to create transactions (or simply supply information). There are new electronic health monitoring devices, electronic utility meters, electronic sensors in automobiles, and so on. We are only at the beginning of this era of machine-generated data and Big Data. Just think back 5 or 10 years at the volume of data that your IT department was processing. And now apply that growth level to the next 5 or 10 years. If we follow similar patterns, then such large systems might be here sooner than you think. And it will be good if you can be confident that your systems can scale to meet those needs.

Why IBM didn't use pureScale for its TPC-C Benchmark Run

Posted on by
Reply

I got a Twitter Direct Message from Ian Bjorhovde (@idbjorh) yesterday, asking why IBM’s latest TPC-C benchmark result uses shared-nothing partitioning (Database Partitioning Feature) rather than shared-disk clustering (DB2 pureScale). After all, partitioning is typically used in warehouse situations, while clustering is typically used in OLTP situations like those being measured in the TPC-C benchmark. He also went on to remind me that IBM spent the last year touting DB2 pureScale as the Oracle RAC killer. So why is IBM not using DB2 pureScale in this situation?

The answer is actually quite straightforward. IBM evaluated the various options—including DB2 pureScale—and determined that for the purposes of reclaiming TPC-C throughput leadership, a system that uses shared-nothing partitioning would achieve the best results. In other words, IBM chose a system that is optimized for the intended workload, just as you would do with any project. You see, the TPC-C data set is highly partitionable (90% of transactions for a given customer are serviced by the same sales warehouse). And it only makes sense for IBM to take advantage of this fact in its system design (just as Oracle takes advantage of that fact to route transactions to specific nodes in its Oracle RAC cluster). So the bottom line here is that IBM chose the system that best suits the intended workload, and in this case that system is based on shared-nothing partitioning.

IBM DB2 Beats Oracle in the Race to Break 10 Million tpmC

Posted on by
2

Remember the infamous advertisement from Oracle where they claimed they would be announcing a TPC-C benchmark result with XX million tpmC, hinting at a massive double digit TPC-C result. And then they ended up announcing a result with more than 7 million tpmC. At the time, a few people joked that they didn’t expect the first X in Oracle’s boast to be a zero. Well, last week IBM became the first vendor to break the 10 million tpmC barrier for the TPC-C benchmark.

The new benchmark result is impressive in many regards. It provides 35% greater throughput1 than the Oracle/Sun result. But this great leap forward in throughput is only part of the story. An equally important consideration is system efficiency, with the IBM result having only half the number of CPU cores when compared with the Oracle/Sun system. If you do the math, the IBM result gets 2.7 times more productivity per CPU core than the Oracle/Sun result. And because software is typically licensed by CPU core, that has the potential to add up to a lot of savings, both for initial purchase price and for ongoing maintenance costs.

But the aspect of this result that really astounded me was the price/performance metric. The IBM result achieved 41% lower cost per transaction than the best Oracle/Sun TPC-C performance result. And don’t forget the games that Oracle played in their price/performance metric. This means that IBM is 41% better even when Oracle resorts to these tactics!

You may recall that, when Oracle talked about their benchmark result, they boasted that their system needed less energy than IBM’s system. Of course, what they neglected to mention was that their system used Solid State Disk (SSD) technology and, at the time, IBM’s used only spinning disk. Well, now the IBM system also uses SSD technology. With this level playing field, the IBM system requires 35% less energy per transaction (Watts/tpmC) than published Oracle energy usage data2.

I’m sure that, like me, everyone in the DB2 community is delighted that DB2 has reclaimed its position at the top of the TPC-C benchmark in such an emphatic manner. It is further proof of DB2′s performance leadership. In fact, since 1 January 2003, DB2 has enjoyed more time leading the TPC-C, TPC-H 10TB, and SAP 3-Tier SD benchmarks than all other vendors combined.

TPC-C, TPC-H, and SAP Benchmark Leadership

1 IBM POWER7 TPC-C Result: IBM Power 780: 10,366,254 tpmC at $1.38USD/tpmC avail 2010/10/13, (24proc/192core/768thread) Oracle Sun TPC-C Result: Sun SPARC Enterprise T5440: 7,646,486 tpmC at $2.36USD/tpmC, avail 2010/03/19, (48proc/384core/3072thread). TPC-C results available at www.tpc.org.

2 Energy claims for either system are not related to official TPC-Energy results and should not be compared to TPC-Energy results. Energy comparisons are between IBM and Oracle/Sun system configurations referenced above. IBM POWER7 energy consumption = 65130 Watts, 0.006282 Watts/tpmC; Oracle/Sun system consumption = 73932 Watts, 0.009668 Watts/tpmC. IBM energy estimate based in IBM calculations using customer-available energy estimation tools for IBM servers, storage energy estimation reports available from IBM Techline services, and published component active power consumption specifications. Oracle energy estimate from Oracle-published results available at http://www.oracle.com/features/strategic-focus-report.pdf.