Category Archives: Benchmark

Truth in Advertising – Oracle's Claims about Performance and Energy Consumption

Posted on by
12

Last year, I blogged about some issues regarding Truth in Advertising – Advanced Data Compression in Oracle 11g. Well, our friends over at Oracle are at it again. This time thay are making some questionable advertising claims. Here is the ad in question. From looking at the ad, you would think that an Oracle/Sun system gives you seven times the performance, while consuming one sixth the amount of power. Let me explain why this is MISLEADING.

Truth in Advertising - Oracle Claims about Performance and Energy Consumption

There are two claims here, both under the banner of being “independently verified.” The independent verification refers to the fact that it is drawing from TPC benchmark data.

The first claim pertains to performance. When someone mentions performance to me in relation to the TPC-C benchmark, I immediately think of the primary metrics, and in tpmC in particular. After all, this is a primary metric for a reason. tpmC represents the performance of the systems for all workloads (for the record, Oracle did outperform IBM by about 20% for tpmC). But, Oracle obviously aren’t looking at tpmC when devising this claim. Instead, they are focusing only on one subset of the performance numbers in the benchmark. In other words, if the TPC-C benchmark were like a triathalon, then Oracle did really well in one of the events. It is downright misleading for them to claim that a 7x lead in one event is indicative of their performance in the overall race.

By the way, you should also be aware that Oracle are comparing an older IBM system to their latest and greatest, which is questionable in its own right. With the rate of change in the industry, IBM’s 2008 result is not indicative of its performance levels today. In addition, the Oracle configuration actually uses 115 TB of solid-state disk (for a database size of 6TB). The IBM result does not use any solid-state disk, instead working with mechanical disks. Solid state disk manufacturers claim that their products are hundreds of times faster than mechanical disk. However, for Oracle, that translated into only a 20% lead over IBM.

But, believe it or not, this is not as misleading as the second claim pertaining to energy consumption. First of all, the TPC results being touted here were posted before the TPC-Energy metric was introduced and reported. This energy data is not coming from the TPC results. Putting this claim under the “independently verified” banner is simply misleading.

Let’s dig a little deeper and do some math with the server specs. Note that Oracle needed a cluster of 12 SPARC Enterprise T5440 servers for their benchmark result, whereas IBM needed only one IBM Power 595 server.

If you go to the Sun SPARC T5440 Power Calculator, you can see that a single server consumes between 1551 watts (idle) and 2002 watts (100% active). There are 12 of these servers in Oracle’s benchmark, which results between 18.612 KW and 24.024 KW of power consumption.

If you look at the same information for the IBM POWER 595, you will see that during typical usage a P595 consumes 18.5kW. At 100% utilization, it consumes 27.7kW.

That’s right, the Oracle configuration in an idle state consumes more power than the IBM configuration performing a typical workload. Oracle, please explain how you arrived at the 6x number in the ad…

Comparing IBM DB2 and Oracle Database for SAP

Posted on by
5

Last week, at an event that IBM hosted for analysts and press, there were some very interesting Twitter messages from John Rymer, Merv Adrian, and Carl Olofson, including:

Event Tweets

In case you are interested in the charts they were referring to, I will include a couple of them here.

The first chart compares two systems that achieve comparable performance for the 2-tier SAP benchmarks (well, actually the IBM system provides more then 15% better performance). In this chart, the focus is on the number of CPU cores needed to achieve these results. You can see that the IBM system requires 1/4 the number of CPU cores. This is important because software is typically licensed based upon the number of CPU cores, and therefore the efficiency of the system has a big determination on the price you end up paying.

IBM DB2 on Power 780 versus Oracle Database on Sun M9000

The second chart compares how many SAP end users are supported per CPU core for IBM and Oracle systems. The per-core efficiency of the system is an important consideration for initial purchase, system maintenance, system upgrades, and system growth.

IBM DB2 on Power 750 versus Oracle Database on Sun T5440

The first chart is based upon the following 2-tier SAP EHP 4 for SAP ERP 6.0 (Unicode) benchmark results, which are valid as of 4/7/2010:

  • DB2 9.7 on IBM Power System 780, 8p / 64–c / 256–t, POWER7, 3.8 GHz, 1024 GB memory, 37,000 SD users, dialog resp.: 0.98s, line items/hour: 4,043,670, Dialog steps/hour: 12,131,000, SAPS: 202,180, DB time (dialog/ update):0.013s / 0.031s, CPU utilization: 99%, OS: AIX 6.1, cert# 2010013. For more details, see http://www.sap.com/benchmark.
  • Oracle 10g on SUN M9000, 64p / 256-c / 512–t, 1156 GB memory, 32,000 SD users, SPARC64 VII, 2.88 GHz, Solaris 10, cert# 2009046. For more details, see http://www.sap.com/benchmark.

The second chart is based upon the following 2-tier SAP EHP 4 for SAP ERP 6.0 (Unicode) benchmark results, which are valid as of 4/7/2010:

  • IBM SAP 2-Tier SD result of 15,600 SD (Sales & Distribution) users (Average dialog response time: 0.98 second), running DB2 9.7 on AIX 6.1 and SAP enhancement package 4 for SAP ERP 6.0 on the IBM Power System 750 with 4 POWER7 3.55 GHz processor chips (32 cores, 128 threads) and 256 GB main memory, certification Number: 2010004. For more details, see http://www.sap.com/benchmark.
  • Sun Microsystems SAP 2-Tier SD result of 4,720 SD (Sales & Distribution) users (Average dialog response time: 0.97 second), running Oracle 10g on Solaris 10 and SAP enhancement package 4 for SAP ERP 6.0 (Unicode) on the SPARC Enterprise T5440 with 4 UltraSPARC T2 Plus 1.6 GHz processor chips (32 cores, 256 threads) and 256 GB main memory, certification Number: 2009026. For more details, see http://www.sap.com/benchmark.

Clarifying Some Recent Oracle Benchmark Claims

Posted on by
5

Earlier this month, Oracle issued a press release regarding their latest results for the SAP® Business Intelligence-Data Mart (BI-D) Standard Application Benchmark. In the press release, Oracle claim that the new Oracle Database result surpasses the best IBM DB2 result by more than six times. While this statement is true, it is a little misleading. You see IBM does not perform the SAP BI-D benchmark tests for DB2 on the Linux, Unix, Windows, or z platforms. IBM performs these benchmarks only for DB2 on the i platform. So, when Oracle says that its result surpasses the best IBM DB2 result by more than six times, it is comparing a 4-node Oracle RAC cluster with a single DB2 for i system. Oracle is also comparing a recent benchmark result against an IBM result that is more than a year old. So, while these comparisons seem spectacular at first glance, once you look at the details you realize that they are not entirely fair.

Also, please note that the BI-D benchmark is a read-only benchmark. We have increasingly heard from clients that this benchmark is not realistic for their environments. IBM clients tell us that their systems typically have mixed workloads. For instance, demands for more current data typically result in trickle feeding, and they often need to rebuild cubes while queries are running. For these reasons, IBM is increasingly unlikely to perform the BI-D benchmark tests. Instead, clients are interested in the more realistic SAP BI-MXL which includes inserts, updates, and deletes with the queries.

The other thing to be aware of is that these benchmark results further illustrate the gap in levels of support for SAP applications provided by these database software products. With IBM DB2, you simply set one variable (DB2_WORKLOAD=SAP) and all the required settings are configured automatically for you. However, if you look at the benchmark submission package, you will see that these new benchmark results from Oracle use some interesting configuration settings. Undocumented and unsupported configuration parameters in Oracle Database begin with an underscore. Well, Oracle uses six of these undocumented and unsupported configuration parameters in these benchmark results. I’m sure its not reassuring for Oracle Database users to learn that Oracle needs to use undocumented and unsupported parameters to get optimal performance. The parameters in question are:

  • _optimizer_cost_based_transformation= off
  • _query_rewrite_fudge = 1
  • _improved_row_length_enabled= FALSE
  • _optim_peek_user_binds = FALSE
  • _optimizer_autostats_job = FALSE
  • _optimizer_save_stats = FALSE

There are some interesting settings in here. Apparently, they don’t trust their cost-based optimizer because they disable it. And they appear to be setting some sort of query rewrite fudge factor. I wonder what fudge factor you should choose for optimal performance in your environment. And, if they are disabling it, perhaps the row length setting does not improve things as its name suggests it might :-)

PS. Many thanks to Chris Eaton for his expertise and help with this blog post.

Presentation for Smackdown: IBM DB2 vs. Oracle Database

Posted on by
Reply

I delivered a session titled “Smackdown: IBM DB2 vs. Oracle Database” at the IBM Information on Demand Conference in Las Vegas a little more than a week ago. It is a business case analysis of IBM DB2 and Oracle Database on distributed systems. Since the session, I have had a large number of requests for the charts from attendees, so I thought I would also post the presentation here…

[slideshare id=1691440&doc=bld-2580abusinesscaseupdated-090707082010-phpapp01]

Sun and Oracle TPC Price/Performance Tactics Revealed

Posted on by
10

Sun and Oracle have a TPC-C benchmark result that delivers 7,646,486 tpmC with a price performance of $2.36 USD/tpmC*. Here are a couple of things about that result (and other recent results from Oracle) that you may not be aware of:

  1. The Oracle benchmark result does not use perpetual software licenses
  2. The Oracle benchmark result uses a Web-based incident support contract

Oracle are comparing their result to the IBM TPC-C result with $2.81 USD/tpmC**. However, this may not be an apples-to-apples comparison because the IBM result includes pricing for 24×7 support, upgrade protection, and perpetual licenses; the Oracle result does not include any of these features. If you include 24×7 support, upgrade protection, and perpetual licensing, you’ll find that the prices most customers will pay are significantly different than what Oracle includes in the benchmark. Let’s see why this is so…

When Oracle prepares TPC-C benchmark results, they typically use a special license called the Oracle Term License (denoted by the ‘Unlimited Users for 3 Years’ text below):

Oracle TPC-C benchmark result uses term licenses

The Oracle Term License is not a perpetual license, like the software license that organizations typically purchase. Instead, it is like a lease. This term license for three years costs 45% less than a perpetual license. After the three year period, you no longer own the right to use the software. At that stage, to keep using the software, you either have to purchase the software license for an additional term or purchase a perpetual license. To truely compare this result with TPC-C results that use perpetual licenses, like the IBM results, you need to do some math with the Oracle Database license costs.

When it comes to support, things get a little more interesting. First of all, you should note that the cost of support for a term license is the same as the cost of support for a perpetual license. If you have a look at the Oracle Web site and do some math, the costs for support work out to be more than 40% of the term license cost. But, support costs for this benchmark are not more than 40% of the software license costs. They are actually a little more than 1% of the software license costs.

You see, for this benchmark and many others, Oracle uses something called the Oracle Incident Server Support Package (OISP). The OISP is a support package that has no telephone support. Instead, it allows you up to 10 Web-based incident requests per server (that expire within one year). What’s more, OISP has no upgrade protection and does not entitle you to future upgrades of the Oracle Database software. The cost of the OISP is $2,300 per server, which is why you see the cost of Oracle support for this benchmark at $82,800 (12 nodes in the cluster for 3 years of support). This represents a little more than 1% of the costs of the term license costs.

As you can see, Oracle manage to significantly improve their price performance result by using term licenses and by using a limited support offering. In the event that you do not use term licenses or this limited support offering, you will need to do additional work to see what the systems would cost for you, or to compare the Oracle results with other TPC-C results.

All TPC results available on the Transaction Processing Performance Council Web site at www.tpc.org.
* 12-Node Sun SPARC Enteprise T5440 server cluster; 7,646,486 tmpC; $2.36/tpmC; available 03/19/10.
** IBM Power 595 Server Model 9119-FHA; 6,085,166 tpmC; $2.81/tpmC; available 12/10/08.

Oracle Accused of Being a "Shameless Truth-Bender"

Posted on by
5

Larry EllisonRyan Tate claims that “Larry Ellison will always be a shameless truth-bender” in an article titled Larry Ellison Can’t Be Bothered With the Facts. The article talks about an advertisement that appeared recently in the Wall Street Journal and goes on to share some interesting anecdotes about Oracle advertising.

The advertisement in question suggests that Oracle plan to announce a new TPC-C result using Sun hardware on October 14. One thing this article does not mention is that, as far as I can tell, the advertisement does not follow the TPC rules. If you read Section 8.2.2 Unfair Use, you will see that Oracle could be interpreted to have violated several of these rules and unfairly used the TPC benchmark. You see, the TPC does not allow organizations to “make TPC-related claims or lead the reader to TPC-related conclusions which are untrue or cannot be substantiated by the entire body of results.” TPC does this to prevent misleading advertising. You may wonder how this could be misleading. Well, it could be misleading because it does not tell the full story. For instance, what if their upcoming benchmark result requires an inordinately expensive hardware configuration. Without certain primary metrics, TPC results cannot reasonably be compared.

Aside from Oracle’s apparent flaunting of the TPC rules, this advertisement is also notable because Oracle indicates that the upcoming TPC result will be on Sun hardware. The reason that this is interesting is that Sun has publicly derided the TPC-C benchmark in the past. In fact, they have not participated in the TPC-C benchmarks for several years, claiming that the benchmark does not reflect real world transactional workloads, and that the benchmark is easily gamed by vendors. It is an interesting about-face for Sun. I wonder what prompted this change in what was a very firm stance on their part.

DB2 Keynote Presentation for Break Free Roadshow

Posted on by
Reply

Here is a keynote presentation that I recently put together for the Break Free roadshow events. This presentation describes the staff, server, and storage costs for DB2 when compared with its major competitors. There is also a section that talks about moving from Oracle Database to DB2.

[slideshare id=1691474&doc=breakfreekeynote0619-090707082905-phpapp01]