You are currently on IBM Systems Media’s archival website. Click here to view our new website.

MAINFRAME > Tips & Techniques > Systems Management

An Accurate Benchmark Is Important

There are frequently situations when you might consider running a benchmark to compare performance on System z with another hardware platform. It is typical for benchmarking to be part of a decision on which platform to deploy an application to or when you are looking at acquiring new software. Most people think that a benchmark is a simple task of comparing the overall speed of an application on two different systems. However, there is much more planning that is required to design a benchmark that provides enough information for an enterprise to make a really informed decision.

Design A Realistic Benchmark

Whenever you consider running a benchmark, you should start by determining what you want to learn. For example, it does not matter how fast an application runs if the test you design is nothing like your typical daily workload. While a simple race can tell you which system is faster, you have not learned anything that will help you make an educated choice.

A well-designed benchmark needs to be representative of your typical workload. If you are an online retailer, you know what percentage of people simply browse your store versus what percentage actually place orders. Your benchmark should show approximately the same workload distribution to give you really valuable information to make a decision.

Another significant consideration is that a benchmark only provides insight when you are at the limits of what you are comparing. This point is best described through a simple analogy. Let's assume that you are designing a weight-lifting competition with two competitors. In this competition, we will start with a small weight and keep increasing the weight until one competitor is unable to lift the weight. This provided us with some insight into what the limits of one competitor were, but we have not yet learned anything about the second competitor. As you can see, the competition only provides enough information to make a decision when both competitors are pushed to their limits.

Bringing this analogy back to technology, a benchmark is only relevant if it pushes both systems to their limit, or 100 percent CPU utilization. Any benchmark which does not stress both systems will not provide you enough insight to make a decision between the competitors.

Now that we have designed a benchmark that is both typical of our normal workload and stresses both competitors, are we ready to run the comparison? The answer to this question is no, there are still some considerations to ensure a fair comparison that we should consider.

Don’t Create Bias

The first of these considerations is if the benchmark we designed is biased by design. What we mean by biased by design is that the benchmark is designed to take advantage of a feature of one competitor over the other. As you can imagine, a biased design will lead to a biased benchmark, which provides us information that leads to a wrong decision.

So, what are some of the common biases that we should be aware of? For the purposes of this article, we will discuss the following examples, but there are several others that you could imagine. We need to know:

  1. Will the workload we are benchmarking run by itself on dedicated or shared hardware when it is in production?
  2. Does the benchmark have the same I/O characteristics as it would when in production?

Now, let's take a look at these biases in a bit more detail.

Whether the workload will run on dedicated or shared hardware is a significant bias in many benchmarks. Most organizations have a dedicated performance test environment. This means that during a benchmark, the computing resources are dedicated to a single workload. However, when the workload is deployed, most companies are using shared hardware, such as a Linux on z virtual machine, an AIX LPAR or a VMWare image on Intel. Placing a workload into a shared environment introduces overhead that affects performance as the CPU tries to juggle many workloads.

The way that systems try to minimize this overhead is by having special cache memory directly on the CPU. The more cache that is available closer to the CPU means that the overhead of switching tasks is reduced. Just think of this cache memory as space on your desk right by your keyboard. Work that is more important stays on top of your desk. Tasks that are less important get pushed into piles on your desk or the desk drawers. When you need to bring back a task from a file cabinet, is that faster or slower than the paper that is visible on your desktop?

According to the IBM Redbooks “IBM zEnterprise System Technical Introduction,” the cache structure on a zEC12 is described as “each PU has an L1 cache that is divided into 64 KB cache for instructions and a 98 KB cache for data. Each PU also has a private L2 cache, with 2 MB in size, split into 1 MB D-cache and 1 MB I-cache. In addition, each PU contains an L3 cache, which is shared by all six PUs on the chip. The shared L3 cache uses eDRAM. The zEC12 has a 48 MB L3 cache and zBC12 has a 24MB L3 cache.”

In contrast, the Intel Core i7-4960K Processor has an L1 cache of 32 KB per core for data and instructions, an L2 cache of 256KB per core and an L3 cache of 15 MB shared between 6 cores. For those that like comparative numbers, this means that a zEnterprise CPU has two times the L1 cache, four times the L2 cache and three times the L3 cache.

In caching structures, L1 is the fastest cache and the closest to the CPU. Having more cache at the L1 level means that more workloads can be stored closer to the CPU, which leads to better overall performance of multiple simultaneous workloads. The same performance benefits apply to L2 and L3 caches. Basically, more caches means that more workloads can run without having to be swapped to system memory—which is orders of magnitude slower.

The second bias that we want to examine further is about I/O characteristics of a benchmark. The amount of time that is spent moving data between the CPU, memory, spinning disks and other storage has a direct impact on the overall performance of an application.

When designing a benchmark, you might find yourself wanting to use a smaller pool of data because it is a pilot. How many of us want to bring over hundreds of gigabytes for a performance test? Not many, probably, but we also need to make sure we remove biases from our benchmark. Smaller data sets pose a bias for several reasons, including that I/O operations are removed from the timing and that there are caching mechanisms that will keep these smaller data sets in memory.

Understand Performance Characteristics

When creating a benchmark, we need to understand the performance characteristics of the underlying I/O subsystem. For example, on System z there is a dedicated I/O subsystem. This means that when z/OS encounters an I/O request, the request is sent to dedicated processors. Once the I/O operation has completed, the main CPU continues processing the workload. On other platforms, the primary CPU handles managing I/O operations, leaving less CPU for the computational workload. As you can imagine, workloads with larger numbers of I/O operations benefit more from the additional resources.

If we are using a smaller data set, we have artificially lowered the number of I/O operations and biased our benchmark. Additionally, the dedicated I/O subsystem reduces load on the CPU, which means that we need to increase the size of our benchmark to keep both system stressed.

Designing a benchmark is not as simple as you might imagine. You need to ensure that you have designed a benchmark that will provide you with new insights to help you make the correct business decision. The benchmark you need to design needs to be representative of your production workload, deployed on the same physical or virtual environment as it would be in production and be free of biases.

David Morlitz is an IBM Certified Client Architect and Open Group Master Certified Architect who has been designing solutions for customers since 2001.

Like what you just read? To receive technical tips and articles directly in your inbox twice per month, sign up for the EXTRA e-newsletter here.



2019 Solutions Edition

A Comprehensive Online Buyer's Guide to Solutions, Services and Education.

Optimal Service Delivery

Overcome eight key challenges and reduce costs


An Accurate Benchmark Is Important

Advice for the Lazy Administrator

Steps you can take to avoid late nights and system frights.

IBM Systems Magazine Subscribe Box Read Now Link Subscribe Now Link iPad App Google Play Store
Mainframe News Sign Up Today! Past News Letters