When we released ZXTM 5.1, we included a useful benchmarking tool named 'ZeusBench' that we use for our internal performance testing and as a load generation tool on the training courses that we run. The very first incarnation of ZeusBench was donated to the Apache project a long time ago, and is now well known as ApacheBench. The new incarnation starts from a completely fresh codebase (it shares a lot of core code with ZWS and ZXTM) and has a new 'rate' mode of operation, is less resource intensive and is often more accurate in its reporting.

You’ll find ZeusBench in the admin/bin directory of your ZXTM installation. Run ZeusBench with the -–help option to display the comprehensive help documentation:

$ZEUSBENCH/admin/bin/zeusbench --help

Using ZeusBench

ZeusBench generates HTTP load by sending requests to the desired server. It can run in two different modes:

Concurrency

When run with the –c N option, ZeusBench simulates N concurrent HTTP clients. Each client makes a request, reads the response and then repeats, as quickly as possible.

Concurrency mode is very stable and will quickly push a web server to its maximum capacity if sufficient concurrent connections are used. It's very difficult to overpower a web server unless you select far too many concurrent connections, so it's a good way to get stable, repeatable transactions-per-second results. This makes it suitable for experimenting with different performance tunings, or looking at the effect of adding TrafficScript rules.

Rate

When run with the –r N option, ZeusBench sends new HTTP requests at the specified rate (requests per second) and reads responses when they are returned.

Rate mode is suitable to test whether a web server can cope with a desired transaction rate, but it is very easy to overwhelm a server with requests. It's great for testing how a service copes with a flash-crowd - try running one ZeusBench instance for background traffic, then fire off a second instance to simulate a short flash crowd effect. Rate-based tests are very variable; it's difficult to get repeatable results when the server is overloaded, and it’s difficult to determine the maximum capacity of the server (use a concurrency test for this).

Comparing concurrency and rate

The charts below illustrate two ZeusBench tests against the same service; one where the concurrency is varied, and one where the rate is varied:

Measuring transactions-per-second (left hand axis, blue) and response times (right hand axis, red) in concurrency and rate-based tests

The concurrency-based tests apply load in a stable manner, so are effective at measuring the maximum achievable transactions-per-second. However, they can create a backlog of requests at high concurrencies, so the response time will grow accordingly.

The rate-based tests are less prone to creating a backlog of requests so long as the request rate is lower then the maximum transactions-per-second. For lower request rates, they give a good estimate of the best achievable response time, but they quickly overload the service when the request rate nears or exceeds the maximum sustainable transaction rate.

Controlling the tests

1. The –n N option instructs ZeusBench to run until it has sent N requests, then stop and report the results.

2. The –t N option instructs ZeusBench to run for N seconds, then stop and report the results.

3. The –f option instructs ZeusBench to run forever (or until you hit Ctrl-C, at which point ZeusBench stops and reports the results).

4. The –v option instructs ZeusBench to report each second on the progress of the tests, including the number of connections started and completed, and the number of timeouts or unexpected error responses.

Keepalives

Keepalives can make a very large difference to the nature of a test. By default, ZeusBench will open a TCP connection and use it for one request and response before closing it. If ZeusBench is instructed to use keepalives (with the –k flag), it will reuse TCP connections indefinitely; the –K N option can specify the number of times a connection is reused before it is closed.

Other ZeusBench options

We’ve just scratched the surface of the options that ZeusBench offers. ZeusBench gives you great control over timeouts, the HTTP requests that it issues, the ability to ramp up concurrency or rate values over time, the SSL parameters and more.

Run

$ZEUSBENCH/admin/bin/zeusbench --help

for the detailed help documentation:

Usage: zeusbench [OPTION]... >URL|File of URLs<
 
Simple examples:
 
Make 1000 requests, 10 at a time, to www.zeus.com:
  zeusbench -n 1000 -c 10 http://www.zeus.com/
 
Request pages from zeus.com at a rate of 50 a second, for 10 seconds:
  zeusbench -r 50 -t 10 http://zeus.com/
 
Run forever, printing stats every second, reading URLs from a file:
  zeusbench -f -v /tmp/url-list
 
Make 50 requests a second, stepping to 250, 450, 650 and 850, running
for a total of thirty seconds:
  zeusbench -t 30 -r 50,200,3 http://www.zeus.com/
 
Request pages for 30 seconds with a concurrency of 10, keeping
connections alive for 3 requests; do not report an error if the
server drops connections after 5000 or more ms of idle time:
  zeusbench -c 10 -k -K 3 -t 30 -i 5000 http://www.zeus.com/
 
Full options:
  -B, --browser=NAME           Behave like a particular browser
  -c, --concurrency=LEVEL      Set concurrency level
  -k, --keepalive              Use HTTP keep-alive requests
  -K, --keepalivesome=NUM      Max keepalives per connection
  -i, --minkeepaliveperiod=NUM Period to expect server to keep connections alive [ms]
  -v, --verbose                Be verbose (reports progress)
  -V, --verbosity=MSECS        Time of each verbose report
  -n, --requests=COUNT         Stop after this many requests
  -R, --random                 Fetch URLs in random order
  -N, --network                Estimate network traffic
  -r, --rate=PERSECOND         Send requests at this rate
  -t, --timelimit=SECS         Stop after this many seconds
  -e, --errorstop              Stop on an error
  -f, --forever                Run forever
  -S, --cookies                Keep cookies
  -T, --timeout=SECS[,SECS]    Set connection [+ data] timeout
  -m, --method=METHOD          Set HTTP method
  -M, --multiplexer=MX         Set multiplexer (e.g. select)
  -H, --header='Key: Value'    Set a HTTP header
  -0, --http1.0                Use HTTP/1.0
  -b, --body                   Read body data from stdin
  -p, --proxy=HOST:Port        Use HTTP Proxy
  -w, --warmup=SECS            Warm-up period (no results used)
  -4, --ipv4                   Use only IPv4 addresses
  -6, --ipv6                   Use only IPv6 addresses
  -P, --procs=PROCS            Use this many processes
  -h, --help                   Print this help message
      --version                Show version information
 
SSL Options:
  -C, --ciphers=CIPHERLIST     Set SSL3 ciphers
  -s, --sessionids             Re-use SSL session IDs
  -2, --ssl2                   Use only SSLv2 for secure requests
  -3, --ssl3                   Use only SSLv3 for secure requests
      --tls1                   Use only TLSv1 for secure requests
      --tls1-sname             Use only TLSv1 + server_name extension.
      --tls1.1                 Use only TLSv1.1 for secure requests
 
For concurrency and rates, you can either specify a single value,
or give START,STEPSIZE,COUNT[,TIME] to step the value up a specified
number of times.

There's a detailed README document in the admin/bin directory; also here.

Running benchmarks

When running a benchmark, it is always wise to sanity-check the results you obtain by comparing several different measurements. For example, you can compare the results reported by ZeusBench with the connection counts charted in ZXTM's Activity Monitor. Some discrepancies are inevitable, due to differences in per-second counting, or differences in data transfers and network traffic counts.

Benchmarking requires a careful, scientific approach, and you need to understand what load you are generating and what you are measuring before drawing any detailed conclusions. Furthermore, the performance you measure over a low-latency local network may be very different to the performance you can achieve when you put a service online at the edge of a high-latency, lossy WAN.

For more information, you may like to refer to the following benchmarking articles on www.zeus.com:

• Benchmarking Guidelines for Zeus Traffic Manager
• Improving the performance of Apache with Zeus Traffic Manager
• Improving the performance of WebLogic with Zeus Traffic Manager

The Apache and WebLogic papers describe a set of tests where a network simulator was used to slow connections down; they investigated the likely real-world performance of the applications over WAN links.

The article Dynamic Rate Shaping of Slow Applications gives a worked example of the use of zeusBench to determine how to rate-shape traffic to a slow application.

Support for ZeusBench

Please note that ZeusBench is provided free-of-charge and is not supported by Zeus Technology. We use it internally and will make best efforts to ensure that it is reliable, stable, efficient and accurate.

Also note that the ZXTM Development Licenses impose a maximum request rate of 10 requests per second; this will obviously impede any benchmarking attempts. You can commence a managed evaluation and obtain a license key that gives unrestricted performance if required.

Kudos

Credits for ZeusBench are due in a large part to Ben, one of Zeus' distinguished engineers, who created ZeusBench for an internal automated benchmarking project out of frustration with ApacheBench. ZeusBench is a key part of our internal automated benchmarking framework which runs continually on development builds to detect performance regressions, and which is used for the majority of performance figures that Zeus publishes.

Owen Garrett [Zeus Dev Team] 26 March 2009