Independent tests demonstrate ZXTM's ability to inspect, process and manipulate complex layer 7 traffic with a performance/price ratio that far exceeds any competitive product.

ZXTM's strengths come from its ability to use low-cost, commodity hardware and operating systems where competitors struggle with proprietary hardware and systems. This article describes the ZXTM and system tuning that can be used in a benchmark environment to squeeze the very best performance from the available hardware.

Hardware Selection

ZXTM is supported on a range of x86, SPARC and Itanium-based platforms. For optimal performance, Zeus strongly recommend using AMD Opteron processors; Zeus' SSL stack performs significantly better on this hardware than on competing processors. For non-SSL processing, the EM64T-capable Intel processors provide similar performance.

Memory and network bandwidth is critical; Zeus recommend 1-4 Gb memory and 2 or more 1Gb interfaces.

Tuning ZXTM

Several changes can be made to the default configuration to improve performance if necessary. Collectively, they may give a 5-10% performance increase, depending on the specific test.

• Virtual Server: Connection Management

  • X-Cluster-Client-IP: For HTTP traffic, ZXTM adds an 'X-Cluster-Client-IP' header containing the remote client's IP address; disable this feature.

  • Client and Server buffers: these read/write buffers are normally sized to 64K; if your benchmark involves large requests or responses, you may wish to increase these.

  • UDP Port SMP: set this to 'yes' if you are testing simple UDP protocols such as DNS.

• Pool: Connection Management

  • Max Connect Time: For benchmarks against busy servers or over a congested network, it may be necessary to increase this.

  • Max Reply Time: The default value (30 seconds) is already high, but it may be necessary to increase this, particularly if the server applications are very slow.

  • Max keepalives per node: If your servers can handle lots of concurrent connections, raising this value can yield a performance increase. However, some application servers may be overwhelmed if this value is set too high.

• Global Settings

  • MaxFDs: If you see file descriptor errors in your error log during testing, you may need to increase this setting. ZXTM uses approximately two file descriptors for each connection (it's approximate because ZXTM multiplexes requests down shared keepalive channels to the servers, but may also maintain a number of idle keepalive connections).

  • chunk_size: This parameter controls how much data ZXTM can read and write at once and defaults to 4096. If you are serving lots of large pages, consider setting it to 8192 or 16384. Doing so will increase the amount of memory ZXTM requires, so drastic increments should be avoided and memory usage should be monitored closely after an increase.

  • SSL Caches: If your SSL tests do not involve any session reuse, you can set these to be small, and can disable the external shared cache. If your tests are using session reuse (a more realistic scenario), you may need to increase the cache sizes and you should use a memory-mapped external shared cache.

  • Recent Conns: Set this to 0 to reduce some internal processing (ZXTM v4 only).

  • Max Retries: Setting this to 1 reduces memory usage, and will help to identify connection problems during the tests that will affect performance.

  • listen_queue_size: Increase this to 1024 (requires a restart). On linux, this value is limited by value of the sysctl /proc/sys/net/core/somaxconn (see below)

Additionally, ensure that you disable or remove any ZXTM features that you do not need to use, such as health monitors, session persistence, TrafficScript rules, logging and activity monitors. Use a simple load-balancing method such as round robin.

After you run each benchmark test, you should review ZXTM's error log ($ZEUSHOME/zxtm/log/errors); any software or resource problems will be logged here.

Operating System Tuning

Linux

The linux kernel is good at self-tuning, but there are a number of limits for the sizes of various kernel resources that you will need to increase. Some of these are obvious - file descriptors and ephemeral ports for example - but many are subtle and can manifest themselves in strange ways. We suggest the following tunables as a starting point, but do monitor kernel logs (use 'dmesg') for unexpected problems.

The following tunings assume a machine with 2Gb or more of memory.

• Ephemeral port range:

  echo "1024 65535" > /proc/sys/net/ipv4/ip_local_port_range

• Listen queue:

  SYN Cookies are recommended:

  echo 1 > /proc/sys/net/ipv4/tcp_syncookies
  echo 8192 > /proc/sys/net/ipv4/tcp_max_syn_backlog
  echo 1024 > /proc/sys/net/core/somaxconn  # restart ZXTM software after changing this

  Without SYN cookies, a much larger value for tcp_max_syn_backlog is required, but this consumes additional kernel memory and scales poorly (the hash table that stores the SYN records is of a fixed size).

You may also change the following settings, although the default kernel values on recent Linux kernels are generally well sized:

• TIME_WAIT:

  echo 1800000 > /proc/sys/net/ipv4/tcp_max_tw_buckets

• Network buffer size (1Gb):

  echo "128000 200000 262144" > /proc/sys/net/ipv4/tcp_mem

• File descriptors:

  echo 2097152 > /proc/sys/fs/file-max

• Netfilter conntrack table size:

  If you're getting the error message: "ip_conntrack: table full, dropping packet." in dmesg, then increase the table size, by adding the lines

  options ip_conntrack hashsize=1310719
  options nf_conntrack hashsize=1310719

  to /etc/modprobe.conf or /etc/modules.conf (this depends on your linux distro), and reboot.

• Spirent workaround:

  If you are using older Spirent test kit, you absolutely must set the following tunables to work around 'optimizations' in their TCP stack:

  echo 0 > /proc/sys/net/ipv4/tcp_timestamps
  echo 0 > /proc/sys/net/ipv4/tcp_window_scaling

These guidelines refer to recent Linux kernels (2.6.10 and later), but equivalent tunings can be made on other platforms. SpecWEB submissions are good sources of tunings that have been performed by various vendors on their platforms.

For other Linux tuning advice, including NIC tunings, please see our FAQ entry Tuning Linux Networking.

FreeBSD

FreeBSD is a stable, widely-deployed, enterprise-level platform from which to run ZXTM. Please reference this document for BSD specific advice.

Other Considerations

Ethernet Auto-negotiation

If one side of a physical ethernet connection is set to autonegotiate and the other has fixed settings, duplex mismatches can result, causing a large number of collisions and very poor network performance. Similar problems can occur with network cards and switches from different manufacturers when both sides are set to autonegotiate.

It is important to ensure, therefore, that the ports on your ZXTM server or appliance are configured with the same settings as thoses on the the switches and routers to which they are connected. If both sides are set to autongetiate and performance problems persist, diabling autonegotation and setting the same fixed settings on both the server and switch/router usually resolves the issue.

Note that a connection between ports with mismatched settings often appears to work when little traffic is flowing through it, but performs increasingly poorly under higher loads. You will typically see a large number of network collisions (in the output of ifconfig for example).

On Linux, you can use the ethtool command to set the network device settings - see the ethtool (8) man page for full details. Some examples follow below:

# Force eth0 to 1 gigabit, full duplex
ethtool -s eth0 speed 1000 duplex full autoneg off
 
# Turn auto-negotation on on eth1
ethtool -s eth1 autoneg 0

Owen Garrett [Zeus Dev Team] 02 September 2005 2 comments