Independent tests demonstrate Zeus Traffic Manager's ability to inspect, process and manipulate complex layer 7 traffic with a performance/price ratio that far exceeds any competitive product.

Zeus's strengths come from its ability to use low-cost, commodity hardware and operating systems where competitors struggle with proprietary hardware and systems. This article describes the Zeus and system tuning that can be used in a benchmark environment to squeeze the very best performance from the available hardware.

Hardware Selection

Zeus software is supported on a range of x86 and SPARC-based platforms.

Memory and network bandwidth is critical; Zeus recommend 1-4 Gb memory and 2 or more 1Gb interfaces; 10 Gb interfaces when the network infrastructure allows.

Tuning Zeus Traffic Manager

Several changes can be made to the default configuration to improve performance if necessary. Collectively, they may give a 5-10% performance increase, depending on the specific test.

• Virtual Server: Connection Management

  • X-Cluster-Client-IP: For HTTP traffic, Zeus Traffic Manager adds an 'X-Cluster-Client-IP' header containing the remote client's IP address; disable this feature.

  • Client and Server buffers: these read/write buffers are normally sized to 64K; if your benchmark involves large requests or responses, you may wish to increase these.

  • UDP Port SMP: set this to 'yes' if you are testing simple UDP protocols such as DNS.

• Pool: Connection Management

  • Max Connect Time: For benchmarks against busy servers or over a congested network, it may be necessary to increase this.

  • Max Reply Time: The default value (30 seconds) is already high, but it may be necessary to increase this, particularly if the server applications are very slow.

  • max_idle_connections_pernode: On systems with large numbers (16+) of CPU threads, such as those based on Oracle's UltraSPARC T2 processors, this should be set to '0' (unlimited), to avoid connections being closed too frequently, resulting in large numbers of connections between Zeus Traffic Manager and its back-end nodes in the TIME_WAIT state.

• Global Settings

  • MaxFDs: If you see file descriptor errors in your error log during testing, you may need to increase this setting. Zeus uses approximately two file descriptors for each connection (it's approximate because Zeus multiplexes requests down shared keepalive channels to the servers, but may also maintain a number of idle keepalive connections).

  • chunk_size: This parameter controls how much data Zeus can read and write at once and defaults to 4096. If you are serving lots of large pages, consider setting it to 8192 or 16384. Doing so will increase the amount of memory Zeus requires, so drastic increments should be avoided and memory usage should be monitored after an increase.

  • SSL Caches: If your SSL tests do not involve any session reuse, you can set these to be small, and can disable the external shared cache. If your tests are using session reuse (a more realistic scenario), you may need to increase the cache sizes and you should use a memory-mapped external shared cache.

  • Max Retries: Setting this to 1 reduces memory usage, and will help to identify connection problems during the tests that will affect performance.

  • listen_queue_size: Increase this to 1024 (requires a restart). On linux, this value is limited by value of the sysctl /proc/sys/net/core/somaxconn (see below)

Additionally, ensure that you disable or remove any Zeus features that you do not need to use, such as health monitors, session persistence, TrafficScript rules, logging and activity monitors. Use a simple load-balancing method such as round robin.

After you run each benchmark test, you should review Zeus's error log ($ZEUSHOME/zxtm/log/errors); any software or resource problems will be logged here.

Operating System Tuning

Linux

The linux kernel is good at self-tuning, but there are a number of limits for the sizes of various kernel resources that you will need to increase. Some of these are obvious - file descriptors and ephemeral ports for example - but many are subtle and can manifest themselves in strange ways. We suggest the following tunables as a starting point, but do monitor kernel logs (use 'dmesg') for unexpected problems.

The following tunings assume a machine with 2Gb or more of memory.

• Ephemeral port range:

  echo "1024 65535" > /proc/sys/net/ipv4/ip_local_port_range

• Listen queue:

  SYN Cookies are recommended:

  echo 1 > /proc/sys/net/ipv4/tcp_syncookies
  echo 8192 > /proc/sys/net/ipv4/tcp_max_syn_backlog
   
  # restart Zeus software after changing somaxconn
  echo 1024 > /proc/sys/net/core/somaxconn

  Without SYN cookies, a much larger value for tcp_max_syn_backlog is required, but this consumes additional kernel memory and scales poorly (the hash table that stores the SYN records is of a fixed size).

You may also change the following settings, although the default kernel values on recent Linux kernels are generally well sized:

• TIME_WAIT:

  echo 1800000 > /proc/sys/net/ipv4/tcp_max_tw_buckets

• Network buffer size (1Gb):

  echo "128000 200000 262144" > /proc/sys/net/ipv4/tcp_mem

• File descriptors:

  echo 2097152 > /proc/sys/fs/file-max

• Netfilter conntrack table size:

  If you're getting the error message: "ip_conntrack: table full, dropping packet." in dmesg, and you need to use the conntrack module or something that depends on it, then increase the table size by adding the lines

  options ip_conntrack hashsize=1310719
  options nf_conntrack hashsize=1310719

  to /etc/modprobe.conf or /etc/modules.conf (this depends on your linux distro), and reboot.

• Spirent workaround:

  If you are using older Spirent test kit, you absolutely must set the following tunables to work around 'optimizations' in their TCP stack:

  echo 0 > /proc/sys/net/ipv4/tcp_timestamps
  echo 0 > /proc/sys/net/ipv4/tcp_window_scaling

These guidelines refer to recent Linux kernels (2.6.10 and later), but equivalent tunings can be made on other platforms. SpecWEB submissions are good sources of tunings that have been performed by various vendors on their platforms.

Review the kernel modules that are loaded, and seek to remove any networking-related modules that are not required. The conntrack module in particular adds significant additional processing in the kernel for each network connection.

For other Linux tuning advice, including NIC tunings, please see our FAQ entry Tuning Linux Networking.

Solaris

For tunables for Solaris and Solaris X86, please see the separate FAQ entry Tuning Solaris for Zeus Traffic Manager.

Other Considerations

Ethernet Auto-negotiation

If one side of a physical ethernet connection is set to autonegotiate and the other has fixed settings, duplex mismatches can result, causing a large number of collisions and very poor network performance. Similar problems can occur with network cards and switches from different manufacturers when both sides are set to autonegotiate.

It is important to ensure, therefore, that the ports on your Zeus server or appliance are configured with the same settings as thoses on the the switches and routers to which they are connected. If both sides are set to autongetiate and performance problems persist, diabling autonegotation and setting the same fixed settings on both the server and switch/router usually resolves the issue.

Note that a connection between ports with mismatched settings often appears to work when little traffic is flowing through it, but performs increasingly poorly under higher loads. You will typically see a large number of network collisions (in the output of ifconfig for example).

On Linux, you can use the ethtool command to set the network device settings - see the ethtool (8) man page for full details. Some examples follow below:

# Force eth0 to 1 gigabit, full duplex
ethtool -s eth0 speed 1000 duplex full autoneg off
 
# Turn auto-negotation on on eth1
ethtool -s eth1 autoneg 0

Owen Garrett [Zeus Dev Team] 02 September 2005 2 comments