Tuning Linux networking

Here are some suggestions when tweaking Linux machines for maximum network performance. For full tuning advice, including non-OS tunings, please see our main article, Tuning ZXTM for Maximum Performance.

Interrupts

Interrupts (IRQs) are wake-up calls to the CPU when new network traffic arrives. The CPU stops what it is doing and is diverted to handle the new network data.

Most NICs will tune their interrupts to be as efficient as possible - for the full details, you will need to consult the documentation for the drivers. For instance, here is the documentation for the e1000 Intel Gigabit NIC. In general, the defaults are quite sensible.

If you are on a machine with multiple CPUs/cores, the interrupt work should be spread out over as many CPUs as possible. Otherwise, one CPU can be the bottleneck in high network traffic. In Linux, you should install the irqbalance program, which will dynamically adjust how interrupts are handled by each CPU. Irqbalance is available as an installable package with most Linux distributions.

Sometimes the IRQ balancing doesn't work out well. If under high load you see one or more 'ksoftirqd' processes using lots of CPU (run top to check), then something is wrong. Tracking down the problem can be difficult, but changing Linux kernel versions, NIC drivers or installing a different version of irqbalance can help.

Network Card Features

Check that any supported NIC features are enabled, using ethtool. Network offload features can be shown with ethtool -k eth0. See the manpage for ethtool to see how to enable supported features. ethtool -S eth0 will show network statistics - check that you aren't getting any packet errors, overruns, collisions, etc, as these are all signs of bad NICs or cabling. (Check that all the cables are plugged in fully - it really does help!)

Finally, check that the NICs are all running at full speed. If the cabling isn't good, then gigabit cards may fall back to 100MBits or less when there is lots of traffic. Always check NIC speeds before and after testing to ensure that the network is reliable. If your NIC is running at half-duplex, chances are that something is terribly wrong!

iptables

iptables performs IP filtering / firewalling for Linux. If you aren't using such features, then be sure to:

1. Turn it off
2. REMOVE THE MODULES

The last step is very important. The mere presence of iptables modules can cause up to 30% performance loss, even when their features are not in use! Run lsmod and check for the following modules. rmmod modulename to get rid of them:

• ip_conntrack
• iptable_filter
• ip_tables
• Anything else with iptable in the name

SYN cookies

SYN cookies are a form of protection against a low-level network denial-of-service attack. If SYN cookies are enabled, Linux will start using them when it sees a high rate of new connections. This is great in normal use on a typically idle machine. However when you are pushing a machine to run as fast as possible, it means that the SYN cookie mechanism turns on just when you don't want it! To disable SYN cookies, run

sysctl -w net.ipv4.tcp_syncookies=0

Other network tunings

These are some suggested tunings - please see the full list of options - but don't go overboard, most options have sensible defaults.

# Widen the range of local ports - needed when
# making lots of outgoing connections
sysctl -w "net.ipv4.ip_local_port_range=1024 65535"
 
# Bigger backlog of SYN packets
sysctl -w net.ipv4.tcp_max_syn_backlog=10240
 
# Increase maximum backlog for accepting new connections
sysctl -w net.core.somaxconn=1024
 
# More efficient handling of lots of old connections
# in the TIME_WAIT state
sysctl -w net.ipv4.tcp_max_tw_buckets=1800000

Ben [Zeus Dev Team] 29 April 2009