100,000 TCP connections - kernel tuning advice wanted
simon at synatech.com.au
Fri Sep 3 05:07:37 PDT 2004
As part of a team, I am working on a TCP multiplexor using FreeBSD. On side A
we have 100,000 TCP connections accepting packets, which are multiplexed
onto a single TCP connection on Side B. Packets going B->A are
demultiplexed in the reverse way. Info -
- freebsd version is 5.2-RELEASE. Kernel has been recompiled to
use DEVICE_POLLING and unused devices removed. The
HZ parameter has been varied through 1000,2000,4000 but this
does not significantly alter our results. We have also played with
the idle and trap sysctl's for polling.
- our network card is an Intel EtherExpress Pro, running at 100Mbits
- UDP is not an option for us
- Average payload size is 50-100 bytes. The payload is preceeded
by a 32 bit value, which is the size of the payload, so reading
is a matter of grabbing the size, allocating a buffer and then
doing the read. Minimal processing is done on the packet.
- We are using our own specialized memory management. We use writev and
readv whereever possible.
- socket buffers have been increased to 1MB on the B side, but are the
default size on side A.
- we are using kevent/kqueue - this task would be impossible without them
- our current test box has 1.5GB RAM and a 1GHZ Athlon CPU. While we might
go for a faster CPU, we would like to keep within our current RAM constraints.
- Side A is connected to a test client, which has 20% idle time.
- Side B is connected via a switch to another test box, which just echos the
packets back for testing purposes. It has significant idle time.
- Our current rough measurements, using top, show 30% user time, and 60%
kernel time, when this app is running. This multiplexing app is the only
app running on the machine. The machine is CPU bound - the multiplexing
requires no disk I/O.
Currently we are getting 4000-6000 packets/sec unidirectional throughput,
depending upon the mix of packet types/sizes. This goes up to
5000-7000 packets/sec for 50,000 connections.
We are seeking advice on what kernel tunables we can tweak to improve
packet throughput. Constants are TCP, 100,000 connections, 50-100 byte
All help appreciated.
More information about the freebsd-questions