Network performance 6.0 with netperf

[Michael VInce]

Hey all,
I been doing some network benchmarking using netperf and just simple 
'fetch' on a new network setup to make sure I am getting the most out of 
the router and servers,  I thought I would post some results in case 
some one can help me with my problems or if others are just interested 
to see the results.

The network is currently like this, where machines A and B are the Dell 
1850s and C is the 2850 x 2 CPU (Server C has Apache2 worker MPM on it) 
and server B is the gateway and A is acting as a client for fetch and 
netperf tests.
A --- B --- C
The 2 1850s are running AMD64 Freebsd 6.0rc1 (A and B) while C is 
running 5.4-stable i386 from Oct 12

My main problem is that if I compile SMP into the machine C (5.4stable) 
the network speed goes down to a range between 6mbytes/sec to 
15mbytes/sec on SMP.
If I use GENERIC kernel the performance goes up to what I have show 
below which is around 65megabytes/sec for a 'fetch' get test from Apache 
server and 933mbits/sec for netperf.
Does any know why why network performance would be so bad on SMP?

Does any one think that if I upgrade the i386 SMP server to 6.0RC1 the 
SMP network performance would improve? This server will be running java 
so I need it to be stable and is the the reason I am using i386 and Java 1.4

I am happy with performance of direct machine to machine (non SMP) which 
is pretty much full 1gigabit/sec speeds.
Going through the gateway server-B seems to drop its speed down a bit 
for in and out direction tcp speed tests using netperf I get around 
266mbits/sec from server A through gateway Server-B to server-C which is 
quite adequate for the link I currently have for it.

Doing a 'fetch' get for a 1gig file from the Apache server gives good 
speeds of close to 600mbits/sec but netperf shows its weakness with 
266mbits/sec.
This is as fast as I need it to be but does any one know the weak points 
on the router gateway to make it faster? Is this the performance I 
should expect for FreeBSD as a router with gigabit ethers?

I have seen 'net.inet.ip.fastforwarding' in some peoples router setups 
on the list but nothing about what it does or what it can affect.
I haven't done any testing with polling yet but if I can get over 
900mbits/sec on the interfaces does polling help with passing packets 
from one interface to the other?
All machines have PF running other then that they don't really have any 
sysctls or special kernel options.

Here are some speed benchmarks using netperf and 'fetch' gets.

Server A to server C with server C using SMP kernel and just GENERIC 
kernel further below

B# /usr/local/netperf/netperf -l 10 -H server-C -t TCP_STREAM -i 10,2 -I 
99,5 -- -m 4096 -s 57344 -S 57344
TCP STREAM TEST to server-C : +/-2.5% @ 99% conf. : histogram
Recv   Send    Send
Socket Socket  Message  Elapsed
Size   Size    Size     Time     Throughput
bytes  bytes   bytes    secs.    10^6bits/sec

 57344  57344   4096    10.06     155.99
tank# fetch -o - > /dev/null http://server-C/file1gig.iso
-                                             100% of 1055 MB   13 MBps 
00m00s

##### Using generic non SMP kernel
Server A to server C with server C using GENERIC kernel.
A# fetch -o - > /dev/null http://server-C/file1gig.iso
-                                             100% of 1055 MB   59 MBps 
00m00s

A# ./tcp_stream_script server-C

/usr/local/netperf/netperf -l 60 -H server-C -t TCP_STREAM -i 10,2 -I 
99,5 -- -m 4096 -s 57344 -S 57344

Recv   Send    Send
Socket Socket  Message  Elapsed
Size   Size    Size     Time     Throughput
bytes  bytes   bytes    secs.    10^6bits/sec

 57344  57344   4096    60.43     266.92

------------------------------------
###############################################
Connecting from server-A to B (gateway)
A# ./tcp_stream_script server-B

------------------------------------

/usr/local/netperf/netperf -l 60 -H server-B -t TCP_STREAM -i 10,2 -I 
99,5 -- -m 4096 -s 57344 -S 57344

TCP STREAM TEST to server-B : +/-2.5% @ 99% conf. : histogram
Recv   Send    Send
Socket Socket  Message  Elapsed
Size   Size    Size     Time     Throughput
bytes  bytes   bytes    secs.    10^6bits/sec

 57344  57344   4096    61.80     926.82

------------------------------------
##########################################
Connecting from server B (gateway) to server C
Fetch and Apache2 test
B# fetch -o - > /dev/null http://server-C/file1gig.iso
-                                             100% of 1055 MB   74 MBps 
00m00s

Netperf test
B# /usr/local/netperf/tcp_stream_script server-C

/usr/local/netperf/netperf -l 60 -H server-C -t TCP_STREAM -i 10,2 -I 
99,5 -- -m 4096 -s 57344 -S 57344

TCP STREAM TEST to server-C : +/-2.5% @ 99% conf. : histogram
Recv   Send    Send
Socket Socket  Message  Elapsed
Size   Size    Size     Time     Throughput
bytes  bytes   bytes    secs.    10^6bits/sec

 57344  57344   4096    62.20     933.94

------------------------------------

Cheers,
Mike