Network performance 6.0 with netperf

Fri Oct 14 12:56:12 PDT 2005

On Fri, 14 Oct 2005, Michael VInce wrote:

> I been doing some network benchmarking using netperf and just simple 
> 'fetch' on a new network setup to make sure I am getting the most out of 
> the router and servers, I thought I would post some results in case some 
> one can help me with my problems or if others are just interested to see 
> the results.

Until recently (or maybe still), netperf was compiled with -DHISTOGRAM by 
our port/package, which resulted in a significant performance drop.  I 
believe that the port maintainer and others have agreed to change it, but 
I'm not sure if it's been committed yet, or which packages have been 
rebuilt.  You may want to manually rebuild it to make sure -DHISTOGRAM 
isn't set.

You may want to try setting net.isr.direct=1 and see what performance 
impact that has for you.

Robert N M Watson

>
> The network is currently like this, where machines A and B are the Dell 1850s 
> and C is the 2850 x 2 CPU (Server C has Apache2 worker MPM on it) and server 
> B is the gateway and A is acting as a client for fetch and netperf tests.
> A --- B --- C
> The 2 1850s are running AMD64 Freebsd 6.0rc1 (A and B) while C is running 
> 5.4-stable i386 from Oct 12
>
> My main problem is that if I compile SMP into the machine C (5.4stable) the 
> network speed goes down to a range between 6mbytes/sec to 15mbytes/sec on 
> SMP.
> If I use GENERIC kernel the performance goes up to what I have show below 
> which is around 65megabytes/sec for a 'fetch' get test from Apache server and 
> 933mbits/sec for netperf.
> Does any know why why network performance would be so bad on SMP?
>
> Does any one think that if I upgrade the i386 SMP server to 6.0RC1 the SMP 
> network performance would improve? This server will be running java so I need 
> it to be stable and is the the reason I am using i386 and Java 1.4
>
> I am happy with performance of direct machine to machine (non SMP) which is 
> pretty much full 1gigabit/sec speeds.
> Going through the gateway server-B seems to drop its speed down a bit for in 
> and out direction tcp speed tests using netperf I get around 266mbits/sec 
> from server A through gateway Server-B to server-C which is quite adequate 
> for the link I currently have for it.
>
> Doing a 'fetch' get for a 1gig file from the Apache server gives good speeds 
> of close to 600mbits/sec but netperf shows its weakness with 266mbits/sec.
> This is as fast as I need it to be but does any one know the weak points on 
> the router gateway to make it faster? Is this the performance I should expect 
> for FreeBSD as a router with gigabit ethers?
>
> I have seen 'net.inet.ip.fastforwarding' in some peoples router setups on the 
> list but nothing about what it does or what it can affect.
> I haven't done any testing with polling yet but if I can get over 
> 900mbits/sec on the interfaces does polling help with passing packets from 
> one interface to the other?
> All machines have PF running other then that they don't really have any 
> sysctls or special kernel options.
>
> Here are some speed benchmarks using netperf and 'fetch' gets.
>
> Server A to server C with server C using SMP kernel and just GENERIC kernel 
> further below
>
> B# /usr/local/netperf/netperf -l 10 -H server-C -t TCP_STREAM -i 10,2 -I 99,5 
> -- -m 4096 -s 57344 -S 57344
> TCP STREAM TEST to server-C : +/-2.5% @ 99% conf. : histogram
> Recv   Send    Send
> Socket Socket  Message  Elapsed
> Size   Size    Size     Time     Throughput
> bytes  bytes   bytes    secs.    10^6bits/sec
>
> 57344  57344   4096    10.06     155.99
> tank# fetch -o - > /dev/null http://server-C/file1gig.iso
> -                                             100% of 1055 MB   13 MBps 
> 00m00s
>
> ##### Using generic non SMP kernel
> Server A to server C with server C using GENERIC kernel.
> A# fetch -o - > /dev/null http://server-C/file1gig.iso
> -                                             100% of 1055 MB   59 MBps 
> 00m00s
>
> A# ./tcp_stream_script server-C
>
> /usr/local/netperf/netperf -l 60 -H server-C -t TCP_STREAM -i 10,2 -I 99,5 -- 
> -m 4096 -s 57344 -S 57344
>
> Recv   Send    Send
> Socket Socket  Message  Elapsed
> Size   Size    Size     Time     Throughput
> bytes  bytes   bytes    secs.    10^6bits/sec
>
> 57344  57344   4096    60.43     266.92
>
> ------------------------------------
> ###############################################
> Connecting from server-A to B (gateway)
> A# ./tcp_stream_script server-B
>
> ------------------------------------
>
> /usr/local/netperf/netperf -l 60 -H server-B -t TCP_STREAM -i 10,2 -I 99,5 -- 
> -m 4096 -s 57344 -S 57344
>
> TCP STREAM TEST to server-B : +/-2.5% @ 99% conf. : histogram
> Recv   Send    Send
> Socket Socket  Message  Elapsed
> Size   Size    Size     Time     Throughput
> bytes  bytes   bytes    secs.    10^6bits/sec
>
> 57344  57344   4096    61.80     926.82
>
> ------------------------------------
> ##########################################
> Connecting from server B (gateway) to server C
> Fetch and Apache2 test
> B# fetch -o - > /dev/null http://server-C/file1gig.iso
> -                                             100% of 1055 MB   74 MBps 
> 00m00s
>
> Netperf test
> B# /usr/local/netperf/tcp_stream_script server-C
>
> /usr/local/netperf/netperf -l 60 -H server-C -t TCP_STREAM -i 10,2 -I 99,5 -- 
> -m 4096 -s 57344 -S 57344
>
> TCP STREAM TEST to server-C : +/-2.5% @ 99% conf. : histogram
> Recv   Send    Send
> Socket Socket  Message  Elapsed
> Size   Size    Size     Time     Throughput
> bytes  bytes   bytes    secs.    10^6bits/sec
>
> 57344  57344   4096    62.20     933.94
>
> ------------------------------------
>
> Cheers,
> Mike
>
> _______________________________________________
> freebsd-net at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org"
>