Network performance 6.0 with netperf
Michael VInce
mv at roq.com
Thu Oct 20 01:25:59 PDT 2005
Here is my probably final round of tests that I thought could possible
be useful to others.
I have enabled polling on the interfaces and discovered some of the
master secret holy grail sysctls that really make this stuff work.
I now get over 900mbits/sec router performance with polling.
Having sysctl either net.isr.direct=1 or net.inet.ip.fastforwarding=1
gave roughly an extra 445mbits performance increase according to netperf
tests, because my tests aren't really lab strict enough I still haven't
been able to easily see a difference between having net.isr.direct=1 or
0 while also having net.inet.ip.fastforwarding set to 1, it does appear
that having net.isr.direct=1 might be stealing the job of the
net.inet.ip.fastforwarding sysctl because when
net.inet.ip.fastforwarding=0 and net.isr.direct=1 on the gateway I still
get the 905.48mbit/sec route speed listed below.
From the client machine (A) through the gateway (B with polling
enabled) to the server (C)
With net.isr.direct=1 and net.inet.ip.fastforwarding=1
A> /usr/local/netperf/netperf -l 10 -H server-C -t TCP_STREAM -i 10,2 -I
99,5 -- -m 4096 -s 57344 -S 57344
Elapsed Throughput - 10^6bits/sec: 905.48
With net.isr.direct=0 and net.inet.ip.fastforwarding=0
Elapsed Throughput - 10^6bits/sec: 460.15
Apache get 'fetch' test.
A> fetch -o - > /dev/null http://server-C/file1gig.iso
- 100% of 1055 MB 67 MBps
00m00s
Interestingly when testing from the gateway it self (B) direct to server
(C) having 'net.isr.direct=1' slowed down performance to 583mbits/sec
B> /usr/local/netperf/netperf -l 10 -H server-C -t TCP_STREAM -i 10,2 -I
99,5 -- -m 4096 -s 57344 -S 57344
Elapsed Throughput - 10^6bits/sec: 583.57
Same test with 'net.isr.direct=0'
Elapsed Throughput - 10^6bits/sec: 868.94
I have to ask how can this be possible if when its being used as a
router with net.isr.direct=1 it passes traffic at over 900mbits/sec
Having net.inet.ip.fastforwarding=1 doesn't affect the performance in
these B to C tests.
I believe faster performance may still be possible as another rack of
gear I have that has another AMD64 6.0 RC1 Dell 2850 (Kes) gives me up
to 930mbits/sec in apache fetch tests, I believe its even faster here
because its an AMD64 Apache server or its possible it could just have a
bit better quality ether cables, as I mentioned before the Apache server
for box "C" in above tests is i386 on 6.0RC1.
This fetch test is only on a switch with no router between them.
spin> fetch -o - > /dev/null http://kes/500megs.zip
- 100% of 610 MB 93 MBps
So far from this casual testing I have discovered these things on my
servers.
Using 6.0 on SMP servers gives a big boost in network performance over
5.x SMP using i386 or AMD64
FreeBSD as router on gigabit ethernet with the use of polling gives over
x2 performance with the right sysctls.
Needs more testing but it appears using AMD64 FreeBSD might be better
then i386 for Apache2 network performance on SMP kernels.
Single interface speeds tests from the router with polling enabled and
with 'net.isr.direct=1' appears to affect performance.
Regards,
Mike
Michael VInce wrote:
> Robert Watson wrote:
>
>>
>> On Fri, 14 Oct 2005, Michael VInce wrote:
>>
>>> I been doing some network benchmarking using netperf and just simple
>>> 'fetch' on a new network setup to make sure I am getting the most
>>> out of the router and servers, I thought I would post some results
>>> in case some one can help me with my problems or if others are just
>>> interested to see the results.
>>
>>
>>
>> Until recently (or maybe still), netperf was compiled with
>> -DHISTOGRAM by our port/package, which resulted in a significant
>> performance drop. I believe that the port maintainer and others have
>> agreed to change it, but I'm not sure if it's been committed yet, or
>> which packages have been rebuilt. You may want to manually rebuild
>> it to make sure -DHISTOGRAM isn't set.
>>
>> You may want to try setting net.isr.direct=1 and see what performance
>> impact that has for you.
>>
>> Robert N M Watson
>
>
> I reinstalled the netperf to make sure its the latest.
>
> I have also decided to upgrade Server-C (the i386 5.4 box) to 6.0RC1
> and noticed it gave a large improvement of network performance with a
> SMP kernel.
>
> As with the network setup ( A --- B --- C ) with server B being the
> gateway, doing a basic 'fetch' from the gateway (B) to the Apache
> server (C) it gives up to 700mbits/sec transfer performance, doing a
> fetch from server A thus going through the gateway gives slower but
> still decent performance of up to 400mbits/sec.
>
> B> fetch -o - > /dev/null http://server-c/file1gig.iso
> - 100% of 1055 MB 69
> MBps 00m00s
>
>
> A> fetch -o - > /dev/null http://server-c/file1gig.iso
> - 100% of 1055 MB 39
> MBps 00m00s
>
> Netperf from the gateway directly to the apache server (C) 916mbits/sec
> B> /usr/local/netperf/netperf -l 20 -H server-C -t TCP_STREAM -i 10,2
> -I 99,5 -- -m 4096 -s 57344 -S 57344
> Elapsed Throughput - 10^6bits/sec: 916.50
>
> Netperf from the client machine through the gateway to the apache
> server (C) 315mbits/sec
> A> /usr/local/netperf/netperf -l 10 -H server-C -t TCP_STREAM -i 10,2
> -I 99,5 -- -m 4096 -s 57344 -S 57344
> Elapsed Throughput - 10^6bits/sec: 315.89
>
> Client to gateway netperf test shows the direct connection between
> these machines is fast. 912mbits/sec
> A> /usr/local/netperf/netperf -l 30 -H server-B -t TCP_STREAM -i 10,2
> -I 99,5 -- -m 4096 -s 57344 -S 5734
> Elapsed Throughput - 10^6bits/sec: 912.11
>
> The strange thing now is in my last post I was able to get faster
> speeds from server A to C with 'fetch' tests on non-smp kernels and
> slower speeds with netperf tests. Now I get speeds a bit slower with
> fetch tests but faster netperf speed tests with or without SMP on
> server-C.
>
> I was going to test with 'net.isr.dispatch' but the sysctl doesn't
> appear to exist, doing this returns nothing.
> 'sysctl -a | grep 'net.isr.dispatch'
>
> I also tried polling but its also like that doesn't exist either.
> ifconfig em3 inet 192.168.1.1 netmask 255.255.255.224 polling
> ifconfig: polling: Invalid argument
>
> When doing netperf tests there was high interrupt usage.
> CPU states: 0.7% user, 0.0% nice, 13.5% system, 70.0% interrupt,
> 15.7% idle
>
> Also the server B is using its last 2 gigabit ethernet ports which are
> listed from pciconf -lv as '82547EI Gigabit Ethernet Controller'
> While the first 2 are listed as 'PRO/1000 P'
> Does any one know if the PRO/1000P would be better?
>
> em0 at pci5:4:0: class=0x020000 card=0x118a8086 chip=0x108a8086
> rev=0x03 hdr=0x00
> vendor = 'Intel Corporation'
> device = 'PRO/1000 P'
>
> em3 at pci9:8:0: class=0x020000 card=0x016d1028 chip=0x10768086
> rev=0x05 hdr=0x00
> vendor = 'Intel Corporation'
> device = '82547EI Gigabit Ethernet Controller'
>
> Cheers,
> Mike
>
>>
>>>
>>> The network is currently like this, where machines A and B are the
>>> Dell 1850s and C is the 2850 x 2 CPU (Server C has Apache2 worker
>>> MPM on it) and server B is the gateway and A is acting as a client
>>> for fetch and netperf tests.
>>> A --- B --- C
>>> The 2 1850s are running AMD64 Freebsd 6.0rc1 (A and B) while C is
>>> running 5.4-stable i386 from Oct 12
>>>
>>> My main problem is that if I compile SMP into the machine C
>>> (5.4stable) the network speed goes down to a range between
>>> 6mbytes/sec to 15mbytes/sec on SMP.
>>> If I use GENERIC kernel the performance goes up to what I have show
>>> below which is around 65megabytes/sec for a 'fetch' get test from
>>> Apache server and 933mbits/sec for netperf.
>>> Does any know why why network performance would be so bad on SMP?
>>>
>>> Does any one think that if I upgrade the i386 SMP server to 6.0RC1
>>> the SMP network performance would improve? This server will be
>>> running java so I need it to be stable and is the the reason I am
>>> using i386 and Java 1.4
>>>
>>> I am happy with performance of direct machine to machine (non SMP)
>>> which is pretty much full 1gigabit/sec speeds.
>>> Going through the gateway server-B seems to drop its speed down a
>>> bit for in and out direction tcp speed tests using netperf I get
>>> around 266mbits/sec from server A through gateway Server-B to
>>> server-C which is quite adequate for the link I currently have for it.
>>>
>>> Doing a 'fetch' get for a 1gig file from the Apache server gives
>>> good speeds of close to 600mbits/sec but netperf shows its weakness
>>> with 266mbits/sec.
>>> This is as fast as I need it to be but does any one know the weak
>>> points on the router gateway to make it faster? Is this the
>>> performance I should expect for FreeBSD as a router with gigabit
>>> ethers?
>>>
>>> I have seen 'net.inet.ip.fastforwarding' in some peoples router
>>> setups on the list but nothing about what it does or what it can
>>> affect.
>>> I haven't done any testing with polling yet but if I can get over
>>> 900mbits/sec on the interfaces does polling help with passing
>>> packets from one interface to the other?
>>> All machines have PF running other then that they don't really have
>>> any sysctls or special kernel options.
>>>
>>> Here are some speed benchmarks using netperf and 'fetch' gets.
>>>
>>> Server A to server C with server C using SMP kernel and just GENERIC
>>> kernel further below
>>>
>>> B# /usr/local/netperf/netperf -l 10 -H server-C -t TCP_STREAM -i
>>> 10,2 -I 99,5 -- -m 4096 -s 57344 -S 57344
>>> TCP STREAM TEST to server-C : +/-2.5% @ 99% conf. : histogram
>>> Recv Send Send
>>> Socket Socket Message Elapsed
>>> Size Size Size Time Throughput
>>> bytes bytes bytes secs. 10^6bits/sec
>>>
>>> 57344 57344 4096 10.06 155.99
>>> tank# fetch -o - > /dev/null http://server-C/file1gig.iso
>>> - 100% of 1055 MB 13
>>> MBps 00m00s
>>>
>>> ##### Using generic non SMP kernel
>>> Server A to server C with server C using GENERIC kernel.
>>> A# fetch -o - > /dev/null http://server-C/file1gig.iso
>>> - 100% of 1055 MB 59
>>> MBps 00m00s
>>>
>>> A# ./tcp_stream_script server-C
>>>
>>> /usr/local/netperf/netperf -l 60 -H server-C -t TCP_STREAM -i 10,2
>>> -I 99,5 -- -m 4096 -s 57344 -S 57344
>>>
>>> Recv Send Send
>>> Socket Socket Message Elapsed
>>> Size Size Size Time Throughput
>>> bytes bytes bytes secs. 10^6bits/sec
>>>
>>> 57344 57344 4096 60.43 266.92
>>>
>>> ------------------------------------
>>> ###############################################
>>> Connecting from server-A to B (gateway)
>>> A# ./tcp_stream_script server-B
>>>
>>> ------------------------------------
>>>
>>> /usr/local/netperf/netperf -l 60 -H server-B -t TCP_STREAM -i 10,2
>>> -I 99,5 -- -m 4096 -s 57344 -S 57344
>>>
>>> TCP STREAM TEST to server-B : +/-2.5% @ 99% conf. : histogram
>>> Recv Send Send
>>> Socket Socket Message Elapsed
>>> Size Size Size Time Throughput
>>> bytes bytes bytes secs. 10^6bits/sec
>>>
>>> 57344 57344 4096 61.80 926.82
>>>
>>> ------------------------------------
>>> ##########################################
>>> Connecting from server B (gateway) to server C
>>> Fetch and Apache2 test
>>> B# fetch -o - > /dev/null http://server-C/file1gig.iso
>>> - 100% of 1055 MB 74
>>> MBps 00m00s
>>>
>>> Netperf test
>>> B# /usr/local/netperf/tcp_stream_script server-C
>>>
>>> /usr/local/netperf/netperf -l 60 -H server-C -t TCP_STREAM -i 10,2
>>> -I 99,5 -- -m 4096 -s 57344 -S 57344
>>>
>>> TCP STREAM TEST to server-C : +/-2.5% @ 99% conf. : histogram
>>> Recv Send Send
>>> Socket Socket Message Elapsed
>>> Size Size Size Time Throughput
>>> bytes bytes bytes secs. 10^6bits/sec
>>>
>>> 57344 57344 4096 62.20 933.94
>>>
>>> ------------------------------------
>>>
>>> Cheers,
>>> Mike
>>>
>>> _______________________________________________
>>> freebsd-net at freebsd.org mailing list
>>> http://lists.freebsd.org/mailman/listinfo/freebsd-net
>>> To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org"
>>>
>> _______________________________________________
>> freebsd-stable at freebsd.org mailing list
>> http://lists.freebsd.org/mailman/listinfo/freebsd-stable
>> To unsubscribe, send any mail to
>> "freebsd-stable-unsubscribe at freebsd.org"
>
>
>
> _______________________________________________
> freebsd-stable at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to "freebsd-stable-unsubscribe at freebsd.org"
More information about the freebsd-net
mailing list