Freebsd IP Forwarding performance (question, and some
info) [7-stable, current, em, smp]
Mike Tancsa
mike at sentex.net
Tue Jul 1 01:29:34 UTC 2008
At 05:05 PM 6/30/2008, Paul wrote:
>With hours and days of tweaking i can't even get 500k pps :/ no
>firewall no anything else..
>What is your kernel config? Sysctl configs?
The only thing that makes a difference is
net.inet.ip.fastforwarding=1
>My machine i'm testing on is dual opteron 2212 , with intel 2 port
>82571 nic..
xeon dual core on a supermicro MB. I am using one NIC on the MB and
one on the dual port.
em0 at pci0:10:1:0: class=0x020000 card=0x11798086
chip=0x10798086 rev=0x03 hdr=0x00
vendor = 'Intel Corporation'
device = '82546EB Dual Port Gigabit Ethernet Controller'
class = network
subclass = ethernet
cap 01[dc] = powerspec 2 supports D0 D3 current D0
cap 07[e4] = PCI-X 64-bit supports 133MHz, 2048 burst read, 1
split transaction
cap 05[f0] = MSI supports 1 message, 64 bit
em1 at pci0:10:1:1: class=0x020000 card=0x11798086
chip=0x10798086 rev=0x03 hdr=0x00
vendor = 'Intel Corporation'
device = '82546EB Dual Port Gigabit Ethernet Controller'
class = network
subclass = ethernet
cap 01[dc] = powerspec 2 supports D0 D3 current D0
cap 07[e4] = PCI-X 64-bit supports 133MHz, 2048 burst read, 1
split transaction
cap 05[f0] = MSI supports 1 message, 64 bit
em2 at pci0:13:0:0: class=0x020000 card=0x108c15d9
chip=0x108c8086 rev=0x03 hdr=0x00
vendor = 'Intel Corporation'
device = '82573E Intel Corporation 82573E Gigabit Ethernet
Controller (Copper)'
class = network
subclass = ethernet
cap 01[c8] = powerspec 2 supports D0 D3 current D0
cap 05[d0] = MSI supports 1 message, 64 bit enabled with 1 message
cap 10[e0] = PCI-Express 1 endpoint
em3 at pci0:14:0:0: class=0x020000 card=0x109a15d9
chip=0x109a8086 rev=0x00 hdr=0x00
vendor = 'Intel Corporation'
device = '82573L Intel PRO/1000 PL Network Adaptor'
class = network
subclass = ethernet
cap 01[c8] = powerspec 2 supports D0 D3 current D0
cap 05[d0] = MSI supports 1 message, 64 bit enabled with 1 message
cap 10[e0] = PCI-Express 1 endpoint
>Using 7-STABLE and I tried 6-stable and -current
>I get the RTM_MISS with 7 and current but only with certain types of
>packets at a certain rate.. :/
I wonder if its a bug with the em driver ? I dont have any other
dual port cards handy right now to test with
>I can not get more than 500kpps.. i tried everything I could think
>of... lowering the rx descriptors on EM to 512 instead of 2048 gave
>me some more.. I was stuck at 400kpps until i changed those and i
>lowered the rx processing limit.
>My tests are going incoming em0 and outgoing em1 in one direction
>only and it has major errors when em0 taskq gets close to 80% cpu..
I now have 3 boxes now generating traffic through the box acting as a
router. I will try some other operating systems as well to see how
they compare when back at the office on Wednesday
>I am pretty disappointed that it maxes out a little over 400kpps and
>even then it gets some errors here and there , mainly missed packets
>due to no buffer and rx overruns (dev.em.0.stats=1)
Something about the MB you are using perhaps ? Just for rough
comparison, how long does
# time make -j4 buildkernel > /var/log/build.out.k
670.485u 66.061s 8:29.54 144.5% 5962+1087k 9185+7419io 380pf+0w
take on your machine ?
The above value is with inet6 and sctp commented out from the kernel.
---Mike
>Mike Tancsa wrote:
>>At 04:04 AM 6/29/2008, Paul wrote:
>>>This is just a question but who can get more than 400k pps
>>>forwarding performance ?
>>
>>
>>OK, I setup 2 boxes on either end of a RELENG_7 box from about May
>>7th just now, to see with 2 boxes blasting across it how it would work.
>>*However*, this is with no firewall loaded and, I must enable ip
>>fast forwarding. Without that enabled, the box just falls over.
>>
>>even at 20Kpps, I start seeing all sorts of messages spewing to
>>route -n monitor
>>
>>
>>got message of size 96 on Mon Jun 30 15:39:10 2008
>>RTM_MISS: Lookup failed on this address: len 96, pid: 0, seq 0,
>>errno 0, flags:<DONE>
>>locks: inits:
>>sockaddrs: <DST>
>> default
>>
>>I am starting to wonder if those messages are the results of
>>corrupted packets the machine just cant keep up with ?
>>
>>
>>CPU is
>>
>>CPU: Intel(R) Xeon(R) CPU 3070 @ 2.66GHz (2660.01-MHz
>>686-class CPU)
>>
>>
>> input (Total) output
>> packets errs bytes packets errs bytes colls
>> 611945 0 77892098 611955 0 77013002 0
>> 616727 0 78215508 616742 0 77303454 0
>> 617066 0 78162130 617082 0 77238434 0
>> 618238 0 78302314 618225 0 77377582 0
>> 617035 0 78141000 617038 0 77215672 0
>> 617625 0 78225600 617588 0 77301734 0
>> 616190 0 78017320 616165 0 77091774 0
>> 615583 0 78064130 615628 0 77152800 0
>> 617662 0 78254388 617658 0 77332340 0
>> 618000 0 78269912 617950 0 77344554 0
>> 617248 0 78183136 617315 0 77259588 0
>> 617325 0 78204566 617289 0 77282094 0
>> 618391 0 78337734 618357 0 77413756 0
>> 616025 0 78116070 616082 0 77203116 0
>>
>>
>>To generate the packets, I am just using
>>/usr/src/tools/tools/netblast on 2 endpoints starting at about the same time
>>
>># ./netblast 10.10.1.2 500 100 40
>>
>>start: 1214854131.083679919
>>finish: 1214854171.084668592
>>send calls: 20139141
>>send errors: 0
>>approx send rate: 503478
>>approx error rate: 0
>>
>>
>># ./netblast 10.10.1.3 500 10 40
>>
>>start: 1214854273.882202815
>>finish: 1214854313.882319031
>>send calls: 23354971
>>send errors: 18757223
>>approx send rate: 114943
>>approx error rate: 0
>>
>>The box in the middle doing the forwarding
>>
>>1[spare-r7]# ifconfig -u
>>em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
>>
>>options=19b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4>
>> ether 00:1b:21:08:32:a8
>> inet 10.20.1.1 netmask 0xffffff00 broadcast 10.20.1.255
>> media: Ethernet autoselect (1000baseTX <full-duplex>)
>> status: active
>>em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
>> options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
>> ether 00:1b:21:08:32:a9
>> inet 192.168.43.193 netmask 0xffffff00 broadcast 192.168.43.255
>> media: Ethernet autoselect (100baseTX <full-duplex>)
>> status: active
>>em3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
>>
>>options=19b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4>
>> ether 00:30:48:90:4c:ff
>> inet 10.10.1.1 netmask 0xffffff00 broadcast 10.10.1.255
>> media: Ethernet autoselect (1000baseTX <full-duplex>)
>> status: active
>>lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
>> inet 127.0.0.1 netmask 0xff000000
>>
>>
>>I am going to try a few more tests with and without, firewall rules
>>etc as well as an updated kernel to RELENG_7 as of today and see how that goes.
>>
>> ---Mike
>>
More information about the freebsd-net
mailing list