Poor performance with natd/ipfw and TSO enabled on bce(4) card and 8.1-PRERELEASE

[Garrett Cooper]

On Thu, Jul 1, 2010 at 4:54 PM, Pyun YongHyeon <pyunyh at gmail.com> wrote:
> On Wed, Jun 30, 2010 at 07:00:53PM -0700, Garrett Cooper wrote:
>> Hi,
>>     Just an observation I made while transferring a file:
>>
>> # time scp floppy.img somehost:
>> Password:
>> floppy.img                                    100% 1440KB  13.7KB/s   01:45
>>
>> real  1m59.400s
>> user  0m0.031s
>> sys   0m0.028s
>> # sysctl net.inet.tcp.tso=0
>> net.inet.tcp.tso: 1 -> 0
>> # time scp floppy.img somehost:
>> floppy.img                                    100% 1440KB   1.4MB/s   00:00
>>
>> real  0m0.712s
>> user  0m0.018s
>> sys   0m0.018s
>>
>>     Going ISDN speeds transferring a 1.44MB file is sad when you have
>> a gigabit uplink :(... natd seems to be doing a LOT of spinning when
>> TSO is enabled (it's going up to 73% CPU on a dual-proc quad-core
>> machine).
>
> I would use pf(4) if I have to handle lots of NAT rules.
>
>>     Here are some other details:
>>
>> # ipfw list
>> 00050 divert 8668 ip4 from any to any via bce1
>> 00100 allow ip from any to any via lo0
>> 00200 deny ip from any to 127.0.0.0/8
>> 00300 deny ip from 127.0.0.0/8 to any
>> 00400 deny ip from any to ::1
>> 00500 deny ip from ::1 to any
>> 00600 allow ipv6-icmp from :: to ff02::/16
>> 00700 allow ipv6-icmp from fe80::/10 to fe80::/10
>> 00800 allow ipv6-icmp from fe80::/10 to ff02::/16
>> 00900 allow ipv6-icmp from any to any ip6 icmp6types 1
>> 01000 allow ipv6-icmp from any to any ip6 icmp6types 2,135,136
>> 65000 allow ip from any to any
>> 65535 deny ip from any to any
>> # ls /etc/natd*
>> ls: /etc/natd*: No such file or directory
>> # uname -a
>> FreeBSD tameshi.cisco.com 8.1-PRERELEASE FreeBSD 8.1-PRERELEASE #0
>> r209169: Mon Jun 14 12:41:49 PDT 2010
>> root@:/usr/obj/data/scratch/src/stable/8/sys/TAMESHI_STABLE  amd64
>> # pciconf -lv | grep -A 4 bce
>> bce1 at pci0:7:0:0:      class=0x020000 card=0x01b21028 chip=0x164c14e4
>> rev=0x12 hdr=0x00
>>     vendor     = 'Broadcom Corporation'
>>     device     = 'Broadcom NetXtreme II Gigabit Ethernet Adapter (BCM5708)'
>>     class      = network
>>     subclass   = ethernet
>> --
>> bce0 at pci0:3:0:0:      class=0x020000 card=0x01b21028 chip=0x164c14e4
>> rev=0x12 hdr=0x00
>>     vendor     = 'Broadcom Corporation'
>>     device     = 'Broadcom NetXtreme II Gigabit Ethernet Adapter (BCM5708)'
>>     class      = network
>>     subclass   = ethernet
>>
>>     Let me know what other info is required.
>
> Can you reproduce this issue on other TSO capable drivers?
> I'm not aware of any TSO issues on bce(4).

Hi Pyun!

I'll have to pop in a Copper Intel card that we have laying around in
the lab. I think it's em(4) compatible.. I forget... I have a few
things to test network wise this weekend, so I'll try and repro a few
things this weekend (say, Sunday?).

I also have my msk(4) enabled machine in the lab I can test with, but
I'll have to install the machine to spec with the Poweredge 2950 I
have in the lab.

I'm using ipfw because it was easy to setup according to the handbook,
but in reality if ipfw is this bad dealing with nat rules, then I need
to work with someone to improve how it scales.

Thanks!
-Garrett