Poor performance with natd/ipfw and TSO enabled on bce(4) card and 8.1-PRERELEASE

[Ian Smith]

On Thu, 1 Jul 2010, Garrett Cooper wrote:
 > On Thu, Jul 1, 2010 at 4:54 PM, Pyun YongHyeon <pyunyh at gmail.com> wrote:
 > > On Wed, Jun 30, 2010 at 07:00:53PM -0700, Garrett Cooper wrote:
 > >> Hi,
 > >>     Just an observation I made while transferring a file:
 > >>
 > >> # time scp floppy.img somehost:
 > >> Password:
 > >> floppy.img                                    100% 1440KB  13.7KB/s   01:45
 > >>
 > >> real  1m59.400s
 > >> user  0m0.031s
 > >> sys   0m0.028s
 > >> # sysctl net.inet.tcp.tso=0
 > >> net.inet.tcp.tso: 1 -> 0
 > >> # time scp floppy.img somehost:
 > >> floppy.img                                    100% 1440KB   1.4MB/s   00:00
 > >>
 > >> real  0m0.712s
 > >> user  0m0.018s
 > >> sys   0m0.018s
 > >>
 > >>     Going ISDN speeds transferring a 1.44MB file is sad when you have
 > >> a gigabit uplink :(... natd seems to be doing a LOT of spinning when
 > >> TSO is enabled (it's going up to 73% CPU on a dual-proc quad-core
 > >> machine).
 > >
 > > I would use pf(4) if I have to handle lots of NAT rules.

There's only one NAT rule here, not clear how many active NAT sessions 
are involved.  I'm tending to doubt this is really a natd issue; natd 
has no interaction with interface issues like TSO, that I know of, 
hopefully someone will correct me if I'm wrong about that.

 > >>     Here are some other details:
 > >>
 > >> # ipfw list
 > >> 00050 divert 8668 ip4 from any to any via bce1
 > >> 00100 allow ip from any to any via lo0
 > >> 00200 deny ip from any to 127.0.0.0/8
 > >> 00300 deny ip from 127.0.0.0/8 to any
 > >> 00400 deny ip from any to ::1
 > >> 00500 deny ip from ::1 to any
 > >> 00600 allow ipv6-icmp from :: to ff02::/16
 > >> 00700 allow ipv6-icmp from fe80::/10 to fe80::/10
 > >> 00800 allow ipv6-icmp from fe80::/10 to ff02::/16
 > >> 00900 allow ipv6-icmp from any to any ip6 icmp6types 1
 > >> 01000 allow ipv6-icmp from any to any ip6 icmp6types 2,135,136
 > >> 65000 allow ip from any to any
 > >> 65535 deny ip from any to any
 > >> # ls /etc/natd*
 > >> ls: /etc/natd*: No such file or directory

I assume that's the 'open' rc.firewall ruleset?  So you have no 
natd.conf, and are taking all defaults?  Just to check the config:

# grep natd_ /etc/rc.conf

# ps axw | grep "[n]atd"

Do you have options IPFIREWALL and IPDIVERT in kernel, or are you 
loading these as modules?

 > >> # uname -a
 > >> FreeBSD tameshi.cisco.com 8.1-PRERELEASE FreeBSD 8.1-PRERELEASE #0
 > >> r209169: Mon Jun 14 12:41:49 PDT 2010
 > >> root@:/usr/obj/data/scratch/src/stable/8/sys/TAMESHI_STABLE  amd64
 > >> # pciconf -lv | grep -A 4 bce
 > >> bce1 at pci0:7:0:0:      class=0x020000 card=0x01b21028 chip=0x164c14e4
 > >> rev=0x12 hdr=0x00
 > >>     vendor     = 'Broadcom Corporation'
 > >>     device     = 'Broadcom NetXtreme II Gigabit Ethernet Adapter (BCM5708)'
 > >>     class      = network
 > >>     subclass   = ethernet
 > >> --
 > >> bce0 at pci0:3:0:0:      class=0x020000 card=0x01b21028 chip=0x164c14e4
 > >> rev=0x12 hdr=0x00
 > >>     vendor     = 'Broadcom Corporation'
 > >>     device     = 'Broadcom NetXtreme II Gigabit Ethernet Adapter (BCM5708)'
 > >>     class      = network
 > >>     subclass   = ethernet
 > >>
 > >>     Let me know what other info is required.
 > >
 > > Can you reproduce this issue on other TSO capable drivers?
 > > I'm not aware of any TSO issues on bce(4).
 > 
 > Hi Pyun!
 > 
 > I'll have to pop in a Copper Intel card that we have laying around in
 > the lab. I think it's em(4) compatible.. I forget... I have a few
 > things to test network wise this weekend, so I'll try and repro a few
 > things this weekend (say, Sunday?).
 > 
 > I also have my msk(4) enabled machine in the lab I can test with, but
 > I'll have to install the machine to spec with the Poweredge 2950 I
 > have in the lab.
 > 
 > I'm using ipfw because it was easy to setup according to the handbook,
 > but in reality if ipfw is this bad dealing with nat rules, then I need
 > to work with someone to improve how it scales.

Unless there's something weird with tagging or something going on with 
divert sockets, this looks like something else; natd usually works fine 
at much higher rates, but I can't talk about gigabit.  Though in-kernel 
NAT should be better at the higher throughput end, your 'ISDN' rate and 
the high CPU usage for natd is certainly not typical.

Does this box have a public IP address on bce1?  It's not clear whether 
you're doing this transfer from this box, or from another, through it, 
ie what address translation is expected?  Where is 'somehost'?  Hence, 
knowing natd's config options and net topology might be helpful.

cheers, Ian