Poor performance with natd/ipfw and TSO enabled on bce(4) card and 8.1-PRERELEASE

[Garrett Cooper]

Hi,
    Just an observation I made while transferring a file:

# time scp floppy.img somehost:
Password:
floppy.img                                    100% 1440KB  13.7KB/s   01:45

real	1m59.400s
user	0m0.031s
sys	0m0.028s
# sysctl net.inet.tcp.tso=0
net.inet.tcp.tso: 1 -> 0
# time scp floppy.img somehost:
floppy.img                                    100% 1440KB   1.4MB/s   00:00

real	0m0.712s
user	0m0.018s
sys	0m0.018s

    Going ISDN speeds transferring a 1.44MB file is sad when you have
a gigabit uplink :(... natd seems to be doing a LOT of spinning when
TSO is enabled (it's going up to 73% CPU on a dual-proc quad-core
machine).
    Here are some other details:

# ipfw list
00050 divert 8668 ip4 from any to any via bce1
00100 allow ip from any to any via lo0
00200 deny ip from any to 127.0.0.0/8
00300 deny ip from 127.0.0.0/8 to any
00400 deny ip from any to ::1
00500 deny ip from ::1 to any
00600 allow ipv6-icmp from :: to ff02::/16
00700 allow ipv6-icmp from fe80::/10 to fe80::/10
00800 allow ipv6-icmp from fe80::/10 to ff02::/16
00900 allow ipv6-icmp from any to any ip6 icmp6types 1
01000 allow ipv6-icmp from any to any ip6 icmp6types 2,135,136
65000 allow ip from any to any
65535 deny ip from any to any
# ls /etc/natd*
ls: /etc/natd*: No such file or directory
# uname -a
FreeBSD tameshi.cisco.com 8.1-PRERELEASE FreeBSD 8.1-PRERELEASE #0
r209169: Mon Jun 14 12:41:49 PDT 2010
root@:/usr/obj/data/scratch/src/stable/8/sys/TAMESHI_STABLE  amd64
# pciconf -lv | grep -A 4 bce
bce1 at pci0:7:0:0:	class=0x020000 card=0x01b21028 chip=0x164c14e4
rev=0x12 hdr=0x00
    vendor     = 'Broadcom Corporation'
    device     = 'Broadcom NetXtreme II Gigabit Ethernet Adapter (BCM5708)'
    class      = network
    subclass   = ethernet
--
bce0 at pci0:3:0:0:	class=0x020000 card=0x01b21028 chip=0x164c14e4
rev=0x12 hdr=0x00
    vendor     = 'Broadcom Corporation'
    device     = 'Broadcom NetXtreme II Gigabit Ethernet Adapter (BCM5708)'
    class      = network
    subclass   = ethernet

    Let me know what other info is required.
Thanks,
-Garrett

PS Please keep me CCed in all emails.