kern/116330: network problems under -current, nfe(4) and jumbo packets

[Michael Reifenberger]

>Number:         116330
>Category:       kern
>Synopsis:       network problems under -current, nfe(4) and jumbo packets
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Thu Sep 13 11:20:01 GMT 2007
>Closed-Date:
>Last-Modified:
>Originator:     Michael Reifenberger
>Release:        FreeBSD 7.0-CURRENT amd64
>Organization:
>Environment:
System: FreeBSD fs.reifenberger.com 7.0-CURRENT FreeBSD 7.0-CURRENT #3: Mon Sep
10 23:21:52 CEST 2007 root at fs.reifenberger.com:/usr/obj/usr/src/sys/fs amd64

>Description:
I have two identical Asus N2M32WS Motherboards with builtin 
NVIDIA nForce MCP55 Networking Adapter NIC's and Marvell 88E1116 Gigabit PHY's.

both nfe0's are connected via GB-switch, nfe1's are connected directly (for testing purposes).
One machine acts as a samba and NFS fileserver.

After turning on a MTU of 9000 on the NIC's of the fileserver I found that
various clients (Linux SLES10, Gentoo, FreeBSD) with various applications 
(NFS over TCP/UDP, iSCSI) where producing errors and corruptions when
transferring from/to the fileserver.

So for further testing I switched to use the direct connected NIC's of the
two computers.
When using benchmarks/netpipe the integrity check hangs using
packetsizes greater 1500, later terminates with error:

(fs)(root) NPtcp -i -h 10.0.1.2                                                 
Doing an integrity check instead of measuring performance
Send and receive buffers are 1048576 and 1048576 bytes
(A bug in Linux doubles the requested buffer sizes)
Now starting the main loop
  0:       5 bytes   2758 times -->  Integrity check passed
  1:       7 bytes   2838 times -->  Integrity check passed
  2:       9 bytes   1978 times -->  Integrity check passed
  3:      13 bytes   2381 times -->  Integrity check passed
  4:      17 bytes   2015 times -->  Integrity check passed
  5:      25 bytes   2585 times -->  Integrity check passed
  6:      33 bytes   1759 times -->  Integrity check passed
  7:      49 bytes   2406 times -->  Integrity check passed
  8:      65 bytes   2172 times -->  Integrity check passed
  9:      97 bytes   3138 times -->  Integrity check passed
 10:     129 bytes   2055 times -->  Integrity check passed
 11:     193 bytes   2842 times -->  Integrity check passed
 12:     257 bytes   1774 times -->  Integrity check passed
 13:     385 bytes   2583 times -->  Integrity check passed
 14:     513 bytes   1667 times -->  Integrity check passed
 15:     769 bytes   2298 times -->  Integrity check passed
 16:    1025 bytes   1341 times -->  Integrity check passed
 17:    1537 bytes   1765 times --> NetPIPE: read: error encountered, errno=60

- How to debug this issue?
- Is this a known problem?

my /etc/sysctl.conf:
#security.bsd.see_other_uids=0
compat.linux.osrelease=2.6.16
#kern.maxvnodes=400000
vfs.lookup_shared=1
kern.coredump=0
#net.inet.tcp.path_mtu_discovery=0
net.inet.udp.recvspace=65536
net.inet.raw.recvspace=16384
#hw.pci.enable_msix=0
#hw.pci.enable_msi=0
kern.ipc.nmbclusters=50000
#kern.timecounter.hardware=ACPI-fast
kern.ipc.maxsockbuf=16777216
net.inet.tcp.rfc1323=1
net.inet.tcp.sendspace=1048576
net.inet.tcp.recvspace=1048576                                                  
net.inet.tcp.log_debug=0

NFS mountoptions are:
.. rw,noatime,bg,soft,-i,-3,-T,async,-r=32768,-w=32768

dmesg and kernel config is available on request.

>How-To-Repeat:
Use NFS with big send/receive blocksizes or benchmarks/netpipe integrity check.


>Fix:
The only known workaround is to lower the MTU to 1500.

>Release-Note:
>Audit-Trail:
>Unformatted: