bizarre em + TSO + MSS issue in RELENG_7

[Mike Andrews]

Has anyone run into problems with MSS not being respected when using TSO, 
specifically on em cards?

I've run into a weird 3-way interaction with TSO, users with small MTU's 
(like 576), and a Cisco firewall that by default drops packets larger than 
the MSS for that session.  Most users can connect to our web servers 
behind this firewall just fine, but a very few couldn't since we moved 
from RELENG_6 to RELENG_7 about two weeks ago.  One of them could connect 
again after removing a Linksys box at his end that was clamping the MSS at 
536.  Some of the others were on dialup which tends to use 576 MTU / 536 
MSS.  The firewall was dropping a lot of *outbound* traffic to these users 
because the MSS was exceeded -- ~1000 byte packets for a 536 MSS 
connection.  All of those users' problems disappeared after disabling TSO 
on our end.  Pretty weird. :)

I'm quite happy running without TSO and I really don't expect this to get 
looked at before 7.0-RELEASE -- I just thought the info that MSS appears 
(to me anyway) to not always be respected might be useful for those 
looking at TSO-related throughput issues.  In our case throughput isn't 
the issue, it's certain connections stalling out -- it actually behaves 
sort of like (but isn't) a PMTUD issue.

I've got a lot more detail for whoever wants it (Jack Vogel, probably) 
including tcpdumps and firewall logs but I thought I'd put it out here 
first in case it wasn't em-specific rather than clutter the list right 
away, or if it's a known issue that's already fixed in HEAD or something. 
This is with em driver version 6.5.3, 7.0-BETA2 from Nov 2.