TCP Reassembly Issues

[Lawrence Stewart]

On 11/25/11 14:19, Kris Bauer wrote:
> On Thu, Nov 24, 2011 at 8:00 PM, Jeremy Chadwick
> <freebsd at jdc.parodius.com>wrote:
>
>> On Thu, Nov 24, 2011 at 07:13:39PM -0600, Kris Bauer wrote:
>>> On Thu, Nov 24, 2011 at 5:35 PM, Adrian Chadd<adrian at freebsd.org>
>> wrote:
>>>
>>>> Have you tried disabling the tcp offload features of your NIC?
>>>>
>>>>
>>>> Adrian
>>>>
>>>
>>> To test this, I added net.inet.tcp.tso=0 to sysctl.conf and restarted the
>>> box; it didn't work.  net.inet.tcp.reass.cursegments immediately started
>>> climbing up and were exhausted within an hour.
>>
>> I think Adrian was referring to RXCSUM and TXCSUM on your NIC; TSO is
>> another offloading feature.
>>
>> See ifconfig(8) for how to disable those.
>>
>> Be aware that disabling them in real-time (e.g. ifconfig xxx -rxcsum
>> -txcsum) may cause problems; there are some NIC drivers on FreeBSD which
>> do not like you doing this once the NIC has established link (meaning
>> "reloading the driver" (for lack of better term) results in wonky
>> behaviour).  So you may instead want to add those hyphen-options to your
>> ifconfig_XXX lines in /etc/rc.conf and reboot the box.
>>
>> If none of this solves the problem, then I consider this a priority 0
>> blocker (read: "all hands on deck") issue with the IP stack in FreeBSD
>> 9.x and will need immediate attention.
>>
>> I would strongly recommend a developer or clueful end-user begin
>> tracking down who committed all of these bits and CC them into the
>> thread.  I would start by looking who implemented the
>> net.inet.tcp.reass.cursegments sysctl, because that isn't in RELENG_8 at
>> all.
>>
>
> I have added -rxcsum -txcsum -tso to rc.conf and rebooted the box.  This
> has not solved the problem.  After a half-hour usage, I'm already up to
> reass.cursegments=2182 and it keeps climbing.

This is pretty much guaranteed to be an accounting problem in the TCP 
reassembly code (netinet/tcp_reass.c), not a driver related issue. I 
would not expect any amount of tweaking, tuning or driver option 
twiddling to change the outcome (but if you do find something which 
alleviates it, do let us know).

Kris, are you in a position to test kernel patches on the machine which 
is experiencing this problem?

Cheers,
Lawrence