Is there any way to limit the amount of data in an mbuf chain submitted to a driver?

[Richard Sharpe]

On Sat, May 4, 2013 at 10:39 AM, Adrian Chadd <adrian at freebsd.org> wrote:
> On 4 May 2013 06:52, Richard Sharpe <realrichardsharpe at gmail.com> wrote:
>> Hi folks,
>>
>> I understand better why I am seeing EINVAL intermittently when sending
>> data from Samba via SMB2.
>>
>> The ixgbe driver, for TSO reasons, limits the amount of data that can
>> be DMA'd to 65535 bytes. It returns EINVAL for any mbuf chain larger
>> than that.
>>
>> The SO_SNDBUF for that socket is set to 131972. Mostly there is less
>> than 64kiB of space available, so that is all TCP etc can put into the
>> socket in one chain of mbufs. However, every now and then there is
>> more than 65535 bytes available in the socket buffers, and we have an
>> SMB packet that is larger than 65535 bytes, and we get hit.
>>
>> To confirm this I am going to set SO_SNDBUF back to the default of
>> 65536 and test again. My repros are very reliable.
>>
>> However, I wondered if my only way around this if I want to continue
>> to use SO_SNDBUF sizes larger than 65536 is to fragment large mbuf
>> chains in the driver?
>
> Hm, is this is a problem without TSO?

We are using the card without TSO, so I am thinking of changing that
limit to 131072 and retesting.

I am currently testing with SO_SNDBUF=32768 and have not hit the problem.

> Is the problem that the NIC can't handle a frame that big, or a buffer that big?
> Ie - if you handed the hardware two descriptors of 64k each, for the
> same IP datagram, will it complain?

I can't find any documentation, but it seems that with TSO it cannot
handle a frame that big. Actually, since we are not using TSO, there
really should not be a problem with larger frames.

> Or do you need to break it up into two separate IP datagrams, facing
> the driver, with a maximum size of 64k each?

Not sure, but it looks like we need to do that.


-- 
Regards,
Richard Sharpe
(何以解憂？唯有杜康。--曹操)