Is there any way to limit the amount of data in an mbuf chain submitted to a driver?

[Jack Vogel]

Hmmm, so its the stack, can that be easily increased Andre?

Regards,

Jack



On Sat, May 4, 2013 at 2:00 PM, Andre Oppermann <andre at freebsd.org> wrote:

> On 04.05.2013 22:47, Jack Vogel wrote:
>
>> Yes, I checked:   #define IXGBE_TSO_SIZE 262140
>>
>> So, the driver is not limiting  you to 64K assuming you are using a
>> version of recent vintage.
>>
>
> The stack won't generate TCP and IP packets larger than 64K.  However
> the ethernet header gets prepended to it and bumps the overall packet
> size as seen by the driver to 64K + sizeof(ether_hdr) and possibly
> additional VLAN headers.  If the bus_dma_tag size is set to 64K then
> bus_dma mapping will fail for such maximum size packets.
>
> --
> Andre
>
>
>  Jack
>>
>>
>>
>> On Sat, May 4, 2013 at 1:41 PM, Jack Vogel <jfvogel at gmail.com> wrote:
>>
>>  If you don't use TSO you will hurt your TX performance significantly from
>>> the tests that I've run. What exactly is the device you are using, I
>>> don't
>>> have the source in front of me now, but I'm almost sure that the limit is
>>> not 64K but 256K, or are you using some ancient version of the driver?
>>>
>>> Jack
>>>
>>>
>>>
>>> On Sat, May 4, 2013 at 1:30 PM, Richard Sharpe <
>>> realrichardsharpe at gmail.com> wrote:
>>>
>>>  On Sat, May 4, 2013 at 10:39 AM, Adrian Chadd <adrian at freebsd.org>
>>>> wrote:
>>>>
>>>>> On 4 May 2013 06:52, Richard Sharpe <realrichardsharpe at gmail.com>
>>>>>
>>>> wrote:
>>>>
>>>>> Hi folks,
>>>>>>
>>>>>> I understand better why I am seeing EINVAL intermittently when sending
>>>>>> data from Samba via SMB2.
>>>>>>
>>>>>> The ixgbe driver, for TSO reasons, limits the amount of data that can
>>>>>> be DMA'd to 65535 bytes. It returns EINVAL for any mbuf chain larger
>>>>>> than that.
>>>>>>
>>>>>> The SO_SNDBUF for that socket is set to 131972. Mostly there is less
>>>>>> than 64kiB of space available, so that is all TCP etc can put into the
>>>>>> socket in one chain of mbufs. However, every now and then there is
>>>>>> more than 65535 bytes available in the socket buffers, and we have an
>>>>>> SMB packet that is larger than 65535 bytes, and we get hit.
>>>>>>
>>>>>> To confirm this I am going to set SO_SNDBUF back to the default of
>>>>>> 65536 and test again. My repros are very reliable.
>>>>>>
>>>>>> However, I wondered if my only way around this if I want to continue
>>>>>> to use SO_SNDBUF sizes larger than 65536 is to fragment large mbuf
>>>>>> chains in the driver?
>>>>>>
>>>>>
>>>>> Hm, is this is a problem without TSO?
>>>>>
>>>>
>>>> We are using the card without TSO, so I am thinking of changing that
>>>> limit to 131072 and retesting.
>>>>
>>>> I am currently testing with SO_SNDBUF=32768 and have not hit the
>>>> problem.
>>>>
>>>>  Is the problem that the NIC can't handle a frame that big, or a buffer
>>>>>
>>>> that big?
>>>>
>>>>> Ie - if you handed the hardware two descriptors of 64k each, for the
>>>>> same IP datagram, will it complain?
>>>>>
>>>>
>>>> I can't find any documentation, but it seems that with TSO it cannot
>>>> handle a frame that big. Actually, since we are not using TSO, there
>>>> really should not be a problem with larger frames.
>>>>
>>>>  Or do you need to break it up into two separate IP datagrams, facing
>>>>> the driver, with a maximum size of 64k each?
>>>>>
>>>>
>>>> Not sure, but it looks like we need to do that.
>>>>
>>>>
>>>> --
>>>> Regards,
>>>> Richard Sharpe
>>>> (何以解憂？唯有杜康。--曹操)
>>>> ______________________________**_________________
>>>> freebsd-net at freebsd.org mailing list
>>>> http://lists.freebsd.org/**mailman/listinfo/freebsd-net<http://lists.freebsd.org/mailman/listinfo/freebsd-net>
>>>> To unsubscribe, send any mail to "freebsd-net-unsubscribe@**freebsd.org<freebsd-net-unsubscribe at freebsd.org>
>>>> "
>>>>
>>>
>>>
>>>
>>>  ______________________________**_________________
>> freebsd-net at freebsd.org mailing list
>> http://lists.freebsd.org/**mailman/listinfo/freebsd-net<http://lists.freebsd.org/mailman/listinfo/freebsd-net>
>> To unsubscribe, send any mail to "freebsd-net-unsubscribe@**freebsd.org<freebsd-net-unsubscribe at freebsd.org>
>> "
>>
>>
>