[please review] TSO mbuf chain length limiting patch
Andrew Gallatin
gallatin at cs.duke.edu
Thu May 31 00:18:07 UTC 2012
On 05/30/12 18:35, Colin Percival wrote:
> On 05/30/12 08:30, Andrew Gallatin wrote:
>> On 05/30/12 10:59, Colin Percival wrote:
>>> The Xen virtual network interface has an issue (ok, really the issue is with
>>> the linux back-end, but that's what most people are using) where it can't
>>> handle scatter-gather writes with lots of pieces, aka. long mbuf chains.
>>> This currently bites us hard with TSO enabled, since it produces said long
>>> mbuf chains.
>>
>> I've never been clear about what the max TSO size supported by FreeBSD
>> is. The NIC I maintain (mxge) is limited to 64K - epsilon for both
>> IPv4 *AND* IPv6. Up until now, this has been enforced by the 16-bit
>> ip length limit of IPv4 and we have not had IPv6 TSO until this week.
>> With IPv6, I'm worried that FreeBSD may now send packets down larger
>> than I could handle. In my case, however, the problem is not s/g list
>> length, but rather it is internal limits in the NIC which limit us to
>> 64K - epsilon for IPv6 as well. I think there may be other NICs in
>> the same boat for IPv6 (and maybe even some which cannot handle the
>> full 64K for IPv4).
>>
>> Your approach would not work well for my size limit. For
>> example, I'd have to set the limit to 4 mbufs to stay under 64KB.
>> This would be assuming the worst case of 16KB jumbo mbufs, so
>> that would limit me to ~8KB per TSO if 2KB mbufs were used.
>
> Right, the problem you describe isn't the one I was trying to solve. :-)
>
>> I think a better approach would be to have a limit on the size of the
>> pre-segmented TCP payload size sent to the driver. I tend to think
>> that this would be more generically useful, and it is a better match
>> for the NDIS APIs, where a driver must specify the max TSO size. I
>> think the changes to the TCP stack might be simpler (eg, they
>> would seem to jive better with the existing "maxmtu" approach).
>>
>> I think this could work for you as well. You could set the Xen max
>> tso size to be 32K (derived from 18 pages/skb, multiplied by a typical
>> 2KB mbuf size, with some slack built in). If the chain was too large,
>> you could m_defrag it down to size.
>
> Sounds good -- I don't want to m_defrag too often, but I imagine in most
> cases when TSO is being invoked most of the mbufs will have 2 kB each.
> This should also make the patch simpler by avoiding the need to modify
> uipc_mbuf.c; if we just limit the TSO payload size then the TCP stack can
> figure things out by itself.
>
> Are you working on a patch, or should I put one together?
>
No, I'd like to, but I'm afraid that I just don't have the time
right now. I would very much appreciate it if you could put it
together. I'd be happy to review it.
Thanks,
Drew
More information about the freebsd-net
mailing list