status of autotuning freebsd for 9.2

[Steven Hartland]

----- Original Message ----- 
From: "Andre Oppermann" <andre at freebsd.org>
To: "Steven Hartland" <killing at multiplay.co.uk>
Cc: "Alfred Perlstein" <alfred at ixsystems.com>; <stable at freebsd.org>; <re at freebsd.org>; <nonesuch at longcount.org>
Sent: Thursday, July 11, 2013 1:47 PM
Subject: Re: status of autotuning freebsd for 9.2


> On 11.07.2013 11:08, Steven Hartland wrote:
>> ----- Original Message ----- From: "Andre Oppermann" <andre at freebsd.org>
>>
>>> On 08.07.2013 16:37, Andre Oppermann wrote:
>>>> On 07.07.2013 20:24, Alfred Perlstein wrote:
>>>>> On 7/7/13 1:34 AM, Andre Oppermann wrote:
>>>>>> Can you help me with with testing?
>>>>>>
>>>>> Yes.  Please give me your proposed changes and I'll stand up a machine and give feedback.
>>>>
>>>> http://people.freebsd.org/~andre/mfc-autotuning-20130708.diff
>>>
>>> Any feedback from testers on this?  The MFC window is closing soon.
>>
>> Few things I've noticed most of which look like issues against the original
>> patch and not the MFC but worth mentioning.
>>
>> 1. You've introduced a new tunable kern.maxmbufmem which is autosized but
>>    doesnt seem to be exposed via a sysctl so it looks like there is no way
>>    to determine what its actually set to?
> 
> Good point.  I've made it global and exposed as kern.ipc.maxmbufmem (RDTUN).
> 
>> 2. There's a missmatch between the tuneable kern.ipc.nmbufs in tunable_mbinit
>>    and the sysctl kern.ipc.nmbuf i.e. no 's'.
> 
> That's a typo, fixed.
> 
>> 3. Should kern.maxmbufmem be kern.ipc.maxmbufmem to sit along side all of
>>    the other sysctls?
> 
> Yes, see above.
> 
>> 4. style issues:
>> * @@ -178,11 +202,13 @@
>>   ...
>>   if (newnmbjumbo9 > nmbjumbo9&&
> 
> Thanks.  All fixed in r253204.
> 
>> Finally out of interest what made us arrive at the various defaults for each
>> type as it looks like the ratios have changed?
> 
> Before it was an arbitrary mess.  Mbufs were not limited at all and the others
> to some random multiple of maxusers with the net limit ending up at some 25,000
> mbuf clusters by default.
> 
> Now default overall limit is set at 50% of all available min(physical, kmem_map)
> memory to prevent mbufs from monopolizing kernel memory and leave some space for
> other kernel structures and buffers as well as user-space programs.  It can be
> raised to 3/4 of available memory by the tunable.
> 
> 2K and 4K (page size) mbuf clusters can each go up to 25% of this mbuf memory.
> The former is dominantly used on the receive path and the latter in the send path.
> 9K and 16K jumbo mbuf clusters can each go up to 12.5% of mbuf memory.  They are
> only used in the receive path if large jumbo MTUs on a network interface are active.
> Both are special in that their memory is contiguous in KVM and physical memory.
> This becomes problematic due to memory fragmentation after a short amount of heavy
> system use.  I hope to deprecate them for 10.0.  Network interfaces should use 4K
> clusters instead and chain them together for larger packets.  All modern NICs
> support that.  Only the early and limited DMA engines without scatter-gather
> capabilities required contiguous physical memory.  They are long gone by now.
> 
> The limit for mbufs itselfs is 12.5% of mbuf memory and should be at least as
> many as the sum of the cluster types.  Each cluster requires an mbuf to which
> it is attached.
> 
> Two examples on the revised mbuf sizing limits:
> 
>   1GB KVM:
>    512MB limit for mbufs
>    419,430 mbufs
>     65,536 2K mbuf clusters
>     32,768 4K mbuf clusters
>      9,709 9K mbuf clusters
>      5,461 16K mbuf clusters
> 
>   16GB RAM:
>    8GB limit for mbufs
>    33,554,432 mbufs
>     1,048,576 2K mbuf clusters
>       524,288 4K mbuf clusters
>       155,344 9K mbuf clusters
>        87,381 16K mbuf clusters
> 
> These defaults should be sufficient for even the most demanding network loads.
> 
> For additional information see:
> 
>  http://svnweb.freebsd.org/changeset/base/243631

Thanks for that Andre and thanks for doing this work its something thats been sorely
needed for a long time.

Am I right is saying there is now the new concept of max mbufs? If so can we get
this added to the output of netstat -m if it already isn't?

I also think this is worth a mention in UPDATING too given the "defaults" are
now becoming sensible ;-)

    Regards
    Steve

    Regards
    Steve

================================================
This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. 

In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337
or return the E.mail to postmaster at multiplay.co.uk.