svn commit: r343030 - in head/sys: cam conf dev/md dev/nvme fs/fuse fs/nfsclient fs/smbfs kern sys ufs/ffs vm

[Bruce Evans]

On Thu, 14 Feb 2019, Mark Johnston wrote:

> On Thu, Feb 14, 2019 at 06:56:42PM +1100, Bruce Evans wrote:
>* ...
>> The only relevant commit between the good and bad versions seems to be
>> r343453.  This fixes uma_prealloc() to actually work.  But it is a feature
>> for it to not work when its caller asks for too much.
>
> I guess you meant r343353.  In any case, the pbuf keg is _NOFREE, so
> even without preallocation the large pbuf zone limits may become
> problematic if there are bursts of allocation requests.

Oops.

>* ...
>> I don't understand how pbuf_preallocate() allocates for the other
>> pbuf pools.  When I debugged this for clpbufs, the preallocation was
>> not used.  pbuf types other than clpbufs seem to be unused in my
>> configurations.  I thought that pbufs were used during initialization,
>> since they end up with a nonzero FREE count, but their only use seems
>> to be to preallocate them.
>
> All of the pbuf zones share a common slab allocator.  The zones have
> individual limits but can tap in to the shared preallocation.

It seems to be working as intended now (except the allocation count is
3 higher than expected):

XX ITEM                   SIZE  LIMIT     USED     FREE      REQ FAIL SLEEP
XX
XX swrbuf:                 336,    128,       0,       0,       0,   0,   0
XX swwbuf:                 336,     64,       0,       0,       0,   0,   0
XX nfspbuf:                336,    128,       0,       0,       0,   0,   0
XX mdpbuf:                 336,     25,       0,       0,       0,   0,   0
XX clpbuf:                 336,    128,       0,      35,    2918,   0,   0
XX vnpbuf:                 336,   2048,       0,       0,       0,   0,   0
XX pbuf:                   336,     16,       0,    2505,       0,   0,   0

pbuf should har 2537 preallocated and FREE initially, but seems to actually
have 2540.  pbufs were only used for clustering, and 35 of them were moved
from pbuf to clpbuf.

In the buggy version, the preallocations stopped after 4.  Then clustering
presumably moved these 4 to clpbuf.  After that, clustering presumably used
non-preallocated buffers until it reached its limit, and then recycled its
own buffers.

What should happen to recover the old overcommit behaviour with better
debugging is 256 preallocated buffers (a few more for large systems) in
pbuf and moving these to other pools, but never allocating from other
pools (keep buffers in other pools only as an optimization and release
them to the main pool under pressure).  Also allow dynamic tuning of
the pool[s] size[s].  The vnode cache does essentially this by using
1 overcommitted pool with unlimited size in uma and external management
of the size.  The separate pools correspond to separate file systems.
These are too hard to manage, so the vnode cache throws everything into
the main pool and depends on locality for the overcommit to not be too
large.

Bruce