svn commit: r343030 - in head/sys: cam conf dev/md dev/nvme fs/fuse fs/nfsclient fs/smbfs kern sys ufs/ffs vm

[Justin Hibbits]

On Tue, 15 Jan 2019 01:02:17 +0000 (UTC)
Gleb Smirnoff <glebius at FreeBSD.org> wrote:

> Author: glebius
> Date: Tue Jan 15 01:02:16 2019
> New Revision: 343030
> URL: https://svnweb.freebsd.org/changeset/base/343030
> 
> Log:
>   Allocate pager bufs from UMA instead of 80-ish mutex protected
> linked list. 
>   o In vm_pager_bufferinit() create pbuf_zone and start accounting on
> how many pbufs are we going to have set.
>     In various subsystems that are going to utilize pbufs create
> private zones via call to pbuf_zsecond_create(). The latter calls
> uma_zsecond_create(), and sets a limit on created zone. After startup
> preallocate pbufs according to requirements of all pbuf zones.
>   
>     Subsystems that used to have a private limit with old allocator
> now have private pbuf zones: md(4), fusefs, NFS client, smbfs, VFS
> cluster, FFS, swap, vnode pager.
>   
>     The following subsystems use shared pbuf zone: cam(4), nvme(4),
> physio(9), aio(4). They should have their private limits, but
> changing that is out of scope of this commit.
>   
>   o Fetch tunable value of kern.nswbuf from init_param2() and while
> here move NSWBUF_MIN to opt_param.h and eliminate opt_swap.h, that
> was holding only this option.
>     Default values aren't touched by this commit, but they probably
> should be reviewed wrt to modern hardware.
>   
>   This change removes a tight bottleneck from sendfile(2) operation,
> that uses pbufs in vnode pager. Other pagers also would benefit from
> faster allocation.
>   
>   Together with:	gallatin
>   Tested by:	pho
> 
> Modified:
>   head/sys/cam/cam_periph.c
>   head/sys/conf/options
>   head/sys/dev/md/md.c
>   head/sys/dev/nvme/nvme_ctrlr.c
>   head/sys/fs/fuse/fuse_main.c
>   head/sys/fs/fuse/fuse_vnops.c
>   head/sys/fs/nfsclient/nfs_clbio.c
>   head/sys/fs/nfsclient/nfs_clport.c
>   head/sys/fs/smbfs/smbfs_io.c
>   head/sys/fs/smbfs/smbfs_vfsops.c
>   head/sys/kern/kern_physio.c
>   head/sys/kern/subr_param.c
>   head/sys/kern/vfs_aio.c
>   head/sys/kern/vfs_bio.c
>   head/sys/kern/vfs_cluster.c
>   head/sys/sys/buf.h
>   head/sys/ufs/ffs/ffs_rawread.c
>   head/sys/vm/swap_pager.c
>   head/sys/vm/vm_pager.c
>   head/sys/vm/vnode_pager.c
> 

Hi Gleb,

This seems to break 32-bit platforms, or at least 32-bit book-e
powerpc, which has a limited KVA space (~500MB).  It preallocates I've
seen over 2500 pbufs, at 128kB each, eating up over 300MB KVA,
leaving very little left for the rest of runtime.

I spent a couple hours earlier today debugging with Mark Johnston, and
his consensus is that the vnode_pbuf_zone is too big on 32-bit
platforms.  Unfortunately I know very little about this area, so can't
provide much extra insight, but can readily reproduce the issues I see
triggered by this change, so am willing to help where I can.

- Justin