vm.kmem_size settings doesn't affect loader?

[Jeremy Chadwick]

On Thu, Sep 25, 2008 at 11:46:41AM -0400, Alexandre Sunny Kovalenko wrote:
> On Thu, 2008-09-25 at 16:10 +0100, Pete French wrote:
> > > These are the tuning settings I use:
> > >
> > > vm.kmem_size="1536M"
> > > vm.kmem_size_max="1536M"
> > > vfs.zfs.arc_min="16M"
> > > vfs.zfs.arc_max="64M"
> > 
> > >
> > > The entire copying process took almost 2 hours.  Not once did I
> > > experience kmem exhaustion.  I can *guarantee* that I would have crashed
> > > the box numerous times had I not tuned the machine with the values
> > > above.
> > 
> > I am interested in the last two values you have there - I also use ZFS and
> > tune to prevent memory exhaustion. I tune kmem_size, but I have not bothered
> > toi set the arc_min - and the default appears to be 16M anyway. The arc_max
> > value I do set, but I got dreadful performance when I had it at 6M, so
> > I upped it to 128M. This has not resulted in any panics, and gives much better
> > performance. As I understand it there is only one ARC, and it comes from
> > kernel space too, so setting it to an extra 64 meg should not use that much
> > more memory, yes ?
> On my RELENG_7 (circa August 14th) 
> 
> vfs.zfs.arc_min: 16777216
> vfs.zfs.arc_max: 402653184
> 
> seem to be set by default. Yes, that is "384M".

The default arc_max value is calculated based on numerous things which
are system-specific.  The code is in arc_init() here, and the variable
is called arc_c_max:

src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c

> I have 
> 
> vm.kmem_size_max="512M"
> vm.kmem_size="512M"
> 
> set in my /boot/loader.conf, and

Your kmem is set to 512MB, yet your maximum ARC size is 384MB.  That
leaves 128MB for the entire rest of the kmap, including other pieces
of ZFS.  I can assure you that with a heavier ZFS load, your machine
will easily panic.

> vfs.ufs.dirhash_maxmem=16777216
> kern.maxvnodes=100000
> 
> set in /etc/sysctl.conf

vfs.ufs.dirhash_maxmem is for UFS/UFS2.

Speculative:

kern.maxvnodes is vfs-related, and ZFS does use vfs (duh).  Increasing
kern.maxvnodes means using more kernel memory (no idea if that's kmem or
not), which could mean you're decreasing the amount of memory available
for ZFS/ARC as well (e.g. panic more likely).

> This is amd64 with 3GB of real memory. ZFS pool is used to serve music,
> digital pictures and video to the rest of household, as well as provide
> target for the weekly backups from couple of Windows desktops, so use is
> not that heavy. Machine has been up for 22 days and has shown no ill
> effects that I can notice.

The lesser load on ZFS, and possibly the system in general, is probably
what's keeping you afloat.

-- 
| Jeremy Chadwick                                jdc at parodius.com |
| Parodius Networking                       http://www.parodius.com/ |
| UNIX Systems Administrator                  Mountain View, CA, USA |
| Making life hard for others since 1977.              PGP: 4BD6C0CB |