ZFS kmem_map too small.

[Dan Nelson]

In the last episode (Oct 05), Pawel Jakub Dawidek said:
> We'are about to branch RELENG_7 and I'd like to start discussion with
> folks that experience 'kmem_map too small' panic with the latest
> HEAD.
> 
> I'm trying hard to reproduce it and I can't, so I need to gather more
> info how you are able to provoke this panic.
> 
> What I did was to rsync 200 FreeBSD src trees from one directory to
> another on the same ZFS file system. It worked fine.
> 
> The system I'm using is i386 and the only tuning I did is bigger
> kmem_map. From my /boot/loader.conf:
> 
> vm.kmem_size=629145600
> vm.kmem_size_max=629145600

I'm running on a 1gb i386 system with

vfs.zfs.arc_min="64M"
vfs.zfs.arc_max="192M"
vm.kmem_size_min="640M"
vm.kmem_size_max="640M"

and no panics.  Set arc_max any higher (256M for example) and I end up
panicing on things like reading a 500M mailbox in mutt, etc. Instead of
playing with huge amounts of tiny files like the src tree, try with a
small amount of large files like ports/distfiles, or create a couple
dozen 500MB files and copy them from place to place.  

I'd really like to set arc_max up to 512M at least.  Sort of feels like
a waste having all that RAM and not being able to use it for disk
cache. Is there any way to make the arc truly dynamic, being able to
shrink and grow as needed just like the old buffer cache?  Why does it
have to live in kmem?

-- 
	Dan Nelson
	dnelson at allantgroup.com