ZFS slowness (not using cache?)

[Kris Kennaway]

David Taylor wrote:
> On Sun, 04 Nov 2007, Peter Schuller wrote:
> 
>>> For example, pkg_delete seems to be _extremely_ slow and ^T reports that
>>> it is stuck waiting on zfs:(&zio->io_cv) for an unreasonable (IMO) amount
>>> of time.
>> FWIW, I have seen pkg_install (and possibly other pkg_* tools) being extremely 
>> slow seemingly as a result of the active set of files it touches exceededing 
>> the amount cached. In particular I had this problem after converting to ZFS, 
>> but prior to switching to amd64 and more RAM.
>>
>> It would sit and churn on disk I/O forever, entirely seek bound. Tracing the 
>> processes showed it traversing the package database over and over (presumably 
>> recursively following dependencies or some such). So the same files were 
>> touched any number of times. As a result, with too little cached, runtime 
>> exploded (it took hours and hours upgrading my desktop using *binary* 
>> pre-built packages because the larger packages with a lot of dependencies 
>> would take forever to install and delete).
> 
> Hmm.  That prompted me to have a look at the arcstats, and I'm now rather
> confused.  It seems to have plenty of cache free, but not actually using
> it properly.
> 
> I'm running i386 for now (on an amd64 motherboard), with 4GB of RAM
> (~3.5Gb usable) and the following settings in loader.conf:
> 
> geom_stripe_load="YES"
> geom_label_load="YES"
> zfs_load="YES"
> snd_driver_load="YES"
> nvidia_load="YES"
> 
> hw.ata.atapi_dma="1"
> kern.maxfiles="25000"
> kern.ipc.shmmax=67108864
> kern.ipc.shmall=32768
> 
> kern.maxdsiz="900M"
> vm.kmem_size=1450M
> vfs.zfs.arc_max=500M
> vfs.root.mountfrom="zfs:tank/i386_root"
> 
> This results in (sysctl):
> 
> vm.kmem_size_scale: 3
> vm.kmem_size_max: 335544320  # I have just noticed this is about 300MB, 
>                              # far lower than vm.kmem_size.  Is that a problem?

No but you can't set kmem to 1500MB without first increasing the kernel 
address space (1GB on i386) to some suitably larger value via KVA_PAGES.

> vm.kmem_size_min: 0
> vm.kmem_size: 1520435200
> 
> vfs.zfs.arc_min: 47513600
> vfs.zfs.arc_max: 524288000
> kstat.zfs.misc.arcstats.c_min: 47513600
> kstat.zfs.misc.arcstats.c_max: 524288000
> kstat.zfs.misc.arcstats.size: 86967808
> 
> Regardless of what I do, I can't seem to get arcstats.size to exceed 100MB.
> It initially goes up with disk usage, but then starts to drop again, causing
> it to hold steady around 70-90MB.  Something seems to be agressively pushing
> data out of the cache, despite it being <20% full, which seems rather fishy.
> Does anyone know what's going on?

Try increasing kern.maxvnodes.

Kris


>