Bad performance when accessing a lot of small files

[Alexandre Biancalana]

On 12/20/07, dima <_pppp at mail.ru> wrote:
> >> Increase the dirhash maxmem
> >> I have 'sysctl -w vfs.ufs.dirhash_maxmem=268435456' (256Mb)in my
> >> rc.local on a (maildir) fileserver here
> >>
> > This is a zfs filesystem, by the way I raise this limit as suggested.
> >
> > I let you know
>
> Can you provide
> sysctl vfs.zfs output then?
>
> ZFS can be tuned by increasing kern.maxvnodes sysctl (don't forget to increase vm.kmem_size and vm.kmem_size_max accordingly). Do it until vfs.numvnodes would stabilize below the kern.maxvnodes value during peak load patterns.

$ sysctl -A | grep zfs
vfs.zfs.arc_min: 49152000
vfs.zfs.arc_max: 1073741824
vfs.zfs.mdcomp_disable: 0
vfs.zfs.prefetch_disable: 1
vfs.zfs.zio.taskq_threads: 0
vfs.zfs.recover: 0
vfs.zfs.vdev.cache.size: 10485760
vfs.zfs.vdev.cache.max: 16384
vfs.zfs.cache_flush_disable: 0
vfs.zfs.zil_disable: 0
vfs.zfs.debug: 0
kstat.zfs.misc.arcstats.hits: 59096254
kstat.zfs.misc.arcstats.misses: 21362360
kstat.zfs.misc.arcstats.demand_data_hits: 21209779
kstat.zfs.misc.arcstats.demand_data_misses: 17688110
kstat.zfs.misc.arcstats.demand_metadata_hits: 37886475
kstat.zfs.misc.arcstats.demand_metadata_misses: 3674250
kstat.zfs.misc.arcstats.prefetch_data_hits: 0
kstat.zfs.misc.arcstats.prefetch_data_misses: 0
kstat.zfs.misc.arcstats.prefetch_metadata_hits: 0
kstat.zfs.misc.arcstats.prefetch_metadata_misses: 0
kstat.zfs.misc.arcstats.mru_hits: 18311543
kstat.zfs.misc.arcstats.mru_ghost_hits: 16169
kstat.zfs.misc.arcstats.mfu_hits: 40784711
kstat.zfs.misc.arcstats.mfu_ghost_hits: 621909
kstat.zfs.misc.arcstats.deleted: 25885007
kstat.zfs.misc.arcstats.recycle_miss: 9413044
kstat.zfs.misc.arcstats.mutex_miss: 9253
kstat.zfs.misc.arcstats.evict_skip: 4418321
kstat.zfs.misc.arcstats.hash_elements: 232462
kstat.zfs.misc.arcstats.hash_elements_max: 489237
kstat.zfs.misc.arcstats.hash_collisions: 12684452
kstat.zfs.misc.arcstats.hash_chains: 56900
kstat.zfs.misc.arcstats.hash_chain_max: 20
kstat.zfs.misc.arcstats.p: 1023016585
kstat.zfs.misc.arcstats.c: 1040187392
kstat.zfs.misc.arcstats.c_min: 49152000
kstat.zfs.misc.arcstats.c_max: 1073741824
kstat.zfs.misc.arcstats.size: 1040184320

$ cat /etc/sysctl.conf
kern.maxvnodes=400000
vfs.ufs.dirhash_maxmem=268435456


$ cat /boot/loader.conf
kern.maxdsiz="2G"       # Set the max data size to 4GB
kern.maxssiz="1G"       # Set the max stack size 2GB

vfs.zfs.prefetch_disable="1"
vfs.zfs.arc_max="1G"

vm.kmem_size_max="1500M"
vm.kmem_size="1500M"

kern.ipc.nmbclusters="32768"