ZFS: Strange performance issues

[Christopher Key]

Hello,

I'm seeing exactly the same issues on my 5 disk raidz pool again.

In brief:

I can only write to the filesystem at ~20MB/s, and read from it at
~25MB/s.  Normally, I'd expect to be able to write at nearer 50-100
MB/s, and read at ~200MB/s.
The values from "zpool iostat -v" are also strange.  When writing to the
pool, I get sensible values, whereby the pool write rate matches the
rate reported by dd, and each of the five drives reports a quarter of
this rate.  When reading from the filesystem, the pool read rate is
~200MB/s, with each drive reading at ~40MB/s.  The read rate that dd
reports is still 25MB/s however.


The last time that this happened, I concluded that ZFS had simply ceased
to perform any caching, which would apparently explain the above
observations.  A reboot fixed this issue, and the assumption of caching
problems seemed borne out by the change in kstat.zfs.misc.arcstats.p, .c
and .size:

Before reboot*:

kstat.zfs.misc.arcstats.p: 21353344
kstat.zfs.misc.arcstats.c: 21353344
kstat.zfs.misc.arcstats.size: 29774848


After reboot:

kstat.zfs.misc.arcstats.p: 412187380
kstat.zfs.misc.arcstats.c: 415807010
kstat.zfs.misc.arcstats.size: 416037376


I had guessed that this caching problem was due to some unexpected
pressure on memory, but had rebooted the system before thinking to check!

Seeing the same issues again, I am now in a position to investigate
further.  I see the same reduced values in kstat.zfs.misc.arcstats.*:

kstat.zfs.misc.arcstats.p: 21353344
kstat.zfs.misc.arcstats.c: 21353344
kstat.zfs.misc.arcstats.size: 29774848

I'm not sure exactly how to interpret stats from top, but there doesn't
appear to be anying amiss:

Mem: 340M Active, 765M Inact, 686M Wired, 976K Cache, 211M Buf, 162M Free
Swap: 8192M Total, 1152K Used, 8191M Free


Before I restart this system, can anyone suggest anything to further
diagnose whats going on here?  What exactly do arcstats.(p|c) mean, and
would increasing arcstats.c_min perhaps help?


Kind Regards,


Christopher Key


* The stats before the reboot in my previous post were incorrect.  I
still have them archived, and include them here for future reference.

vfs.zfs.arc_min: 21353344
vfs.zfs.arc_max: 512480256
vfs.zfs.mdcomp_disable: 0
vfs.zfs.prefetch_disable: 0
vfs.zfs.zio.taskq_threads: 0
vfs.zfs.recover: 0
vfs.zfs.vdev.cache.size: 10485760
vfs.zfs.vdev.cache.max: 16384
vfs.zfs.cache_flush_disable: 0
vfs.zfs.zil_disable: 0
vfs.zfs.debug: 0
kstat.zfs.misc.arcstats.hits: 512541171
kstat.zfs.misc.arcstats.misses: 39560635
kstat.zfs.misc.arcstats.demand_data_hits: 156634097
kstat.zfs.misc.arcstats.demand_data_misses: 5743443
kstat.zfs.misc.arcstats.demand_metadata_hits: 303622664
kstat.zfs.misc.arcstats.demand_metadata_misses: 5425923
kstat.zfs.misc.arcstats.prefetch_data_hits: 4293179
kstat.zfs.misc.arcstats.prefetch_data_misses: 28037490
kstat.zfs.misc.arcstats.prefetch_metadata_hits: 47991231
kstat.zfs.misc.arcstats.prefetch_metadata_misses: 353779
kstat.zfs.misc.arcstats.mru_hits: 43042878
kstat.zfs.misc.arcstats.mru_ghost_hits: 16196037
kstat.zfs.misc.arcstats.mfu_hits: 420079251
kstat.zfs.misc.arcstats.mfu_ghost_hits: 3400520
kstat.zfs.misc.arcstats.deleted: 29558675
kstat.zfs.misc.arcstats.recycle_miss: 14747429
kstat.zfs.misc.arcstats.mutex_miss: 12390
kstat.zfs.misc.arcstats.evict_skip: 330353811
kstat.zfs.misc.arcstats.hash_elements: 1410
kstat.zfs.misc.arcstats.hash_elements_max: 30816
kstat.zfs.misc.arcstats.hash_collisions: 10467654
kstat.zfs.misc.arcstats.hash_chains: 31
kstat.zfs.misc.arcstats.hash_chain_max: 8
kstat.zfs.misc.arcstats.p: 21353344
kstat.zfs.misc.arcstats.c: 21353344
kstat.zfs.misc.arcstats.c_min: 21353344
kstat.zfs.misc.arcstats.c_max: 512480256
kstat.zfs.misc.arcstats.size: 29774848