ZFS "stalls" -- and maybe we should be talking about defaults?
fjwcash at gmail.com
Tue Mar 5 18:17:52 UTC 2013
On Tue, Mar 5, 2013 at 7:22 AM, Gary Palmer <gpalmer at freebsd.org> wrote:
> Just as a note that there was a page I read in the past few months
> that pointed out that having a huge ARC may not always be in the best
> interests of the system. Some operation on the filesystem (I forget
> what, apologies) caused the system to churn through the ARC and discard
> most of it, while regular I/O was blocked
Huh. What timing. I've been fighting with our largest ZFS box (128 GB of
RAM, 16 CPU cores, 2x SSD for SLOG, 2x SSD for L2ARC, 45x 2 TB HD for pool
in 6-driive raidz2 vdevs) for the past week trying to figure out why ZFS
send/recv just hangs after awhile. Everything is stuck in "D" in "ps ax"
output, and top show the l2arc_feed_ thread using 100% of one CPU. Even
removing the L2ARC devices from the pool doesn't help, just slows the
amount of time until the "hang".
ARC was configured for 120 GB, with arc_meta_limit set to 90 GB. Yes,
dedup and compression are enabled (it's a backups storage box, and we get
over 5x combined dedup/compress ratio). After several hours of running,
the ARC and wired would get up to 100+ GB, and the box would spend most of
its time "spinning", with almost 0 I/O to the pool (only a few KB/s of
reads in "zpool iostat 1" or "gstat").
ZFS send/recv would eventually complete, but what used to take 15-20
minutes would take 6-8 hours to complete.
I've reduced the ARC to only 32 GB, with arc_meta set to 28 GB, and things
are running much smoother now (50-200 MB/s writes for 3-5 seconds every
10s), and send/recv is back down to 10-15 minutes.
Who would have thought "too much RAM" would be an issue?
Will play with this over the next couple of days with different ARC max
settings to see where the problems start. All of our ZFS boxes until this
one had under 64 GB of RAM. (And we had issues with dedupe enabled on
boxes with too little RAM, as in under 32 GB.)
fjwcash at gmail.com
More information about the freebsd-stable