Strange slowdown when cache devices enabled in ZFS

[Freddie Cash]

The following settings in /etc/sysctl.conf prevent the "stalls" completely,
even when the L2ARC devices are 100% full and all RAM is wired into the
ARC.  Been running without issues for 5 days now:

vfs.zfs.l2arc_norw=0                                  # Default is 1
vfs.zfs.l2arc_feed_again=0                         # Default is 1
vfs.zfs.l2arc_noprefetch=0                          # Default is 0
vfs.zfs.l2arc_feed_min_ms=1000                 # Default is 200
vfs.zfs.l2arc_write_boost=320000000           # Default is 8 MBps
vfs.zfs.l2arc_write_max=160000000             # Default is 8 MBps

With these settings, I'm also able to expand the ARC to use the full 128 GB
of RAM in the biggest box, and to use both L2ARC devices (60 GB in total).
And, can set primarycache and secondarycache to all (the default) instead
of just metadata.

The only two settings in sysctl.conf that I've changed since the "stalls"
began are:
  vfs.zfs.l2arc_norw
  vfs.zfs.l2arc_feed_again

The other settings were already set when the boxes went live a few months
back.

I've run the dtrace hotkernel script from -HEAD on each of the 9-STABLE
boxes.  Not really sure how to interpret the results.  I ran the script (on
the receiving box) while doing a full zfs send/recv with the norw set to 1
(default) and then again with it set to 0.  Here's the top 20 entries from
each run.  There doesn't seem to be much of a difference in the output:

==> hotkernel.norw0 <==
kernel`_sx_try_xlock                                    48996   0.1%
kernel`bzero                                            50683   0.1%
kernel`_mtx_lock_sleep                                  71089   0.1%
kernel`hpet_get_timecount                               81257   0.2%
kernel`atomic_add_long                                  97927   0.2%
kernel`_sx_xunlock                                     122009   0.2%
kernel`_sx_xlock                                       122743   0.2%
zfs.ko`l2arc_write_eligible                            134717   0.3%
kernel`bcopy                                           157756   0.3%
zfs.ko`buf_hash                                        184670   0.4%
zfs.ko`l2arc_feed_thread                               295029   0.6%
zfs.ko`list_next                                       309683   0.6%
kernel`sched_idletd                                    316410   0.6%
kernel`spinlock_exit                                   337550   0.7%
zfs.ko`SHA256_Transform                                427619   0.9%
zfs.ko`lzjb_compress                                   500649   1.0%
kernel`_sx_xlock_hard                                  537433   1.1%
kernel`cpu_idle_mwait                                 3214405   6.5%
kernel`acpi_cpu_c1                                   41039557  83.2%


==> hotkernel.norw1 <==
kernel`copyin                                           49078   0.2%
kernel`_mtx_lock_sleep                                  54643   0.2%
kernel`_rw_wlock_hard                                   60637   0.2%
kernel`hpet_get_timecount                               64973   0.2%
kernel`atomic_add_long                                  83166   0.3%
zfs.ko`l2arc_write_eligible                            104574   0.3%
kernel`_sx_xunlock                                     108864   0.4%
kernel`_sx_xlock                                       112804   0.4%
zfs.ko`buf_hash                                        151566   0.5%
kernel`bcopy                                           161269   0.5%
zfs.ko`l2arc_feed_thread                               240827   0.8%
zfs.ko`list_next                                       252949   0.8%
kernel`spinlock_exit                                   361583   1.2%
kernel`sched_idletd                                    407961   1.3%
zfs.ko`SHA256_Transform                                546927   1.8%
kernel`_sx_xlock_hard                                  629281   2.0%
zfs.ko`lzjb_compress                                   634820   2.1%
kernel`cpu_idle_mwait                                 3359336  10.9%
kernel`acpi_cpu_c1                                   22177795  71.9%

-- 
Freddie Cash
fjwcash at gmail.com