tuning vfs.zfs.vdev.max_pending and solving the issue of ZFS writes choking read IO

[Ivan Voras]

Dan Naumov wrote:
> Hello
> 
> I am having a slight issue (and judging by Google results, similar
> issues have been seen by other FreeBSD and Solaris/OpenSolaris users)
> with writes choking the read IO. The issue I am having is described
> pretty well here:
> http://opensolaris.org/jive/thread.jspa?threadID=106453 It seems that
> under heavy write load, ZFS likes to aggregate a really huge amount of
> data before actually writing it to disks, resulting in sudden 10+
> second stalls where it frantically tries to commit everything,
> completely choking read IO in the process and sometimes even the
> network (with a large enough write to a mirror pool using DD, I can
> cause my SSH sessions to drop dead, without actually running out of
> RAM. As soon as the data is committed, I can reconnect back).

Mostly a wild guess, but can you test if this patch will help with 
choking your network and ssh:

http://people.freebsd.org/~ivoras/diffs/spa.c.diff

?

You can then fiddle with the vfs.zfs.zio_worker_threads_count loader 
tunable to see if it helps more.