ZFS resilvering strangles IO

[Michael Gmelin]

On May 9, 2012, at 00:06, Artem Belevich wrote:

> On Tue, May 8, 2012 at 2:33 PM, Freddie Cash <fjwcash at gmail.com> wrote:
>> On Tue, May 8, 2012 at 2:31 PM, Bob Friesenhahn
>> <bfriesen at simple.dallas.tx.us> wrote:
>>> On Tue, 8 May 2012, Michael Gmelin wrote:
>>>> 
>>>> Do you think it would make sense to try to play with zfs_resilver_delay
>>>> directly in the ZFS kernel module?
>>> 
>>> This may be the wrong approach if the issue is really that there are too
>>> many I/Os queued for the device.  Finding a tunable which reduces the
>>> maximum number of I/Os queued for a disk device may help reduce write
>>> latencies by limiting the backlog.
>>> 
>>> On my Solaris 10 system, I accomplished this via a tunable in /etc/system:
>>> set zfs:zfs_vdev_max_pending = 5
>>> 
>>> What is the equivalent for FreeBSD?
>> 
>> Setting vfs.zfs.vdev_max_pending="4" in /boot/loader.conf (or whatever
>> value you want).  The default is 10.
> 

Do you think this will actually make a difference. As far as I
understand my primary problem is not latency but throughput. Simple
example is dd if=/dev/zero of=filename bs=1m, which gave me 500kb/s.
Latency might be an additional problem (or am I mislead and a shorter
queue would raise the processes chance to get data through?).

> You may also want to look at vfs.zfs.scrub_limit sysctl. According to
> description it's "Maximum scrub/resilver I/O queue" which sounds like
> something that may help in this case.
> 
> --Artem

Very good point, thank you. I also found this entry in the FreeBSD
forums indicating that this might ease the pain (even though he's also
talking about scrub, not resilver, hopefully the tunable does both as
indicated in the comments):

http://forums.freebsd.org/showthread.php?t=31628

/* maximum scrub/resilver I/O queue per leaf vdev */ int
zfs_scrub_limit = 10;

TUNABLE_INT("vfs.zfs.scrub_limit", &zfs_scrub_limit);
SYSCTL_INT(_vfs_zfs, OID_AUTO, scrub_limit, CTLFLAG_RDTUN,
&zfs_scrub_limit, 0, "Maximum scrub/resilver I/O queue");    

I will try lowering the value zfs_scrub_limit to 6 in loader.conf
and replace the drive once more later this month.

-- 
Michael