ZFS, compression, system load, pauses (livelocks?)

[Ivan Voras]

2009/12/15 Wiktor Niesiobedzki <bsd at w.evip.pl>:
> 2009/12/15 Ivan Voras <ivoras at freebsd.org>:
>> The context of this post is file servers running FreeBSD 8 and ZFS with
>> compressed file systems on low-end hardware, or actually high-end hardware
>> on VMWare ESX 3.5 and 4, which kind of makes it low-end as far as storage is
>> concerned. The servers are standby backup mirrors of production servers -
>> thus many writes, few reads.
>>
>> Running this setup I notice two things:
>>
>> 1) load averages get very high, though the only usage these systems get is
>> file system usage:
>> 2) long pauses, in what looks like vfs.zfs.txg.timeout second intervals,
>> which seemengly block everything, or at least the entire userland. These
>> pauses are sometimes so long that file transfers fail, which must be
>> avoided.
>>
>> Looking at the list of processes it looks like a large number of kernel and
>> userland processes are woken up at once. From the kernel side there are
>> regularily all g_* threads, but also unrelated threads like bufdaemon,
>> softdepflush, etc. and from the userland - top, syslog, cron, etc. It is
>> like ZFS livelocks everything else.
>>
>> Any ideas on the "pauses" issue?
>>
>
> Hi,
>
> I've a bit striped your post. It's kind of "me too" message (more
> details here: http://lists.freebsd.org/pipermail/freebsd-geom/2009-December/003810.html).
> What I've figured out so far is, that lowering the kernel thread
> priority (as pjd@ suggested) gives quite promising results (no
> livelocks at all). Though my bottleneck were caused by GELI thread.
>
> The pattern there is like this:
>
> sched_prio(curthread, PRIBIO);
> [...]
> msleep(sc, &sc->sc_queue_mtx, PDROP | PRIBIO,  "geli:w", 0);

I have tried before reducing priority of ZFS taskqueues but only to
PRIBIO, not below it - not much effect wrt "pauses".