Two ZFS pools on one system: load on one pool badly affects other one

[Zaphod Beeblebrox]

[ regarding zfs arc cache on two pools, where one pool can clear the cache
of the other ]

... have you tried introducing an l2 cache on an SSD for the "private"
pool?  It seems to me that size of the l2 ARC are pool specific (where the
division of memory in the l1 ARC is not).  It doesn't surprise me that this
happens.  I have a large RAIZ2 pool for media (currently 40T, expanding to
60T shortly) on an otherwise general home server with 32G RAM.  There is
also a mirrored pool (root, home, a few things) on this machine for various
reasons (like not booting from such a large raid) and things got overall
better when I added a few hundred gig of l2 cache to each of the two pools.


On Mon, Apr 20, 2020 at 7:11 AM Lev Serebryakov <lev at freebsd.org> wrote:

>
>  I have system with two ZFS pools. Both pools are RADIZ on HDDs (not
> SSDs), and don't share any storage hardware. One pool is "my private space"
> and other one is storage for torrents.
>
>  Private space contains mirrors of FreeBSD SVN, my home directory, etc.
> Storage for torrents contains torrents.
>
>  My home directory contains, among other data, working copies of FreeBSD
> src and ports.
>
>  When torrent-client is stopped, disk-depended operations on PRIVATE pool
> like "svn up ~/FreeBSD/ports" starts immediately on cold caches.
>
>  When torrent-client are running (and reads about 5-6MB/s in small
> chunks), "svn up ~/FreeBSD/ports" could wait up to 5 minutes before start
> output of updated paths. ^T shows this:
>
> load: 0.48  cmd: svn 18126 [zio->io_cv] 275.95r 3.14u 6.14s 2% 21388k
>
>  Is it normal? I understand, that ARC is shared between pools, but it
> looks like too much interaction: there are NO shared I/O resources (disks,
> controllers) between pools and these operations, but "svn" waits 60-300
> seconds for some condition variable!
>
> --
> // Lev Serebryakov
>
>