Serious zfs slowdown when mixed with another file system (ufs/msdosfs/etc.).

Jeremy Chadwick freebsd at
Sun Jul 11 20:48:00 UTC 2010

On Sun, Jul 11, 2010 at 11:25:12AM -0700, Richard Lee wrote:
> This is on clean FreeBSD 8.1 RC2, amd64, with 4GB memory.
> The closest I found by Googling was this:
> And it talks about all kinds of little tweaks, but in the end, the
> only thing that actually works is the stupid 1-line perl code that
> forces the kernal to free the memory allocated to (non-zfs) disk
> cache, which is the "Inact"ive memory in "top."
> I have a 4-disk raidz pool, but that's unlikely to matter.
> Try to copy large files from non-zfs disk to zfs disk.  FreeBSD will
> cache the data read from non-zfs disk in memory, and free memory will
> go down.  This is as expected, obviously.
> Once there's very little free memory, one would expect whatever is
> more important to kick out the cached data (Inact) and make memory
> available.
> But when almost all of the memory is taken by disk cache (of non-zfs
> file system), ZFS disks start threshing like mad and the write
> throughput goes down in 1-digit MB/second.
> I believe it should be extremely easy to duplicate.  Just plug in a
> big USB drive formatted in UFS (msdosfs will likely do the same), and
> copy large files from that USB drive to zfs pool.
> Right after clean boot, gstat will show something like 20+MB/s
> movement from USB device (da*), and occasional bursts of activity on
> zpool devices at very high rate.  Once free memory is exhausted, zpool
> devices will change to constant low-speed activity, with disks
> threshing about constantly.
> I tried enabling/disabling prefetch, messing with vnode counts,
> zfs.vdev.min/max_pending, etc.  The only thing that works is that
> stupid perl 1-liner (perl -e '$x="x"x1500000000'), which returns the
> activity to that seen right after a clean boot.  It doesn't last very
> long, though, as the disk cache again consumes all the memory.
> Copying files between zfs devices doesn't seem to affect anything.
> I understand zfs subsystem has its own memory/cache management.
> Can a zfs expert please comment on this?
> And is there a way to force the kernel to not cache non-zfs disk data?

I believe you may be describing two separate issues:

1) ZFS using a lot of memory but not freeing it as you expect
2) Lack of disk I/O scheduler

For (1), try this in /boot/loader.conf and reboot:

# Disable UMA (uma(9)) for ZFS; amd64 was moved to exclusively use UMA
# on 2010/05/24.

For (2), may try gsched_rr:

| Jeremy Chadwick                                   jdc at |
| Parodius Networking              |
| UNIX Systems Administrator                  Mountain View, CA, USA |
| Making life hard for others since 1977.              PGP: 4BD6C0CB |

More information about the freebsd-stable mailing list