Serious zfs slowdown when mixed with another file system (ufs/msdosfs/etc.).

Scott Sanbeg ssanbeg at gmail.com
Sun Jul 11 21:02:12 UTC 2010


Using Jeremy's suggestion as follows:
1) ZFS using a lot of memory but not freeing it as you expect
For (1), try this in /boot/loader.conf and reboot:
vfs.zfs.zio.use_uma="0"

... works like a charm for me.  Thank you.

Scott

-----Original Message-----
From: owner-freebsd-stable at freebsd.org
[mailto:owner-freebsd-stable at freebsd.org] On Behalf Of Jeremy Chadwick
Sent: Sunday, July 11, 2010 1:48 PM
To: Richard Lee
Cc: freebsd-stable at freebsd.org
Subject: Re: Serious zfs slowdown when mixed with another file system
(ufs/msdosfs/etc.).

On Sun, Jul 11, 2010 at 11:25:12AM -0700, Richard Lee wrote:
> This is on clean FreeBSD 8.1 RC2, amd64, with 4GB memory.
> 
> The closest I found by Googling was this:
> http://forums.freebsd.org/showthread.php?t=9935
> 
> And it talks about all kinds of little tweaks, but in the end, the
> only thing that actually works is the stupid 1-line perl code that
> forces the kernal to free the memory allocated to (non-zfs) disk
> cache, which is the "Inact"ive memory in "top."
> 
> I have a 4-disk raidz pool, but that's unlikely to matter.
> 
> Try to copy large files from non-zfs disk to zfs disk.  FreeBSD will
> cache the data read from non-zfs disk in memory, and free memory will
> go down.  This is as expected, obviously.
> 
> Once there's very little free memory, one would expect whatever is
> more important to kick out the cached data (Inact) and make memory
> available.
> 
> But when almost all of the memory is taken by disk cache (of non-zfs
> file system), ZFS disks start threshing like mad and the write
> throughput goes down in 1-digit MB/second.
> 
> I believe it should be extremely easy to duplicate.  Just plug in a
> big USB drive formatted in UFS (msdosfs will likely do the same), and
> copy large files from that USB drive to zfs pool.
> 
> Right after clean boot, gstat will show something like 20+MB/s
> movement from USB device (da*), and occasional bursts of activity on
> zpool devices at very high rate.  Once free memory is exhausted, zpool
> devices will change to constant low-speed activity, with disks
> threshing about constantly.
> 
> I tried enabling/disabling prefetch, messing with vnode counts,
> zfs.vdev.min/max_pending, etc.  The only thing that works is that
> stupid perl 1-liner (perl -e '$x="x"x1500000000'), which returns the
> activity to that seen right after a clean boot.  It doesn't last very
> long, though, as the disk cache again consumes all the memory.
> 
> Copying files between zfs devices doesn't seem to affect anything.
> 
> I understand zfs subsystem has its own memory/cache management.
> Can a zfs expert please comment on this?
> 
> And is there a way to force the kernel to not cache non-zfs disk data?

I believe you may be describing two separate issues:

1) ZFS using a lot of memory but not freeing it as you expect
2) Lack of disk I/O scheduler

For (1), try this in /boot/loader.conf and reboot:

# Disable UMA (uma(9)) for ZFS; amd64 was moved to exclusively use UMA
# on 2010/05/24.
# http://lists.freebsd.org/pipermail/freebsd-stable/2010-June/057162.html
vfs.zfs.zio.use_uma="0"

For (2), may try gsched_rr:

http://svnweb.freebsd.org/viewvc/base/releng/8.1/sys/geom/sched/README?view=
markup

-- 
| Jeremy Chadwick                                   jdc at parodius.com |
| Parodius Networking                       http://www.parodius.com/ |
| UNIX Systems Administrator                  Mountain View, CA, USA |
| Making life hard for others since 1977.              PGP: 4BD6C0CB |

_______________________________________________
freebsd-stable at freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscribe at freebsd.org"



More information about the freebsd-stable mailing list