kern/146410: [PATCH] bad file copy performance from UFS to ZFS

[Juha-Matti Tilli]

>Number:         146410
>Category:       kern
>Synopsis:       [PATCH] bad file copy performance from UFS to ZFS
>Confidential:   no
>Severity:       non-critical
>Priority:       medium
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Sat May 08 21:00:10 UTC 2010
>Closed-Date:
>Last-Modified:
>Originator:     Juha-Matti Tilli
>Release:        FreeBSD 8.0-STABLE amd64
>Organization:
>Environment:
System: FreeBSD chernobyl.jmtilli.iki.fi 8.0-STABLE FreeBSD 8.0-STABLE #0: Sat May 8 18:10:57 EEST 2010 root at chernobyl:/usr/obj/usr/src/sys/CHERNOBYL amd64

% dmesg|grep ' MB'   
real memory  = 4294967296 (4096 MB)
avail memory = 3829776384 (3652 MB)

In loader.conf:
vfs.zfs.arc_min=671088640
vfs.zfs.arc_max=2684354560
vfs.zfs.prefetch_disable=1
vm.kmem_size=3221225472

>Description:

When copying files from UFS to ZFS, performance decreases quickly to a small
fraction of the transfer rate of disks. The problem seems to be that caching of
files on UFS decreases the available memory so much that ZFS decreases ARC size
to the minimum and starts to throttle writes.

>How-To-Repeat:

Mount a ZFS filesystem and a UFS filesystem. Copy large files from UFS to ZFS.
Wait until top(1) shows that there is little memory free and that
kstat.zfs.misc.arcstats.size has decreased to vfs.zfs.arc_min. Observe the bad
performance and the continuous rapid increase of
kstat.zfs.misc.arcstats.memory_throttle_count.

>Fix:

I've managed to get better performance by modifying arc_memory_throttle to
include v_cache_count in addition to v_free_count to the number of available
pages, and increasing vm.v_cache_min and vm.v_cache_max so that more cached
file data is included in v_cache_count instead of v_inactive_count.

Here's a patch to sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c:

--- arc.c.orig  2010-05-08 17:53:38.343964308 +0300
+++ arc.c       2010-05-08 17:57:34.756952644 +0300
@@ -3516,7 +3516,8 @@
 {
 #ifdef _KERNEL
        uint64_t inflight_data = arc_anon->arcs_size;
-       uint64_t available_memory = ptoa((uintmax_t)cnt.v_free_count);
+       uint64_t available_memory = ptoa((uintmax_t)cnt.v_free_count +
+                                        (uintmax_t)cnt.v_cache_count);
        static uint64_t page_load = 0;
        static uint64_t last_txg = 0;

With this patch, vfs.zfs.arc_min=671088640, vm.v_cache_min=300000 and
vm.v_cache_max=400000, performance is adequate and I haven't seen a single
increase of kstat.zfs.misc.arcstats.memory_throttle_count.

This still doesn't solve the problem that ZFS decreases ARC size when reading
from UFS. One way to fix that would be to implement a configurable maximum
limit for the amount of cached file data for UFS, like vfs.zfs.arc_max for ZFS.

Another way to prevent ARC from decreasing is increasing vfs.zfs.arc_min when
copying data from a UFS partition and decreasing it to the original value after
all data is copied. However, vfs.zfs.arc_min can't be changed with sysctl, so
that approach requires two reboots. Perhaps it should be made possible to
change vfs.zfs.arc_min without requiring a reboot?
>Release-Note:
>Audit-Trail:
>Unformatted: