misc/145229: Vast differences in ZFS ARC behavior between 8.0-RC1 and 8.0-RELEASE

[Rich Ercolani]

>Number:         145229
>Category:       misc
>Synopsis:       Vast differences in ZFS ARC behavior between 8.0-RC1 and 8.0-RELEASE
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Wed Mar 31 07:00:20 UTC 2010
>Closed-Date:
>Last-Modified:
>Originator:     Rich Ercolani
>Release:        8.0-RC1
>Organization:
JHU ACM
>Environment:
FreeBSD manticore.acm.jhu.edu 8.0-RC1 FreeBSD 8.0-RC1 #3: Tue Sep 29 17:31:37 EDT 2009     root@:/usr/obj/usr/local/ncvs/src/sys/DTRACE  amd64

>Description:
Recently, I was forced back into using 8.0-RC1 temporarily
when the root disk of the server failed, and we switched
to an old copy of the root (we'd swapped hard drives six
months ago, and had kept the old one in case of emergency).

We were surprised to find that, while the pathological cases
of latency were still bad, the peak performance of ZFS on
8.0-RC1 was strictly nicer than 8.0-RELEASE! In particular:
 Operation      Count    AvgLat    MaxLat
 ----------------------------------------
 NTCreateX      59220     2.745  1584.125
 Close          43455     2.551  1583.580
 Rename          2487     5.284  1128.664
 Unlink         12066     2.016  1581.133
 Qpathinfo      53524     1.337  1582.991
 Qfileinfo       9209     1.528   818.264
 Qfsinfo         9764     2.469   545.334
 Sfileinfo       4862     2.084   311.741
 Find           20457     3.734  1983.271
 WriteX         28984     0.051     3.099
 ReadX          91122     0.638  1579.555
 LockX            182     0.554     2.514
 UnlockX          182     1.233     9.976
 Flush           4165     7.339  1998.383

Throughput 30.3449 MB/sec  10 clients  10 procs
max_latency=1998.389 ms


In addition to the above numbers, we found that, more
concretely, the free page count on our server was now
stable at an order of magnitude higher than it had
been on 8.0-RELEASE (3.1 million versus ~300k), and
the ZFS ARC was now stable at approximately 1.2G
(whereas on 8.0-RELEASE, regardless of how we modified
the _min and _max tunables, the ARC would always
stabilize at ~220M and not budge).

And, concretely, while performance on non-local NFS
still has horrible latency in the worst case, all
of the throughput cases rapidly accelerate up from
the 1-2 MB/s observed (see PR 145189) toward 8-30 MB/s,
which is not ideal, but still far better than extant.

I'll have a stock 8.0-RELEASE install disk for this
machine within a few days, and will be able to switch
between them freely for instrumenting this - but it's
a fascinating....if not regression, difference in
performance.
>How-To-Repeat:
1) Compare local and NFS-based ZFS performance on RC1 and RELEASE
>Fix:


>Release-Note:
>Audit-Trail:
>Unformatted: