misc/142914: ZFS performance degratation over time

[Miks]

>Number:         142914
>Category:       misc
>Synopsis:       ZFS performance degratation over time
>Confidential:   no
>Severity:       serious
>Priority:       low
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Sun Jan 17 18:00:07 UTC 2010
>Closed-Date:
>Last-Modified:
>Originator:     Miks
>Release:        Freebsd 8.0 Release
>Organization:
>Environment:
FreeBSD host1 8.0-RELEASE FreeBSD 8.0-RELEASE #0: Sat Nov 21 15:02:08 UTC 2009     root at mason.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC  amd64
>Description:
After migrating servers to from UFS to ZFS, we got constant file system performance degradation in couple time by week.
System load has not been changed since migration to ZFS, so only reason of this problems can be ZFS itself.
We had this problems on Intel server systems with 8GB and more RAM. We also tried to disable prefetch, but problem still existed.
Even with latest FreeBSD 8.0 Stable we had same problem.
One solution to this problem is - stop all httpd, php-cgi and mysql processes, wait 1-2 minutes and then start it again. 
On other servers we just migrated back to UFS and problem was gone.

One strange thing we noticed is - when server have performance problem, "zpool iostat" is showing like this
"
pool         used  avail   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
zroot        113G  26.1G     53    392  2.80M  20.3M
"

after processes restart:
"
pool         used  avail   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
zroot       99.9G  39.1G      6     40   367K  1.12M

"

How can "used" value differ?
>How-To-Repeat:

>Fix:


>Release-Note:
>Audit-Trail:
>Unformatted: