zfs very poor performance compared to ufs due to lack of cache?

[jhell]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 09/03/2010 21:09, Artem Belevich wrote:
>> For reference top shows:-
>> Mem: 42M Active, 3129M Inact, 565M Wired, 3188K Cache, 19M Buf, 203M Free
> 
> Here's your problem -- inactive list got all your memory and starved ZFS ARC.
> Easy workaround is to set vfs.zfs.arc_min to a value that would
> guarantee that ARC does not give up too much memory. Let's say - 2GB.
> Be warned that it would effectively make those 2GB unavailable to
> applications.
> 
> Long term, though, there were number of patches posted on
> freebsd-current and freebsd-hackers recently that do improve that
> particular issue with ZFS. This patch in particular may help you:
> http://lists.freebsd.org/pipermail/freebsd-hackers/2010-August/032731.html
> 
>> Swap: 4096M Total, 1180K Used, 4095M Free
>>
>> So what are we missing and how to we get zfs to perform like ufs
>> and use all free ram as cache?
> 

Attached is the needfree patch mentioned in the URL alongside a local
system patch to adjust kern.maxusers to no more than 512 on systems that
can support it. Modern systems would always raise to a value of 512 or
more so I do not see a need to keep it limited to 384. This setting also
effects the outcome of other values in the system.

Not saying that these will directly help your case in anyway shape or
form but if you would like to give them a shot then they are here for
the taking.

Apply with:
cd /usr/src
patch -p1 </path/to/patch


Regards,

- -- 

 jhell,v
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.16 (FreeBSD)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQEcBAEBAgAGBQJMgl1kAAoJEJBXh4mJ2FR+oggH/2Rq2C2vakQkUr9tGraeR+BN
9aIjTPjqAaUIZ/uwrb0q8UBVbb/5pgPDknKlGQUs3S84UwbS1bZ91NbQwoadqqwx
MzUFmvETCCKa1GuU0KE6ozd61J9IN7uEm2ZaZhHtcfqsAe5oRMBYhSKNAvOi19H4
4ePWQI0Cgk3lhzAesbG5YD23dh9mOe7EU2cAR8mbwGA8C56bfLK3VSz1mjq2VfVf
rkjerZoIkPtrswtXqhdC3G1Zwe+PkPIUfR5IvVWauaLVlt++nywVoB6gZ8JzoWMR
kECtVn+/TBbr+69p0XcL6R31DDhtM/QhdisMdpuf2gLakHRXtm4EMBWSTgGon7Y=
=YpPh
-----END PGP SIGNATURE-----
-------------- next part --------------
changeset:   181:deccbf75a766
user:        J. Hellenthal <jhell at DataIX.net>
date:        Fri Sep 03 12:50:37 2010 -0400
files:       sys/kern/subr_param.c
description:
Increase maxusers to 512 by default if it equals 512 or higher.


diff -r 627cf9b49e0f -r deccbf75a766 sys/kern/subr_param.c
--- a/sys/kern/subr_param.c	Fri Sep 03 12:42:00 2010 -0400
+++ b/sys/kern/subr_param.c	Fri Sep 03 12:50:37 2010 -0400
@@ -257,8 +257,8 @@
 		maxusers = physpages / (2 * 1024 * 1024 / PAGE_SIZE);
 		if (maxusers < 32)
 			maxusers = 32;
-		if (maxusers > 384)
-			maxusers = 384;
+		if (maxusers > 512)
+			maxusers = 512;
 	}
 
 	/*

-------------- next part --------------
changeset:   188:dafc39874132
user:        J. Hellenthal <jhell at DataIX.net>
date:        Sat Sep 04 07:34:28 2010 -0400
files:       sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c
description:
PFC: r211762 arc.c needfree patch

------------------------------------------------------------------------
r211762 | avg | 2010-08-24 13:48:22 -0400 (Tue, 24 Aug 2010) | 20 lines

zfs arc_reclaim_thread: no need to call arc_reclaim_needed when
resetting needfree

needfree is checked at the very start of arc_reclaim_needed.
This change makes code easier to follow and maintain in face of
potential changed in arc_reclaim_needed.

Also, put the whole sub-block under _KERNEL because needfree can be set
only in kernel code.

To do: rename needfree to something else to aovid confusion with
OpenSolaris global variable of the same name which is used in the same
code, but has different meaning (page deficit).

Note: I have an impression that locking around accesses to this variable
as well as mutual notifications between arc_reclaim_thread and
arc_lowmem are not proper.

MFC after:      1 week

------------------------------------------------------------------------


diff -r 34a823aff8f6 -r dafc39874132 sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c
--- a/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c	Sat Sep 04 07:23:09 2010 -0400
+++ b/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c	Sat Sep 04 07:34:28 2010 -0400
@@ -2317,12 +2317,12 @@
 		if (arc_eviction_list != NULL)
 			arc_do_user_evicts();
 
-		if (arc_reclaim_needed()) {
+#ifdef _KERNEL
+		if (needfree) {
 			needfree = 0;
-#ifdef _KERNEL
 			wakeup(&needfree);
+		}
 #endif
-		}
 
 		/* block until needed, or one second, whichever is shorter */
 		CALLB_CPR_SAFE_BEGIN(&cpr);

-------------- next part --------------
A non-text attachment was scrubbed...
Name: maxusers.patch.sig
Type: application/octet-stream
Size: 287 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-fs/attachments/20100904/fedcc8dc/maxusers.patch.obj
-------------- next part --------------
A non-text attachment was scrubbed...
Name: needfree.patch.sig
Type: application/octet-stream
Size: 287 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-fs/attachments/20100904/fedcc8dc/needfree.patch.obj