zfs very poor performance compared to ufs due to lack of cache?

[Artem Belevich]

> For reference top shows:-
> Mem: 42M Active, 3129M Inact, 565M Wired, 3188K Cache, 19M Buf, 203M Free

Here's your problem -- inactive list got all your memory and starved ZFS ARC.
Easy workaround is to set vfs.zfs.arc_min to a value that would
guarantee that ARC does not give up too much memory. Let's say - 2GB.
Be warned that it would effectively make those 2GB unavailable to
applications.

Long term, though, there were number of patches posted on
freebsd-current and freebsd-hackers recently that do improve that
particular issue with ZFS. This patch in particular may help you:
http://lists.freebsd.org/pipermail/freebsd-hackers/2010-August/032731.html

> Swap: 4096M Total, 1180K Used, 4095M Free
>
> So what are we missing and how to we get zfs to perform like ufs
> and use all free ram as cache?

--Artem