zfs very poor performance compared to ufs due to lack of cache?

[Steven Hartland]

Ok the results are in, the conclusions I can see from the data, others may see
more, are:-

=== Common ===
* arc size on boot is ~180M with a target size of 6.5G

=== sendfile on ===
* arc size increases on demand but peaks the min value
* The difference between min and max arc is taken up by inactive pages
* vm page daemon wakeups sit at a constant level once the machine has
  filled memory and never fully empties swap.

=== sendfile off ===
* arc size increases on demand all the way up the the max value
* vm cache count stays at almost zero all the time

=== conclusion ===
The interaction of zfs and sendfile is causing large amounts of memory
to end up in the inactive pool and only the use of a hard min arc limit is
ensuring that zfs forces the vm to release said memory so that it can be
used by zfs arc.

The source data, xls's and exported graphs can be found here:-
http://www.multiplaygameservers.com/dropzone/zfs-sendfile-results.zip

    Regards
    Steve

================================================
This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. 

In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337
or return the E.mail to postmaster at multiplay.co.uk.