Performance issues with 8.0 ZFS and sendfile/lighttpd

[Ben Kelly]

On Nov 6, 2009, at 2:53 PM, Michael Loftis wrote:
> --On Monday, November 02, 2009 12:55 PM +0100 Ivan Voras <ivoras at freebsd.org 
> > wrote:
>
>> Do you have actual disk IO or is the vast majority of your data  
>> served
>> from the caches? (actually - the same question to the OP)
>
> That's the problem 64GB of RAM and ZFS doesn't seem to use any  
> cache.  It also seems to not be realizing when multiple reads are on  
> the same block (same issue sorta) and dispatches the same I/O request.


Have you tried adjusting vfs.zfs.arc_meta_limit?  I've noticed the  
default value for this is set poorly when the overall ARC size is  
small.  This happens because various structure's not actually  
allocated from the ARC like dnodes and dbufs are included in the  
metadata usage stats.  When the ARC is large this is somewhat  
negligible, but for small ARCs it overwhelms the calculated default  
metadata limit and you end up not caching any real file system metadata.

You could try increasing the metadata limit or even better increase  
your maximum ARC size to something over 1GB.  (I seem to remember your  
ARC size is 128M from earlier in the thread).

Hope that helps.

- Ben