ZFS v13 performance drops with low memory on FreeBSD-7 STABLE

[Nathanael Hoyle]

Kai Gallasch wrote:
> Hi.
>
> I upgraded a server with 7-STABLE-amd64 and the MFC'd ZFS v13 about 8
> days ago. Since then the machine is running stable and this without
> manually tuning vm.kmem_size, vfs.zfs.arc, etc. in loader.conf - so far
> so good :-)
>
> In the last few days I noticed some performance issues with zfs, as some
> customers complained about slow mysql database responses.
>
> MySQL is running in a database jail on a zfs v13 zpool, websites using
> the mysql database are also running on zfs on the same server.
>
>   
Last I knew, ZFS on FreeBSD still wasn't considered production. I'd be 
careful using it on customer-facing systems.
> The server is running about 30 in production jails, has 16GB RAM and 8GB
> swap. Swap usage is about only 1% currently.
>   
More important than usage is swap activity.  Is the box actually 
swapping under load?  A little disk I/O thrown in can destroy performance.
> After debugging the mysql settings for a while I found out, that when I
> stopped some processes on the server that were using high amounts of
> RAM, the datbase response times for queries were almost back to normal
> again..
>
> So for me this looks like when running applications and ZFS compete for
> free RAM, ZFS looses. Is that so?
>   
Yes, the ARC cache is designed to back off and give the other 
applications RAM to run.  Potentially, rather than tuning the ZFS ARC 
size (which may not be appropriate, if other applications are really 
using up that RAM legitimately), if the MySQL performance is the issue, 
try increasing the shared pool memory buffers for the MySQL instance (I 
use postgres, I'm expecting MySQL to have similar options).
> Is there anything I can do (besides buying more RAM :) to help ZFS to
> secure it's share of RAM, to prevent a performance drop?
>
> I was thinking about setting vm.kmem_size_min to about 2 GB, would that
> help zfs performance?
>   
That is the sysctl to tune the minimum ARC size.  Increasing it would 
guarantee more RAM to the ZFS ARC, which might or might not increase 
performance (it would speed up ZFS, potentially at the expense of other 
apps), depending on working-set memory pressure versus disk I/O patterns.

<snip>

-Nathanael