zfs l2arc issue

[Robert Schulze]

Hi,

we are running an NFS server with the following pool setup:

home        ONLINE       0     0     0
	  raidz2    ONLINE       0     0     0
	    da1     ONLINE       0     0     0
	    da2     ONLINE       0     0     0
	    da3     ONLINE       0     0     0
	    da4     ONLINE       0     0     0
	    da5     ONLINE       0     0     0
	  raidz2    ONLINE       0     0     0
	    da6     ONLINE       0     0     0
	    da7     ONLINE       0     0     0
	    da8     ONLINE       0     0     0
	    da9     ONLINE       0     0     0
	    da10    ONLINE       0     0     0
	logs        ONLINE       0     0     0
	  mirror    ONLINE       0     0     0
	    da12    ONLINE       0     0     0
	    da13    ONLINE       0     0     0
	cache
	  ad4       ONLINE       0     0     0
	  ad8       ONLINE       0     0     0


All drives except the caching SSDs are attached to a LSI 9690SA-8I.
The system is equipped with 32 GB RAM, and runs with a load of <1, 
please note: we are running 8.0, yet, since there was one issue with ZFS 
which blocked the upgrade to 8-STABLE.

After about 100d uptime, we had a sudden large increase in load of about 
5-7, nfsd had 100-400% WCPU. Also an rsync downloading files from that 
machine was very slow.

We didn't really narrow down the problem, we had to reboot the machine 
because performance was nearly completely absent. After reboot, system 
performance became normal.

Could this problem be related to the caching SSDs beeing full? Cache 
consists of two 76 GB SSDs, after warming up, only 8 MB are free on each 
disk.
Is ZFS supposed to fill arbitrary large caches? I think of doubling the 
cache and then ending up with fully filled SSDs again. For if, could 
l2arc be limited somehow, so that SSDs don't get written full?

Could this behaviour also appear in 8-STABLE?

With kind regards,
Robert Schulze