ARC size constantly shrinks, then ZFS slows down extremely

[Attila Nagy]

Hello,

I'm using FreeBSD 8 (previously 7) on a machine with a lot of disks and 
32 GB RAM. With 7.x it ran very well for about 50 days, but suddenly 
every operation have slowed down.
gstat showed that the disks are working a lot more than usual the 
zpool/zfs was pretty unusable.

I've rebooted the machine then with FreeBSD 8 in the hope the new ZFS 
fixes will correct this issue (no 50 days have passed since then, so I 
don't know yet) and started to monitor ZFS's statistics.

It seems that after a reboot, the ARC size starts to grow, then 
something flips the switch and it changes to shrinking, instead of 
maintaining the size.

Please see the pictures here: 
http://people.fsn.hu/~bra/freebsd/20090929-zfs-arcsize/

Before the 27th, the machine ran FreeBSD 7, after that date it runs 8.

As you can see, no user process tooks the memory, so I don't know why 
the ARC size grows first and then start to decrease.

Could it be that the ARC size decreases such a big amount that it 
effectively disappears and this causes the IO activity go up and kill 
the machine?

Thanks,