ZFS performance gains real or imaginary?

[Adam McDougall]

On Tue, Dec 23, 2008 at 12:43:47PM -0800, Matt Simerson wrote:

  
  On Dec 18, 2008, at 4:12 PM, Andrew Snow wrote:
  
  >> If so, then I really should be upgrading my production ZFS servers  
  >> to the latest -HEAD.
  >
  > Thats correct, that is the only way to get the best working version  
  > of ZFS.  Of course, then everything is unstable and broken - eg.  
  > SMBFS became unusable for me and would crash the server.
  
  I got bit by the ARP bug (kern/129730) which really annoyed our  
  network manager. After applying a patch for that, I've got a working  
  kernel and upgraded ZFS.
  
  Unfortunately, the newer kernel hangs much more frequently. Previously  
  I was getting nearly a month between reboots. Now I don't even get 1  
  day between hangs. Worse yet, I upgraded the ZFS version of the pools,  
  to see if that would make any difference. It did not, and now I can't  
  revert. :-(
  
  I have these settings in /boot/loader.conf
  
  	vm.kmem_size="1536M"
  	vm.kmem_size_max="1536M"
  	vfs.zfs.arc_max="100M"
  
  I have also experimented with vfs.zfs.prefetch_disable,  
  vfs.zfs.arc_min in the past, and I'm open to suggestions on what might  
  help under this workload (multiple concurrent rsync processes from  
  remote systems to this one).
  
  Matt
  
  Specs: SuperMicro 24 disk, 8-core Xeon, 16GB RAM, 2 x 12 disk HW RAID  
  striped (RAID 0).
  
  back01# uname -a
  FreeBSD back01.int.spry.com 8.0-CURRENT FreeBSD 8.0-CURRENT #2: Fri  
  Dec 19 15:37:12 PST 2008     root at back01.int.spry.com:/usr/obj/usr/src/ 
  sys/BACK01  amd64

Can you try:

vm.kmem_size=2G
vm.kmem_size_max=2G
vfs.zfs.arc_max=512M

This has been working for me on one amd64 system that only has 2G of ram
but had similar problem frequency to yours.  I don't know if its coincidence
with the data that I am rsyncing lately, but: 10:47PM  up 22 days,  7:12