Swap on ZFS - bad idea? (swap_pager: indefinite wait buffer)

[Benjamin Lutz]

We have storage server that I've found unresponsive this morning. The 
console was filled with these messages:

swap_pager: indefinite wait buffer: bufobj: 0, blkno: 17344, size: 61440

The lines were all exactly the same, no variation in the numbers. 
Ctrl-Alt-Delete had no effect.

Now, the only thing related to swap that's special about this machine, is 
that we use swap on ZFS, because we figured, silent disk corruption is 
just as bad when it happens to swapped memory as it is when it happens to 
stored data. However, might this be the problem? One thought I was having 
was: ZFS hits its memory bounds, a process makes the kernel want to 
allocate a swap buffer which causes more ZFS operations which never 
complete because ZFS would need to do some housework to free memory, but 
the kernel is waiting for swap to become available, some sort of deadlock? 
I'm just speculating here, not knowing the internals of this subsystem.

The machine is a quad core system with 12GiB RAM and a RAIDZ3 consisting 
of 12x2TB disks. It's used as a storage system, with accesses mainly 
through netatalk (an AFP server, what OS X uses) and a bit of iSCSI. I've 
set the following sysctls:

# 
http://www.solarisinternals.com/wiki/index.php/ZFS_Evil_Tuning_Guide#Device_I.2FO_Queue_Size_.28I.2FO_Concurrency.29
vfs.zfs.vdev.max_pending=20
# limit kernel size to 10GB to leave some RAM for processes
vm.kmem_size_max="10737418240"
# increase number of mbuf clusters, see igb(4). The default is 25600
kern.ipc.nmbclusters="102400"

This sort of hang seems to happen every couple of months. I'd really like 
some pointers on how to increase the stability of this system. Maybe there 
are some more tunables I need to adjust? Or is swap on ZFS maybe just not 
a great idea?

Cheers,
Benjamin
--
Benjamin Lutz | Software Engineer | BIOLAB Technology AG
Dufourstr. 80 | CH-8008 Zurich | www.biolab.ch | benjamin.lutz at biolab.ch
PHONE +41 44 295 97 13 | MOBILE +41 79 558 57 13 | FAX +41 44 295 97 19



This e-mail and the information it contains including attachments are 
confidential and meant
only for use by the intended recipient(s); disclosure or copying is 
strictly prohibited. If you
are not addressed, but in the possession of this e-mail, please notify the 
sender immediately.