SWAP / ZFS / SSD question

[Paul Kraus]

Hi all,
	I’ve got a box running Freebsd 9.1 with VirtualBox and have run into a couple odd hangs. The system is a SuperMicro with two quad-core Xeon, 32GB RAM, and 3 1TB SAS drives mirrored via an internal LSI 2208 controller (the drives are in JBOD mode) via ZFS.  The system is all ZFS with a 32GB zvol for swap.

	I am occasionally running into the dreaded "kernel: swap_pager: indefinite wait buffer: bufobj: 0” issue. In the past it was always when I was either adding a VM or changing a VM’s configuration. I assumed low RAM. But it hit last night (at 03:00 this morning in fact) and nothing was changing at that time. There was about 770MB RAM free and the ZFS ARC was using about 4GB, so the system was not really under heavy memory pressure at the time.

	In order to reduce the impact, until we put a more permanent fix in place, I have set sync=disable for the zvol … I figure that if the system crashes and I lose a swap write I really don’t care (I am assuming here that swapd commits it’s writes as sync).

	Our long term plan is a change in overall architecture that will separate storage and compute services and have multiple compute nodes. We started with one box assuming we would grown to more than one as the business need arose. So our mid-term plan is to install a dedicated swap device. The question is whether swapd (under 9.1) will be happy with an SSD or whether there are unintended consequences to that route. Small SSDs are cheap enough these days, we really don’t need more than 32GB (64GB if we are feeling extravagant).

	I know about the major performance differences between SSD vendors and models (I heard all about that on the various ZFS lists I subscribe to), so we will have to choose an SSD carefully.

--
Paul Kraus
paul at kraus-haus.org