Swapping on ZFS - stability issue

[Peter Schuller]

Hello,

Yesterday a machine went unresponsive (pinged, TCP connections died instantly 
after establishment) died after someone started MySQL with cranked up buffer 
sizes.

When I got to the machine there was no sensible message on the console, but I 
did see output to the effect of "swap", "pager" and such all garbled.

The machine had swap configured on a ZVOL which made me suspicious, so after 
rebooting I tried running a simple loop:

  for (int i =0 ; i < 4000; i++) {
    char* mem = (char*)malloc(1024*1024);
    memset(mem, NULL, 1024*1024);
  }

The machine swaped a select few megs and then died completely (no messages; 
couldn't kill top or the test application, though the console still worked). 
This was done on a machine with 4 GB of RAM and 4 GB of swap on a ZVOL on 
7-CURRENT.

Later on I tried this on my workstation too (3 GB of RAM). Swapping to 
glabel-on-disk behaved as you would expect; it swapped hundreds of megs to 
swap device without the system every being adversely affected beyond the 
obvious result of disk activity and memory pressue.

However, swapping either to a ZVOL or a file (through md) on ZFS exhibited 
similar behavior to that observed on the original server; it would get off a 
few megs fairly slowly followed by the system becoming completely unusable, 
with both top and the test app not being killable by CTRL-C, though virtual 
console switching was operational.

Swapping to ZFS was significantly slower (on the order of factor 10 or so), 
yet I do not believe slowness in and of itself is the only explanation for 
the difference (as I would still have expected to be able to recover, even if 
it might have taken a bit longer).

I'll see if I can confirm behavior on a more recent CURRENT. The above was on 
CURRENT:s from 1-2 months ago in both cases.

-- 
/ Peter Schuller

PGP userID: 0xE9758B7D or 'Peter Schuller <peter.schuller at infidyne.com>'
Key retrieval: Send an E-Mail to getpgpkey at scode.org
E-Mail: peter.schuller at infidyne.com Web: http://www.scode.org