NFS-exported ZFS instability

[Rick Macklem]

Hiroki Sato wrote:
> Hello,
> 
> I have been in a trouble about my NFS server for a long time. The
> symptom is that it stops working in one or two weeks after a boot. I
> could not track down the cause yet, but it is reproducible and only
> occurred under a very high I/O load.
> 
> It did not panic, just stopped working---while it responded to ping,
> userland programs seemed not working. I could break it into DDB and
> get a kernel dump. The following URLs are a log of ps, trace, and
> etc.:
> 
> http://people.allbsd.org/~hrs/FreeBSD/pool.log.20130102
> http://people.allbsd.org/~hrs/FreeBSD/pool.dmesg.20130102
> 
> Does anyone see how to debug this? I guess this is due to a deadlock
> somewhere. I have suffered from this problem for almost two years.
> The above log is from stable/9 as of Dec 19, but this have persisted
> since 8.X.
> 
Well, I took a quick glance at the log and there are a lot of processes
sleeping on "pfault" (in vm_waitpfault() in sys/vm/vm_page.c). I'm no
vm guy, so I'm not sure when/why that will happen. The comment on the
function suggests they are waiting for free pages.

Maybe something as simple as running out of swap space or a problem
talking to the disk(s) that has the swap partition(s) or ???
(I'm talking through my hat here, because I'm not conversant with
 the vm side of things.)

I might take a closer look this evening and see if I can spot anything
in the log, rick
ps: I hope Alan and Kostik don't mind being added to the cc list.

> -- Hiroki