ZFS and deadlock with {nullfs,NFS}
Kris Kennaway
kris at obsecurity.org
Wed Jun 20 16:28:55 UTC 2007
On Wed, Jun 20, 2007 at 12:17:54PM -0400, Joe Marcus Clarke wrote:
> On Wed, 2007-06-20 at 12:03 -0400, Kris Kennaway wrote:
> > On Wed, Jun 20, 2007 at 11:53:43AM -0400, Joe Marcus Clarke wrote:
> > > I've resurrected by amd64 Tinderbox with a ZFS base, and I've been
> > > seeing a 100% reproducible deadlock when I use it with either localhost
> > > NFS or nullfs. When this occurs, the CPU is 100% idle, but I can no
> > > longer connect via SSH, and the box will only reboot from the debugger.
> > > I know there are some tuning bits I can tweak, but all I've run across
> > > is for memory consumption. Any pointers would be helpful. I'm also at
> > > the debugger, so if there is anything I can do to help troubleshoot why
> > > this is happening, please let me know.
> > >
> > > This box is -CURRENT as of June 19, 2007. It has a GENERIC kernel minus
> > > devices I do not have (i.e. SMP kernel). I am currently using nullfs
> > > for the Tinderbox. The process that most regularly locks up is mtree.
> > > Here is the trace:
> >
> > > A full process list from the debugger can be found at
> > > http://www.marcuscom.com/downloads/cobbler_proc.txt .
> >
> > 404 at the moment, but look for processes involving zil* in the
> > backtrace. I had to disable zil (vfs.zfs.zil_disable=1 tunable) to
> > prevent low-memory deadlocks on my machines. Since then it's been
> > fine.
>
> Fixed, sorry.
>
> >
> > You may also wish to use my patches (see the archives) to improve
> > performance and low-memory behaviour.
>
> Thanks for the advice. I'll check. I didn't think low memory since it
> didn't look like I was using much. Even now with the box locked, I have
> 1035 MB free with no swap in use (this box has 2 GB total).
By default there is only a 320 MB kmem_map into which all of zfs
(including its buffer cache and I/O buffers) has to cram itself, so
that is where the low memory condition may be happening. This is one
of the things that should be tuned to give non-terrible performance by
actually allowing some caching to occur.
Kris
More information about the freebsd-current
mailing list