Reproducible filesystem deadlock on RELENG_5

[Craig Reyenga]

On Mon, Jan 10, 2005 at 03:47:18PM -0800, Matt Reimer wrote:
> On a UP machine (P4, 128M RAM) running RELENG_5 (as of Friday), I am seeing 
> what looks like a hang or deadlock on a filesystem with 10 snapshots. Our 
> problems began when we ran out of disk space, resulting in a series of 
> these log messages:
> 
> kernel: pid 39 (bufdaemon), uid 0 inumber 7277783 on /backup: filesystem 
> full
> kernel: initiate_write_filepage: already started
> 
> So I tried to delete a snapshot to free up some space, but then the kernel 
> began panicking. In my effort to workaround the panic, I disabled 
> softupdates. Then I came across the identical panic in a post by Kris 
> Kennaway 
> (http://lists.freebsd.org/pipermail/freebsd-current/2004-September/036946.html), 
> which he fixed by increasing KSTACK_PAGES. After increasing it to 31, the 
> kernel no longer panics, but instead filesystem access seems to deadlock: 
> if I try to even touch a file into existence on that partition, the touch 
> command hangs in state 'wdrain', and other attempts to access that 
> filesystem hang as well. This problem is 100% reproducible.
> 
> How to proceed? Serial console access is available if someone wants to 
> tackle it.
> 
> Matt
> 
[sniiip]

Hi,

I am seeing this too, and it would appear that I've been beaten to sending
a message about it. :)

This is what my /var/log/kernel had to say, after rebooting from a
(live|dead)lock.

Jan 11 00:23:08 burnout kernel: >pid 44 (pagedaemon), uid 0 inumber 8298 on /var: filesystem full
Jan 11 00:23:08 burnout kernel: vnode_pager_putpages: I/O error 28
Jan 11 00:23:08 burnout kernel: vnode_pager_putpages: residual I/O 65536 at 21872
Jan 11 00:23:08 burnout kernel: pid 44 (pagedaemon), uid 0 inumber 8298 on /var: filesystem full
Jan 11 00:23:08 burnout kernel: vnode_pager_putpages: I/O error 28
Jan 11 00:23:08 burnout kernel: vnode_pager_putpages: residual I/O 65536 at 21872

Over and over and over. Of course, this log is On the /var FS itself.

FreeBSD burnout 5.3-RELEASE-p2 FreeBSD 5.3-RELEASE-p2 #1: Wed Jan  5 18:44:27 EST 2005     craig at burnout:/usr/obj/usr/src/sys/BURNOUT5  i386

I'm not sure what other info to paste, my vfs.* sysctls are all defaults,
except for vfs.usermount=1.

-Craig