Swapfile on ZFS & Deadlock
Benjamin.Close at clearchain.com
Sat Jun 16 09:57:33 UTC 2007
Kris Kennaway wrote:
> On Sat, Jun 16, 2007 at 12:51:41PM +0930, Benjamin Close wrote:
>> Kris Kennaway wrote:
>>> On Fri, Jun 15, 2007 at 11:00:04PM +0930, Benjamin Close wrote:
>>>> Hi All,
>>>> Whilst running out of memory compiling Xorg (scanPCI is a killer) I
>>>> discovered a quick way to deadlock the system:
>>>> dd if=/dev/zero of=somefileonzfs bs=something count=something
>>>> mdconfig -a -f something
>>>> swapon /dev/md0
>>>> Then do something that needs swap.. instant deadlock. The system is
>>>> still responsive but all disk access become hung.
>>>> Known issue? If so is there a way we can warn users/prevent users from
>>>> doing it?
>>> Enable DEBUG_VFS_LOCKS and DEBUG_LOCKS, then break to DDB when the
>>> deadlock occurs and do 'show lockedvnods'.
>> Ok, enabled the above and this time got:
>> swap_pager: indefinite wait buffer: bufobj: 0, blkno: 312865, size: 16384
>> just before the deadlock:
>> Show locked vnods returns (hand transcribed)
>> 0xffffff002c3ab5d0: tagz zfs, type VREG
>> usecout 1, writecount 1, refcount 2 mountedhere 0
>> v_object 0xffffff002f047c80 ref 0 pages 0
>> lock type zfs: EXCL (count 1) by thread 0xffffff0030573360 (pid 1188)
> What was needed was the backtrace that should have been also displayed
oops, made the kernel, forgot to install it, new trace below - looks
like this one is not directly zil related.
>> Pid 1188 is:
>> 1188 0 0 0 SL zfs:(&zi 0xffffff000208fd58 [md0]
> Also a trace of the current backtrace of this. However based on the
> single byte of information 'i' I predict that this is due to zil (ZFS
> intent logging). I have turned this off on my machines because of
> deadlock conditions during low memory, which is also going to be true
> on your system. Try adding to /boot/loader.conf
So this is a known issue then.
>> called doadump and though it went through the motions, savecore didn't
>> find anything saved.
>> Not sure what you need debugging wise, let me know.
>> System is Intel Core 2 duo, running in SMP amd64, updated Friday 15th June.
>> The box has 1G physical ram, 1G dedicated swap partition, but needs an
>> extra 300M to compile xf86ScanPCI.c (freaky!).
> Or just use -O0 on this file.
In the end I did use -O0 but figured the report would be more
worthwhile, it's certainly a issue for when zfs becomes non experimental.
Here's the latest trace:
0xffffff0023daa20: tag zfs, type VREG
usecount 1, writecount 1, refcount 2, mounted here0
vobject 0xffffff002df1cbb8 ref 0 pages 0
lock type zfs: EXCL (count 1) by thread 0xffffff002ed5b000 (pid 1033)
#0 0xfffffff8045f62b at _lockmgr+0x4cb
#1 at ... VOP_LOCK1_APV+0x79
#2 at ... _vn_lock+0x94
#3 at ... mdstart_vnode+0x15f
#4 at ... md_kthread+0x111
#5 at ... fork_exit
#6 at ... fork_trampoline
1033 0 0 0 SL vmwait 0xffffffff80a573b8 [md0]
Looks like zfs is wanting to write but in order to do it needs memory so
asks md for it. md needs to page something out in order to be able to
provide it. Since md is zfs file backed - deadlock.
Got a dump this time but the stack appears corrupt .... hmm, haven't got
a valid dump for a while now, wonder if something else is up.
564 vn_lock(vp,KL_EXCLUSIVE | LK_RETY, td );
More information about the freebsd-current