Locked up processes after upgrade to ZFS v15

Andriy Gapon avg at freebsd.org
Sun Oct 10 09:48:16 UTC 2010


on 08/10/2010 19:24 Andriy Gapon said the following:
> on 06/10/2010 21:51 Kai Gallasch said the following:
>>
>> Am 06.10.2010 um 19:32 schrieb Martin Simmons:
>>
>>>>>>>> On Wed, 6 Oct 2010 14:28:31 +0200, Kai Gallasch said:
>>>>
>>>> How can I debug this and get further information?
>>>
>>> procstat -k -k $pid will generate a backtrace (or replace $pid by -a for all
>>> processes).
>>
>> procstat for process 12111 (state: zfs)
>> sonnenkraft:~ # procstat -k -k 12111
>>   PID    TID COMM             TDNAME           KSTACK                       
>> 12111 102385 httpd            -                mi_switch+0x21b sleepq_switch+0x123 sleepq_wait+0x4d __lockmgr_args+0x7ae vop_stdlock+0x39 VOP_LOCK1_APV+0x9b _vn_lock+0x57 vget+0x7b cache_lookup+0x4e0 vfs_cache_lookup+0xc0 VOP_LOOKUP_APV+0xb7 lookup+0x3d3 namei+0x457 vn_open_cred+0x1e3 kern_openat+0x181 syscall+0x102 Xfast_syscall+0xe2
>>
>> procstat for process 24731 (state: zfsmrb)
>> # procstat -k -k 24731
>>   PID    TID COMM             TDNAME           KSTACK                       
>> 24731 102273 httpd            -                mi_switch+0x21b sleepq_switch+0x123 sleepq_wait+0x4d _sleep+0x369 zfs_freebsd_read+0x2a6 VOP_READ_APV+0xaf vnode_pager_generic_getpages+0x3ea VOP_GETPAGES_APV+0xb5 vnode_pager_getpages+0x8c vm_fault+0x685 trap_pfault+0x128 trap+0x52c calltrap+0x8

Hm, I think that we actually shouldn't see a stack like that.
vm_fault sets VPO_BUSY on a page before calling vnode_pager_generic_getpages, so
the thread gets stuck forever in zfs mappedread.
It seems like the page that was seen as invalid in vm_fault becomes valid while
call flow reaches mappedread.

>> In my original post I wrote that only apache httpd processes would lock up..
>> This is wrong. Several other non-httpd processes also got stuck in state zfs or zfsmrb.
> 
> Interesting.
> It's possible that TID 102385 might be waiting on a vnode lock held by TID 102273.
> But TID 102273 seems to be waiting on a vnode's page lock.
> It would be very interesting to learn what process has that page busy, for how
> long and why.
> Perhaps there is a code path that busies a page, but never un-busies it...
> 


-- 
Andriy Gapon


More information about the freebsd-fs mailing list