Freebsd 11.0 RELEASE - ZFS deadlock

[Andriy Gapon]

On 10/11/2016 12:30, Henri Hennebert wrote:
> On 11/10/2016 11:21, Andriy Gapon wrote:
>> On 09/11/2016 15:58, Eric van Gyzen wrote:
>>> On 11/09/2016 07:48, Henri Hennebert wrote:
>>>> I encounter a strange deadlock on
>>>>
>>>> FreeBSD avoriaz.restart.bel 11.0-RELEASE-p3 FreeBSD 11.0-RELEASE-p3 #0 r308260:
>>>> Fri Nov  4 02:51:33 CET 2016
>>>> root at avoriaz.restart.bel:/usr/obj/usr/src/sys/AVORIAZ  amd64
>>>>
>>>> This system is exclusively running on zfs.
>>>>
>>>> After 3 or 4 days, `periodic daily` is locked in the directory
>>>> /usr/local/news/bin
>>>>
>>>> [root at avoriaz ~]# ps xa|grep find
>>>> 85656  -  D        0:01.13 find / ( ! -fstype local -o -fstype rdonly ) -prune
>>>> -o ( -name [#,]* -o -name .#* -o -name a.out -o -nam
>>>>   462  1  S+       0:00.00 grep find
>>>> [root at avoriaz ~]# procstat -f 85656
>>>>   PID COMM                FD T V FLAGS    REF  OFFSET PRO NAME
>>>> 85656 find              text v r r-------   -       - - /usr/bin/find
>>>> 85656 find               cwd v d r-------   -       - - /usr/local/news/bin
>>>> 85656 find              root v d r-------   -       - - /
>>>> 85656 find                 0 v c r-------   3       0 - /dev/null
>>>> 85656 find                 1 p - rw------   1       0 - -
>>>> 85656 find                 2 v r -w------   7      17 - -
>>>> 85656 find                 3 v d r-------   1       0 - /home/root
>>>> 85656 find                 4 v d r-------   1       0 - /home/root
>>>> 85656 find                 5 v d r----n--   1 533545184 - /usr/local/news/bin
>>>> [root at avoriaz ~]#
>>>>
>>>> If I try `ls /usr/local/news/bin` it is also locked.
>>>>
>>>> After `shutdown -r now` the system remain locked after the line '0 0 0 0 0 0'
>>>>
>>>> After a reset and reboot  I can access /usr/local/news/bin.
>>>>
>>>> I delete this directory and reinstall the package `portupgrade -fu news/inn`
>>>>
>>>> 5 days later `periodic daily`is locked on the same directory :-o
>>>>
>>>> Any idea?
>>>
>>> I can't help with the deadlock, but someone who _can_ help will probably ask for
>>> the output of "procstat -kk PID" with the PID of the "find" process.
>>
>> In fact, it's procstat -kk -a.  With just one thread we would see that a thread
>> is blocked on something, but we won't see why that something can not be acquired.
>>
>>
> I attach the result,

Interesting.  I can not spot any suspicious thread that would hold the vnode
lock.  Could you please run kgdb (just like that, no arguments), then execute
'bt' command and then select a frame when _vn_lock is called with 'fr N'
command.  Then please 'print *vp' and share the result.



-- 
Andriy Gapon