kldload zfs spins the system after upgrading from 12.2 to 13-BETA
avg at FreeBSD.org
Sat Mar 20 16:27:44 UTC 2021
On 20/03/2021 05:01, Yoshihiro Ota wrote:
> On Tue, 9 Mar 2021 11:24:53 +0200
> Andriy Gapon <avg at FreeBSD.org> wrote:
>> On 08/03/2021 05:24, Yoshihiro Ota wrote:
>>> On Sun, 7 Mar 2021 00:09:33 +0200
>>> Andriy Gapon <avg at FreeBSD.org> wrote:
>>>> On 06/03/2021 20:09, Yoshihiro Ota wrote:
>>>>> Hi all,
>>>>> I'm upgrading fron 12.2-RELEASE to 13-BETA/RC one by one.
>>>>> After upgrading one in VMWare, 'zfs mount -a' hangs the system.
>>>>> I don't have boottime zfs mount on nor don't have zfsroot.
>>>>> I just simply ran install world/kernel and mergemaster.
>>>> Please use procstat -kk to capture a kernel stack trace of the hung process.
>>> Actually, spining was 'kldload zfs'.
>>> Console doesn't response but ping and sshd sessions still work.
>>> procstat output is below.
>>> In addition, this doesn't happen to systems that I've been following 13-CURRENT
>>> but rather happen only wiht a system upgraded from 12.2-RELEASE to 13-RC.
>>> # procstat -kk 1049
>>> PID TID COMM TDNAME KSTACK
>>> 1049 100215 kldload - spa_init+0xc6 zfs_kmod_init+0x1a
>>> zfs_modevent+0x34 module_register_init+0x8c linker_load_module+0xaab kern_kldload+0xc1
>>> sys_kldload+0x50 syscall+0x17d g_ctx+0xe280bf29
>> If you could use kgdb to find out what source code line spa_init+0xc6
>> corresponds to that may help to see what's going on.
> It look me a while to get kgdb working properly.
> At last, I got the output.
> It looks it is spining on a mutex.
> I have few other machines run the same kernel but they can load zfs.ko.
> It is only vmware VM that spins with 'kldload zfs'.
> vmware# kgdb101 /usr/usr/lib/debug/boot/kernel/zfs.ko.debug
> GNU gdb (GDB) 10.1 [GDB v10.1 for FreeBSD]
> Copyright (C) 2020 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.
> Type "show copying" and "show warranty" for details.
> This GDB was configured as "i386-portbld-freebsd13.0".
> Type "show configuration" for configuration details.
> For bug reporting instructions, please see:
> Find the GDB manual and other documentation resources online at:
> For help, type "helpType "apropos word" to search for commands related to "word"...
> Reading symbols from zfs.ko.debug...
> (kgdb) info line *spa_init+0xc6
> Line 2345 of "/usr/src/sys/contrib/openzfs/module/zfs/spa_misc.c"
> starts at address 0x2b0461 <spa_init+193>
> and ends at 0x2b0467 <spa_init+199>.
> spa_init(spa_mode_t mode)
> mutex_init(&spa_namespace_lock, NULL, MUTEX_DEFAULT, NULL);
> mutex_init(&spa_spare_lock, NULL, MUTEX_DEFAULT, NULL); // <- line 2345
It would highly unusual (and unexplainable) for a thread to get stuck here.
Are you sure that your source tree matches the binary?
Can you disassemble the function to check instructions at and near 0xc6 offset?
More information about the freebsd-stable