PHP-FPM processes hang in UFS-state

[Christos Chatzaras]

> On 24 Oct 2020, at 21:13, Konstantin Belousov <kostikbel at gmail.com> wrote:
> I do not remember about 12.1.  I certainly not aware of anything in 12.2.
> Quota is less tested than non-quota.  One of the backtraces in your dump
> indicates that a quota file lock was preventing other thread to make a
> progress.

Can you tell me where I see this file lock about quota? I search https://www.coderz.gr/ufs/procstat.txt <https://www.coderz.gr/ufs/procstat.txt> and didn't find it.


Also "10.6. Debugging Deadlocks" write "The recipe below is especially useful if you suspect that a deadlock occurs in the VFS layer. Add these options to the kernel configuration file.".  Do I have to compile new kernel with these options enabled or I can try with my GENERIC kernel?

If I understand correctly because the system doesn't panic but keeps running I have to make it panic using "sysctl debug.kdb.panic=1" to create the core dump. Then after the system boots I run "kgdb /boot/kernel/kernel /var/crash/vmcore.0". Then I run the commands described in "10.6. Debugging Deadlocks". Am I right?

Here is procstat --k for all stuck processes:

procstat -kk 18239  39430  54990  63240  66176  73548  83476 84493  92011  95629

  PID    TID COMM                TDNAME              KSTACK
18239 100333 php-fpm             -                   mi_switch+0xe2 sleepq_wait+0x2c sleeplk+0x185 lockmgr_slock_hard+0x1ae __lockmgr_args+0xe7 ffs_lock+0xa5 VOP_LOCK1_APV+0x7c _vn_lock+0x44 vget+0x4a cache_lookup+0x9e7 vfs_cache_lookup+0xac VOP_LOOKUP_APV+0x76 lookup+0x6d1 namei+0x437 kern_statat+0x72 sys_fstatat+0x2f amd64_syscall+0x364 fast_syscall_common+0x101

39430 100747 php-fpm             -                   mi_switch+0xe2 sleepq_wait+0x2c sleeplk+0x185 lockmgr_slock_hard+0x1ae __lockmgr_args+0xe7 ffs_lock+0xa5 VOP_LOCK1_APV+0x7c _vn_lock+0x44 vget+0x4a cache_lookup+0x9e7 vfs_cache_lookup+0xac VOP_LOOKUP_APV+0x76 lookup+0x6d1 namei+0x437 kern_statat+0x72 sys_fstatat+0x2f amd64_syscall+0x364 fast_syscall_common+0x101

54990 100232 php-fpm             -                   mi_switch+0xe2 sleepq_wait+0x2c sleeplk+0x185 lockmgr_slock_hard+0x1ae __lockmgr_args+0xe7 ffs_lock+0xa5 VOP_LOCK1_APV+0x7c _vn_lock+0x44 vget+0x4a cache_lookup+0x9e7 vfs_cache_lookup+0xac VOP_LOOKUP_APV+0x76 lookup+0x6d1 namei+0x437 kern_statat+0x72 sys_fstatat+0x2f amd64_syscall+0x364 fast_syscall_common+0x101

63240 100218 php-fpm             -                   mi_switch+0xe2 sleepq_wait+0x2c sleeplk+0x185 lockmgr_slock_hard+0x1ae __lockmgr_args+0xe7 ffs_lock+0xa5 VOP_LOCK1_APV+0x7c _vn_lock+0x44 vget+0x4a cache_lookup+0x9e7 vfs_cache_lookup+0xac VOP_LOOKUP_APV+0x76 lookup+0x6d1 namei+0x437 kern_statat+0x72 sys_fstatat+0x2f amd64_syscall+0x364 fast_syscall_common+0x101

66176 100256 php-fpm             -                   mi_switch+0xe2 sleepq_wait+0x2c sleeplk+0x185 lockmgr_slock_hard+0x1ae __lockmgr_args+0xe7 ffs_lock+0xa5 VOP_LOCK1_APV+0x7c _vn_lock+0x44 vget+0x4a cache_lookup+0x9e7 vfs_cache_lookup+0xac VOP_LOOKUP_APV+0x76 lookup+0x6d1 namei+0x437 kern_statat+0x72 sys_fstatat+0x2f amd64_syscall+0x364 fast_syscall_common+0x101

73548 100630 php-fpm             -                   mi_switch+0xe2 sleepq_wait+0x2c sleeplk+0x185 lockmgr_xlock_hard+0x1f5 __lockmgr_args+0x2bf ffs_lock+0xa5 VOP_LOCK1_APV+0x7c _vn_lock+0x44 vget+0x4a vfs_hash_get+0xb0 ffs_vgetf+0x3f softdep_sync_buf+0xbee ffs_syncvnode+0x26f ffs_truncate+0x71b ufs_direnter+0x5f8 ufs_makeinode+0x5fb ufs_create+0x34 VOP_CREATE_APV+0x76

83476 100710 php-fpm             -                   mi_switch+0xe2 sleepq_wait+0x2c sleeplk+0x185 lockmgr_slock_hard+0x1ae __lockmgr_args+0xe7 ffs_lock+0xa5 VOP_LOCK1_APV+0x7c _vn_lock+0x44 vget+0x4a cache_lookup+0x9e7 vfs_cache_lookup+0xac VOP_LOOKUP_APV+0x76 lookup+0x6d1 namei+0x437 kern_statat+0x72 sys_fstatat+0x2f amd64_syscall+0x364 fast_syscall_common+0x101

84493 100835 php-fpm             -                   mi_switch+0xe2 sleepq_wait+0x2c sleeplk+0x185 lockmgr_slock_hard+0x1ae __lockmgr_args+0xe7 ffs_lock+0xa5 VOP_LOCK1_APV+0x7c _vn_lock+0x44 vget+0x4a cache_lookup+0x9e7 vfs_cache_lookup+0xac VOP_LOOKUP_APV+0x76 lookup+0x6d1 namei+0x437 kern_accessat+0x108 amd64_syscall+0x364 fast_syscall_common+0x101

92011 100586 php-fpm             -                   mi_switch+0xe2 sleepq_wait+0x2c sleeplk+0x185 lockmgr_slock_hard+0x1ae __lockmgr_args+0xe7 ffs_lock+0xa5 VOP_LOCK1_APV+0x7c _vn_lock+0x44 vget+0x4a cache_lookup+0x9e7 vfs_cache_lookup+0xac VOP_LOOKUP_APV+0x76 lookup+0x6d1 namei+0x437 kern_statat+0x72 sys_fstatat+0x2f amd64_syscall+0x364 fast_syscall_common+0x101

95629 101591 php-fpm             -                   mi_switch+0xe2 sleepq_wait+0x2c sleeplk+0x185 lockmgr_slock_hard+0x1ae __lockmgr_args+0xe7 ffs_lock+0xa5 VOP_LOCK1_APV+0x7c _vn_lock+0x44 vget+0x4a cache_lookup+0x9e7 vfs_cache_lookup+0xac VOP_LOOKUP_APV+0x76 lookup+0x6d1 namei+0x437 kern_statat+0x72 sys_fstatat+0x2f amd64_syscall+0x364 fast_syscall_common+0x101