8.0-RC2: ZFS deadlock with zfs receive

Wed Oct 28 16:24:50 UTC 2009

Hello,

I've been sending some alltraces to pjd about this easy to reproduce  
problem. I am using zfs send/zfs receive to replicate a dataset from  
one server to another. At 1 minute intervals, an incremental snapshot  
is sent to update the dataset copy. If there is reading activity on  
the dataset copy, a deadlock can happen rendering ZFS and all the FS  
subsystem unusable. I've tried with 8.0RC2 and it still happens.

How to reproduce:

For example, while doing a make buildworld, replicate the dataset  
containing /usr/obj to the second machine. On the second machine a  
script just runs an endless loop copying all the contents of /usr/obj  
(which, remember, are being copied from the first machine using zfs  
receive) to another location, thus generating quite a lot of read  
activity. The time to deadlock is random, but soon the deadlock appears.

The problem can be triggered by a backup accessing the destination  
dataset or when the security periodic scripts run at night.

I tried to mitigate the problem by cloning a given snapshot at the  
destination and doing the backup from it, but the problem persists.

Any ideas? Any way to mitigate this problem?

Borja.

This is the alltrace after breaking into ddb:

>> PID USERNAME  THR PRI NICE   SIZE    RES STATE   C   TIME   WCPU
>> COMMAND
>> 1914 root        1  62    0 11932K  2564K zfsvfs  0   0:51  0.00%  
>> bsdtar
>> 1093 borjam      1  44    0  8304K  2464K CPU1    1   0:32  0.00% top
>> 1913 root        1  54    0 11932K  2600K rrl->r  0   0:19  0.00%  
>> bsdtar
>> 1019 root        1  44    0 25108K  4812K select  0   0:05  0.00%  
>> sshd
>> 2008 root        1  76    0 13600K  1904K tx->tx  0   0:04  0.00% zfs
>> 1089 borjam      1  44    0 37040K  5216K select  1   0:04  0.00%  
>> sshd
>>  995 root        1  76    0  8252K  2652K pause   0   0:02  0.00% csh
>>  840 root        1  44    0 11044K  3828K select  1   0:02  0.00%

debug.ddb.capture.data: db> alltrace

Tracing command sysctl pid 1373 tid 100183 td 0xffffff00177e5ab0
kdb_enter() at kdb_enter+0x3d
kdb_sysctl_enter() at kdb_sysctl_enter+0x89
sysctl_root() at sysctl_root+0x113
userland_sysctl() at userland_sysctl+0x158
__sysctl() at __sysctl+0xaa
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (202, FreeBSD ELF64, __sysctl), rip = 0x80073d9dc, rsp =  
0x7fffffffe2a8, rbp = 0x7fffffffe37c ---

Tracing command csh pid 1369 tid 100181 td 0xffffff00177e6390
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
kern_sigsuspend() at kern_sigsuspend+0xca
sigsuspend() at sigsuspend+0x34
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (341, FreeBSD ELF64, sigsuspend), rip = 0x80094e75c, rsp =  
0x7fffffffe638, rbp = 0x800c59400 ---

Tracing command su pid 1368 tid 100159 td 0xffffff001284aab0
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
kern_wait() at kern_wait+0x3f7
wait4() at wait4+0x35
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (7, FreeBSD ELF64, wait4), rip = 0x8009e8edc, rsp =  
0x7fffffffe598, rbp = 0x559 ---

Tracing command unlink pid 1363 tid 100162 td 0xffffff001284a000
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_wait() at sleepq_wait+0x4d
__lockmgr_args() at __lockmgr_args+0x79b
ffs_lock() at ffs_lock+0x9c
VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b
_vn_lock() at _vn_lock+0x57
lookup() at lookup+0x5f0
namei() at namei+0x4a9
kern_unlinkat() at kern_unlinkat+0xab
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (10, FreeBSD ELF64, unlink), rip = 0x80072be5c, rsp =  
0x7fffffffec68, rbp = 0 ---

Tracing command sh pid 1361 tid 100100 td 0xffffff000270bab0
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
kern_wait() at kern_wait+0x3f7
wait4() at wait4+0x35
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (7, FreeBSD ELF64, wait4), rip = 0x800935edc, rsp =  
0x7fffffffe618, rbp = 0x551 ---

Tracing command cron pid 1360 tid 100122 td 0xffffff0002af0390
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
pipe_read() at pipe_read+0x4a3
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x80095a7ac, rsp =  
0x7fffffffe4b8, rbp = 0x800a9e700 ---

Tracing command unlink pid 1355 tid 100127 td 0xffffff0002aea000
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_wait() at sleepq_wait+0x4d
__lockmgr_args() at __lockmgr_args+0x79b
ffs_lock() at ffs_lock+0x9c
VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b
_vn_lock() at _vn_lock+0x57
lookup() at lookup+0x5f0
namei() at namei+0x4a9
kern_unlinkat() at kern_unlinkat+0xab
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (10, FreeBSD ELF64, unlink), rip = 0x80072be5c, rsp =  
0x7fffffffec68, rbp = 0 ---

Tracing command sh pid 1353 tid 100171 td 0xffffff00177e8ab0
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
kern_wait() at kern_wait+0x3f7
wait4() at wait4+0x35
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (7, FreeBSD ELF64, wait4), rip = 0x800935edc, rsp =  
0x7fffffffe618, rbp = 0x549 ---

Tracing command cron pid 1352 tid 100155 td 0xffffff001284d000
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
pipe_read() at pipe_read+0x4a3
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x80095a7ac, rsp =  
0x7fffffffe4b8, rbp = 0x800a9e700 ---

Tracing command unlink pid 1347 tid 100157 td 0xffffff001284c720
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_wait() at sleepq_wait+0x4d
__lockmgr_args() at __lockmgr_args+0x79b
ffs_lock() at ffs_lock+0x9c
VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b
_vn_lock() at _vn_lock+0x57
lookup() at lookup+0x5f0
namei() at namei+0x4a9
kern_unlinkat() at kern_unlinkat+0xab
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (10, FreeBSD ELF64, unlink), rip = 0x80072be5c, rsp =  
0x7fffffffec68, rbp = 0 ---

Tracing command sh pid 1345 tid 100108 td 0xffffff0002916ab0
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
kern_wait() at kern_wait+0x3f7
wait4() at wait4+0x35
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (7, FreeBSD ELF64, wait4), rip = 0x800935edc, rsp =  
0x7fffffffe618, rbp = 0x541 ---

Tracing command cron pid 1344 tid 100101 td 0xffffff000270b720
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
pipe_read() at pipe_read+0x4a3
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x80095a7ac, rsp =  
0x7fffffffe4b8, rbp = 0x800a9e700 ---

Tracing command unlink pid 1337 tid 100172 td 0xffffff00177e8720
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_wait() at sleepq_wait+0x4d
__lockmgr_args() at __lockmgr_args+0x79b
ffs_lock() at ffs_lock+0x9c
VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b
_vn_lock() at _vn_lock+0x57
lookup() at lookup+0x5f0
namei() at namei+0x4a9
kern_unlinkat() at kern_unlinkat+0xab
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (10, FreeBSD ELF64, unlink), rip = 0x80072be5c, rsp =  
0x7fffffffec68, rbp = 0 ---

Tracing command sh pid 1334 tid 100098 td 0xffffff000270e390
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
kern_wait() at kern_wait+0x3f7
wait4() at wait4+0x35
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (7, FreeBSD ELF64, wait4), rip = 0x800935edc, rsp =  
0x7fffffffe618, rbp = 0x536 ---

Tracing command newsyslog pid 1333 tid 100065 td 0xffffff00026d6ab0
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_wait() at sleepq_wait+0x4d
_sx_slock_hard() at _sx_slock_hard+0x1b7
_sx_slock() at _sx_slock+0xc1
zfs_freebsd_reclaim() at zfs_freebsd_reclaim+0x4a
VOP_RECLAIM_APV() at VOP_RECLAIM_APV+0xb5
vgonel() at vgonel+0x119
vnlru_free() at vnlru_free+0x345
getnewvnode() at getnewvnode+0x24f
ffs_vgetf() at ffs_vgetf+0xdf
ufs_lookup_() at ufs_lookup_+0xb46
VOP_CACHEDLOOKUP_APV() at VOP_CACHEDLOOKUP_APV+0xaf
vfs_cache_lookup() at vfs_cache_lookup+0xf0
VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xb7
lookup() at lookup+0x2eb
namei() at namei+0x4a9
kern_statat_vnhook() at kern_statat_vnhook+0x8f
kern_statat() at kern_statat+0x15
stat() at stat+0x2a
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (188, FreeBSD ELF64, stat), rip = 0x80073251c, rsp =  
0x7fffffffc2b8, rbp = 0x8009075e0 ---

Tracing command cron pid 1331 tid 100147 td 0xffffff0002cb9000
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
pipe_read() at pipe_read+0x4a3
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x80095a7ac, rsp =  
0x7fffffffe4b8, rbp = 0x800a9e700 ---

Tracing command cron pid 1330 tid 100178 td 0xffffff00177e7000
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
pipe_read() at pipe_read+0x4a3
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x80095a7ac, rsp =  
0x7fffffffe4b8, rbp = 0x800a9e700 ---

Tracing command unlink pid 1329 tid 100177 td 0xffffff00177e7390
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_wait() at sleepq_wait+0x4d
__lockmgr_args() at __lockmgr_args+0x79b
ffs_lock() at ffs_lock+0x9c
VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b
_vn_lock() at _vn_lock+0x57
lookup() at lookup+0x5f0
namei() at namei+0x4a9
kern_unlinkat() at kern_unlinkat+0xab
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (10, FreeBSD ELF64, unlink), rip = 0x80072be5c, rsp =  
0x7fffffffec68, rbp = 0 ---

Tracing command sh pid 1326 tid 100176 td 0xffffff00177e7720
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
kern_wait() at kern_wait+0x3f7
wait4() at wait4+0x35
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (7, FreeBSD ELF64, wait4), rip = 0x800935edc, rsp =  
0x7fffffffe618, rbp = 0x52e ---

Tracing command cron pid 1324 tid 100115 td 0xffffff0002915000
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
pipe_read() at pipe_read+0x4a3
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x80095a7ac, rsp =  
0x7fffffffe4b8, rbp = 0x800a9e700 ---

Tracing command unlink pid 1319 tid 100059 td 0xffffff000256a720
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_wait() at sleepq_wait+0x4d
__lockmgr_args() at __lockmgr_args+0x79b
ffs_lock() at ffs_lock+0x9c
VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b
_vn_lock() at _vn_lock+0x57
lookup() at lookup+0x5f0
namei() at namei+0x4a9
kern_unlinkat() at kern_unlinkat+0xab
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (10, FreeBSD ELF64, unlink), rip = 0x80072be5c, rsp =  
0x7fffffffec68, rbp = 0 ---

Tracing command sh pid 1317 tid 100058 td 0xffffff0002650000
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
kern_wait() at kern_wait+0x3f7
wait4() at wait4+0x35
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (7, FreeBSD ELF64, wait4), rip = 0x800935edc, rsp =  
0x7fffffffe618, rbp = 0x525 ---

Tracing command cron pid 1316 tid 100055 td 0xffffff0002650ab0
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
pipe_read() at pipe_read+0x4a3
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x80095a7ac, rsp =  
0x7fffffffe4b8, rbp = 0x800a9e700 ---

Tracing command unlink pid 1309 tid 100112 td 0xffffff0002915ab0
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_wait() at sleepq_wait+0x4d
__lockmgr_args() at __lockmgr_args+0x79b
ffs_lock() at ffs_lock+0x9c
VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b
_vn_lock() at _vn_lock+0x57
lookup() at lookup+0x5f0
namei() at namei+0x4a9
kern_unlinkat() at kern_unlinkat+0xab
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (10, FreeBSD ELF64, unlink), rip = 0x80072be5c, rsp =  
0x7fffffffec68, rbp = 0 ---

Tracing command sh pid 1307 tid 100167 td 0xffffff0002cb8000
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
kern_wait() at kern_wait+0x3f7
wait4() at wait4+0x35
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (7, FreeBSD ELF64, wait4), rip = 0x800935edc, rsp =  
0x7fffffffe618, rbp = 0x51b ---

Tracing command cron pid 1306 tid 100166 td 0xffffff0002af0720
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
pipe_read() at pipe_read+0x4a3
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x80095a7ac, rsp =  
0x7fffffffe4b8, rbp = 0x800a9e700 ---

Tracing command unlink pid 1301 tid 100164 td 0xffffff00026e7ab0
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_wait() at sleepq_wait+0x4d
__lockmgr_args() at __lockmgr_args+0x79b
ffs_lock() at ffs_lock+0x9c
VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b
_vn_lock() at _vn_lock+0x57
lookup() at lookup+0x5f0
namei() at namei+0x4a9
kern_unlinkat() at kern_unlinkat+0xab
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (10, FreeBSD ELF64, unlink), rip = 0x80072be5c, rsp =  
0x7fffffffec68, rbp = 0 ---

Tracing command sh pid 1299 tid 100103 td 0xffffff000270a000
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
kern_wait() at kern_wait+0x3f7
wait4() at wait4+0x35
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (7, FreeBSD ELF64, wait4), rip = 0x800935edc, rsp =  
0x7fffffffe618, rbp = 0x513 ---

Tracing command cron pid 1298 tid 100160 td 0xffffff001284a720
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
pipe_read() at pipe_read+0x4a3
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x80095a7ac, rsp =  
0x7fffffffe4b8, rbp = 0x800a9e700 ---

Tracing command sh pid 1289 tid 100161 td 0xffffff001284a390
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_wait() at sleepq_wait+0x4d
_sx_slock_hard() at _sx_slock_hard+0x1b7
_sx_slock() at _sx_slock+0xc1
zfs_freebsd_reclaim() at zfs_freebsd_reclaim+0x4a
VOP_RECLAIM_APV() at VOP_RECLAIM_APV+0xb5
vgonel() at vgonel+0x119
vnlru_free() at vnlru_free+0x345
getnewvnode() at getnewvnode+0x24f
ffs_vgetf() at ffs_vgetf+0xdf
ufs_lookup_() at ufs_lookup_+0xb46
VOP_CACHEDLOOKUP_APV() at VOP_CACHEDLOOKUP_APV+0xaf
vfs_cache_lookup() at vfs_cache_lookup+0xf0
VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xb7
lookup() at lookup+0x2eb
namei() at namei+0x4a9
kern_statat_vnhook() at kern_statat_vnhook+0x8f
kern_statat() at kern_statat+0x15
stat() at stat+0x2a
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (188, FreeBSD ELF64, stat), rip = 0x8009ae51c, rsp =  
0x7fffffffe798, rbp = 0x800c06b80 ---

Tracing command cron pid 1288 tid 100063 td 0xffffff000264f720
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
pipe_read() at pipe_read+0x4a3
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x80095a7ac, rsp =  
0x7fffffffe4b8, rbp = 0x800a9e700 ---

Tracing command zfs pid 1285 tid 100143 td 0xffffff0002cbb000
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_wait() at sleepq_wait+0x4d
_cv_wait() at _cv_wait+0x17a
txg_wait_synced() at txg_wait_synced+0x7c
zfsvfs_teardown() at zfsvfs_teardown+0x1b2
zfs_suspend_fs() at zfs_suspend_fs+0x2b
zfs_ioc_recv() at zfs_ioc_recv+0x28b
zfsdev_ioctl() at zfsdev_ioctl+0x8d
devfs_ioctl_f() at devfs_ioctl_f+0x76
kern_ioctl() at kern_ioctl+0xc5
ioctl() at ioctl+0xfd
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (54, FreeBSD ELF64, ioctl), rip = 0x800fe874c, rsp =  
0x7fffffffbbf8, rbp = 0x7fffffffc930 ---

Tracing command csh pid 1283 tid 100163 td 0xffffff0009979ab0
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
kern_sigsuspend() at kern_sigsuspend+0xca
sigsuspend() at sigsuspend+0x34
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (341, FreeBSD ELF64, sigsuspend), rip = 0x80094e75c, rsp =  
0x7fffffffe598, rbp = 0x800c1d300 ---

Tracing command sshd pid 1281 tid 100124 td 0xffffff0002aeaab0
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_cv_wait_sig() at _cv_wait_sig+0x17e
seltdwait() at seltdwait+0xac
kern_select() at kern_select+0x620
select() at select+0x5d
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (93, FreeBSD ELF64, select), rip = 0x8013d772c, rsp =  
0x7fffffffdcb8, rbp = 0x7fffffffdd40 ---

Tracing command zpool pid 1256 tid 100109 td 0xffffff0002916720
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19
_sleep() at _sleep+0x23f
kern_nanosleep() at kern_nanosleep+0x118
nanosleep() at nanosleep+0x6e
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (240, FreeBSD ELF64, nanosleep), rip = 0x8010ce92c, rsp =  
0x7fffffffaac8, rbp = 0 ---

Tracing command csh pid 1252 tid 100158 td 0xffffff001284c000
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
kern_sigsuspend() at kern_sigsuspend+0xca
sigsuspend() at sigsuspend+0x34
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (341, FreeBSD ELF64, sigsuspend), rip = 0x80094e75c, rsp =  
0x7fffffffe638, rbp = 0x800c59400 ---

Tracing command su pid 1248 tid 100156 td 0xffffff001284cab0
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
kern_wait() at kern_wait+0x3f7
wait4() at wait4+0x35
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (7, FreeBSD ELF64, wait4), rip = 0x8009e8edc, rsp =  
0x7fffffffe598, rbp = 0x4e4 ---

Tracing command csh pid 1243 tid 100137 td 0xffffff0002917720
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
kern_sigsuspend() at kern_sigsuspend+0xca
sigsuspend() at sigsuspend+0x34
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (341, FreeBSD ELF64, sigsuspend), rip = 0x80094e75c, rsp =  
0x7fffffffe558, rbp = 0x800c1f700 ---

Tracing command sshd pid 1242 tid 100154 td 0xffffff001284d390
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_cv_wait_sig() at _cv_wait_sig+0x17e
seltdwait() at seltdwait+0xac
kern_select() at kern_select+0x620
select() at select+0x5d
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (93, FreeBSD ELF64, select), rip = 0x8013d772c, rsp =  
0x7fffffffdcb8, rbp = 0x7fffffffdd40 ---

Tracing command sshd pid 1234 tid 100060 td 0xffffff000264fab0
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
soreceive_generic() at soreceive_generic+0xf99
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x8013d77ac, rsp =  
0x7fffffffdcd8, rbp = 0 ---

Tracing command bsdtar pid 1203 tid 100153 td 0xffffff000270aab0
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
pipe_read() at pipe_read+0x4a3
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x800e447ac, rsp =  
0x7fffffffe7a8, rbp = 0x80101a240 ---

Tracing command bsdtar pid 1202 tid 100057 td 0xffffff0002650390
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_wait() at sleepq_wait+0x4d
__lockmgr_args() at __lockmgr_args+0x79b
vop_stdlock() at vop_stdlock+0x39
VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b
_vn_lock() at _vn_lock+0x57
vacl_get_acl() at vacl_get_acl+0x55
__acl_get_file() at __acl_get_file+0xcf
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (347, FreeBSD ELF64, __acl_get_file), rip = 0x800dbde0c,  
rsp = 0x7fffffffe1d8, rbp = 0x2 ---

Tracing command csh pid 1200 tid 100152 td 0xffffff000270a720
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
kern_sigsuspend() at kern_sigsuspend+0xca
sigsuspend() at sigsuspend+0x34
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (341, FreeBSD ELF64, sigsuspend), rip = 0x80094e75c, rsp =  
0x7fffffffe3a8, rbp = 0x800c43200 ---

Tracing command bsdtar pid 1199 tid 100151 td 0xffffff0002af0ab0
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
pipe_read() at pipe_read+0x4a3
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x800e447ac, rsp =  
0x7fffffffe7a8, rbp = 0x80101a240 ---

Tracing command bsdtar pid 1198 tid 100105 td 0xffffff000264c000
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_wait() at sleepq_wait+0x4d
_cv_wait() at _cv_wait+0x17a
rrw_enter() at rrw_enter+0xb1
zfs_freebsd_read() at zfs_freebsd_read+0x55
VOP_READ_APV() at VOP_READ_APV+0xaf
vn_read() at vn_read+0x254
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x800e447ac, rsp =  
0x7fffffffe898, rbp = 0x1a87 ---

Tracing command csh pid 1196 tid 100056 td 0xffffff0002650720
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
kern_sigsuspend() at kern_sigsuspend+0xca
sigsuspend() at sigsuspend+0x34
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (341, FreeBSD ELF64, sigsuspend), rip = 0x80094e75c, rsp =  
0x7fffffffe3a8, rbp = 0x800c43200 ---

Tracing command bsdtar pid 1195 tid 100067 td 0xffffff00026d6720
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_wait() at sleepq_wait+0x4d
_sx_slock_hard() at _sx_slock_hard+0x1b7
_sx_slock() at _sx_slock+0xc1
zfs_freebsd_reclaim() at zfs_freebsd_reclaim+0x4a
VOP_RECLAIM_APV() at VOP_RECLAIM_APV+0xb5
vgonel() at vgonel+0x119
vnlru_free() at vnlru_free+0x345
getnewvnode() at getnewvnode+0x24f
zfs_znode_cache_constructor() at zfs_znode_cache_constructor+0x2c
zfs_znode_alloc() at zfs_znode_alloc+0x39
zfs_mknode() at zfs_mknode+0x205
zfs_freebsd_create() at zfs_freebsd_create+0x583
VOP_CREATE_APV() at VOP_CREATE_APV+0xb3
vn_open_cred() at vn_open_cred+0x473
kern_openat() at kern_openat+0x179
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (5, FreeBSD ELF64, open), rip = 0x800e3557c, rsp =  
0x7fffffffe758, rbp = 0x1a4 ---

Tracing command bsdtar pid 1194 tid 100150 td 0xffffff0002cb8390
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_wait() at sleepq_wait+0x4d
__lockmgr_args() at __lockmgr_args+0x79b
vop_stdlock() at vop_stdlock+0x39
VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b
_vn_lock() at _vn_lock+0x57
vacl_get_acl() at vacl_get_acl+0x55
__acl_get_file() at __acl_get_file+0xcf
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (347, FreeBSD ELF64, __acl_get_file), rip = 0x800dbde0c,  
rsp = 0x7fffffffe1d8, rbp = 0x2 ---

Tracing command csh pid 1192 tid 100138 td 0xffffff0002cbc390
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
kern_sigsuspend() at kern_sigsuspend+0xca
sigsuspend() at sigsuspend+0x34
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (341, FreeBSD ELF64, sigsuspend), rip = 0x80094e75c, rsp =  
0x7fffffffe3a8, rbp = 0x800c43200 ---

Tracing command bsdtar pid 1191 tid 100113 td 0xffffff0002915720
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
pipe_read() at pipe_read+0x4a3
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x800e447ac, rsp =  
0x7fffffffe7a8, rbp = 0x80101a240 ---

Tracing command bsdtar pid 1190 tid 100118 td 0xffffff0002914390
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_wait() at sleepq_wait+0x4d
__lockmgr_args() at __lockmgr_args+0x79b
vop_stdlock() at vop_stdlock+0x39
VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b
_vn_lock() at _vn_lock+0x57
vacl_get_acl() at vacl_get_acl+0x55
__acl_get_file() at __acl_get_file+0xcf
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (347, FreeBSD ELF64, __acl_get_file), rip = 0x800dbde0c,  
rsp = 0x7fffffffe1d8, rbp = 0x2 ---

Tracing command csh pid 1188 tid 100144 td 0xffffff0002cb9ab0
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
kern_sigsuspend() at kern_sigsuspend+0xca
sigsuspend() at sigsuspend+0x34
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (341, FreeBSD ELF64, sigsuspend), rip = 0x80094e75c, rsp =  
0x7fffffffe3a8, rbp = 0x800c43200 ---

Tracing command bsdtar pid 1187 tid 100139 td 0xffffff0002cbc000
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
pipe_read() at pipe_read+0x4a3
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x800e447ac, rsp =  
0x7fffffffe7a8, rbp = 0x80101a240 ---

Tracing command bsdtar pid 1186 tid 100123 td 0xffffff0002af0000
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_wait() at sleepq_wait+0x4d
__lockmgr_args() at __lockmgr_args+0x79b
vop_stdlock() at vop_stdlock+0x39
VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b
_vn_lock() at _vn_lock+0x57
vacl_get_acl() at vacl_get_acl+0x55
__acl_get_file() at __acl_get_file+0xcf
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (347, FreeBSD ELF64, __acl_get_file), rip = 0x800dbde0c,  
rsp = 0x7fffffffe1d8, rbp = 0x2 ---

Tracing command csh pid 1184 tid 100146 td 0xffffff0002cb9390
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
kern_sigsuspend() at kern_sigsuspend+0xca
sigsuspend() at sigsuspend+0x34
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (341, FreeBSD ELF64, sigsuspend), rip = 0x80094e75c, rsp =  
0x7fffffffe3a8, rbp = 0x800c43200 ---

Tracing command bsdtar pid 1183 tid 100126 td 0xffffff0002aea390
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_wait() at sleepq_wait+0x4d
__lockmgr_args() at __lockmgr_args+0x7b2
vop_stdlock() at vop_stdlock+0x39
VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b
_vn_lock() at _vn_lock+0x57
vget() at vget+0x7b
cache_lookup() at cache_lookup+0x4e0
vfs_cache_lookup() at vfs_cache_lookup+0xc0
VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xb7
lookup() at lookup+0x2eb
namei() at namei+0x4a9
kern_statat_vnhook() at kern_statat_vnhook+0x8f
kern_statat() at kern_statat+0x15
lstat() at lstat+0x2a
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (190, FreeBSD ELF64, lstat), rip = 0x800e354fc, rsp =  
0x7fffffffe778, rbp = 0x801023380 ---

Tracing command bsdtar pid 1182 tid 100066 td 0xffffff000264f000
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_wait() at sleepq_wait+0x4d
__lockmgr_args() at __lockmgr_args+0x79b
vop_stdlock() at vop_stdlock+0x39
VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b
_vn_lock() at _vn_lock+0x57
vacl_get_acl() at vacl_get_acl+0x55
__acl_get_file() at __acl_get_file+0xcf
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (347, FreeBSD ELF64, __acl_get_file), rip = 0x800dbde0c,  
rsp = 0x7fffffffe1d8, rbp = 0x2 ---

Tracing command csh pid 1180 tid 100120 td 0xffffff0002714ab0
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
kern_sigsuspend() at kern_sigsuspend+0xca
sigsuspend() at sigsuspend+0x34
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (341, FreeBSD ELF64, sigsuspend), rip = 0x80094e75c, rsp =  
0x7fffffffe3a8, rbp = 0x800c43200 ---

Tracing command csh pid 1147 tid 100142 td 0xffffff0002cbb390
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_cv_wait_sig() at _cv_wait_sig+0x17e
tty_wait() at tty_wait+0x48
ttydisc_read() at ttydisc_read+0x2f1
ttydev_read() at ttydev_read+0xab
devfs_read_f() at devfs_read_f+0x86
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x8009ec7ac, rsp =  
0x7fffffffe658, rbp = 0x1 ---

Tracing command sshd pid 1146 tid 100111 td 0xffffff0002916000
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_cv_wait_sig() at _cv_wait_sig+0x17e
seltdwait() at seltdwait+0xac
kern_select() at kern_select+0x620
select() at select+0x5d
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (93, FreeBSD ELF64, select), rip = 0x8013d772c, rsp =  
0x7fffffffdcb8, rbp = 0x7fffffffdd40 ---

Tracing command sshd pid 1133 tid 100116 td 0xffffff0002914ab0
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
soreceive_generic() at soreceive_generic+0xf99
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x8013d77ac, rsp =  
0x7fffffffdcd8, rbp = 0 ---

Tracing command csh pid 959 tid 100149 td 0xffffff0002cb8720
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_cv_wait_sig() at _cv_wait_sig+0x17e
tty_wait() at tty_wait+0x48
ttydisc_read() at ttydisc_read+0x2f1
ttydev_read() at ttydev_read+0xab
devfs_read_f() at devfs_read_f+0x86
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x8009ec7ac, rsp =  
0x7fffffffe738, rbp = 0x1 ---

Tracing command su pid 958 tid 100148 td 0xffffff0002cb8ab0
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
kern_wait() at kern_wait+0x3f7
wait4() at wait4+0x35
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (7, FreeBSD ELF64, wait4), rip = 0x8009e8edc, rsp =  
0x7fffffffe598, rbp = 0x3bf ---

Tracing command csh pid 952 tid 100125 td 0xffffff0002aea720
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
kern_sigsuspend() at kern_sigsuspend+0xca
sigsuspend() at sigsuspend+0x34
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (341, FreeBSD ELF64, sigsuspend), rip = 0x80094e75c, rsp =  
0x7fffffffe558, rbp = 0x800c1f500 ---

Tracing command sshd pid 951 tid 100130 td 0xffffff0002ae8390
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_cv_wait_sig() at _cv_wait_sig+0x17e
seltdwait() at seltdwait+0xac
kern_select() at kern_select+0x620
select() at select+0x5d
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (93, FreeBSD ELF64, select), rip = 0x8013d772c, rsp =  
0x7fffffffdcb8, rbp = 0x7fffffffdd40 ---

Tracing command csh pid 946 tid 100141 td 0xffffff0002cbb720
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
kern_sigsuspend() at kern_sigsuspend+0xca
sigsuspend() at sigsuspend+0x34
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (341, FreeBSD ELF64, sigsuspend), rip = 0x80094e75c, rsp =  
0x7fffffffe558, rbp = 0x800c1f500 ---

Tracing command sshd pid 945 tid 100140 td 0xffffff0002cbbab0
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_cv_wait_sig() at _cv_wait_sig+0x17e
seltdwait() at seltdwait+0xac
kern_select() at kern_select+0x620
select() at select+0x5d
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (93, FreeBSD ELF64, select), rip = 0x8013d772c, rsp =  
0x7fffffffdcb8, rbp = 0x7fffffffdd40 ---

Tracing command sshd pid 942 tid 100128 td 0xffffff0002ae8ab0
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
soreceive_generic() at soreceive_generic+0xf99
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x8013d77ac, rsp =  
0x7fffffffdcd8, rbp = 0 ---

Tracing command sshd pid 939 tid 100132 td 0xffffff0002ae7ab0
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
soreceive_generic() at soreceive_generic+0xf99
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x8013d77ac, rsp =  
0x7fffffffdcd8, rbp = 0 ---

Tracing command getty pid 908 tid 100117 td 0xffffff0002914720
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_cv_wait_sig() at _cv_wait_sig+0x17e
tty_wait() at tty_wait+0x48
ttydisc_read() at ttydisc_read+0x2f1
ttydev_read() at ttydev_read+0xab
devfs_read_f() at devfs_read_f+0x86
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x80084f7ac, rsp =  
0x7fffffffecc8, rbp = 0 ---

Tracing command getty pid 907 tid 100129 td 0xffffff0002ae8720
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_cv_wait_sig() at _cv_wait_sig+0x17e
tty_wait() at tty_wait+0x48
ttydisc_read() at ttydisc_read+0x2f1
ttydev_read() at ttydev_read+0xab
devfs_read_f() at devfs_read_f+0x86
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x80084f7ac, rsp =  
0x7fffffffecc8, rbp = 0 ---

Tracing command getty pid 906 tid 100106 td 0xffffff0002917390
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_cv_wait_sig() at _cv_wait_sig+0x17e
tty_wait() at tty_wait+0x48
ttydisc_read() at ttydisc_read+0x2f1
ttydev_read() at ttydev_read+0xab
devfs_read_f() at devfs_read_f+0x86
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x80084f7ac, rsp =  
0x7fffffffecc8, rbp = 0 ---

Tracing command getty pid 905 tid 100121 td 0xffffff0002714720
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_cv_wait_sig() at _cv_wait_sig+0x17e
tty_wait() at tty_wait+0x48
ttydisc_read() at ttydisc_read+0x2f1
ttydev_read() at ttydev_read+0xab
devfs_read_f() at devfs_read_f+0x86
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x80084f7ac, rsp =  
0x7fffffffecc8, rbp = 0 ---

Tracing command getty pid 904 tid 100136 td 0xffffff0002917ab0
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_cv_wait_sig() at _cv_wait_sig+0x17e
tty_wait() at tty_wait+0x48
ttydisc_read() at ttydisc_read+0x2f1
ttydev_read() at ttydev_read+0xab
devfs_read_f() at devfs_read_f+0x86
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x80084f7ac, rsp =  
0x7fffffffecc8, rbp = 0 ---

Tracing command getty pid 903 tid 100135 td 0xffffff0002ae7000
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_cv_wait_sig() at _cv_wait_sig+0x17e
tty_wait() at tty_wait+0x48
ttydisc_read() at ttydisc_read+0x2f1
ttydev_read() at ttydev_read+0xab
devfs_read_f() at devfs_read_f+0x86
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x80084f7ac, rsp =  
0x7fffffffecc8, rbp = 0 ---

Tracing command getty pid 902 tid 100134 td 0xffffff0002ae7390
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_cv_wait_sig() at _cv_wait_sig+0x17e
tty_wait() at tty_wait+0x48
ttydisc_read() at ttydisc_read+0x2f1
ttydev_read() at ttydev_read+0xab
devfs_read_f() at devfs_read_f+0x86
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x80084f7ac, rsp =  
0x7fffffffecc8, rbp = 0 ---

Tracing command getty pid 901 tid 100133 td 0xffffff0002ae7720
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_cv_wait_sig() at _cv_wait_sig+0x17e
tty_wait() at tty_wait+0x48
ttydisc_read() at ttydisc_read+0x2f1
ttydev_read() at ttydev_read+0xab
devfs_read_f() at devfs_read_f+0x86
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x80084f7ac, rsp =  
0x7fffffffecc8, rbp = 0 ---

Tracing command cron pid 853 tid 100119 td 0xffffff0002914000
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19
_sleep() at _sleep+0x23f
kern_nanosleep() at kern_nanosleep+0x118
nanosleep() at nanosleep+0x6e
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (240, FreeBSD ELF64, nanosleep), rip = 0x80093d92c, rsp =  
0x7fffffffeb28, rbp = 0x3c ---

Tracing command sendmail pid 847 tid 100061 td 0xffffff000256a390
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
kern_sigsuspend() at kern_sigsuspend+0xca
sigsuspend() at sigsuspend+0x34
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (341, FreeBSD ELF64, sigsuspend), rip = 0x800d3975c, rsp =  
0x7fffffffcd68, rbp = 0x1 ---

Tracing command sendmail pid 841 tid 100107 td 0xffffff0002917000
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19
_cv_timedwait_sig() at _cv_timedwait_sig+0x18c
seltdwait() at seltdwait+0x56
kern_select() at kern_select+0x620
select() at select+0x5d
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (93, FreeBSD ELF64, select), rip = 0x800dd772c, rsp =  
0x7fffffffc1e8, rbp = 0x7fffffffc280 ---

Tracing command sshd pid 834 tid 100110 td 0xffffff0002916390
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_cv_wait_sig() at _cv_wait_sig+0x17e
seltdwait() at seltdwait+0xac
kern_select() at kern_select+0x620
select() at select+0x5d
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (93, FreeBSD ELF64, select), rip = 0x8013d772c, rsp =  
0x7fffffffddd8, rbp = 0x2 ---

Tracing command syslogd pid 610 tid 100064 td 0xffffff000264f390
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_cv_wait_sig() at _cv_wait_sig+0x17e
seltdwait() at seltdwait+0xac
kern_select() at kern_select+0x620
select() at select+0x5d
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (93, FreeBSD ELF64, select), rip = 0x80085272c, rsp =  
0x7fffffffdcb8, rbp = 0x1 ---

Tracing command devd pid 503 tid 100114 td 0xffffff0002915390
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_cv_wait_sig() at _cv_wait_sig+0x17e
seltdwait() at seltdwait+0xac
kern_select() at kern_select+0x620
select() at select+0x5d
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (93, FreeBSD ELF64, select), rip = 0x440cfc, rsp =  
0x7fffffffe898, rbp = 0x7fffffffe8b0 ---

Tracing command flowcleaner pid 22 tid 100054 td 0xffffff000256aab0
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_timedwait() at sleepq_timedwait+0x4d
_cv_timedwait() at _cv_timedwait+0x18c
flowtable_
debug.ddb.capture.bufsize: 49152
debug.ddb.capture.inprogress: 0
debug.ddb.capture.maxbufsize: 5242880
debug.ddb.capture.bufoff: 49152
debug.ddb.scripting.unscript:
debug.ddb.scripting.scripts:
debug.ddb.textdump.do_version: 1
debug.ddb.textdump.do_panic: 1
debug.ddb.textdump.do_msgbuf: 1
debug.ddb.textdump.do_ddb: 1
debug.ddb.textdump.pending: 0