Re: NFSv4 client hung
- Go to: [ bottom of page ] [ top of archives ] [ this month ]
Date: Tue, 02 Sep 2025 01:25:52 UTC
It has happened again. Please let me know if it's the wrong list to handle this kind of content. What's different now is that the hang occurred in the client mounting nfs with v3. Here is the information that I managed to collect: The machine has plenty of resources available: last pid: 58749; load averages: 4.78, 4.85, 4.87 up 7+14:01:09 00:32:57 77 processes: 1 running, 65 sleeping, 10 stopped, 1 zombie CPU: 1.0% user, 0.0% nice, 15.2% system, 0.0% interrupt, 83.7% idle Mem: 4141M Active, 18G Inact, 4365M Wired, 981M Buf, 36G Free ARC: 1305M Total, 904M MFU, 272M MRU, 20M Header, 106M Other 1083M Compressed, 4084M Uncompressed, 3.77:1 Ratio Swap: 16G Total, 16G Free Checking network related memory allocations looks fine: root@bhyve01:~ # netstat -m 94864/30371/125235 mbufs in use (current/cache/total) 28841/22665/51506/4070747 mbuf clusters in use (current/cache/total/max) 94/9050 mbuf+clusters out of packet secondary zone in use (current/cache) 57341/12763/70104/2035373 4k (page size) jumbo clusters in use (current/cache/total/max) 0/0/0/603073 9k jumbo clusters in use (current/cache/total/max) 0/0/0/339228 16k jumbo clusters in use (current/cache/total/max) 310762K/103974K/414736K bytes allocated to network (current/cache/total) 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/0/0 requests for mbufs delayed (mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters delayed (4k/9k/16k) 0/0/0 requests for jumbo clusters denied (4k/9k/16k) 0 sendfile syscalls 0 sendfile syscalls completed without I/O request 0 requests for I/O initiated by sendfile 0 pages read by sendfile as part of a request 0 pages were valid at time of a sendfile request 0 pages were valid and substituted to bogus page 0 pages were requested for read ahead by applications 0 pages were read ahead by sendfile 0 times sendfile encountered an already busy page 0 requests for sfbufs denied 0 requests for sfbufs delayed As before, the machine is responsive, I can ssh into it and do everything without hang except operations on mounted nfs volumes, there are some processes blocked on newbuf: root@bhyve01:~ # ps -axHl|grep newbuf you have mail 0 14734 13899 0 20 0 3208884 2875980 newbuf DC 1 1:06.89 bhyve: fw (bhyve) 0 25671 14483 4 20 0 4272676 4035220 newbuf DC 1 5:23.05 bhyve: haos (bhyve) 0 25671 14483 6 20 0 4272676 4035220 newbuf DC 1 5:25.20 bhyve: haos (bhyve) 0 52892 52785 13 20 0 14012 2560 newbuf TC 6 0:00.00 head -n1 /vms/mediaDeduper-worker2/run.lock 0 53375 52734 17 20 0 13796 2428 newbuf T+ 6 0:00.00 find . 0 58317 56402 23 20 0 13836 2460 piperd S+ 8 0:00.00 grep newbuf Here's the result of procstat -kk of those pids: root@bhyve01:~ # procstat -kk 53375 PID TID COMM TDNAME KSTACK 53375 102264 find - mi_switch+0xbd _sleep+0x1f3 bufspace_wait+0x19f getnewbuf+0x2b0 getblkx+0x4da getblk+0x22 nfs_getcacheblk+0x40 ncl_bioread+0x6ef nfs_readdir+0x217 vop_sigdefer+0x2a VOP_READDIR_APV+0x20 kern_getdirentries+0x229 sys_getdirentries+0x29 amd64_syscall+0x117 fast_syscall_common+0xf8 root@bhyve01:~ # procstat -kk 52892 PID TID COMM TDNAME KSTACK 52892 101046 head - mi_switch+0xbd _sleep+0x1f3 bufspace_wait+0x19f getnewbuf+0x2b0 getblkx+0x4da getblk+0x22 nfs_getcacheblk+0x40 ncl_bioread+0xd41 vop_sigdefer+0x2a VOP_READ_APV+0x20 vn_read+0x1f2 vn_io_fault_doio+0x45 vn_io_fault1+0x167 vn_io_fault+0x151 dofileread+0x80 sys_read+0xb3 amd64_syscall+0x117 fast_syscall_common+0xf8 root@bhyve01:~ # procstat -kk 14734 PID TID COMM TDNAME KSTACK 14734 100826 bhyve mevent mi_switch+0xbd sleepq_catch_signals+0x2a2 sleepq_wait_sig+0x9 _sleep+0x1e6 kqueue_scan+0xa11 kqueue_kevent+0x13b kern_kevent_fp+0x4b kern_kevent_generic+0xdf sys_kevent+0x61 amd64_syscall+0x117 fast_syscall_common+0xf8 14734 101935 bhyve blk-4:0-0 mi_switch+0xbd sleeplk+0x10a lockmgr_xlock_hard+0x398 nfs_lock+0x29 vop_sigdefer+0x2a vn_write+0x180 vn_io_fault_doio+0x45 vn_io_fault1+0x167 vn_io_fault+0x151 dofilewrite+0x81 kern_pwritev+0x61 sys_pwritev+0x38 amd64_syscall+0x117 fast_syscall_common+0xf8 14734 101936 bhyve blk-4:0-1 mi_switch+0xbd _sleep+0x1f3 bufspace_wait+0x19f getnewbuf+0x2b0 getblkx+0x4da getblk+0x22 nfs_getcacheblk+0x40 ncl_write+0x929 vop_sigdefer+0x2a VOP_WRITE_APV+0x94 vn_write+0x287 vn_io_fault_doio+0x45 vn_io_fault1+0x167 vn_io_fault+0x151 dofilewrite+0x81 kern_pwritev+0x61 sys_pwritev+0x38 amd64_syscall+0x117 14734 101937 bhyve blk-4:0-2 mi_switch+0xbd sleepq_catch_signals+0x2a2 sleepq_wait_sig+0x9 _sleep+0x1e6 umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x117 fast_syscall_common+0xf8 14734 101938 bhyve blk-4:0-3 mi_switch+0xbd sleepq_catch_signals+0x2a2 sleepq_wait_sig+0x9 _sleep+0x1e6 umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x117 fast_syscall_common+0xf8 14734 101939 bhyve blk-4:0-4 mi_switch+0xbd sleepq_catch_signals+0x2a2 sleepq_wait_sig+0x9 _sleep+0x1e6 umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x117 fast_syscall_common+0xf8 14734 101940 bhyve blk-4:0-5 mi_switch+0xbd sleeplk+0x10a lockmgr_xlock_hard+0x398 nfs_lock+0x29 vop_sigdefer+0x2a vn_write+0x180 vn_io_fault_doio+0x45 vn_io_fault1+0x167 vn_io_fault+0x151 dofilewrite+0x81 kern_pwritev+0x61 sys_pwritev+0x38 amd64_syscall+0x117 fast_syscall_common+0xf8 14734 101941 bhyve blk-4:0-6 mi_switch+0xbd sleepq_catch_signals+0x2a2 sleepq_wait_sig+0x9 _sleep+0x1e6 umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x117 fast_syscall_common+0xf8 14734 101942 bhyve blk-4:0-7 mi_switch+0xbd sleepq_catch_signals+0x2a2 sleepq_wait_sig+0x9 _sleep+0x1e6 umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x117 fast_syscall_common+0xf8 14734 101943 bhyve nvme-aen-4:0 mi_switch+0xbd sleepq_catch_signals+0x2a2 sleepq_wait_sig+0x9 _sleep+0x1e6 umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x117 fast_syscall_common+0xf8 14734 101944 bhyve vtnet-5:0 tx mi_switch+0xbd sleepq_catch_signals+0x2a2 sleepq_wait_sig+0x9 _sleep+0x1e6 umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x117 fast_syscall_common+0xf8 14734 101945 bhyve vtnet-5:1 tx mi_switch+0xbd sleepq_catch_signals+0x2a2 sleepq_wait_sig+0x9 _sleep+0x1e6 umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x117 fast_syscall_common+0xf8 14734 101946 bhyve vtnet-5:2 tx mi_switch+0xbd sleepq_catch_signals+0x2a2 sleepq_wait_sig+0x9 _sleep+0x1e6 umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x117 fast_syscall_common+0xf8 14734 101947 bhyve vtnet-5:3 tx mi_switch+0xbd sleepq_catch_signals+0x2a2 sleepq_wait_sig+0x9 _sleep+0x1e6 umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x117 fast_syscall_common+0xf8 14734 101948 bhyve vtnet-5:4 tx mi_switch+0xbd sleepq_catch_signals+0x2a2 sleepq_wait_sig+0x9 _sleep+0x1e6 umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x117 fast_syscall_common+0xf8 14734 101949 bhyve vtnet-5:5 tx mi_switch+0xbd sleepq_catch_signals+0x2a2 sleepq_wait_sig+0x9 _sleep+0x1e6 umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x117 fast_syscall_common+0xf8 14734 101950 bhyve vtnet-5:6 tx mi_switch+0xbd sleepq_catch_signals+0x2a2 sleepq_wait_sig+0x9 _sleep+0x1e6 umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x117 fast_syscall_common+0xf8 14734 101951 bhyve vtnet-5:7 tx mi_switch+0xbd sleepq_catch_signals+0x2a2 sleepq_wait_sig+0x9 _sleep+0x1e6 umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x117 fast_syscall_common+0xf8 14734 101952 bhyve vtnet-6:0 tx mi_switch+0xbd sleepq_catch_signals+0x2a2 sleepq_wait_sig+0x9 _sleep+0x1e6 umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x117 fast_syscall_common+0xf8 14734 101953 bhyve vtnet-6:1 tx mi_switch+0xbd sleepq_catch_signals+0x2a2 sleepq_wait_sig+0x9 _sleep+0x1e6 umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x117 fast_syscall_common+0xf8 14734 101954 bhyve vcpu 0 vmx_run+0x24ab vm_run+0x172 vmmdev_ioctl+0xd59 devfs_ioctl+0xcb vn_ioctl+0xc8 devfs_ioctl_f+0x1e kern_ioctl+0x255 sys_ioctl+0x101 amd64_syscall+0x117 fast_syscall_common+0xf8 14734 101955 bhyve vcpu 1 sys_ioctl+0xed amd64_syscall+0x117 fast_syscall_common+0xf8 14734 101956 bhyve vcpu 2 vmx_run+0x24ab vm_run+0x172 vmmdev_ioctl+0xd59 devfs_ioctl+0xcb vn_ioctl+0xc8 devfs_ioctl_f+0x1e kern_ioctl+0x255 sys_ioctl+0x101 amd64_syscall+0x117 fast_syscall_common+0xf8 14734 101957 bhyve vcpu 3 mi_switch+0xbd sleepq_catch_signals+0x2a2 sleepq_wait_sig+0x9 _sleep+0x1e6 umtxq_sleep+0x2cd __umtx_op_sem2_wait+0x470 sys__umtx_op+0x7e amd64_syscall+0x117 fast_syscall_common+0xf8 root@bhyve01:~ # procstat -kk 25671 PID TID COMM TDNAME KSTACK 25671 101830 bhyve mevent mi_switch+0xbd sleepq_catch_signals+0x2a2 sleepq_wait_sig+0x9 _sleep+0x1e6 kqueue_scan+0xa11 kqueue_kevent+0x13b kern_kevent_fp+0x4b kern_kevent_generic+0xdf sys_kevent+0x61 amd64_syscall+0x117 fast_syscall_common+0xf8 25671 104064 bhyve blk-4:0-0 mi_switch+0xbd sleeplk+0x10a lockmgr_xlock_hard+0x398 nfs_lock+0x29 vop_sigdefer+0x2a vn_write+0x180 vn_io_fault_doio+0x45 vn_io_fault1+0x167 vn_io_fault+0x151 dofilewrite+0x81 kern_pwritev+0x61 sys_pwritev+0x38 amd64_syscall+0x117 fast_syscall_common+0xf8 25671 104065 bhyve blk-4:0-1 mi_switch+0xbd sleeplk+0x10a lockmgr_xlock_hard+0x398 nfs_lock+0x29 vop_sigdefer+0x2a vn_write+0x180 vn_io_fault_doio+0x45 vn_io_fault1+0x167 vn_io_fault+0x151 dofilewrite+0x81 kern_pwritev+0x61 sys_pwritev+0x38 amd64_syscall+0x117 fast_syscall_common+0xf8 25671 104066 bhyve blk-4:0-2 mi_switch+0xbd _sleep+0x1f3 bufspace_wait+0x19f getnewbuf+0x2b0 getblkx+0x4da getblk+0x22 nfs_getcacheblk+0x40 ncl_bioread+0xd41 vop_sigdefer+0x2a VOP_READ_APV+0x20 vn_read+0x1f2 vn_io_fault_doio+0x45 vn_io_fault1+0x167 vn_io_fault+0x151 dofileread+0x80 kern_preadv+0x61 sys_preadv+0x38 amd64_syscall+0x117 25671 104067 bhyve blk-4:0-3 mi_switch+0xbd sleeplk+0x10a lockmgr_xlock_hard+0x398 nfs_lock+0x29 vop_sigdefer+0x2a vn_write+0x180 vn_io_fault_doio+0x45 vn_io_fault1+0x167 vn_io_fault+0x151 dofilewrite+0x81 kern_pwritev+0x61 sys_pwritev+0x38 amd64_syscall+0x117 fast_syscall_common+0xf8 25671 104068 bhyve blk-4:0-4 mi_switch+0xbd sleeplk+0x10a lockmgr_xlock_hard+0x398 nfs_lock+0x29 vop_sigdefer+0x2a vn_write+0x180 vn_io_fault_doio+0x45 vn_io_fault1+0x167 vn_io_fault+0x151 dofilewrite+0x81 kern_pwritev+0x61 sys_pwritev+0x38 amd64_syscall+0x117 fast_syscall_common+0xf8 25671 104069 bhyve blk-4:0-5 mi_switch+0xbd sleeplk+0x10a lockmgr_xlock_hard+0x398 nfs_lock+0x29 vop_sigdefer+0x2a vn_write+0x180 vn_io_fault_doio+0x45 vn_io_fault1+0x167 vn_io_fault+0x151 dofilewrite+0x81 kern_pwritev+0x61 sys_pwritev+0x38 amd64_syscall+0x117 fast_syscall_common+0xf8 25671 104070 bhyve blk-4:0-6 mi_switch+0xbd _sleep+0x1f3 bufspace_wait+0x19f getnewbuf+0x2b0 getblkx+0x4da getblk+0x22 nfs_getcacheblk+0x40 ncl_bioread+0xd41 vop_sigdefer+0x2a VOP_READ_APV+0x20 vn_read+0x1f2 vn_io_fault_doio+0x45 vn_io_fault1+0x167 vn_io_fault+0x151 dofileread+0x80 kern_preadv+0x61 sys_preadv+0x38 amd64_syscall+0x117 25671 104071 bhyve blk-4:0-7 mi_switch+0xbd sleeplk+0x10a lockmgr_xlock_hard+0x398 nfs_lock+0x29 vop_sigdefer+0x2a vn_write+0x180 vn_io_fault_doio+0x45 vn_io_fault1+0x167 vn_io_fault+0x151 dofilewrite+0x81 kern_pwritev+0x61 sys_pwritev+0x38 amd64_syscall+0x117 fast_syscall_common+0xf8 25671 104072 bhyve nvme-aen-4:0 mi_switch+0xbd sleepq_catch_signals+0x2a2 sleepq_wait_sig+0x9 _sleep+0x1e6 umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x117 fast_syscall_common+0xf8 25671 104073 bhyve vtnet-5:0 tx mi_switch+0xbd sleepq_catch_signals+0x2a2 sleepq_wait_sig+0x9 _sleep+0x1e6 umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x117 fast_syscall_common+0xf8 25671 104074 bhyve rfb mi_switch+0xbd sleepq_catch_signals+0x2a2 sleepq_wait_sig+0x9 _sleep+0x1e6 solisten_dequeue+0x65 kern_accept4+0x295 amd64_syscall+0x117 fast_syscall_common+0xf8 25671 104075 bhyve vcpu 0 mi_switch+0xbd sleepq_catch_signals+0x2a2 sleepq_wait_sig+0x9 _sleep+0x1e6 umtxq_sleep+0x2cd __umtx_op_sem2_wait+0x470 sys__umtx_op+0x7e amd64_syscall+0x117 fast_syscall_common+0xf8 25671 104076 bhyve vcpu 1 vmx_run+0x12c3 vm_run+0x172 vmmdev_ioctl+0xd59 devfs_ioctl+0xcb vn_ioctl+0xc8 devfs_ioctl_f+0x1e kern_ioctl+0x255 sys_ioctl+0x101 amd64_syscall+0x117 fast_syscall_common+0xf8 25671 104077 bhyve vcpu 2 mi_switch+0xbd sleepq_catch_signals+0x2a2 sleepq_wait_sig+0x9 _sleep+0x1e6 umtxq_sleep+0x2cd __umtx_op_sem2_wait+0x470 sys__umtx_op+0x7e amd64_syscall+0x117 fast_syscall_common+0xf8 25671 104078 bhyve vcpu 3 mi_switch+0xbd sleepq_catch_signals+0x2a2 sleepq_wait_sig+0x9 _sleep+0x1e6 umtxq_sleep+0x2cd __umtx_op_sem2_wait+0x470 sys__umtx_op+0x7e amd64_syscall+0x117 fast_syscall_common+0xf8 And here follow a complete ps: root@bhyve01:~ # ps -daxHl|more UID PID PPID C PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND 0 0 0 2 -16 0 0 5312 swapin DLs - 1:00.05 [kernel/swapper] 0 1 0 1 21 0 12724 1324 wait SLs - 0:01.78 - /sbin/init 0 4099 1 21 20 0 15444 4200 select Is - 0:22.73 |-- /sbin/devd 0 4276 1 7 20 0 13904 2896 select Ss - 0:45.22 |-- /usr/sbin/syslogd -N -O rfc5424 0 4421 1 12 26 0 14140 2388 pause Is - 0:00.00 |-- nfsuserd: master (nfsuserd) 0 4422 4421 4 20 0 14140 2388 select S - 0:00.26 | |-- nfsuserd: server (nfsuserd) 0 4423 4421 26 20 0 14140 2388 select S - 0:00.26 | |-- nfsuserd: server (nfsuserd) 0 4424 4421 16 20 0 14140 2388 select S - 0:00.26 | |-- nfsuserd: server (nfsuserd) 0 4425 4421 15 20 0 14140 2388 select S - 0:00.25 | `-- nfsuserd: server (nfsuserd) 0 4517 1 6 20 0 13828 2644 select Is - 0:00.38 |-- /usr/sbin/rpcbind 0 4519 1 3 20 0 275952 2508 select Ss - 0:00.31 |-- /usr/sbin/rpc.statd 0 4532 1 13 68 0 13860 2504 rpcsvc Ss - 0:00.63 |-- /usr/sbin/rpc.lockd 123 4575 1 16 20 0 20336 5680 select Ss - 0:20.30 |-- /usr/sbin/ntpd -p /var/db/ntp/ntpd.pid -c /etc/ntp.conf -f /var/db/ntp/ntpd.drift -g 0 4578 1 7 20 0 13848 2348 select Ss - 0:34.46 |-- /usr/sbin/powerd -a adp -N 65534 4595 1 5 68 0 13856 2440 kqread Is - 0:00.00 |-- daemon: /usr/bin/env[4596] (daemon) 65534 4596 4595 16 20 0 1263428 25084 uwait I - 4:05.97 | |-- /usr/local/bin/node_exporter --web.listen-address=:9100 --collector.textfile.directory= 65534 4596 4595 13 20 0 1263428 25084 uwait I - 0:27.61 | |-- /usr/local/bin/node_exporter --web.listen-address=:9100 --collector.textfile.directory= 65534 4596 4595 12 20 0 1263428 25084 kqread I - 4:24.84 | |-- /usr/local/bin/node_exporter --web.listen-address=:9100 --collector.textfile.directory= 65534 4596 4595 8 20 0 1263428 25084 uwait I - 3:58.04 | |-- /usr/local/bin/node_exporter --web.listen-address=:9100 --collector.textfile.directory= 65534 4596 4595 13 68 0 1263428 25084 uwait I - 0:00.00 | |-- /usr/local/bin/node_exporter --web.listen-address=:9100 --collector.textfile.directory= 65534 4596 4595 11 68 0 1263428 25084 uwait I - 0:00.00 | `-- /usr/local/bin/node_exporter --web.listen-address=:9100 --collector.textfile.directory= 0 4612 1 27 68 0 13856 2336 kqread Is - 0:00.00 |-- daemon: /usr/local/bin/zrepl[4613] (daemon) 0 4613 4612 23 64 0 1337260 59296 uwait I - 0:04.80 | |-- /usr/local/bin/zrepl --config /usr/local/etc/zrepl/zrepl.yml daemon 0 4613 4612 13 20 0 1337260 59296 uwait S - 1:09.07 | |-- /usr/local/bin/zrepl --config /usr/local/etc/zrepl/zrepl.yml daemon 0 4613 4612 20 68 0 1337260 59296 uwait I - 0:00.00 | |-- /usr/local/bin/zrepl --config /usr/local/etc/zrepl/zrepl.yml daemon 0 4613 4612 12 21 0 1337260 59296 uwait I - 0:24.48 | |-- /usr/local/bin/zrepl --config /usr/local/etc/zrepl/zrepl.yml daemon 0 4613 4612 8 68 0 1337260 59296 uwait I - 0:00.00 | |-- /usr/local/bin/zrepl --config /usr/local/etc/zrepl/zrepl.yml daemon 0 4613 4612 19 20 0 1337260 59296 uwait I - 0:13.44 | |-- /usr/local/bin/zrepl --config /usr/local/etc/zrepl/zrepl.yml daemon 0 4613 4612 27 20 0 1337260 59296 uwait I - 0:25.96 | |-- /usr/local/bin/zrepl --config /usr/local/etc/zrepl/zrepl.yml daemon 0 4613 4612 0 20 0 1337260 59296 uwait S - 0:23.72 | |-- /usr/local/bin/zrepl --config /usr/local/etc/zrepl/zrepl.yml daemon 0 4613 4612 4 20 0 1337260 59296 uwait S - 0:25.41 | |-- /usr/local/bin/zrepl --config /usr/local/etc/zrepl/zrepl.yml daemon 0 4613 4612 27 20 0 1337260 59296 uwait I - 0:24.03 | |-- /usr/local/bin/zrepl --config /usr/local/etc/zrepl/zrepl.yml daemon 0 4613 4612 25 21 0 1337260 59296 uwait I - 0:25.57 | |-- /usr/local/bin/zrepl --config /usr/local/etc/zrepl/zrepl.yml daemon 0 4613 4612 15 20 0 1337260 59296 uwait S - 0:23.21 | |-- /usr/local/bin/zrepl --config /usr/local/etc/zrepl/zrepl.yml daemon 0 4613 4612 7 20 0 1337260 59296 uwait I - 0:25.65 | |-- /usr/local/bin/zrepl --config /usr/local/etc/zrepl/zrepl.yml daemon 0 4613 4612 19 20 0 1337260 59296 uwait I - 0:23.38 | |-- /usr/local/bin/zrepl --config /usr/local/etc/zrepl/zrepl.yml daemon 0 4613 4612 10 20 0 1337260 59296 uwait I - 0:25.29 | |-- /usr/local/bin/zrepl --config /usr/local/etc/zrepl/zrepl.yml daemon 0 4613 4612 13 20 0 1337260 59296 uwait S - 0:25.18 | |-- /usr/local/bin/zrepl --config /usr/local/etc/zrepl/zrepl.yml daemon 0 4613 4612 21 20 0 1337260 59296 uwait S - 0:25.33 | |-- /usr/local/bin/zrepl --config /usr/local/etc/zrepl/zrepl.yml daemon 0 4613 4612 3 20 0 1337260 59296 uwait I - 0:26.42 | |-- /usr/local/bin/zrepl --config /usr/local/etc/zrepl/zrepl.yml daemon 0 4613 4612 6 20 0 1337260 59296 uwait S - 0:25.35 | |-- /usr/local/bin/zrepl --config /usr/local/etc/zrepl/zrepl.yml daemon 0 4613 4612 15 20 0 1337260 59296 uwait I - 0:26.15 | |-- /usr/local/bin/zrepl --config /usr/local/etc/zrepl/zrepl.yml daemon 0 4613 4612 8 20 0 1337260 59296 uwait I - 0:25.15 | |-- /usr/local/bin/zrepl --config /usr/local/etc/zrepl/zrepl.yml daemon 0 4613 4612 14 20 0 1337260 59296 uwait S - 0:23.08 | |-- /usr/local/bin/zrepl --config /usr/local/etc/zrepl/zrepl.yml daemon 0 4613 4612 15 20 0 1337260 59296 uwait I - 0:26.52 | |-- /usr/local/bin/zrepl --config /usr/local/etc/zrepl/zrepl.yml daemon 0 4613 4612 20 20 0 1337260 59296 uwait S - 0:23.47 | |-- /usr/local/bin/zrepl --config /usr/local/etc/zrepl/zrepl.yml daemon 0 4613 4612 9 20 0 1337260 59296 uwait I - 0:20.30 | |-- /usr/local/bin/zrepl --config /usr/local/etc/zrepl/zrepl.yml daemon 0 4613 4612 13 20 0 1337260 59296 uwait I - 0:22.18 | |-- /usr/local/bin/zrepl --config /usr/local/etc/zrepl/zrepl.yml daemon 0 4613 4612 21 21 0 1337260 59296 uwait I - 0:15.10 | |-- /usr/local/bin/zrepl --config /usr/local/etc/zrepl/zrepl.yml daemon 0 4613 4612 23 20 0 1337260 59296 kqread I - 0:24.54 | |-- /usr/local/bin/zrepl --config /usr/local/etc/zrepl/zrepl.yml daemon 0 4613 4612 24 20 0 1337260 59296 uwait I - 0:24.64 | |-- /usr/local/bin/zrepl --config /usr/local/etc/zrepl/zrepl.yml daemon 0 4613 4612 0 20 0 1337260 59296 uwait I - 0:25.20 | |-- /usr/local/bin/zrepl --config /usr/local/etc/zrepl/zrepl.yml daemon 0 4613 4612 27 20 0 1337260 59296 uwait I - 0:23.79 | |-- /usr/local/bin/zrepl --config /usr/local/etc/zrepl/zrepl.yml daemon 0 4613 4612 5 20 0 1337260 59296 uwait I - 0:19.08 | |-- /usr/local/bin/zrepl --config /usr/local/etc/zrepl/zrepl.yml daemon 0 4613 4612 12 20 0 1337260 59296 uwait S - 0:15.81 | `-- /usr/local/bin/zrepl --config /usr/local/etc/zrepl/zrepl.yml daemon 0 4616 1 1 20 0 13944 2628 nanslp Ss - 0:21.69 |-- /usr/sbin/cron -s 0 4632 1 26 20 0 23508 9644 select Is - 0:21.06 |-- sshd: /usr/sbin/sshd [listener] 0 of 10-100 startups (sshd) 0 4803 4632 7 23 0 24040 10296 select Is - 0:00.02 | |-- sshd-session: ale [priv] (sshd-session) 1001 4814 4803 9 20 0 24040 10588 select I - 0:16.88 | | `-- sshd-session: ale@pts/0 (sshd-session) 1001 4815 4814 9 24 0 14404 3388 wait Is 0 0:00.01 | | `-- -sh (sh) 0 4818 4815 3 20 0 21140 8840 select I+ 0 0:17.09 | | `-- sudo -i 0 4819 4818 0 21 0 21140 8832 select Is 1 0:00.00 | | `-- sudo -i 0 4820 4819 22 20 0 14404 3424 ttyin I+ 1 0:00.04 | | `-- -sh (sh) 0 52717 4632 0 21 0 24040 10632 select Is - 0:00.01 | |-- sshd-session: ale [priv] (sshd-session) 1001 52720 52717 17 20 0 24040 10916 select I - 0:00.25 | | `-- sshd-session: ale@pts/2 (sshd-session) 1001 52721 52720 12 23 0 14404 3308 wait Is 2 0:00.00 | | `-- -sh (sh) 0 52724 52721 16 20 0 21140 9028 select I+ 2 0:00.18 | | `-- sudo -i 0 52733 52724 21 21 0 21140 9024 select Is 6 0:00.00 | | `-- sudo -i 0 52734 52733 1 20 0 14404 3336 wait I 6 0:00.01 | | `-- -sh (sh) 0 52736 52734 6 20 0 14404 3920 - T 6 0:00.01 | | |-- /bin/sh /usr/local/sbin/vm list 0 52782 52736 13 20 0 14404 3924 - T 6 0:00.00 | | | `-- /bin/sh /usr/local/sbin/vm list 0 52785 52782 10 20 0 14404 3924 - T 6 0:00.01 | | | `-- /bin/sh /usr/local/sbin/vm list 0 52892 52785 13 20 0 14012 2560 newbuf TC 6 0:00.00 | | | `-- head -n1 /vms/mediaDeduper-worker2/run.lock 0 52929 52734 6 20 0 14144 2788 - T 6 0:00.21 | | |-- truss -f vm list 0 52930 52929 1 20 0 14404 3920 - TX 6 0:00.01 | | | `-- /bin/sh /usr/local/sbin/vm list 0 52976 52930 18 20 0 14404 3924 - TX 6 0:00.00 | | | `-- /bin/sh /usr/local/sbin/vm list 0 52979 52976 21 20 0 14404 3924 - TX 6 0:00.01 | | | `-- /bin/sh /usr/local/sbin/vm list 0 4613 4612 21 20 0 1337260 59296 uwait S - 0:25.33 | |-- /usr/local/bin/zrepl --config /usr/local/etc/zrepl/zrepl.yml daemon 0 4613 4612 3 20 0 1337260 59296 uwait I - 0:26.42 | |-- /usr/local/bin/zrepl --config /usr/local/etc/zrepl/zrepl.yml daemon 0 4613 4612 6 20 0 1337260 59296 uwait S - 0:25.35 | |-- /usr/local/bin/zrepl --config /usr/local/etc/zrepl/zrepl.yml daemon UID PID PPID C PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND 0 0 0 2 -16 0 0 5312 swapin DLs - 1:00.05 [kernel/swapper] 0 1 0 1 21 0 12724 1324 wait SLs - 0:01.78 - /sbin/init 0 4099 1 21 20 0 15444 4200 select Is - 0:22.73 |-- /sbin/devd 0 4276 1 7 20 0 13904 2896 select Ss - 0:45.22 |-- /usr/sbin/syslogd -N -O rfc5424 0 4421 1 12 26 0 14140 2388 pause Is - 0:00.00 |-- nfsuserd: master (nfsuserd) 0 4422 4421 4 20 0 14140 2388 select S - 0:00.26 | |-- nfsuserd: server (nfsuserd) 0 4423 4421 26 20 0 14140 2388 select S - 0:00.26 | |-- nfsuserd: server (nfsuserd) 0 4424 4421 16 20 0 14140 2388 select S - 0:00.26 | |-- nfsuserd: server (nfsuserd) 0 4425 4421 15 20 0 14140 2388 select S - 0:00.25 | `-- nfsuserd: server (nfsuserd) 0 4517 1 6 20 0 13828 2644 select Is - 0:00.38 |-- /usr/sbin/rpcbind 0 4519 1 3 20 0 275952 2508 select Ss - 0:00.31 |-- /usr/sbin/rpc.statd 0 4532 1 13 68 0 13860 2504 rpcsvc Ss - 0:00.63 |-- /usr/sbin/rpc.lockd 123 4575 1 16 20 0 20336 5680 select Ss - 0:20.30 |-- /usr/sbin/ntpd -p /var/db/ntp/ntpd.pid -c /etc/ntp.conf -f /var/db/ntp/ntpd.drift -g 0 4578 1 7 20 0 13848 2348 select Ss - 0:34.46 |-- /usr/sbin/powerd -a adp -N 65534 4595 1 5 68 0 13856 2440 kqread Is - 0:00.00 |-- daemon: /usr/bin/env[4596] (daemon) 65534 4596 4595 16 20 0 1263428 25084 uwait I - 4:05.97 | |-- /usr/local/bin/node_exporter --web.listen-address=:9100 --collector.textfile.directory= 65534 4596 4595 13 20 0 1263428 25084 uwait I - 0:27.61 | |-- /usr/local/bin/node_exporter --web.listen-address=:9100 --collector.textfile.directory= 65534 4596 4595 12 20 0 1263428 25084 kqread I - 4:24.84 | |-- /usr/local/bin/node_exporter --web.listen-address=:9100 --collector.textfile.directory= 65534 4596 4595 8 20 0 1263428 25084 uwait I - 3:58.04 | |-- /usr/local/bin/node_exporter --web.listen-address=:9100 --collector.textfile.directory= 65534 4596 4595 13 68 0 1263428 25084 uwait I - 0:00.00 | |-- /usr/local/bin/node_exporter --web.listen-address=:9100 --collector.textfile.directory= 65534 4596 4595 11 68 0 1263428 25084 uwait I - 0:00.00 | `-- /usr/local/bin/node_exporter --web.listen-address=:9100 --collector.textfile.directory= 0 4612 1 27 68 0 13856 2336 kqread Is - 0:00.00 |-- daemon: /usr/local/bin/zrepl[4613] (daemon) 0 4613 4612 23 64 0 1337260 59296 uwait I - 0:04.80 | |-- /usr/local/bin/zrepl --config /usr/local/etc/zrepl/zrepl.yml daemon 0 4613 4612 13 20 0 1337260 59296 uwait S - 1:09.07 | |-- /usr/local/bin/zrepl --config /usr/local/etc/zrepl/zrepl.yml daemon 0 4613 4612 20 68 0 1337260 59296 uwait I - 0:00.00 | |-- /usr/local/bin/zrepl --config /usr/local/etc/zrepl/zrepl.yml daemon 0 4613 4612 12 21 0 1337260 59296 uwait I - 0:24.48 | |-- /usr/local/bin/zrepl --config /usr/local/etc/zrepl/zrepl.yml daemon 0 4613 4612 8 68 0 1337260 59296 uwait I - 0:00.00 | |-- /usr/local/bin/zrepl --config /usr/local/etc/zrepl/zrepl.yml daemon 0 4613 4612 19 20 0 1337260 59296 uwait I - 0:13.44 | |-- /usr/local/bin/zrepl --config /usr/local/etc/zrepl/zrepl.yml daemon 0 4613 4612 27 20 0 1337260 59296 uwait I - 0:25.96 | |-- /usr/local/bin/zrepl --config /usr/local/etc/zrepl/zrepl.yml daemon 0 4613 4612 0 20 0 1337260 59296 uwait S - 0:23.72 | |-- /usr/local/bin/zrepl --config /usr/local/etc/zrepl/zrepl.yml daemon 0 4613 4612 4 20 0 1337260 59296 uwait S - 0:25.41 | |-- /usr/local/bin/zrepl --config /usr/local/etc/zrepl/zrepl.yml daemon 0 4613 4612 27 20 0 1337260 59296 uwait I - 0:24.03 | |-- /usr/local/bin/zrepl --config /usr/local/etc/zrepl/zrepl.yml daemon 0 4613 4612 25 21 0 1337260 59296 uwait I - 0:25.57 | |-- /usr/local/bin/zrepl --config /usr/local/etc/zrepl/zrepl.yml daemon 0 4613 4612 15 20 0 1337260 59296 uwait S - 0:23.21 | |-- /usr/local/bin/zrepl --config /usr/local/etc/zrepl/zrepl.yml daemon 0 4613 4612 7 20 0 1337260 59296 uwait I - 0:25.65 | |-- /usr/local/bin/zrepl --config /usr/local/etc/zrepl/zrepl.yml daemon 0 4613 4612 19 20 0 1337260 59296 uwait I - 0:23.38 | |-- /usr/local/bin/zrepl --config /usr/local/etc/zrepl/zrepl.yml daemon 0 4613 4612 10 20 0 1337260 59296 uwait I - 0:25.29 | |-- /usr/local/bin/zrepl --config /usr/local/etc/zrepl/zrepl.yml daemon 0 4613 4612 13 20 0 1337260 59296 uwait S - 0:25.18 | |-- /usr/local/bin/zrepl --config /usr/local/etc/zrepl/zrepl.yml daemon 0 4613 4612 21 20 0 1337260 59296 uwait S - 0:25.33 | |-- /usr/local/bin/zrepl --config /usr/local/etc/zrepl/zrepl.yml daemon 0 4613 4612 3 20 0 1337260 59296 uwait I - 0:26.42 | |-- /usr/local/bin/zrepl --config /usr/local/etc/zrepl/zrepl.yml daemon 0 4613 4612 6 20 0 1337260 59296 uwait S - 0:25.35 | |-- /usr/local/bin/zrepl --config /usr/local/etc/zrepl/zrepl.yml daemon 0 4613 4612 15 20 0 1337260 59296 uwait I - 0:26.15 | |-- /usr/local/bin/zrepl --config /usr/local/etc/zrepl/zrepl.yml daemon 0 4613 4612 8 20 0 1337260 59296 uwait I - 0:25.15 | |-- /usr/local/bin/zrepl --config /usr/local/etc/zrepl/zrepl.yml daemon 0 4613 4612 14 20 0 1337260 59296 uwait S - 0:23.08 | |-- /usr/local/bin/zrepl --config /usr/local/etc/zrepl/zrepl.yml daemon 0 4613 4612 15 20 0 1337260 59296 uwait I - 0:26.52 | |-- /usr/local/bin/zrepl --config /usr/local/etc/zrepl/zrepl.yml daemon 0 4613 4612 20 20 0 1337260 59296 uwait S - 0:23.47 | |-- /usr/local/bin/zrepl --config /usr/local/etc/zrepl/zrepl.yml daemon 0 4613 4612 9 20 0 1337260 59296 uwait I - 0:20.30 | |-- /usr/local/bin/zrepl --config /usr/local/etc/zrepl/zrepl.yml daemon 0 4613 4612 13 20 0 1337260 59296 uwait I - 0:22.18 | |-- /usr/local/bin/zrepl --config /usr/local/etc/zrepl/zrepl.yml daemon 0 4613 4612 21 21 0 1337260 59296 uwait I - 0:15.10 | |-- /usr/local/bin/zrepl --config /usr/local/etc/zrepl/zrepl.yml daemon 0 4613 4612 23 20 0 1337260 59296 kqread I - 0:24.54 | |-- /usr/local/bin/zrepl --config /usr/local/etc/zrepl/zrepl.yml daemon 0 4613 4612 24 20 0 1337260 59296 uwait I - 0:24.64 | |-- /usr/local/bin/zrepl --config /usr/local/etc/zrepl/zrepl.yml daemon 0 4613 4612 0 20 0 1337260 59296 uwait I - 0:25.20 | |-- /usr/local/bin/zrepl --config /usr/local/etc/zrepl/zrepl.yml daemon 0 4613 4612 27 20 0 1337260 59296 uwait I - 0:23.79 | |-- /usr/local/bin/zrepl --config /usr/local/etc/zrepl/zrepl.yml daemon 0 4613 4612 5 20 0 1337260 59296 uwait I - 0:19.08 | |-- /usr/local/bin/zrepl --config /usr/local/etc/zrepl/zrepl.yml daemon 0 4613 4612 12 20 0 1337260 59296 uwait S - 0:15.81 | `-- /usr/local/bin/zrepl --config /usr/local/etc/zrepl/zrepl.yml daemon 0 4616 1 1 20 0 13944 2628 nanslp Ss - 0:21.69 |-- /usr/sbin/cron -s 0 4632 1 26 20 0 23508 9644 select Is - 0:21.06 |-- sshd: /usr/sbin/sshd [listener] 0 of 10-100 startups (sshd) 0 4803 4632 7 23 0 24040 10296 select Is - 0:00.02 | |-- sshd-session: ale [priv] (sshd-session) 1001 4814 4803 9 20 0 24040 10588 select I - 0:16.88 | | `-- sshd-session: ale@pts/0 (sshd-session) 1001 4815 4814 9 24 0 14404 3388 wait Is 0 0:00.01 | | `-- -sh (sh) 0 4818 4815 3 20 0 21140 8840 select I+ 0 0:17.09 | | `-- sudo -i 0 4819 4818 0 21 0 21140 8832 select Is 1 0:00.00 | | `-- sudo -i 0 4820 4819 22 20 0 14404 3424 ttyin I+ 1 0:00.04 | | `-- -sh (sh) 0 52717 4632 0 21 0 24040 10632 select Is - 0:00.01 | |-- sshd-session: ale [priv] (sshd-session) 1001 52720 52717 17 20 0 24040 10916 select I - 0:00.25 | | `-- sshd-session: ale@pts/2 (sshd-session) 1001 52721 52720 12 23 0 14404 3308 wait Is 2 0:00.00 | | `-- -sh (sh) 0 52724 52721 16 20 0 21140 9028 select I+ 2 0:00.18 | | `-- sudo -i 0 52733 52724 21 21 0 21140 9024 select Is 6 0:00.00 | | `-- sudo -i 0 52734 52733 1 20 0 14404 3336 wait I 6 0:00.01 | | `-- -sh (sh) 0 52736 52734 6 20 0 14404 3920 - T 6 0:00.01 | | |-- /bin/sh /usr/local/sbin/vm list 0 52782 52736 13 20 0 14404 3924 - T 6 0:00.00 | | | `-- /bin/sh /usr/local/sbin/vm list 0 52785 52782 10 20 0 14404 3924 - T 6 0:00.01 | | | `-- /bin/sh /usr/local/sbin/vm list 0 52892 52785 13 20 0 14012 2560 newbuf TC 6 0:00.00 | | | `-- head -n1 /vms/mediaDeduper-worker2/run.lock 0 52929 52734 6 20 0 14144 2788 - T 6 0:00.21 | | |-- truss -f vm list 0 52930 52929 1 20 0 14404 3920 - TX 6 0:00.01 | | | `-- /bin/sh /usr/local/sbin/vm list 0 52976 52930 18 20 0 14404 3924 - TX 6 0:00.00 | | | `-- /bin/sh /usr/local/sbin/vm list 0 52979 52976 21 20 0 14404 3924 - TX 6 0:00.01 | | | `-- /bin/sh /usr/local/sbin/vm list 0 53086 52979 27 20 0 0 0 - ZC 6 0:00.00 | | | `-- <defunct> 0 53375 52734 17 20 0 13796 2428 newbuf T+ 6 0:00.00 | | `-- find . 0 56393 4632 3 20 0 24040 10800 select Is - 0:00.01 | `-- sshd-session: ale [priv] (sshd-session) 1001 56396 56393 0 20 0 24040 10932 select S - 0:00.04 | `-- sshd-session: ale@pts/7 (sshd-session) 1001 56397 56396 17 21 0 14404 3308 wait Is 7 0:00.00 | `-- -sh (sh) 0 56400 56397 20 20 0 21140 9160 select S+ 7 0:00.02 | `-- sudo -i 0 56401 56400 26 21 0 21140 9156 select Is 8 0:00.00 | `-- sudo -i 0 56402 56401 2 20 0 14404 3332 wait S 8 0:00.00 | `-- -sh (sh) 0 56422 56402 27 20 0 14484 4244 - R+ 8 0:00.00 | |-- ps -daxHl 0 56423 56402 24 20 0 14260 2924 - R+ 8 0:00.00 | `-- more 0 4642 1 25 20 0 16528 5064 nanslp I - 0:00.04 |-- /usr/local/sbin/smartd -c /usr/local/etc/smartd.conf -p /var/run/smartd.pid 0 74681 1 16 20 0 24416 10744 select Ss - 0:08.08 |-- screen -RR -D (screen-5.0.1) 0 74682 74681 27 68 0 14404 3316 wait Is 4 0:00.05 | |-- -/bin/sh 0 76947 74682 9 68 0 14404 3916 wait I+ 4 0:00.01 | | `-- /bin/sh /usr/local/sbin/vm console git 0 77045 76947 25 20 0 13812 2372 ttyin I+ 4 0:00.00 | | `-- cu -l /dev/nmdm-git.1B 0 77046 77045 0 20 0 13812 2368 ttyin I+ 4 0:00.05 | | `-- cu -l /dev/nmdm-git.1B 0 77054 74681 6 68 0 14404 3324 ttyin Is+ 5 0:00.05 | `-- -/bin/sh 0 96418 1 21 20 0 13872 2388 ttyin Is+ v0 0:00.00 |-- /usr/libexec/getty Pc ttyv0 0 4654 1 11 68 0 13872 2384 ttyin Is+ v1 0:00.00 |-- /usr/libexec/getty Pc ttyv1 0 4655 1 4 68 0 13872 2384 ttyin Is+ v2 0:00.00 |-- /usr/libexec/getty Pc ttyv2 0 4656 1 1 68 0 13872 2388 ttyin Is+ v3 0:00.00 |-- /usr/libexec/getty Pc ttyv3 0 4657 1 20 68 0 13872 2384 ttyin Is+ v4 0:00.00 |-- /usr/libexec/getty Pc ttyv4 0 4658 1 22 68 0 13872 2396 ttyin Is+ v5 0:00.00 |-- /usr/libexec/getty Pc ttyv5 0 4659 1 15 68 0 13872 2384 ttyin Is+ v6 0:00.00 |-- /usr/libexec/getty Pc ttyv6 0 4660 1 13 68 0 13872 2384 ttyin Is+ v7 0:00.00 |-- /usr/libexec/getty Pc ttyv7 0 13899 1 12 68 0 14404 3916 wait I 1 0:00.04 |-- /bin/sh /usr/local/sbin/vm _run fw 0 14734 13899 8 20 0 3208884 2875980 kqread IC 1 260:04.09 | |-- bhyve: fw (bhyve) 0 14734 13899 6 20 0 3208884 2875980 nfs DC 1 1:06.87 | |-- bhyve: fw (bhyve) 0 14734 13899 0 20 0 3208884 2875980 newbuf DC 1 1:06.89 | |-- bhyve: fw (bhyve) 0 14734 13899 1 20 0 3208884 2875980 uwait IC 1 1:06.69 | |-- bhyve: fw (bhyve) 0 14734 13899 5 20 0 3208884 2875980 uwait IC 1 1:06.25 | |-- bhyve: fw (bhyve) 0 14734 13899 27 20 0 3208884 2875980 uwait IC 1 1:06.38 | |-- bhyve: fw (bhyve) 0 14734 13899 2 20 0 3208884 2875980 nfs DC 1 1:06.74 | |-- bhyve: fw (bhyve) 0 14734 13899 16 20 0 3208884 2875980 uwait IC 1 1:06.96 | |-- bhyve: fw (bhyve) 0 14734 13899 20 20 0 3208884 2875980 uwait IC 1 1:06.31 | |-- bhyve: fw (bhyve) 0 14734 13899 24 28 0 3208884 2875980 uwait IC 1 0:00.00 | |-- bhyve: fw (bhyve) 0 14734 13899 17 20 0 3208884 2875980 uwait IC 1 0:38.06 | |-- bhyve: fw (bhyve) 0 14734 13899 15 20 0 3208884 2875980 uwait IC 1 83:25.88 | |-- bhyve: fw (bhyve) 0 14734 13899 16 20 0 3208884 2875980 uwait IC 1 20:07.86 | |-- bhyve: fw (bhyve) Before do a poweroff of the machine I made a savecore -L, so I have a livecore.0 file it that help something I the troubleshooting. Thank you for any help. Regards, Alexandre On Fri, Jul 25, 2025 at 11:53 AM Alexandre Biancalana <biancalana@gmail.com> wrote: > > I've searched for open bugs or threads related to something close to what I'm experiencing but can't find anything. > > My Environment has one nfsserver exporting zfs (nvme mirrored) dataset with bhyve vms raw disks and images. Two nfsclient's (bhyve01 and bhive02) mounting those volumes and running bhyve vms. > > The volumes mounted are: > 10.10.10.10:/mnt/datastore0/bhyve_instances 980G 266G 713G 27% /vms > 10.10.10.10:/mnt/datastore1/iso 369G 62G 307G 17% /vms/.iso > 10.10.10.10:/mnt/ds_ssd_vms_03/disks 431G 306G 125G 71% /vms/.disks > > The sintom is that after some time (12hrs to 48hrs) of uptime and nfs volumes mounted and being used, some of them (apparently the ones that do more io) stop responding. This hang is not happening on both clients at the same time. When I still can log into the nfsclients, commands to read/list the nfs mounted volumes didn't finish, for example a find command in one of those volumes. > > Any ideas/suggestions of what can/need to be done ? > > I'm isolating one of the machines so we can experiment and investigate. > > > root@bhyve01:~ # procstat -kk 13561 > PID TID COMM TDNAME KSTACK > 13561 101619 find - mi_switch+0xbd _sleep+0x1f3 bufspace_wait+0x19f getnewbuf+0x2b0 getblkx+0x4da ge > tblk+0x22 nfs_getcacheblk+0x40 ncl_bioread+0x6ef nfs_readdir+0x217 vop_sigdefer+0x2a VOP_READDIR_APV+0x20 kern_getdirentries+0x229 sy > s_getdirentries+0x29 amd64_syscall+0x117 fast_syscall_common+0xf8 > > Another example the haos vm is hang now: > > root@bhyve01:~ # procstat -kk 5645 > PID TID COMM TDNAME KSTACK > 5645 100741 bhyve mevent mi_switch+0xbd sleepq_catch_signals+0x2a2 sleepq_wait_sig+0x9 _sleep+0x1e6 kqueu > e_scan+0xa11 kqueue_kevent+0x13b kern_kevent_fp+0x4b kern_kevent_generic+0xdf sys_kevent+0x61 amd64_syscall+0x117 fast_syscall_common > +0xf8 > 5645 101155 bhyve blk-4:0-0 mi_switch+0xbd sleeplk+0x10a lockmgr_xlock_hard+0x398 nfs_lock+0x29 vop_sigdefer > +0x2a vn_write+0x180 vn_io_fault_doio+0x45 vn_io_fault1+0x167 vn_io_fault+0x151 dofilewrite+0x81 kern_pwritev+0x61 sys_pwritev+0x38 a > md64_syscall+0x117 fast_syscall_common+0xf8 > 5645 101156 bhyve blk-4:0-1 mi_switch+0xbd sleeplk+0x10a lockmgr_xlock_hard+0x398 nfs_lock+0x29 vop_sigdefer > +0x2a vn_write+0x180 vn_io_fault_doio+0x45 vn_io_fault1+0x167 vn_io_fault+0x151 dofilewrite+0x81 kern_pwritev+0x61 sys_pwritev+0x38 a > md64_syscall+0x117 fast_syscall_common+0xf8 > 5645 101157 bhyve blk-4:0-2 mi_switch+0xbd _sleep+0x1f3 bufspace_wait+0x19f getnewbuf+0x2b0 getblkx+0x4da ge > tblk+0x22 nfs_getcacheblk+0x40 ncl_bioread+0xd41 vop_sigdefer+0x2a VOP_READ_APV+0x20 vn_read+0x1f2 vn_io_fault_doio+0x45 vn_io_fault1 > +0x167 vn_io_fault+0x151 dofileread+0x80 kern_preadv+0x61 sys_preadv+0x38 amd64_syscall+0x117 > 5645 101158 bhyve blk-4:0-3 mi_switch+0xbd sleeplk+0x10a lockmgr_xlock_hard+0x398 nfs_lock+0x29 vop_sigdefer > +0x2a vn_write+0x180 vn_io_fault_doio+0x45 vn_io_fault1+0x167 vn_io_fault+0x151 dofilewrite+0x81 kern_pwritev+0x61 sys_pwritev+0x38 a > md64_syscall+0x117 fast_syscall_common+0xf8 > 5645 101159 bhyve blk-4:0-4 mi_switch+0xbd _sleep+0x1f3 bufspace_wait+0x19f getnewbuf+0x2b0 getblkx+0x4da ge > tblk+0x22 nfs_getcacheblk+0x40 ncl_bioread+0xd41 vop_sigdefer+0x2a VOP_READ_APV+0x20 vn_read+0x1f2 vn_io_fault_doio+0x45 vn_io_fault1 > +0x167 vn_io_fault+0x151 dofileread+0x80 kern_preadv+0x61 sys_preadv+0x38 amd64_syscall+0x117 > 5645 101160 bhyve blk-4:0-5 mi_switch+0xbd sleeplk+0x10a lockmgr_xlock_hard+0x398 nfs_lock+0x29 vop_sigdefer > +0x2a vn_write+0x180 vn_io_fault_doio+0x45 vn_io_fault1+0x167 vn_io_fault+0x151 dofilewrite+0x81 kern_pwritev+0x61 sys_pwritev+0x38 a > md64_syscall+0x117 fast_syscall_common+0xf8 > 5645 101161 bhyve blk-4:0-6 mi_switch+0xbd sleeplk+0x10a lockmgr_xlock_hard+0x398 nfs_lock+0x29 vop_sigdefer > +0x2a vn_write+0x180 vn_io_fault_doio+0x45 vn_io_fault1+0x167 vn_io_fault+0x151 dofilewrite+0x81 kern_pwritev+0x61 sys_pwritev+0x38 a > md64_syscall+0x117 fast_syscall_common+0xf8 > 5645 101162 bhyve blk-4:0-7 mi_switch+0xbd sleeplk+0x10a lockmgr_xlock_hard+0x398 nfs_lock+0x29 vop_sigdefer > +0x2a vn_write+0x180 vn_io_fault_doio+0x45 vn_io_fault1+0x167 vn_io_fault+0x151 dofilewrite+0x81 kern_pwritev+0x61 sys_pwritev+0x38 a > md64_syscall+0x117 fast_syscall_common+0xf8 > 5645 101163 bhyve nvme-aen-4:0 mi_switch+0xbd sleepq_catch_signals+0x2a2 sleepq_wait_sig+0x9 _sleep+0x1e6 umtxq > _sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x117 fast_syscall_common+0xf8 > 5645 101164 bhyve vtnet-5:0 tx mi_switch+0xbd sleepq_catch_signals+0x2a2 sleepq_wait_sig+0x9 _sleep+0x1e6 umtxq > _sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x117 fast_syscall_common+0xf8 > 5645 101165 bhyve rfb mi_switch+0xbd sleepq_catch_signals+0x2a2 sleepq_wait_sig+0x9 _sleep+0x1e6 solis > ten_dequeue+0x65 kern_accept4+0x295 amd64_syscall+0x117 fast_syscall_common+0xf8 > 5645 101167 bhyve vcpu 0 mi_switch+0xbd sleepq_catch_signals+0x2a2 sleepq_wait_sig+0x9 _sleep+0x1e6 umtxq > _sleep+0x2cd __umtx_op_sem2_wait+0x470 sys__umtx_op+0x7e amd64_syscall+0x117 fast_syscall_common+0xf8 > 5645 101168 bhyve vcpu 1 mi_switch+0xbd sleepq_catch_signals+0x2a2 sleepq_wait_sig+0x9 _sleep+0x1e6 umtxq > _sleep+0x2cd __umtx_op_sem2_wait+0x470 sys__umtx_op+0x7e amd64_syscall+0x117 fast_syscall_common+0xf8 > 5645 101169 bhyve vcpu 2 mi_switch+0xbd sleepq_catch_signals+0x2a2 sleepq_wait_sig+0x9 _sleep+0x1e6 umtxq > _sleep+0x2cd __umtx_op_sem2_wait+0x470 sys__umtx_op+0x7e amd64_syscall+0x117 fast_syscall_common+0xf8 > 5645 101170 bhyve vcpu 3 mi_switch+0xbd sleepq_catch_signals+0x2a2 sleepq_wait_sig+0x9 _sleep+0x1e6 umtxq > _sleep+0x2cd __umtx_op_sem2_wait+0x470 sys__umtx_op+0x7e amd64_syscall+0x117 fast_syscall_common+0xf8 > > > > I'm have put in the end of the email the 'ps axHl' of the client with the hung processes (haos vm and find command) > > I've migrated these mountpoints to nfs4 last week when that started hanging, I'm going to do now is rollback the mountpoints of one client to nfs v3, so I can bring some stability to the environment and have time to troubleshoot without problems. > > > > nfsserver: > > # uname -a > FreeBSD alenas.local 13.1-RELEASE-p9 FreeBSD 13.1-RELEASE-p9 n245433-9dc2dc9b081 TRUENAS amd64 > > # zpool list -v > NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT > datastore0 1.81T 1.06T 771G - - 83% 58% 1.16x ONLINE /mnt > mirror-0 1.81T 1.06T 771G - - 83% 58.5% - ONLINE > gpt/kingfast-nvme-data 1.82T - - - - - - - ONLINE > gpt/netac-nvme-data 1.82T - - - - - - - ONLINE > > > clients: > > root@bhyve01:~ # uname -a > FreeBSD bhyve01 14.3-RELEASE FreeBSD 14.3-RELEASE releng/14.3-n271432-8c9ce319fef7 GENERIC amd64 > > root@bhyve01:~ # nfsstat -m > 10.10.10.10:/mnt/datastore0/bhyve_instances on /vms > nfsv4,minorversion=2,tcp,resvport,nconnect=16,hard,cto,sec=sys,acdirmin=3,acdirmax=60,acregmin=5,acregmax=60,nametimeo=60,negnametime > o=60,rsize=1048576,wsize=1048576,readdirsize=1048576,readahead=1,wcommitsize=16777216,timeout=120,retrans=2147483647 > 10.10.10.10:/mnt/datastore1/iso on /vms/.iso > nfsv4,minorversion=2,tcp,resvport,nconnect=16,hard,cto,sec=sys,acdirmin=3,acdirmax=60,acregmin=5,acregmax=60,nametimeo=60,negnametime > o=60,rsize=1048576,wsize=1048576,readdirsize=1048576,readahead=8,wcommitsize=67108864,timeout=120,retrans=2147483647 > 10.10.10.10:/mnt/ds_ssd_vms_03/disks on /vms/.disks > nfsv4,minorversion=2,tcp,resvport,nconnect=16,hard,cto,sec=sys,acdirmin=3,acdirmax=60,acregmin=5,acregmax=60,nametimeo=60,negnametime > o=60,rsize=1048576,wsize=1048576,readdirsize=1048576,readahead=8,wcommitsize=67108864,timeout=120,retrans=2147483647 > > root@bhyve01:~ # ps axHl|more > UID PID PPID C PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND > 0 0 0 23 -16 0 0 5312 swapin DLs - 24268:43.88 [kernel/swapper] > 0 0 0 0 -60 0 0 5312 - DLs - 0:00.00 [kernel/softirq_0] > 0 0 0 1 -60 0 0 5312 - DLs - 0:00.00 [kernel/softirq_1] > 0 0 0 2 -60 0 0 5312 - DLs - 0:00.00 [kernel/softirq_2] > 0 0 0 3 -60 0 0 5312 - DLs - 0:00.00 [kernel/softirq_3] > 0 0 0 4 -60 0 0 5312 - DLs - 0:00.00 [kernel/softirq_4] > 0 0 0 5 -60 0 0 5312 - DLs - 0:00.00 [kernel/softirq_5] > 0 0 0 6 -60 0 0 5312 - DLs - 0:00.00 [kernel/softirq_6] > 0 0 0 7 -60 0 0 5312 - DLs - 0:00.00 [kernel/softirq_7] > 0 0 0 8 -60 0 0 5312 - DLs - 0:00.00 [kernel/softirq_8] > 0 0 0 9 -60 0 0 5312 - DLs - 0:00.00 [kernel/softirq_9] > 0 0 0 10 -60 0 0 5312 - DLs - 0:00.00 [kernel/softirq_10] > 0 0 0 11 -60 0 0 5312 - DLs - 0:00.00 [kernel/softirq_11] > 0 0 0 12 -60 0 0 5312 - DLs - 0:00.00 [kernel/softirq_12] > 0 0 0 13 -60 0 0 5312 - DLs - 0:00.00 [kernel/softirq_13] > 0 0 0 14 -60 0 0 5312 - DLs - 0:00.00 [kernel/softirq_14] > 0 0 0 15 -60 0 0 5312 - DLs - 0:00.00 [kernel/softirq_15] > 0 0 0 16 -60 0 0 5312 - DLs - 0:00.00 [kernel/softirq_16] > 0 0 0 17 -60 0 0 5312 - DLs - 0:00.00 [kernel/softirq_17] > 0 0 0 18 -60 0 0 5312 - DLs - 0:00.00 [kernel/softirq_18] > 0 0 0 19 -60 0 0 5312 - DLs - 0:00.00 [kernel/softirq_19] > 0 0 0 20 -60 0 0 5312 - DLs - 0:00.00 [kernel/softirq_20] > 0 0 0 21 -60 0 0 5312 - DLs - 0:00.00 [kernel/softirq_21] > 0 0 0 22 -60 0 0 5312 - DLs - 0:00.00 [kernel/softirq_22] > 0 0 0 23 -60 0 0 5312 - DLs - 0:00.00 [kernel/softirq_23] > 0 0 0 24 -60 0 0 5312 - DLs - 0:00.00 [kernel/softirq_24] > 0 0 0 25 -60 0 0 5312 - DLs - 0:00.00 [kernel/softirq_25] > 0 0 0 26 -60 0 0 5312 - DLs - 0:00.00 [kernel/softirq_26] > 0 0 0 27 -60 0 0 5312 - DLs - 0:00.00 [kernel/softirq_27] > 0 0 0 0 -60 0 0 5312 - DLs - 3:52.44 [kernel/if_io_tqg_0] > 0 0 0 1 -60 0 0 5312 - DLs - 0:00.00 [kernel/if_io_tqg_1] > 0 0 0 2 -60 0 0 5312 - DLs - 4:50.91 [kernel/if_io_tqg_2] > 0 0 0 3 -60 0 0 5312 - DLs - 0:00.00 [kernel/if_io_tqg_3] > 0 0 0 4 -60 0 0 5312 - DLs - 2:57.04 [kernel/if_io_tqg_4] > 0 0 0 5 -60 0 0 5312 - DLs - 0:00.00 [kernel/if_io_tqg_5] > 0 0 0 6 -60 0 0 5312 - DLs - 2:32.68 [kernel/if_io_tqg_6] > 0 0 0 7 -60 0 0 5312 - DLs - 0:00.00 [kernel/if_io_tqg_7] > 0 0 0 8 -60 0 0 5312 - DLs - 0:14.44 [kernel/if_io_tqg_8] > 0 0 0 9 -60 0 0 5312 - DLs - 0:00.00 [kernel/if_io_tqg_9] > 0 0 0 10 -60 0 0 5312 - DLs - 4:53.15 [kernel/if_io_tqg_10] > 0 0 0 11 -60 0 0 5312 - DLs - 0:00.00 [kernel/if_io_tqg_11] > 0 0 0 12 -60 0 0 5312 - DLs - 2:31.60 [kernel/if_io_tqg_12] > 0 0 0 13 -60 0 0 5312 - DLs - 0:00.00 [kernel/if_io_tqg_13] > 0 0 0 14 -60 0 0 5312 - DLs - 5:01.03 [kernel/if_io_tqg_14] > 0 0 0 15 -60 0 0 5312 - DLs - 0:00.00 [kernel/if_io_tqg_15] > 0 0 0 16 -60 0 0 5312 - DLs - 3:40.18 [kernel/if_io_tqg_16] > 0 0 0 17 -60 0 0 5312 - DLs - 0:00.00 [kernel/if_io_tqg_17] > 0 0 0 18 -60 0 0 5312 - DLs - 0:40.63 [kernel/if_io_tqg_18] > 0 0 0 19 -60 0 0 5312 - DLs - 0:00.00 [kernel/if_io_tqg_19] > 0 0 0 20 -60 0 0 5312 - DLs - 4:58.54 [kernel/if_io_tqg_20] > 0 0 0 21 -60 0 0 5312 - DLs - 0:00.00 [kernel/if_io_tqg_21] > 0 0 0 22 -60 0 0 5312 - DLs - 2:50.16 [kernel/if_io_tqg_22] > 0 0 0 23 -60 0 0 5312 - DLs - 0:00.00 [kernel/if_io_tqg_23] > 0 0 0 24 -60 0 0 5312 - DLs - 3:06.44 [kernel/if_io_tqg_24] > 0 0 0 25 -60 0 0 5312 - DLs - 0:00.00 [kernel/if_io_tqg_25] > 0 0 0 26 -60 0 0 5312 - DLs - 1:28.38 [kernel/if_io_tqg_26] > 0 0 0 27 -60 0 0 5312 - DLs - 0:00.00 [kernel/if_io_tqg_27] > 0 0 0 23 -60 0 0 5312 - DLs - 0:00.00 [kernel/if_config_tqg_0] > 0 0 0 9 8 0 0 5312 - DLs - 0:00.00 [kernel/kqueue_ctx taskq] > 0 0 0 23 8 0 0 5312 - DLs - 0:00.00 [kernel/jail_remove taskq] > 0 0 0 6 8 0 0 5312 - DLs - 0:00.00 [kernel/bus taskq] > 0 0 0 16 8 0 0 5312 - DLs - 0:00.33 [kernel/thread taskq] > 0 0 0 23 8 0 0 5312 - DLs - 0:00.00 [kernel/aiod_kick taskq] > 0 0 0 6 8 0 0 5312 - DLs - 0:00.00 [kernel/deferred_unmount ta] > 0 0 0 23 8 0 0 5312 - DLs - 0:00.00 [kernel/inm_free taskq] > 0 0 0 22 8 0 0 5312 - DLs - 0:00.00 [kernel/in6m_free taskq] > 0 0 0 20 8 0 0 5312 - DLs - 0:00.00 [kernel/linuxkpi_irq_wq] > 0 0 0 6 8 0 0 5312 - DLs - 0:00.00 [kernel/linuxkpi_short_wq_0] > 0 0 0 22 8 0 0 5312 - DLs - 0:00.00 [kernel/linuxkpi_short_wq_1] > 0 0 0 25 8 0 0 5312 - DLs - 0:00.00 [kernel/linuxkpi_short_wq_2] > 0 0 0 6 8 0 0 5312 - DLs - 0:00.00 [kernel/linuxkpi_short_wq_3] > 0 0 0 22 8 0 0 5312 - DLs - 0:00.00 [kernel/linuxkpi_short_wq_4] > 0 0 0 26 8 0 0 5312 - DLs - 0:00.00 [kernel/linuxkpi_short_wq_5] > 0 0 0 24 8 0 0 5312 - DLs - 0:00.00 [kernel/linuxkpi_short_wq_6] > 0 0 0 6 8 0 0 5312 - DLs - 0:00.00 [kernel/linuxkpi_short_wq_7] > 0 0 0 22 8 0 0 5312 - DLs - 0:00.00 [kernel/linuxkpi_short_wq_8] > 0 0 0 15 8 0 0 5312 - DLs - 0:00.00 [kernel/linuxkpi_short_wq_9] > 0 0 0 21 8 0 0 5312 - DLs - 0:00.00 [kernel/linuxkpi_short_wq_1] > 0 0 0 24 8 0 0 5312 - DLs - 0:00.00 [kernel/linuxkpi_short_wq_1] > 0 0 0 22 8 0 0 5312 - DLs - 0:00.00 [kernel/linuxkpi_short_wq_1] > 0 0 0 6 8 0 0 5312 - DLs - 0:00.00 [kernel/linuxkpi_short_wq_1] > 0 0 0 15 8 0 0 5312 - DLs - 0:00.00 [kernel/linuxkpi_short_wq_1] > 0 0 0 21 8 0 0 5312 - DLs - 0:00.00 [kernel/linuxkpi_short_wq_1] > 0 0 0 22 8 0 0 5312 - DLs - 0:00.00 [kernel/linuxkpi_short_wq_1] > 0 0 0 10 8 0 0 5312 - DLs - 0:00.00 [kernel/linuxkpi_short_wq_1] > 0 0 0 24 8 0 0 5312 - DLs - 0:00.00 [kernel/linuxkpi_short_wq_1] > 0 0 0 22 8 0 0 5312 - DLs - 0:00.00 [kernel/linuxkpi_short_wq_1] > 0 0 0 10 8 0 0 5312 - DLs - 0:00.00 [kernel/linuxkpi_short_wq_2] > 0 0 0 24 8 0 0 5312 - DLs - 0:00.00 [kernel/linuxkpi_short_wq_2] > 0 0 0 22 8 0 0 5312 - DLs - 0:00.00 [kernel/linuxkpi_short_wq_2] > 0 0 0 13 8 0 0 5312 - DLs - 0:00.00 [kernel/linuxkpi_short_wq_2] > 0 0 0 18 8 0 0 5312 - DLs - 0:00.00 [kernel/linuxkpi_short_wq_2] > 0 0 0 24 8 0 0 5312 - DLs - 0:00.00 [kernel/linuxkpi_short_wq_2] > 0 0 0 21 8 0 0 5312 - DLs - 0:00.00 [kernel/linuxkpi_short_wq_2] > 0 0 0 16 8 0 0 5312 - DLs - 0:00.00 [kernel/linuxkpi_short_wq_2] > 0 0 0 13 8 0 0 5312 - DLs - 0:00.00 [kernel/linuxkpi_short_wq_2] > 0 0 0 24 8 0 0 5312 - DLs - 0:00.00 [kernel/linuxkpi_long_wq_0] > 0 0 0 18 8 0 0 5312 - DLs - 0:00.00 [kernel/linuxkpi_long_wq_1] > 0 0 0 21 8 0 0 5312 - DLs - 0:00.00 [kernel/linuxkpi_long_wq_2] > 0 0 0 24 8 0 0 5312 - DLs - 0:00.00 [kernel/linuxkpi_long_wq_3] > 0 0 0 13 8 0 0 5312 - DLs - 0:00.00 [kernel/linuxkpi_long_wq_4] > 0 0 0 21 8 0 0 5312 - DLs - 0:00.00 [kernel/linuxkpi_long_wq_5] > 0 0 0 24 8 0 0 5312 - DLs - 0:00.00 [kernel/linuxkpi_long_wq_6] > 0 0 0 13 8 0 0 5312 - DLs - 0:00.00 [kernel/linuxkpi_long_wq_7] > 0 0 0 21 8 0 0 5312 - DLs - 0:00.00 [kernel/linuxkpi_long_wq_8] > 0 0 0 24 8 0 0 5312 - DLs - 0:00.00 [kernel/linuxkpi_long_wq_9] > 0 0 0 13 8 0 0 5312 - DLs - 0:00.00 [kernel/linuxkpi_long_wq_10] > 0 0 0 21 8 0 0 5312 - DLs - 0:00.00 [kernel/linuxkpi_long_wq_11] > 0 0 0 24 8 0 0 5312 - DLs - 0:00.00 [kernel/linuxkpi_long_wq_12] > 0 0 0 13 8 0 0 5312 - DLs - 0:00.00 [kernel/linuxkpi_long_wq_13] > 0 0 0 26 8 0 0 5312 - DLs - 0:00.00 [kernel/linuxkpi_long_wq_14] > 0 0 0 21 8 0 0 5312 - DLs - 0:00.00 [kernel/linuxkpi_long_wq_15] > 0 0 0 24 8 0 0 5312 - DLs - 0:00.00 [kernel/linuxkpi_long_wq_16] > 0 0 0 13 8 0 0 5312 - DLs - 0:00.00 [kernel/linuxkpi_long_wq_17] > 0 0 0 26 8 0 0 5312 - DLs - 0:00.00 [kernel/linuxkpi_long_wq_18] > 0 0 0 21 8 0 0 5312 - DLs - 0:00.00 [kernel/linuxkpi_long_wq_19] > 0 0 0 5 8 0 0 5312 - DLs - 0:00.00 [kernel/linuxkpi_long_wq_20] > 0 0 0 26 8 0 0 5312 - DLs - 0:00.00 [kernel/linuxkpi_long_wq_21] > 0 0 0 24 8 0 0 5312 - DLs - 0:00.00 [kernel/linuxkpi_long_wq_22] > 0 0 0 5 8 0 0 5312 - DLs - 0:00.00 [kernel/linuxkpi_long_wq_23] > 0 0 0 26 8 0 0 5312 - DLs - 0:00.00 [kernel/linuxkpi_long_wq_24] > 0 0 0 24 8 0 0 5312 - DLs - 0:00.00 [kernel/linuxkpi_long_wq_25] > 0 0 0 5 8 0 0 5312 - DLs - 0:00.00 [kernel/linuxkpi_long_wq_26] > 0 0 0 26 8 0 0 5312 - DLs - 0:00.00 [kernel/linuxkpi_long_wq_27] > 0 0 0 24 8 0 0 5312 - DLs - 0:00.00 [kernel/linuxkpi_long_wq_28] > 0 0 0 22 8 0 0 5312 - DLs - 0:00.00 [kernel/firmware taskq] > 0 0 0 26 -20 0 0 5312 - DLs - 0:00.00 [kernel/crypto_0] > 0 0 0 20 -20 0 0 5312 - DLs - 0:00.00 [kernel/crypto_1] > 0 0 0 22 -20 0 0 5312 - DLs - 0:00.00 [kernel/crypto_2] > 0 0 0 16 -20 0 0 5312 - DLs - 0:00.00 [kernel/crypto_3] > 0 0 0 2 -20 0 0 5312 - DLs - 0:00.00 [kernel/crypto_4] > 0 0 0 4 -20 0 0 5312 - DLs - 0:00.00 [kernel/crypto_5] > 0 0 0 26 -20 0 0 5312 - DLs - 0:00.00 [kernel/crypto_6] > 0 0 0 12 -20 0 0 5312 - DLs - 0:00.00 [kernel/crypto_7] > 0 0 0 2 -20 0 0 5312 - DLs - 0:00.00 [kernel/crypto_8] > 0 0 0 8 -20 0 0 5312 - DLs - 0:00.00 [kernel/crypto_9] > 0 0 0 23 -20 0 0 5312 - DLs - 0:00.00 [kernel/crypto_10] > 0 0 0 13 -20 0 0 5312 - DLs - 0:00.00 [kernel/crypto_11] > 0 0 0 5 -20 0 0 5312 - DLs - 0:00.00 [kernel/crypto_12] > 0 0 0 20 -20 0 0 5312 - DLs - 0:00.00 [kernel/crypto_13] > 0 0 0 24 -20 0 0 5312 - DLs - 0:00.00 [kernel/crypto_14] > 0 0 0 19 -20 0 0 5312 - DLs - 0:00.00 [kernel/crypto_15] > 0 0 0 14 -20 0 0 5312 - DLs - 0:00.00 [kernel/crypto_16] > 0 0 0 11 -20 0 0 5312 - DLs - 0:00.00 [kernel/crypto_17] > 0 0 0 3 -20 0 0 5312 - DLs - 0:00.00 [kernel/crypto_18] > 0 0 0 25 -20 0 0 5312 - DLs - 0:00.00 [kernel/crypto_19] > 0 0 0 14 -20 0 0 5312 - DLs - 0:00.00 [kernel/crypto_20] > 0 0 0 17 -20 0 0 5312 - DLs - 0:00.00 [kernel/crypto_21] > 0 0 0 19 -20 0 0 5312 - DLs - 0:00.00 [kernel/crypto_22] > 0 0 0 21 -20 0 0 5312 - DLs - 0:00.00 [kernel/crypto_23] > 0 0 0 2 -20 0 0 5312 - DLs - 0:00.00 [kernel/crypto_24] > 0 0 0 5 -20 0 0 5312 - DLs - 0:00.00 [kernel/crypto_25] > 0 0 0 15 -20 0 0 5312 - DLs - 0:00.00 [kernel/crypto_26] > 0 0 0 7 -20 0 0 5312 - DLs - 0:00.00 [kernel/crypto_27] > 0 0 0 24 -64 0 0 5312 - DLs - 0:00.00 [kernel/nvme taskq_0] > 0 0 0 26 -64 0 0 5312 - DLs - 0:00.00 [kernel/nvme taskq_1] > 0 0 0 15 -64 0 0 5312 - DLs - 0:16.37 [kernel/if_ix0_tq] > 0 0 0 16 -64 0 0 5312 - DLs - 0:18.06 [kernel/if_ix1_tq] > 0 0 0 26 -8 0 0 5312 - DLs - 0:00.00 [kernel/system_taskq_0] > 0 0 0 11 -8 0 0 5312 - DLs - 0:00.00 [kernel/system_taskq_1] > 0 0 0 22 -8 0 0 5312 - DLs - 0:00.00 [kernel/system_taskq_2] > 0 0 0 14 -8 0 0 5312 - DLs - 0:00.00 [kernel/system_taskq_3] > 0 0 0 5 -8 0 0 5312 - DLs - 0:00.00 [kernel/system_taskq_4] > 0 0 0 25 -8 0 0 5312 - DLs - 0:00.01 [kernel/system_taskq_5] > 0 0 0 16 -8 0 0 5312 - DLs - 0:00.00 [kernel/system_taskq_6] > 0 0 0 23 -8 0 0 5312 - DLs - 0:00.00 [kernel/system_taskq_7] > 0 0 0 0 -8 0 0 5312 - DLs - 0:00.00 [kernel/system_taskq_8] > 0 0 0 20 -8 0 0 5312 - DLs - 0:00.00 [kernel/system_taskq_9] > 0 0 0 18 -8 0 0 5312 - DLs - 0:00.00 [kernel/system_taskq_10] > 0 0 0 15 -8 0 0 5312 - DLs - 0:00.00 [kernel/system_taskq_11] > 0 0 0 2 -8 0 0 5312 - DLs - 0:00.00 [kernel/system_taskq_12] > 0 0 0 17 -8 0 0 5312 - DLs - 0:00.00 [kernel/system_taskq_13] > 0 0 0 0 -8 0 0 5312 - DLs - 0:00.00 [kernel/system_taskq_14] > 0 0 0 4 -8 0 0 5312 - DLs - 0:00.00 [kernel/system_taskq_15] > 0 0 0 8 -8 0 0 5312 - DLs - 0:00.00 [kernel/system_taskq_16] > 0 0 0 7 -8 0 0 5312 - DLs - 0:00.00 [kernel/system_taskq_17] > 0 0 0 18 -8 0 0 5312 - DLs - 0:00.00 [kernel/system_taskq_18] > 0 0 0 27 -8 0 0 5312 - DLs - 0:00.00 [kernel/system_taskq_19] > 0 0 0 5 -8 0 0 5312 - DLs - 0:00.00 [kernel/system_taskq_20] > 0 0 0 2 -8 0 0 5312 - DLs - 0:00.00 [kernel/system_taskq_21] > 0 0 0 10 -8 0 0 5312 - DLs - 0:00.00 [kernel/system_taskq_22] > 0 0 0 19 -8 0 0 5312 - DLs - 0:00.00 [kernel/system_taskq_23] > 0 0 0 8 -8 0 0 5312 - DLs - 0:00.00 [kernel/system_taskq_24] > 0 0 0 20 -8 0 0 5312 - DLs - 0:00.00 [kernel/system_taskq_25] > 0 0 0 14 -8 0 0 5312 - DLs - 0:00.00 [kernel/system_taskq_26] > 0 0 0 4 -8 0 0 5312 - DLs - 0:00.00 [kernel/system_taskq_27] > 0 0 0 1 -8 0 0 5312 - DLs - 0:00.00 [kernel/system_delay_taskq_] > 0 0 0 13 -8 0 0 5312 - DLs - 0:00.00 [kernel/system_delay_taskq_] > 0 0 0 3 -8 0 0 5312 - DLs - 0:00.00 [kernel/system_delay_taskq_] > 0 0 0 19 -8 0 0 5312 - DLs - 0:00.00 [kernel/system_delay_taskq_] > 0 0 0 15 -8 0 0 5312 - DLs - 0:00.00 [kernel/system_delay_taskq_] > 0 0 0 8 -8 0 0 5312 - DLs - 0:00.00 [kernel/system_delay_taskq_] > 0 0 0 22 -8 0 0 5312 - DLs - 0:00.00 [kernel/system_delay_taskq_] > 0 0 0 16 -8 0 0 5312 - DLs - 0:00.00 [kernel/system_delay_taskq_] > 0 0 0 6 -8 0 0 5312 - DLs - 0:00.00 [kernel/system_delay_taskq_] > 0 0 0 15 -8 0 0 5312 - DLs - 0:00.00 [kernel/system_delay_taskq_] > 0 0 0 18 -8 0 0 5312 - DLs - 0:00.00 [kernel/system_delay_taskq_] > 0 0 0 8 -8 0 0 5312 - DLs - 0:00.00 [kernel/system_delay_taskq_] > 0 0 0 12 -8 0 0 5312 - DLs - 0:00.00 [kernel/system_delay_taskq_] > 0 0 0 14 -8 0 0 5312 - DLs - 0:00.00 [kernel/system_delay_taskq_] > 0 0 0 4 -8 0 0 5312 - DLs - 0:00.00 [kernel/system_delay_taskq_] > 0 0 0 1 -8 0 0 5312 - DLs - 0:00.00 [kernel/system_delay_taskq_] > 0 0 0 16 -8 0 0 5312 - DLs - 0:00.00 [kernel/system_delay_taskq_] > 0 0 0 22 -8 0 0 5312 - DLs - 0:00.00 [kernel/system_delay_taskq_] > 0 0 0 8 -8 0 0 5312 - DLs - 0:00.00 [kernel/system_delay_taskq_] > 0 0 0 11 -8 0 0 5312 - DLs - 0:00.00 [kernel/system_delay_taskq_] > 0 0 0 7 -8 0 0 5312 - DLs - 0:00.00 [kernel/system_delay_taskq_] > 0 0 0 0 -8 0 0 5312 - DLs - 0:00.00 [kernel/system_delay_taskq_] > 0 0 0 9 -8 0 0 5312 - DLs - 0:00.00 [kernel/system_delay_taskq_] > 0 0 0 22 -8 0 0 5312 - DLs - 0:00.00 [kernel/system_delay_taskq_] > 0 0 0 15 -8 0 0 5312 - DLs - 0:00.00 [kernel/system_delay_taskq_] > 0 0 0 6 -8 0 0 5312 - DLs - 0:00.00 [kernel/system_delay_taskq_] > 0 0 0 16 -8 0 0 5312 - DLs - 0:00.00 [kernel/system_delay_taskq_] > 0 0 0 23 -8 0 0 5312 - DLs - 0:00.00 [kernel/system_delay_taskq_] > 0 0 0 11 -8 0 0 5312 - DLs - 0:00.00 [kernel/arc_prune] > 0 0 0 13 8 0 0 5312 - DLs - 0:00.00 [kernel/acpi_task_0] > 0 0 0 11 8 0 0 5312 - DLs - 0:00.00 [kernel/acpi_task_1] > 0 0 0 21 8 0 0 5312 - DLs - 0:00.00 [kernel/acpi_task_2] > 0 0 0 27 -60 0 0 5312 - DLs - 0:00.05 [kernel/mca taskq] > 0 0 0 12 -8 0 0 5312 - DLs - 0:00.00 [kernel/CAM taskq] > 0 0 0 25 -16 0 0 5312 - DLs - 0:00.04 [kernel/z_null_iss] > 0 0 0 18 -16 0 0 5312 - DLs - 0:00.13 [kernel/z_null_int] > 0 0 0 24 -16 0 0 5312 - DLs - 0:00.00 [kernel/z_rd_iss_0] > 0 0 0 2 -16 0 0 5312 - DLs - 0:00.00 [kernel/z_rd_iss_1] > 0 0 0 22 -16 0 0 5312 - DLs - 0:00.00 [kernel/z_rd_iss_2] > 0 0 0 21 -16 0 0 5312 - DLs - 0:00.00 [kernel/z_rd_iss_3] > 0 0 0 6 -16 0 0 5312 - DLs - 0:00.00 [kernel/z_rd_iss_4] > 0 0 0 25 -16 0 0 5312 - DLs - 0:00.00 [kernel/z_rd_iss_5] > 0 0 0 10 -16 0 0 5312 - DLs - 0:00.00 [kernel/z_rd_iss_6] > 0 0 0 13 -16 0 0 5312 - DLs - 0:00.00 [kernel/z_rd_iss_7] > 0 0 0 11 -16 0 0 5312 - DLs - 0:00.02 [kernel/z_rd_int_0_0] > 0 0 0 9 -16 0 0 5312 - DLs - 0:00.05 [kernel/z_rd_int_0_1] > 0 0 0 0 -16 0 0 5312 - DLs - 0:00.09 [kernel/z_rd_int_0_2] > 0 0 0 5 -16 0 0 5312 - DLs - 0:00.15 [kernel/z_rd_int_0_3] > 0 0 0 3 -16 0 0 5312 - DLs - 0:00.05 [kernel/z_rd_int_0_4] > 0 0 0 3 -16 0 0 5312 - DLs - 0:00.19 [kernel/z_rd_int_1_0] > 0 0 0 4 -16 0 0 5312 - DLs - 0:00.06 [kernel/z_rd_int_1_1] > 0 0 0 16 -16 0 0 5312 - DLs - 0:00.01 [kernel/z_rd_int_1_2] > 0 0 0 2 -16 0 0 5312 - DLs - 0:00.03 [kernel/z_rd_int_1_3] > 0 0 0 11 -16 0 0 5312 - DLs - 0:00.07 [kernel/z_rd_int_1_4] > 0 0 0 26 -16 0 0 5312 - DLs - 0:00.00 [kernel/z_rd_int_2_0] > 0 0 0 15 -16 0 0 5312 - DLs - 0:00.10 [kernel/z_rd_int_2_1] > 0 0 0 10 -16 0 0 5312 - DLs - 0:00.02 [kernel/z_rd_int_2_2] > 0 0 0 8 -16 0 0 5312 - DLs - 0:00.08 [kernel/z_rd_int_2_3] > 0 0 0 8 -16 0 0 5312 - DLs - 0:00.17 [kernel/z_rd_int_2_4] > 0 0 0 12 -16 0 0 5312 - DLs - 0:00.02 [kernel/z_rd_int_3_0] > 0 0 0 0 -16 0 0 5312 - DLs - 0:00.14 [kernel/z_rd_int_3_1] > 0 0 0 19 -16 0 0 5312 - DLs - 0:00.11 [kernel/z_rd_int_3_2] > 0 0 0 14 -16 0 0 5312 - DLs - 0:00.06 [kernel/z_rd_int_3_3] > 0 0 0 16 -16 0 0 5312 - DLs - 0:00.05 [kernel/z_rd_int_3_4] > 0 0 0 22 -12 0 0 5312 - DLs - 0:00.03 [kernel/z_wr_iss_0] > 0 0 0 24 -12 0 0 5312 - DLs - 0:00.01 [kernel/z_wr_iss_1] > 0 0 0 0 -12 0 0 5312 - DLs - 0:00.03 [kernel/z_wr_iss_2] > 0 0 0 14 -12 0 0 5312 - DLs - 0:00.02 [kernel/z_wr_iss_3] > 0 0 0 25 -12 0 0 5312 - DLs - 0:00.03 [kernel/z_wr_iss_4] > 0 0 0 7 -12 0 0 5312 - DLs - 0:00.01 [kernel/z_wr_iss_5] > 0 0 0 3 -12 0 0 5312 - DLs - 0:00.04 [kernel/z_wr_iss_6] > 0 0 0 25 -12 0 0 5312 - DLs - 0:00.03 [kernel/z_wr_iss_7] > 0 0 0 17 -12 0 0 5312 - DLs - 0:00.01 [kernel/z_wr_iss_8] > 0 0 0 20 -12 0 0 5312 - DLs - 0:00.04 [kernel/z_wr_iss_9] > 0 0 0 4 -12 0 0 5312 - DLs - 0:00.03 [kernel/z_wr_iss_10] > 0 0 0 11 -12 0 0 5312 - DLs - 0:00.04 [kernel/z_wr_iss_11] > 0 0 0 3 -12 0 0 5312 - DLs - 0:00.00 [kernel/z_wr_iss_12] > 0 0 0 6 -12 0 0 5312 - DLs - 0:00.02 [kernel/z_wr_iss_13] > 0 0 0 8 -12 0 0 5312 - DLs - 0:00.01 [kernel/z_wr_iss_14] > 0 0 0 13 -12 0 0 5312 - DLs - 0:00.03 [kernel/z_wr_iss_15] > 0 0 0 11 -12 0 0 5312 - DLs - 0:00.03 [kernel/z_wr_iss_16] > 0 0 0 12 -12 0 0 5312 - DLs - 0:00.01 [kernel/z_wr_iss_17] > 0 0 0 14 -12 0 0 5312 - DLs - 0:00.04 [kernel/z_wr_iss_18] > 0 0 0 21 -12 0 0 5312 - DLs - 0:00.03 [kernel/z_wr_iss_19] > 0 0 0 16 -12 0 0 5312 - DLs - 0:00.01 [kernel/z_wr_iss_20] > 0 0 0 4 -12 0 0 5312 - DLs - 0:00.02 [kernel/z_wr_iss_21] > 0 0 0 24 -16 0 0 5312 - DLs - 0:00.00 [kernel/z_wr_iss_h_0] > 0 0 0 19 -16 0 0 5312 - DLs - 0:00.00 [kernel/z_wr_iss_h_1] > 0 0 0 2 -16 0 0 5312 - DLs - 0:00.00 [kernel/z_wr_iss_h_2] > 0 0 0 24 -16 0 0 5312 - DLs - 0:00.00 [kernel/z_wr_iss_h_3] > 0 0 0 10 -16 0 0 5312 - DLs - 0:00.00 [kernel/z_wr_iss_h_4] > 0 0 0 7 -16 0 0 5312 - DLs - 0:00.02 [kernel/z_wr_int_0_0] > 0 0 0 1 -16 0 0 5312 - DLs - 0:00.02 [kernel/z_wr_int_0_1] > 0 0 0 14 -16 0 0 5312 - DLs - 0:00.01 [kernel/z_wr_int_0_2] > 0 0 0 22 -16 0 0 5312 - DLs - 0:00.01 [kernel/z_wr_int_0_3] > 0 0 0 12 -16 0 0 5312 - DLs - 0:00.01 [kernel/z_wr_int_0_4] > 0 0 0 1 -16 0 0 5312 - DLs - 0:00.01 [kernel/z_wr_int_1_0] > 0 0 0 18 -16 0 0 5312 - DLs - 0:00.01 [kernel/z_wr_int_1_1] > 0 0 0 27 -16 0 0 5312 - DLs - 0:00.02 [kernel/z_wr_int_1_2] > 0 0 0 6 -16 0 0 5312 - DLs - 0:00.01 [kernel/z_wr_int_1_3] > 0 0 0 13 -16 0 0 5312 - DLs - 0:00.01 [kernel/z_wr_int_1_4] > 0 0 0 5 -16 0 0 5312 - DLs - 0:00.02 [kernel/z_wr_int_2_0] > 0 0 0 21 -16 0 0 5312 - DLs - 0:00.01 [kernel/z_wr_int_2_1] > 0 0 0 1 -16 0 0 5312 - DLs - 0:00.01 [kernel/z_wr_int_2_2] > 0 0 0 18 -16 0 0 5312 - DLs - 0:00.02 [kernel/z_wr_int_2_3] > 0 0 0 10 -16 0 0 5312 - DLs - 0:00.01 [kernel/z_wr_int_2_4] > 0 0 0 1 -16 0 0 5312 - DLs - 0:00.01 [kernel/z_wr_int_3_0] > 0 0 0 26 -16 0 0 5312 - DLs - 0:00.01 [kernel/z_wr_int_3_1] > 0 0 0 14 -16 0 0 5312 - DLs - 0:00.01 [kernel/z_wr_int_3_2] > 0 0 0 10 -16 0 0 5312 - DLs - 0:00.01 [kernel/z_wr_int_3_3] > 0 0 0 8 -16 0 0 5312 - DLs - 0:00.02 [kernel/z_wr_int_3_4] > 0 0 0 24 -16 0 0 5312 - DLs - 0:00.00 [kernel/z_wr_int_h_0] > 0 0 0 16 -16 0 0 5312 - DLs - 0:00.00 [kernel/z_wr_int_h_1] > 0 0 0 19 -16 0 0 5312 - DLs - 0:00.00 [kernel/z_wr_int_h_2] > 0 0 0 20 -16 0 0 5312 - DLs - 0:00.00 [kernel/z_wr_int_h_3] > 0 0 0 21 -16 0 0 5312 - DLs - 0:00.00 [kernel/z_wr_int_h_4] > 0 0 0 24 -16 0 0 5312 - DLs - 0:00.00 [kernel/z_fr_iss_0_0] > 0 0 0 27 -16 0 0 5312 - DLs - 0:00.00 [kernel/z_fr_iss_0_1] > 0 0 0 6 -16 0 0 5312 - DLs - 0:00.00 [kernel/z_fr_iss_0_2] > 0 0 0 16 -16 0 0 5312 - DLs - 0:00.00 [kernel/z_fr_iss_0_3] > 0 0 0 5 -16 0 0 5312 - DLs - 0:00.00 [kernel/z_fr_iss_0_4] > 0 0 0 24 -16 0 0 5312 - DLs - 0:00.00 [kernel/z_fr_iss_1_0] > 0 0 0 3 -16 0 0 5312 - DLs - 0:00.00 [kernel/z_fr_iss_1_1] > 0 0 0 14 -16 0 0 5312 - DLs - 0:00.00 [kernel/z_fr_iss_1_2] > 0 0 0 5 -16 0 0 5312 - DLs - 0:00.00 [kernel/z_fr_iss_1_3] > 0 0 0 13 -16 0 0 5312 - DLs - 0:00.00 [kernel/z_fr_iss_1_4] > 0 0 0 24 -16 0 0 5312 - DLs - 0:00.00 [kernel/z_fr_iss_2_0] > 0 0 0 27 -16 0 0 5312 - DLs - 0:00.00 [kernel/z_fr_iss_2_1] > 0 0 0 20 -16 0 0 5312 - DLs - 0:00.00 [kernel/z_fr_iss_2_2] > 0 0 0 14 -16 0 0 5312 - DLs - 0:00.00 [kernel/z_fr_iss_2_3] > 0 0 0 6 -16 0 0 5312 - DLs - 0:00.00 [kernel/z_fr_iss_2_4] > 0 0 0 24 -16 0 0 5312 - DLs - 0:00.00 [kernel/z_fr_iss_3_0] > 0 0 0 20 -16 0 0 5312 - DLs - 0:00.00 [kernel/z_fr_iss_3_1] > 0 0 0 17 -16 0 0 5312 - DLs - 0:00.00 [kernel/z_fr_iss_3_2] > 0 0 0 2 -16 0 0 5312 - DLs - 0:00.00 [kernel/z_fr_iss_3_3] > 0 0 0 19 -16 0 0 5312 - DLs - 0:00.00 [kernel/z_fr_iss_3_4] > 0 0 0 24 -16 0 0 5312 - DLs - 0:00.00 [kernel/z_fr_int] > 0 0 0 24 -16 0 0 5312 - DLs - 0:00.00 [kernel/z_cl_iss] > 0 0 0 24 -16 0 0 5312 - DLs - 0:00.00 [kernel/z_cl_int] > 0 0 0 24 -16 0 0 5312 - DLs - 0:00.00 [kernel/z_ioctl_iss] > 0 0 0 24 -16 0 0 5312 - DLs - 0:00.00 [kernel/z_ioctl_int] > 0 0 0 24 -16 0 0 5312 - DLs - 0:00.00 [kernel/z_trim_iss_0] > 0 0 0 7 -16 0 0 5312 - DLs - 0:00.00 [kernel/z_trim_iss_1] > 0 0 0 20 -16 0 0 5312 - DLs - 0:00.00 [kernel/z_trim_iss_2] > 0 0 0 15 -16 0 0 5312 - DLs - 0:00.00 [kernel/z_trim_iss_3] > 0 0 0 24 -16 0 0 5312 - DLs - 0:00.00 [kernel/z_trim_int] > 0 1 0 14 20 0 12724 744 wait ILs - 0:00.04 /sbin/init > 0 2 0 25 -60 0 0 448 - WL - 0:28.36 [clock/clock (0)] > 0 2 0 1 -60 0 0 448 - WL - 0:10.44 [clock/clock (1)] > 0 2 0 2 -60 0 0 448 - WL - 0:10.78 [clock/clock (2)] > 0 2 0 3 -60 0 0 448 - WL - 0:10.93 [clock/clock (3)] > 0 2 0 4 -60 0 0 448 - WL - 0:10.96 [clock/clock (4)] > 0 2 0 5 -60 0 0 448 - WL - 0:11.10 [clock/clock (5)] > 0 2 0 6 -60 0 0 448 - WL - 0:11.02 [clock/clock (6)] > 0 2 0 7 -60 0 0 448 - WL - 0:11.20 [clock/clock (7)] > 0 2 0 8 -60 0 0 448 - WL - 0:11.20 [clock/clock (8)] > 0 2 0 9 -60 0 0 448 - WL - 0:11.26 [clock/clock (9)] > 0 2 0 2 -60 0 0 448 - WL - 0:11.16 [clock/clock (10)] > 0 2 0 11 -60 0 0 448 - WL - 0:11.41 [clock/clock (11)] > 0 2 0 12 -60 0 0 448 - WL - 0:11.35 [clock/clock (12)] > 0 2 0 13 -60 0 0 448 - WL - 0:11.47 [clock/clock (13)] > 0 2 0 14 -60 0 0 448 - WL - 0:11.27 [clock/clock (14)] > 0 2 0 15 -60 0 0 448 - WL - 0:11.45 [clock/clock (15)] > 0 2 0 16 -60 0 0 448 - WL - 0:11.62 [clock/clock (16)] > 0 2 0 17 -60 0 0 448 - WL - 0:11.53 [clock/clock (17)] > 0 2 0 18 -60 0 0 448 - WL - 0:11.65 [clock/clock (18)] > 0 2 0 19 -60 0 0 448 - WL - 0:11.88 [clock/clock (19)] > 0 2 0 20 -60 0 0 448 - WL - 0:11.74 [clock/clock (20)] > 0 2 0 21 -60 0 0 448 - WL - 0:12.03 [clock/clock (21)] > 0 2 0 11 -60 0 0 448 - WL - 0:11.88 [clock/clock (22)] > 0 2 0 23 -60 0 0 448 - WL - 0:12.04 [clock/clock (23)] > 0 2 0 24 -60 0 0 448 - WL - 0:11.89 [clock/clock (24)] > 0 2 0 25 -60 0 0 448 - WL - 0:12.09 [clock/clock (25)] > 0 2 0 26 -60 0 0 448 - WL - 0:12.27 [clock/clock (26)] > 0 2 0 27 -60 0 0 448 - WL - 0:12.17 [clock/clock (27)] > 0 3 0 18 -16 0 0 464 crypto_w DL - 0:00.00 [crypto/crypto] > 0 3 0 18 -16 0 0 464 crypto_r DL - 0:00.00 [crypto/crypto returns 0] > 0 3 0 18 -16 0 0 464 crypto_r DL - 0:00.00 [crypto/crypto returns 1] > 0 3 0 18 -16 0 0 464 crypto_r DL - 0:00.00 [crypto/crypto returns 2] > 0 3 0 18 -16 0 0 464 crypto_r DL - 0:00.00 [crypto/crypto returns 3] > 0 3 0 18 -16 0 0 464 crypto_r DL - 0:00.00 [crypto/crypto returns 4] > 0 3 0 18 -16 0 0 464 crypto_r DL - 0:00.00 [crypto/crypto returns 5] > 0 3 0 18 -16 0 0 464 crypto_r DL - 0:00.00 [crypto/crypto returns 6] > 0 3 0 18 -16 0 0 464 crypto_r DL - 0:00.00 [crypto/crypto returns 7] > 0 3 0 18 -16 0 0 464 crypto_r DL - 0:00.00 [crypto/crypto returns 8] > 0 3 0 18 -16 0 0 464 crypto_r DL - 0:00.00 [crypto/crypto returns 9] > 0 3 0 18 -16 0 0 464 crypto_r DL - 0:00.00 [crypto/crypto returns 10] > 0 3 0 18 -16 0 0 464 crypto_r DL - 0:00.00 [crypto/crypto returns 11] > 0 3 0 18 -16 0 0 464 crypto_r DL - 0:00.00 [crypto/crypto returns 12] > 0 3 0 18 -16 0 0 464 crypto_r DL - 0:00.00 [crypto/crypto returns 13] > 0 3 0 18 -16 0 0 464 crypto_r DL - 0:00.00 [crypto/crypto returns 14] > 0 3 0 18 -16 0 0 464 crypto_r DL - 0:00.00 [crypto/crypto returns 15] > 0 3 0 18 -16 0 0 464 crypto_r DL - 0:00.00 [crypto/crypto returns 16] > 0 3 0 18 -16 0 0 464 crypto_r DL - 0:00.00 [crypto/crypto returns 17] > 0 3 0 18 -16 0 0 464 crypto_r DL - 0:00.00 [crypto/crypto returns 18] > 0 3 0 18 -16 0 0 464 crypto_r DL - 0:00.00 [crypto/crypto returns 19] > 0 3 0 18 -16 0 0 464 crypto_r DL - 0:00.00 [crypto/crypto returns 20] > 0 3 0 18 -16 0 0 464 crypto_r DL - 0:00.00 [crypto/crypto returns 21] > 0 3 0 18 -16 0 0 464 crypto_r DL - 0:00.00 [crypto/crypto returns 22] > 0 3 0 18 -16 0 0 464 crypto_r DL - 0:00.00 [crypto/crypto returns 23] > 0 3 0 18 -16 0 0 464 crypto_r DL - 0:00.00 [crypto/crypto returns 24] > 0 3 0 18 -16 0 0 464 crypto_r DL - 0:00.00 [crypto/crypto returns 25] > 0 3 0 18 -16 0 0 464 crypto_r DL - 0:00.00 [crypto/crypto returns 26] > 0 3 0 18 -16 0 0 464 crypto_r DL - 0:00.00 [crypto/crypto returns 27] > 0 4 0 6 -16 0 0 112 - DL - 0:00.00 [cam/doneq0] > 0 4 0 21 -16 0 0 112 - DL - 0:00.00 [cam/doneq1] > 0 4 0 6 -16 0 0 112 - DL - 0:00.00 [cam/doneq2] > 0 4 0 6 -16 0 0 112 - DL - 0:00.00 [cam/doneq3] > 0 4 0 6 -16 0 0 112 - DL - 0:00.00 [cam/doneq4] > 0 4 0 6 -16 0 0 112 - DL - 0:00.00 [cam/async] > 0 4 0 26 -16 0 0 112 - DL - 0:00.00 [cam/scanner] > 0 5 0 12 -60 0 0 16 - DL - 0:00.00 [busdma] > 0 6 0 17 -8 0 0 3488 t->zthr_ DL - 0:00.19 [zfskern/arc_evict] > 0 6 0 25 -8 0 0 3488 t->zthr_ DL - 0:00.25 [zfskern/arc_reap] > 0 6 0 21 -8 0 0 3488 - DL - 0:00.00 [zfskern/dbu_evict] > 0 6 0 21 -8 0 0 3488 dbuf_evi DL - 0:00.24 [zfskern/dbuf_evict_thread] > 0 6 0 7 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_vdev_file_0] > 0 6 0 18 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_vdev_file_1] > 0 6 0 25 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_vdev_file_2] > 0 6 0 10 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_vdev_file_3] > 0 6 0 13 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_vdev_file_4] > 0 6 0 5 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_vdev_file_5] > 0 6 0 17 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_vdev_file_6] > 0 6 0 11 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_vdev_file_7] > 0 6 0 21 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_vdev_file_8] > 0 6 0 13 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_vdev_file_9] > 0 6 0 4 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_vdev_file_10] > 0 6 0 14 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_vdev_file_11] > 0 6 0 17 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_vdev_file_12] > 0 6 0 6 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_vdev_file_13] > 0 6 0 10 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_vdev_file_14] > 0 6 0 2 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_vdev_file_15] > 0 6 0 13 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_vdev_file_16] > 0 6 0 25 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_vdev_file_17] > 0 6 0 17 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_vdev_file_18] > 0 6 0 11 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_vdev_file_19] > 0 6 0 0 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_vdev_file_20] > 0 6 0 19 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_vdev_file_21] > 0 6 0 22 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_vdev_file_22] > 0 6 0 10 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_vdev_file_23] > 0 6 0 21 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_vdev_file_24] > 0 6 0 8 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_vdev_file_25] > 0 6 0 12 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_vdev_file_26] > 0 6 0 16 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_vdev_file_27] > 0 6 0 23 -8 0 0 3488 l2arc_fe DL - 0:00.19 [zfskern/l2arc_feed_thread] > 0 6 0 1 -8 0 0 3488 - DL - 0:00.00 [zfskern/zfsvfs] > 0 6 0 8 -8 0 0 3488 zevent_c IL - 0:00.00 [zfskern/sysevent] > 0 6 0 20 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_zvol] > 0 6 0 20 -16 0 0 3488 - DL - 0:00.00 [zfskern/z_metaslab_0] > 0 6 0 4 -16 0 0 3488 - DL - 0:00.00 [zfskern/z_metaslab_1] > 0 6 0 15 -16 0 0 3488 - DL - 0:00.00 [zfskern/z_metaslab_2] > 0 6 0 19 -16 0 0 3488 - DL - 0:00.00 [zfskern/z_metaslab_3] > 0 6 0 9 -16 0 0 3488 - DL - 0:00.00 [zfskern/z_metaslab_4] > 0 6 0 10 -16 0 0 3488 - DL - 0:00.00 [zfskern/z_metaslab_5] > 0 6 0 24 -16 0 0 3488 - DL - 0:00.00 [zfskern/z_metaslab_6] > 0 6 0 27 -16 0 0 3488 - DL - 0:00.00 [zfskern/z_metaslab_7] > 0 6 0 17 -16 0 0 3488 - DL - 0:00.00 [zfskern/z_metaslab_8] > 0 6 0 22 -16 0 0 3488 - DL - 0:00.00 [zfskern/z_metaslab_9] > 0 6 0 13 -16 0 0 3488 - DL - 0:00.00 [zfskern/z_metaslab_10] > 0 6 0 21 -16 0 0 3488 - DL - 0:00.00 [zfskern/z_metaslab_11] > 0 6 0 2 -16 0 0 3488 - DL - 0:00.00 [zfskern/z_metaslab_12] > 0 6 0 9 -16 0 0 3488 - DL - 0:00.00 [zfskern/z_metaslab_13] > 0 6 0 20 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_prefetch_0] > 0 6 0 8 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_prefetch_1] > 0 6 0 17 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_prefetch_2] > 0 6 0 25 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_prefetch_3] > 0 6 0 2 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_prefetch_4] > 0 6 0 21 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_prefetch_5] > 0 6 0 9 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_prefetch_6] > 0 6 0 23 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_prefetch_7] > 0 6 0 26 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_prefetch_8] > 0 6 0 24 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_prefetch_9] > 0 6 0 12 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_prefetch_10] > 0 6 0 20 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_prefetch_11] > 0 6 0 18 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_prefetch_12] > 0 6 0 17 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_prefetch_13] > 0 6 0 10 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_prefetch_14] > 0 6 0 24 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_prefetch_15] > 0 6 0 12 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_prefetch_16] > 0 6 0 20 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_prefetch_17] > 0 6 0 4 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_prefetch_18] > 0 6 0 2 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_prefetch_19] > 0 6 0 19 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_prefetch_20] > 0 6 0 10 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_prefetch_21] > 0 6 0 27 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_prefetch_22] > 0 6 0 20 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_prefetch_23] > 0 6 0 15 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_prefetch_24] > 0 6 0 6 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_prefetch_25] > 0 6 0 13 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_prefetch_26] > 0 6 0 11 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_prefetch_27] > 0 6 0 20 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_upgrade_0] > 0 6 0 9 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_upgrade_1] > 0 6 0 19 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_upgrade_2] > 0 6 0 16 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_upgrade_3] > 0 6 0 12 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_upgrade_4] > 0 6 0 20 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_upgrade_5] > 0 6 0 22 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_upgrade_6] > 0 6 0 8 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_upgrade_7] > 0 6 0 13 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_upgrade_8] > 0 6 0 25 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_upgrade_9] > 0 6 0 21 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_upgrade_10] > 0 6 0 10 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_upgrade_11] > 0 6 0 18 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_upgrade_12] > 0 6 0 22 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_upgrade_13] > 0 6 0 20 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_upgrade_14] > 0 6 0 24 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_upgrade_15] > 0 6 0 5 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_upgrade_16] > 0 6 0 13 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_upgrade_17] > 0 6 0 9 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_upgrade_18] > 0 6 0 20 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_upgrade_19] > 0 6 0 15 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_upgrade_20] > 0 6 0 22 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_upgrade_21] > 0 6 0 2 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_upgrade_22] > 0 6 0 10 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_upgrade_23] > 0 6 0 20 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_upgrade_24] > 0 6 0 6 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_upgrade_25] > 0 6 0 23 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_upgrade_26] > 0 6 0 11 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_upgrade_27] > 0 6 0 19 -8 0 0 3488 - DL - 0:00.01 [zfskern/dp_sync_taskq_0] > 0 6 0 14 -8 0 0 3488 - DL - 0:00.02 [zfskern/dp_sync_taskq_1] > 0 6 0 21 -8 0 0 3488 - DL - 0:00.01 [zfskern/dp_sync_taskq_2] > 0 6 0 23 -8 0 0 3488 - DL - 0:00.00 [zfskern/dp_sync_taskq_3] > 0 6 0 17 -8 0 0 3488 - DL - 0:00.01 [zfskern/dp_sync_taskq_4] > 0 6 0 4 -8 0 0 3488 - DL - 0:00.00 [zfskern/dp_sync_taskq_5] > 0 6 0 27 -8 0 0 3488 - DL - 0:00.01 [zfskern/dp_sync_taskq_6] > 0 6 0 13 -8 0 0 3488 - DL - 0:00.01 [zfskern/dp_sync_taskq_7] > 0 6 0 16 -8 0 0 3488 - DL - 0:00.01 [zfskern/dp_sync_taskq_8] > 0 6 0 9 -8 0 0 3488 - DL - 0:00.01 [zfskern/dp_sync_taskq_9] > 0 6 0 26 -8 0 0 3488 - DL - 0:00.01 [zfskern/dp_sync_taskq_10] > 0 6 0 23 -8 0 0 3488 - DL - 0:00.01 [zfskern/dp_sync_taskq_11] > 0 6 0 25 -8 0 0 3488 - DL - 0:00.01 [zfskern/dp_sync_taskq_12] > 0 6 0 8 -8 0 0 3488 - DL - 0:00.02 [zfskern/dp_sync_taskq_13] > 0 6 0 2 -8 0 0 3488 - DL - 0:00.00 [zfskern/dp_sync_taskq_14] > 0 6 0 14 -8 0 0 3488 - DL - 0:00.00 [zfskern/dp_sync_taskq_15] > 0 6 0 15 -8 0 0 3488 - DL - 0:00.02 [zfskern/dp_sync_taskq_16] > 0 6 0 22 -8 0 0 3488 - DL - 0:00.01 [zfskern/dp_sync_taskq_17] > 0 6 0 12 -8 0 0 3488 - DL - 0:00.01 [zfskern/dp_sync_taskq_18] > 0 6 0 24 -8 0 0 3488 - DL - 0:00.01 [zfskern/dp_sync_taskq_19] > 0 6 0 3 -8 0 0 3488 - DL - 0:00.01 [zfskern/dp_sync_taskq_20] > 0 6 0 20 -8 0 0 3488 - DL - 0:00.00 [zfskern/dp_zil_clean_taskq_] > 0 6 0 11 -8 0 0 3488 - DL - 0:00.00 [zfskern/dp_zil_clean_taskq_] > 0 6 0 16 -8 0 0 3488 - DL - 0:00.00 [zfskern/dp_zil_clean_taskq_] > 0 6 0 27 -8 0 0 3488 - DL - 0:00.00 [zfskern/dp_zil_clean_taskq_] > 0 6 0 3 -8 0 0 3488 - DL - 0:00.00 [zfskern/dp_zil_clean_taskq_] > 0 6 0 20 -8 0 0 3488 - DL - 0:00.00 [zfskern/dp_zil_clean_taskq_] > 0 6 0 19 -8 0 0 3488 - DL - 0:00.00 [zfskern/dp_zil_clean_taskq_] > 0 6 0 16 -8 0 0 3488 - DL - 0:00.00 [zfskern/dp_zil_clean_taskq_] > 0 6 0 3 -8 0 0 3488 - DL - 0:00.00 [zfskern/dp_zil_clean_taskq_] > 0 6 0 21 -8 0 0 3488 - DL - 0:00.00 [zfskern/dp_zil_clean_taskq_] > 0 6 0 7 -8 0 0 3488 - DL - 0:00.00 [zfskern/dp_zil_clean_taskq_] > 0 6 0 11 -8 0 0 3488 - DL - 0:00.00 [zfskern/dp_zil_clean_taskq_] > 0 6 0 5 -8 0 0 3488 - DL - 0:00.00 [zfskern/dp_zil_clean_taskq_] > 0 6 0 20 -8 0 0 3488 - DL - 0:00.00 [zfskern/dp_zil_clean_taskq_] > 0 6 0 22 -8 0 0 3488 - DL - 0:00.00 [zfskern/dp_zil_clean_taskq_] > 0 6 0 15 -8 0 0 3488 - DL - 0:00.00 [zfskern/dp_zil_clean_taskq_] > 0 6 0 25 -8 0 0 3488 - DL - 0:00.00 [zfskern/dp_zil_clean_taskq_] > 0 6 0 8 -8 0 0 3488 - DL - 0:00.00 [zfskern/dp_zil_clean_taskq_] > 0 6 0 20 -8 0 0 3488 - DL - 0:00.00 [zfskern/dp_zil_clean_taskq_] > 0 6 0 26 -8 0 0 3488 - DL - 0:00.00 [zfskern/dp_zil_clean_taskq_] > 0 6 0 3 -8 0 0 3488 - DL - 0:00.00 [zfskern/dp_zil_clean_taskq_] > 0 6 0 16 -8 0 0 3488 - DL - 0:00.00 [zfskern/dp_zil_clean_taskq_] > 0 6 0 13 -8 0 0 3488 - DL - 0:00.00 [zfskern/dp_zil_clean_taskq_] > 0 6 0 21 -8 0 0 3488 - DL - 0:00.00 [zfskern/dp_zil_clean_taskq_] > 0 6 0 12 -8 0 0 3488 - DL - 0:00.00 [zfskern/dp_zil_clean_taskq_] > 0 6 0 22 -8 0 0 3488 - DL - 0:00.00 [zfskern/dp_zil_clean_taskq_] > 0 6 0 20 -8 0 0 3488 - DL - 0:00.00 [zfskern/dp_zil_clean_taskq_] > 0 6 0 20 -8 0 0 3488 - DL - 0:00.00 [zfskern/dp_zil_clean_taskq_] > 0 6 0 20 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_zrele_0] > 0 6 0 13 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_zrele_1] > 0 6 0 14 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_zrele_2] > 0 6 0 10 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_zrele_3] > 0 6 0 18 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_zrele_4] > 0 6 0 20 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_zrele_5] > 0 6 0 2 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_zrele_6] > 0 6 0 16 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_zrele_7] > 0 6 0 9 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_zrele_8] > 0 6 0 13 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_zrele_9] > 0 6 0 20 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_zrele_10] > 0 6 0 15 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_zrele_11] > 0 6 0 16 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_zrele_12] > 0 6 0 6 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_zrele_13] > 0 6 0 12 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_zrele_14] > 0 6 0 21 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_zrele_15] > 0 6 0 2 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_zrele_16] > 0 6 0 18 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_zrele_17] > 0 6 0 17 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_zrele_18] > 0 6 0 24 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_zrele_19] > 0 6 0 22 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_zrele_20] > 0 6 0 20 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_zrele_21] > 0 6 0 15 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_zrele_22] > 0 6 0 9 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_zrele_23] > 0 6 0 11 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_zrele_24] > 0 6 0 26 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_zrele_25] > 0 6 0 21 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_zrele_26] > 0 6 0 24 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_zrele_27] > 0 6 0 20 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_unlinked_drain_0] > 0 6 0 5 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_unlinked_drain_1] > 0 6 0 26 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_unlinked_drain_2] > 0 6 0 11 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_unlinked_drain_3] > 0 6 0 12 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_unlinked_drain_4] > 0 6 0 21 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_unlinked_drain_5] > 0 6 0 9 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_unlinked_drain_6] > 0 6 0 22 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_unlinked_drain_7] > 0 6 0 11 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_unlinked_drain_8] > 0 6 0 17 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_unlinked_drain_9] > 0 6 0 20 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_unlinked_drain_10] > 0 6 0 5 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_unlinked_drain_11] > 0 6 0 2 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_unlinked_drain_12] > 0 6 0 9 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_unlinked_drain_13] > 0 6 0 7 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_unlinked_drain_14] > 0 6 0 25 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_unlinked_drain_15] > 0 6 0 20 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_unlinked_drain_16] > 0 6 0 2 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_unlinked_drain_17] > 0 6 0 27 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_unlinked_drain_18] > 0 6 0 12 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_unlinked_drain_19] > 0 6 0 11 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_unlinked_drain_20] > 0 6 0 16 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_unlinked_drain_21] > 0 6 0 20 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_unlinked_drain_22] > 0 6 0 7 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_unlinked_drain_23] > 0 6 0 23 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_unlinked_drain_24] > 0 6 0 11 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_unlinked_drain_25] > 0 6 0 17 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_unlinked_drain_26] > 0 6 0 21 -8 0 0 3488 - DL - 0:00.00 [zfskern/z_unlinked_drain_27] > 0 6 0 24 -8 0 0 3488 tx->tx_q DL - 0:00.09 [zfskern/txg_thread_enter] > 0 6 0 19 -8 0 0 3488 tx->tx_s DL - 0:00.89 [zfskern/txg_thread_enter] > 0 6 0 20 -8 0 0 3488 mmp->mmp DL - 0:00.30 [zfskern/mmp_thread_enter] > 0 6 0 20 -8 0 0 3488 t->zthr_ DL - 0:00.00 [zfskern/z_indirect_condense] > 0 6 0 20 -8 0 0 3488 t->zthr_ DL - 0:00.00 [zfskern/z_livelist_destroy] > 0 6 0 20 -8 0 0 3488 t->zthr_ DL - 0:00.00 [zfskern/z_livelist_condense] > 0 6 0 20 -8 0 0 3488 t->zthr_ DL - 0:00.00 [zfskern/z_checkpoint_discar] > 0 7 0 1 -16 0 0 16 - DL - 0:05.46 [rand_harvestq] > 0 8 0 25 -16 0 0 64 psleep DL - 1:03.76 [pagedaemon/dom0] > 0 8 0 9 -16 0 0 64 psleep DL - 0:00.00 [pagedaemon/dom0 helper0] > 0 8 0 4 -16 0 0 64 launds DL - 0:00.00 [pagedaemon/laundry: dom0] > 0 8 0 15 -16 0 0 64 umarcl DL - 0:00.16 [pagedaemon/uma] > 0 9 0 4 -16 0 0 16 psleep DL - 0:03.93 [vmdaemon] > 0 10 0 6 -16 0 0 16 audit_wo DL - 0:00.00 [audit] > 0 11 0 0 187 0 0 448 - RNL - 889:52.95 [idle/idle: cpu0] > 0 11 0 1 187 0 0 448 - RNL - 884:05.68 [idle/idle: cpu1] > 0 11 0 2 187 0 0 448 - RNL - 885:30.00 [idle/idle: cpu2] > 0 11 0 3 187 0 0 448 - RNL - 884:12.10 [idle/idle: cpu3] > 0 11 0 4 187 0 0 448 - RNL - 886:37.60 [idle/idle: cpu4] > 0 11 0 5 187 0 0 448 - RNL - 883:50.60 [idle/idle: cpu5] > 0 11 0 6 187 0 0 448 - RNL - 887:12.11 [idle/idle: cpu6] > 0 11 0 7 187 0 0 448 - RNL - 883:30.21 [idle/idle: cpu7] > 0 11 0 8 187 0 0 448 - RNL - 886:22.98 [idle/idle: cpu8] > 0 11 0 9 187 0 0 448 - RNL - 883:00.83 [idle/idle: cpu9] > 0 11 0 10 187 0 0 448 - RNL - 885:41.28 [idle/idle: cpu10] > 0 11 0 11 187 0 0 448 - RNL - 882:59.00 [idle/idle: cpu11] > 0 11 0 12 187 0 0 448 - RNL - 885:32.54 [idle/idle: cpu12] > 0 11 0 13 187 0 0 448 - RNL - 882:53.12 [idle/idle: cpu13] > 0 11 0 14 187 0 0 448 - RNL - 884:41.62 [idle/idle: cpu14] > 0 11 0 15 187 0 0 448 - RNL - 882:19.53 [idle/idle: cpu15] > 0 11 0 16 187 0 0 448 - RNL - 888:50.82 [idle/idle: cpu16] > 0 11 0 17 187 0 0 448 - RNL - 881:50.35 [idle/idle: cpu17] > 0 11 0 18 187 0 0 448 - RNL - 887:02.89 [idle/idle: cpu18] > 0 11 0 19 187 0 0 448 - RNL - 881:44.35 [idle/idle: cpu19] > 0 11 0 20 187 0 0 448 - RNL - 883:19.66 [idle/idle: cpu20] > 0 11 0 21 187 0 0 448 - RNL - 882:00.54 [idle/idle: cpu21] > 0 11 0 22 187 0 0 448 - RNL - 884:20.75 [idle/idle: cpu22] > 0 11 0 23 187 0 0 448 - RNL - 881:32.94 [idle/idle: cpu23] > 0 11 0 24 187 0 0 448 - RNL - 883:15.99 [idle/idle: cpu24] > 0 11 0 25 187 0 0 448 - RNL - 881:09.17 [idle/idle: cpu25] > 0 11 0 26 187 0 0 448 - RNL - 889:05.08 [idle/idle: cpu26] > 0 11 0 27 187 0 0 448 - RNL - 880:38.96 [idle/idle: cpu27] > 0 12 0 7 -60 0 0 528 - WL - 0:26.42 [intr/swi6: task queue] > 0 12 0 6 -60 0 0 528 - WL - 0:00.99 [intr/swi6: Giant taskq] > 0 12 0 -1 -60 0 0 528 - WL - 0:00.00 [intr/swi5: fast taskq] > 0 12 0 25 -60 0 0 528 - WL - 0:00.00 [intr/swi1: netisr 0] > 0 12 0 23 -64 0 0 528 - WL - 0:00.00 [intr/irq56: nvme0:admin] > 0 12 0 27 -64 0 0 528 - WL - 0:00.01 [intr/irq71: hdac0] > 0 12 0 2 -64 0 0 528 - WL - 0:00.00 [intr/irq102: xhci0] > 0 12 0 4 -64 0 0 528 - WL - 0:00.89 [intr/irq18: ehci0 ehci1+] > 0 12 0 6 -64 0 0 528 - WL - 0:00.02 [intr/irq103: hdac1] > 0 12 0 -1 -64 0 0 528 - WL - 0:00.00 [intr/irq104: re0] > 0 12 0 10 -60 0 0 528 - WL - 0:00.97 [intr/irq1: atkbd0] > 0 12 0 1 -64 0 0 528 - WL - 0:00.02 [intr/irq57: nvme0:io0] > 0 12 0 3 -64 0 0 528 - WL - 0:00.01 [intr/irq58: nvme0:io1] > 0 12 0 5 -64 0 0 528 - WL - 0:00.02 [intr/irq59: nvme0:io2] > 0 12 0 7 -64 0 0 528 - WL - 0:00.02 [intr/irq60: nvme0:io3] > 0 12 0 9 -64 0 0 528 - WL - 0:00.01 [intr/irq61: nvme0:io4] > 0 12 0 11 -64 0 0 528 - WL - 0:00.01 [intr/irq62: nvme0:io5] > 0 12 0 13 -64 0 0 528 - WL - 0:00.02 [intr/irq63: nvme0:io6] > 0 12 0 15 -64 0 0 528 - WL - 0:00.01 [intr/irq64: nvme0:io7] > 0 12 0 17 -64 0 0 528 - WL - 0:00.01 [intr/irq65: nvme0:io8] > 0 12 0 19 -64 0 0 528 - WL - 0:00.02 [intr/irq66: nvme0:io9] > 0 12 0 21 -64 0 0 528 - WL - 0:00.01 [intr/irq67: nvme0:io10] > 0 12 0 23 -64 0 0 528 - WL - 0:00.02 [intr/irq68: nvme0:io11] > 0 12 0 25 -64 0 0 528 - WL - 0:00.01 [intr/irq69: nvme0:io12] > 0 12 0 27 -64 0 0 528 - WL - 0:00.01 [intr/irq70: nvme0:io13] > 0 12 0 -1 -64 0 0 528 - WL - 0:00.00 [intr/irq105: ioat0] > 0 12 0 -1 -64 0 0 528 - WL - 0:00.00 [intr/irq106: ioat1] > 0 12 0 -1 -64 0 0 528 - WL - 0:00.00 [intr/irq107: ioat2] > 0 12 0 -1 -64 0 0 528 - WL - 0:00.00 [intr/irq108: ioat3] > 0 12 0 -1 -64 0 0 528 - WL - 0:00.00 [intr/irq109: ioat4] > 0 12 0 -1 -64 0 0 528 - WL - 0:00.00 [intr/irq110: ioat5] > 0 12 0 -1 -64 0 0 528 - WL - 0:00.00 [intr/irq111: ioat6] > 0 12 0 -1 -64 0 0 528 - WL - 0:00.00 [intr/irq112: ioat7] > 0 13 0 25 -8 0 0 48 - DL - 0:00.00 [geom/g_event] > 0 13 0 15 -8 0 0 48 - DL - 0:00.00 [geom/g_up] > 0 13 0 15 -8 0 0 48 - DL - 0:00.00 [geom/g_down] > 0 14 0 22 -16 0 0 16 seqstate DL - 0:00.00 [sequencer 00] > 0 15 0 14 -60 0 0 240 - DL - 0:00.99 [usb/usbus0] > 0 15 0 9 -60 0 0 240 - DL - 0:00.00 [usb/usbus0] > 0 15 0 9 -60 0 0 240 - DL - 0:00.00 [usb/usbus0] > 0 15 0 12 -60 0 0 240 - DL - 0:02.56 [usb/usbus0] > 0 15 0 26 -60 0 0 240 - DL - 0:00.00 [usb/usbus0] > 0 15 0 9 -60 0 0 240 - DL - 0:00.00 [usb/usbus1] > 0 15 0 9 -60 0 0 240 - DL - 0:00.00 [usb/usbus1] > 0 15 0 9 -60 0 0 240 - DL - 0:00.00 [usb/usbus1] > 0 15 0 22 -60 0 0 240 - DL - 0:02.22 [usb/usbus1] > 0 15 0 1 -60 0 0 240 - DL - 0:00.59 [usb/usbus1] > 0 15 0 9 -60 0 0 240 - DL - 0:00.00 [usb/usbus2] > 0 15 0 9 -60 0 0 240 - DL - 0:00.00 [usb/usbus2] > 0 15 0 9 -60 0 0 240 - DL - 0:00.00 [usb/usbus2] > 0 15 0 5 -60 0 0 240 - DL - 0:02.25 [usb/usbus2] > 0 15 0 11 -60 0 0 240 - DL - 0:00.71 [usb/usbus2] > 0 16 0 5 20 0 0 128 psleep DL - 0:38.45 [bufdaemon/bufdaemon] > 0 16 0 21 20 0 0 128 - DL - 0:00.29 [bufdaemon/bufspacedaemon-0] > 0 16 0 14 20 0 0 128 - DL - 0:00.19 [bufdaemon/bufspacedaemon-1] > 0 16 0 24 20 0 0 128 - DL - 0:00.69 [bufdaemon/bufspacedaemon-2] > 0 16 0 0 20 0 0 128 - DL - 0:00.27 [bufdaemon/bufspacedaemon-3] > 0 16 0 7 20 0 0 128 - DL - 0:03.60 [bufdaemon/bufspacedaemon-4] > 0 16 0 2 20 0 0 128 bufspace DL - 0:09.28 [bufdaemon/bufspacedaemon-5] > 0 16 0 16 20 0 0 128 - DL - 0:01.20 [bufdaemon/bufspacedaemon-6] > 0 17 0 8 -16 0 0 16 vlruwt DL - 0:00.19 [vnlru] > 0 18 0 23 16 0 0 16 syncer DL - 0:10.31 [syncer] > 0 19 0 0 -16 0 0 16 - DL - 0:22.35 [racctd] > 0 4089 1 3 20 0 15444 2808 select Is - 0:00.06 /sbin/devd > 0 4259 1 23 20 0 13904 2404 select Ss - 0:00.11 /usr/sbin/syslogd -N -O rfc5424 > 0 4294 1 13 24 0 14140 1920 pause Is - 0:00.00 nfsuserd: master (nfsuserd) > 0 4295 4294 11 20 0 14140 1920 select S - 0:00.02 nfsuserd: server (nfsuserd) > 0 4296 4294 9 20 0 14140 1920 select S - 0:00.02 nfsuserd: server (nfsuserd) > 0 4297 4294 6 20 0 14140 1920 select S - 0:00.02 nfsuserd: server (nfsuserd) > 0 4298 4294 11 20 0 14140 1920 select S - 0:00.02 nfsuserd: server (nfsuserd) > 0 4322 0 16 -16 0 0 16 nfscl DL - 0:00.53 [nfscl] > 0 4324 0 16 -16 0 0 16 nfscl DL - 0:00.40 [nfscl] > 0 4326 0 17 -16 0 0 16 nfscl DL - 0:00.45 [nfscl] > 0 4699 1 25 20 0 13828 2168 select Ss - 0:00.04 /usr/sbin/rpcbind > 0 4707 1 24 20 0 275952 2004 select Ss - 0:00.03 /usr/sbin/rpc.statd > 0 4737 1 6 68 0 13860 1956 rpcsvc Ss - 0:00.06 /usr/sbin/rpc.lockd > 123 4815 1 11 20 0 20336 4504 select Ss - 0:01.92 /usr/sbin/ntpd -p /var/db/ntp/ntpd.pid -c /etc/ntp.conf -f > 0 4825 1 27 20 0 13848 1872 select Ss - 0:03.35 /usr/sbin/powerd -a adp -N > 0 4878 1 26 20 0 13944 2148 nanslp Is - 0:00.15 /usr/sbin/cron -s > 0 4970 1 15 20 0 23508 6804 select Is - 0:00.01 sshd: /usr/sbin/sshd [listener] 0 of 10-100 startups (sshd > 65534 5000 1 17 68 0 13856 1944 kqread Is - 0:00.00 daemon: /usr/bin/env[5001] (daemon) > 65534 5001 5000 4 20 0 1263428 19068 uwait S - 0:22.96 /usr/local/bin/node_exporter --web.listen-address=:9100 -- > 65534 5001 5000 27 20 0 1263428 19068 uwait S - 0:02.60 /usr/local/bin/node_exporter --web.listen-address=:9100 -- > 65534 5001 5000 10 20 0 1263428 19068 uwait I - 0:20.45 /usr/local/bin/node_exporter --web.listen-address=:9100 -- > 65534 5001 5000 17 68 0 1263428 19068 uwait I - 0:00.00 /usr/local/bin/node_exporter --web.listen-address=:9100 -- > 65534 5001 5000 0 20 0 1263428 19068 kqread S - 0:21.42 /usr/local/bin/node_exporter --web.listen-address=:9100 -- > 65534 5001 5000 24 68 0 1263428 19068 uwait I - 0:00.00 /usr/local/bin/node_exporter --web.listen-address=:9100 -- > 0 5057 1 3 20 0 16528 3156 nanslp I - 0:00.00 /usr/local/sbin/smartd -c /usr/local/etc/smartd.conf -p /v > 0 5378 0 22 -60 0 0 16 nfsreq DL - 0:35.89 [newnfs 0] > 0 5416 0 2 -16 0 0 16 nfsreq DL - 0:10.91 [newnfs 1] > 0 5428 1 0 68 0 14404 3208 wait I - 0:00.02 /bin/sh /usr/local/sbin/vm _run haos > 0 5645 5428 23 20 0 4272676 3980904 kqread IC - 0:30.51 bhyve: haos (bhyve) > 0 5645 5428 18 20 0 4272676 3980904 nfs DC - 0:24.69 bhyve: haos (bhyve) > 0 5645 5428 3 20 0 4272676 3980904 nfs DC - 0:24.05 bhyve: haos (bhyve) > 0 5645 5428 19 20 0 4272676 3980904 newbuf DC - 0:24.21 bhyve: haos (bhyve) > 0 5645 5428 25 20 0 4272676 3980904 nfs DC - 0:24.51 bhyve: haos (bhyve) > 0 5645 5428 20 20 0 4272676 3980904 newbuf DC - 0:24.46 bhyve: haos (bhyve) > 0 5645 5428 0 20 0 4272676 3980904 nfs DC - 0:24.16 bhyve: haos (bhyve) > 0 5645 5428 26 20 0 4272676 3980904 nfs DC - 0:24.27 bhyve: haos (bhyve) > 0 5645 5428 13 20 0 4272676 3980904 nfs DC - 0:24.07 bhyve: haos (bhyve) > 0 5645 5428 13 32 0 4272676 3980904 uwait IC - 0:00.00 bhyve: haos (bhyve) > 0 5645 5428 25 20 0 4272676 3980904 uwait IC - 0:34.08 bhyve: haos (bhyve) > 0 5645 5428 14 68 0 4272676 3980904 accept IC - 0:00.00 bhyve: haos (bhyve) > 0 5645 5428 27 68 0 4272676 3980904 usem IC - 21:09.26 bhyve: haos (bhyve) > 0 5645 5428 27 21 0 4272676 3980904 usem IC - 24:33.53 bhyve: haos (bhyve) > 0 5645 5428 6 24 0 4272676 3980904 usem IC - 26:55.54 bhyve: haos (bhyve) > 0 5645 5428 7 21 0 4272676 3980904 usem IC - 21:40.88 bhyve: haos (bhyve) > 0 5685 4813 17 20 0 3208884 2745780 kqread SC - 9:14.15 bhyve: fw (bhyve) > 0 5685 4813 17 20 0 3208884 2745780 uwait SC - 0:08.02 bhyve: fw (bhyve) > 0 5685 4813 5 20 0 3208884 2745780 uwait SC - 0:07.93 bhyve: fw (bhyve) > 0 5685 4813 9 20 0 3208884 2745780 uwait SC - 0:07.79 bhyve: fw (bhyve) > 0 5685 4813 25 20 0 3208884 2745780 uwait SC - 0:07.93 bhyve: fw (bhyve) > 0 5685 4813 26 20 0 3208884 2745780 uwait SC - 0:07.92 bhyve: fw (bhyve) > 0 5685 4813 22 20 0 3208884 2745780 uwait SC - 0:07.84 bhyve: fw (bhyve) > 0 5685 4813 3 20 0 3208884 2745780 uwait SC - 0:07.72 bhyve: fw (bhyve) > 0 5685 4813 12 20 0 3208884 2745780 uwait SC - 0:07.72 bhyve: fw (bhyve) > 0 5685 4813 16 28 0 3208884 2745780 uwait IC - 0:00.00 bhyve: fw (bhyve) > 0 5685 4813 2 20 0 3208884 2745780 uwait SC - 0:02.75 bhyve: fw (bhyve) > 0 5685 4813 13 20 0 3208884 2745780 uwait SC - 0:29.38 bhyve: fw (bhyve) > 0 5685 4813 6 20 0 3208884 2745780 uwait SC - 1:53.56 bhyve: fw (bhyve) > 0 5685 4813 20 20 0 3208884 2745780 uwait SC - 3:05.02 bhyve: fw (bhyve) > 0 5685 4813 0 20 0 3208884 2745780 uwait IC - 0:55.14 bhyve: fw (bhyve) > 0 5685 4813 12 20 0 3208884 2745780 uwait SC - 0:12.32 bhyve: fw (bhyve) > 0 5685 4813 2 20 0 3208884 2745780 uwait SC - 0:20.05 bhyve: fw (bhyve) > 0 5685 4813 26 20 0 3208884 2745780 uwait SC - 0:29.10 bhyve: fw (bhyve) > 0 5685 4813 27 20 0 3208884 2745780 uwait SC - 0:07.60 bhyve: fw (bhyve) > 0 5685 4813 26 20 0 3208884 2745780 uwait SC - 0:57.11 bhyve: fw (bhyve) > 0 5685 4813 24 23 0 3208884 2745780 vmidle DC - 57:40.06 bhyve: fw (bhyve) > 0 5685 4813 25 21 0 3208884 2745780 vmidle DC - 38:19.64 bhyve: fw (bhyve) > 0 5685 4813 0 21 0 3208884 2745780 vmidle DC - 44:22.95 bhyve: fw (bhyve) > 0 5685 4813 15 21 0 3208884 2745780 vmidle DC - 43:29.83 bhyve: fw (bhyve) > 0 5760 0 1 -16 0 0 16 nfsreq DL - 0:07.24 [newnfs 2] > 0 5761 0 17 -16 0 0 16 nfsreq DL - 0:05.86 [newnfs 3] > 0 5762 0 14 -16 0 0 16 nfsreq DL - 0:05.11 [newnfs 4] > 0 5763 0 24 -16 0 0 16 nfsreq DL - 0:04.52 [newnfs 5] > 0 5764 0 3 -16 0 0 16 nfsreq DL - 0:04.10 [newnfs 6] > 0 5765 0 8 -60 0 0 16 nfsreq DL - 0:03.78 [newnfs 7] > 0 5766 0 15 -16 0 0 16 nfsreq DL - 0:03.27 [newnfs 8] > 0 5767 0 11 -16 0 0 16 nfsreq DL - 0:02.62 [newnfs 9] > 0 5768 0 22 -16 0 0 16 nfsreq DL - 0:02.28 [newnfs 10] > 0 5769 0 1 -60 0 0 16 nfsreq DL - 0:02.05 [newnfs 11] > 0 5770 0 9 -16 0 0 16 nfsreq DL - 0:01.85 [newnfs 12] > 0 5771 0 19 -16 0 0 16 nfsreq DL - 0:01.73 [newnfs 13] > 0 5772 0 13 -16 0 0 16 nfsreq DL - 0:01.58 [newnfs 14] > 0 5784 1 3 68 0 14404 3208 wait I - 0:00.02 /bin/sh /usr/local/sbin/vm _run sound > 0 5999 5784 0 20 0 2167604 995796 kqread SC - 0:02.35 bhyve: sound (bhyve) > 0 5999 5784 10 20 0 2167604 995796 uwait SC - 0:05.48 bhyve: sound (bhyve) > 0 5999 5784 5 20 0 2167604 995796 uwait SC - 0:05.50 bhyve: sound (bhyve) > 0 5999 5784 25 20 0 2167604 995796 uwait SC - 0:05.53 bhyve: sound (bhyve) > 0 5999 5784 7 20 0 2167604 995796 uwait SC - 0:05.45 bhyve: sound (bhyve) > 0 5999 5784 15 20 0 2167604 995796 uwait SC - 0:05.39 bhyve: sound (bhyve) > 0 5999 5784 3 20 0 2167604 995796 uwait SC - 0:05.54 bhyve: sound (bhyve) > 0 5999 5784 26 20 0 2167604 995796 uwait SC - 0:05.50 bhyve: sound (bhyve) > 0 5999 5784 21 20 0 2167604 995796 uwait SC - 0:05.49 bhyve: sound (bhyve) > 0 5999 5784 13 20 0 2167604 995796 uwait IC - 0:00.00 bhyve: sound (bhyve) > 0 5999 5784 15 20 0 2167604 995796 uwait SC - 0:01.36 bhyve: sound (bhyve) > 0 5999 5784 8 68 0 2167604 995796 accept IC - 0:00.00 bhyve: sound (bhyve) > 0 5999 5784 18 21 0 2167604 995796 vmidle DC - 11:37.76 bhyve: sound (bhyve) > 0 5999 5784 2 20 0 2167604 995796 vmidle DC - 10:05.81 bhyve: sound (bhyve) > 0 6046 1 0 68 0 14404 3208 wait I - 0:00.02 /bin/sh /usr/local/sbin/vm _run git > 0 6281 6046 8 20 0 1107236 901388 kqread SC - 0:02.96 bhyve: git (bhyve) > 0 6281 6046 21 20 0 1107236 901388 uwait SC - 0:05.85 bhyve: git (bhyve) > 0 6281 6046 26 20 0 1107236 901388 uwait SC - 0:05.76 bhyve: git (bhyve) > 0 6281 6046 14 20 0 1107236 901388 uwait SC - 0:05.63 bhyve: git (bhyve) > 0 6281 6046 20 20 0 1107236 901388 uwait SC - 0:05.81 bhyve: git (bhyve) > 0 6281 6046 5 20 0 1107236 901388 uwait SC - 0:05.72 bhyve: git (bhyve) > 0 6281 6046 1 20 0 1107236 901388 uwait SC - 0:05.77 bhyve: git (bhyve) > 0 6281 6046 23 20 0 1107236 901388 uwait SC - 0:05.61 bhyve: git (bhyve) > 0 6281 6046 7 20 0 1107236 901388 uwait SC - 0:05.85 bhyve: git (bhyve) > 0 6281 6046 13 26 0 1107236 901388 uwait IC - 0:00.00 bhyve: git (bhyve) > 0 6281 6046 23 20 0 1107236 901388 uwait IC - 0:00.20 bhyve: git (bhyve) > 0 6281 6046 13 20 0 1107236 901388 uwait IC - 0:00.21 bhyve: git (bhyve) > 0 6281 6046 18 20 0 1107236 901388 uwait IC - 0:00.19 bhyve: git (bhyve) > 0 6281 6046 17 20 0 1107236 901388 uwait IC - 0:00.20 bhyve: git (bhyve) > 0 6281 6046 22 20 0 1107236 901388 uwait SC - 0:00.23 bhyve: git (bhyve) > 0 6281 6046 23 20 0 1107236 901388 uwait IC - 0:00.17 bhyve: git (bhyve) > 0 6281 6046 13 20 0 1107236 901388 uwait SC - 0:00.21 bhyve: git (bhyve) > 0 6281 6046 0 20 0 1107236 901388 uwait IC - 0:00.18 bhyve: git (bhyve) > 0 6281 6046 1 26 0 1107236 901388 uwait IC - 0:00.00 bhyve: git (bhyve) > 0 6281 6046 0 20 0 1107236 901388 uwait SC - 0:02.16 bhyve: git (bhyve) > 0 6281 6046 3 20 0 1107236 901388 vmidle DC - 9:38.26 bhyve: git (bhyve) > 0 6281 6046 20 20 0 1107236 901388 vmidle DC - 9:43.11 bhyve: git (bhyve) > 0 9629 4970 17 21 0 24040 7408 select Is - 0:00.01 sshd-session: ale [priv] (sshd-session) > 1001 9632 9629 9 20 0 24040 7692 select S - 0:00.33 sshd-session: ale@pts/2 (sshd-session) > 0 13365 0 9 -16 0 0 16 - IL - 0:00.03 [newnfs 15] > 0 13556 13555 4 20 0 24416 10556 select Ss - 0:00.05 screen -RR -D (screen-5.0.1) > 0 13740 0 5 -16 0 0 16 - IL - 0:00.00 [newnfs 16] > 0 4813 1 13 68 0 14404 3212 wait I v0- 0:00.05 /bin/sh /usr/local/sbin/vm _run fw > 0 5405 1 21 20 0 13872 1900 ttyin Is+ v0 0:00.00 /usr/libexec/getty Pc ttyv0 > 0 5406 1 3 68 0 13872 1904 ttyin Is+ v1 0:00.00 /usr/libexec/getty Pc ttyv1 > 0 5407 1 19 68 0 13872 1896 ttyin Is+ v2 0:00.00 /usr/libexec/getty Pc ttyv2 > 0 5408 1 16 68 0 13872 1892 ttyin Is+ v3 0:00.00 /usr/libexec/getty Pc ttyv3 > 0 5409 1 4 68 0 13872 1900 ttyin Is+ v4 0:00.00 /usr/libexec/getty Pc ttyv4 > 0 5410 1 27 68 0 13872 1900 ttyin Is+ v5 0:00.00 /usr/libexec/getty Pc ttyv5 > 0 5411 1 20 68 0 13872 1896 ttyin Is+ v6 0:00.00 /usr/libexec/getty Pc ttyv6 > 0 5412 1 9 68 0 13872 1896 ttyin Is+ v7 0:00.00 /usr/libexec/getty Pc ttyv7 > 0 13557 13556 25 21 0 14404 2868 wait Is 0 0:00.05 -/bin/sh > 0 13561 13557 22 21 0 13796 2064 newbuf D+ 0 0:00.00 find . > 0 13562 13556 19 20 0 14404 2896 wait Ss 1 0:00.06 -/bin/sh > 0 13761 13562 27 20 0 14484 3804 - R+ 1 0:00.00 ps axHl > 0 13762 13562 22 20 0 14260 2672 piperd S+ 1 0:00.00 more > 1001 9633 9632 13 24 0 14404 2756 wait Is 2 0:00.00 -sh (sh) > 0 9636 9633 2 20 0 21140 6372 select S+ 2 0:00.04 sudo -i > 0 7694 1 2 68 0 14404 3368 wait I 3- 0:00.03 /bin/sh /usr/local/sbin/vm _run dockge > 0 7972 7694 2 20 0 8540120 4653764 kqread SC 3- 0:22.73 bhyve: dockge (bhyve) > 0 7972 7694 25 20 0 8540120 4653764 uwait IC 3- 0:00.45 bhyve: dockge (bhyve) > 0 7972 7694 10 20 0 8540120 4653764 uwait IC 3- 0:00.47 bhyve: dockge (bhyve) > 0 7972 7694 8 20 0 8540120 4653764 uwait IC 3- 0:00.48 bhyve: dockge (bhyve) > 0 7972 7694 5 20 0 8540120 4653764 uwait IC 3- 0:00.49 bhyve: dockge (bhyve) > 0 7972 7694 5 20 0 8540120 4653764 uwait IC 3- 0:00.48 bhyve: dockge (bhyve) > 0 7972 7694 11 20 0 8540120 4653764 uwait IC 3- 0:00.50 bhyve: dockge (bhyve) > 0 7972 7694 4 20 0 8540120 4653764 uwait IC 3- 0:00.45 bhyve: dockge (bhyve) > 0 7972 7694 22 20 0 8540120 4653764 uwait IC 3- 0:00.50 bhyve: dockge (bhyve) > 0 7972 7694 24 26 0 8540120 4653764 uwait IC 3- 0:00.00 bhyve: dockge (bhyve) > 0 7972 7694 8 20 0 8540120 4653764 uwait IC 3- 0:00.85 bhyve: dockge (bhyve) > 0 7972 7694 3 20 0 8540120 4653764 uwait IC 3- 0:00.86 bhyve: dockge (bhyve) > 0 7972 7694 27 20 0 8540120 4653764 uwait IC 3- 0:00.85 bhyve: dockge (bhyve) > 0 7972 7694 26 20 0 8540120 4653764 uwait IC 3- 0:00.85 bhyve: dockge (bhyve) > 0 7972 7694 12 20 0 8540120 4653764 uwait IC 3- 0:00.86 bhyve: dockge (bhyve) > 0 7972 7694 23 20 0 8540120 4653764 uwait IC 3- 0:00.85 bhyve: dockge (bhyve) > 0 7972 7694 8 20 0 8540120 4653764 uwait IC 3- 0:00.86 bhyve: dockge (bhyve) > 0 7972 7694 5 20 0 8540120 4653764 uwait IC 3- 0:00.85 bhyve: dockge (bhyve) > 0 7972 7694 20 26 0 8540120 4653764 uwait IC 3- 0:00.00 bhyve: dockge (bhyve) > 0 7972 7694 3 20 0 8540120 4653764 uwait IC 3- 0:00.00 bhyve: dockge (bhyve) > 0 7972 7694 12 20 0 8540120 4653764 uwait IC 3- 0:00.01 bhyve: dockge (bhyve) > 0 7972 7694 1 20 0 8540120 4653764 uwait IC 3- 0:00.01 bhyve: dockge (bhyve) > 0 7972 7694 21 20 0 8540120 4653764 uwait IC 3- 0:00.01 bhyve: dockge (bhyve) > 0 7972 7694 9 20 0 8540120 4653764 uwait IC 3- 0:00.01 bhyve: dockge (bhyve) > 0 7972 7694 6 20 0 8540120 4653764 uwait IC 3- 0:00.01 bhyve: dockge (bhyve) > 0 7972 7694 27 20 0 8540120 4653764 uwait IC 3- 0:00.01 bhyve: dockge (bhyve) > 0 7972 7694 13 20 0 8540120 4653764 uwait IC 3- 0:00.00 bhyve: dockge (bhyve) > 0 7972 7694 24 26 0 8540120 4653764 uwait IC 3- 0:00.00 bhyve: dockge (bhyve) > 0 7972 7694 13 20 0 8540120 4653764 uwait SC 3- 0:03.66 bhyve: dockge (bhyve) > 0 7972 7694 24 42 0 8540120 4653764 uwait IC 3- 0:00.00 bhyve: dockge (bhyve) > 0 7972 7694 1 20 0 8540120 4653764 uwait SC 3- 0:27.11 bhyve: dockge (bhyve) > 0 7972 7694 8 68 0 8540120 4653764 accept IC 3- 0:00.00 bhyve: dockge (bhyve) > 0 7972 7694 9 48 0 8540120 4653764 vmidle DC 3- 25:44.20 bhyve: dockge (bhyve) > 0 7972 7694 1 20 0 8540120 4653764 vmidle DC 3- 62:59.81 bhyve: dockge (bhyve) > 0 7972 7694 8 21 0 8540120 4653764 vmidle DC 3- 52:39.38 bhyve: dockge (bhyve) > 0 7972 7694 24 21 0 8540120 4653764 vmidle DC 3- 42:23.71 bhyve: dockge (bhyve) > 0 7972 7694 11 44 0 8540120 4653764 vmidle DC 3- 34:51.91 bhyve: dockge (bhyve) > 0 7972 7694 22 26 0 8540120 4653764 vmidle DC 3- 38:34.85 bhyve: dockge (bhyve) > 0 7972 7694 14 21 0 8540120 4653764 vmidle DC 3- 27:12.23 bhyve: dockge (bhyve) > 0 7972 7694 6 20 0 8540120 4653764 vmidle DC 3- 34:36.53 bhyve: dockge (bhyve) > 0 7972 7694 2 20 0 8540120 4653764 vmidle DC 3- 43:33.37 bhyve: dockge (bhyve) > 0 7972 7694 15 20 0 8540120 4653764 vmidle DC 3- 35:22.08 bhyve: dockge (bhyve) > 0 7972 7694 4 20 0 8540120 4653764 vmidle DC 3- 23:36.09 bhyve: dockge (bhyve) > 0 7972 7694 4 21 0 8540120 4653764 vmidle DC 3- 31:42.67 bhyve: dockge (bhyve) > 0 7972 7694 14 23 0 8540120 4653764 vmidle DC 3- 27:36.15 bhyve: dockge (bhyve) > 0 7972 7694 12 21 0 8540120 4653764 vmidle DC 3- 36:10.16 bhyve: dockge (bhyve) > 0 7972 7694 17 20 0 8540120 4653764 vmidle DC 3- 37:08.96 bhyve: dockge (bhyve) > 0 7972 7694 15 21 0 8540120 4653764 vmidle DC 3- 24:54.26 bhyve: dockge (bhyve) > 0 8303 1 3 68 0 14404 3364 wait I 3- 0:00.02 /bin/sh /usr/local/sbin/vm _run tsdb > 0 8782 8303 2 20 0 3231944 2488900 kqread SC 3- 1:23.71 bhyve: tsdb (bhyve) > 0 8782 8303 8 20 0 3231944 2488900 uwait SC 3- 0:07.00 bhyve: tsdb (bhyve) > 0 8782 8303 27 20 0 3231944 2488900 uwait SC 3- 0:06.87 bhyve: tsdb (bhyve) > 0 8782 8303 11 20 0 3231944 2488900 uwait SC 3- 0:06.96 bhyve: tsdb (bhyve) > 0 8782 8303 20 20 0 3231944 2488900 uwait SC 3- 0:06.84 bhyve: tsdb (bhyve) > 0 8782 8303 20 20 0 3231944 2488900 uwait SC 3- 0:06.86 bhyve: tsdb (bhyve) > 0 8782 8303 7 20 0 3231944 2488900 uwait SC 3- 0:06.96 bhyve: tsdb (bhyve) > 0 8782 8303 0 20 0 3231944 2488900 uwait SC 3- 0:06.89 bhyve: tsdb (bhyve) > 0 8782 8303 3 20 0 3231944 2488900 uwait SC 3- 0:06.91 bhyve: tsdb (bhyve) > 0 8782 8303 25 26 0 3231944 2488900 uwait IC 3- 0:00.00 bhyve: tsdb (bhyve) > 0 8782 8303 21 20 0 3231944 2488900 uwait SC 3- 0:05.54 bhyve: tsdb (bhyve) > 0 8782 8303 22 20 0 3231944 2488900 uwait SC 3- 0:05.44 bhyve: tsdb (bhyve) > 0 8782 8303 25 20 0 3231944 2488900 uwait SC 3- 0:05.36 bhyve: tsdb (bhyve) > 0 8782 8303 8 20 0 3231944 2488900 uwait SC 3- 0:05.61 bhyve: tsdb (bhyve) > 0 8782 8303 20 20 0 3231944 2488900 uwait SC 3- 0:05.49 bhyve: tsdb (bhyve) > 0 8782 8303 19 20 0 3231944 2488900 uwait SC 3- 0:05.65 bhyve: tsdb (bhyve) > 0 8782 8303 9 20 0 3231944 2488900 uwait SC 3- 0:05.48 bhyve: tsdb (bhyve) > 0 8782 8303 20 20 0 3231944 2488900 uwait SC 3- 0:05.51 bhyve: tsdb (bhyve) > 0 8782 8303 3 26 0 3231944 2488900 uwait IC 3- 0:00.00 bhyve: tsdb (bhyve) > 0 8782 8303 18 20 0 3231944 2488900 uwait SC 3- 0:08.44 bhyve: tsdb (bhyve) > 0 8782 8303 25 20 0 3231944 2488900 uwait SC 3- 0:08.53 bhyve: tsdb (bhyve) > 0 8782 8303 21 20 0 3231944 2488900 uwait SC 3- 0:08.50 bhyve: tsdb (bhyve) > 0 8782 8303 5 20 0 3231944 2488900 uwait SC 3- 0:08.46 bhyve: tsdb (bhyve) > 0 8782 8303 6 20 0 3231944 2488900 uwait SC 3- 0:08.50 bhyve: tsdb (bhyve) > 0 8782 8303 19 20 0 3231944 2488900 uwait SC 3- 0:08.29 bhyve: tsdb (bhyve) > 0 8782 8303 20 20 0 3231944 2488900 uwait SC 3- 0:08.42 bhyve: tsdb (bhyve) > 0 8782 8303 24 20 0 3231944 2488900 uwait SC 3- 0:08.46 bhyve: tsdb (bhyve) > 0 8782 8303 25 26 0 3231944 2488900 uwait IC 3- 0:00.00 bhyve: tsdb (bhyve) > 0 8782 8303 18 20 0 3231944 2488900 uwait IC 3- 0:01.78 bhyve: tsdb (bhyve) > 0 8782 8303 17 20 0 3231944 2488900 uwait IC 3- 0:02.62 bhyve: tsdb (bhyve) > 0 8782 8303 19 20 0 3231944 2488900 uwait IC 3- 0:02.95 bhyve: tsdb (bhyve) > 0 8782 8303 22 20 0 3231944 2488900 uwait IC 3- 0:01.69 bhyve: tsdb (bhyve) > 0 8782 8303 10 20 0 3231944 2488900 uwait IC 3- 0:01.96 bhyve: tsdb (bhyve) > 0 8782 8303 25 20 0 3231944 2488900 uwait IC 3- 0:02.24 bhyve: tsdb (bhyve) > 0 8782 8303 7 20 0 3231944 2488900 uwait IC 3- 0:01.65 bhyve: tsdb (bhyve) > 0 8782 8303 11 20 0 3231944 2488900 uwait IC 3- 0:01.98 bhyve: tsdb (bhyve) > 0 8782 8303 10 26 0 3231944 2488900 uwait IC 3- 0:00.00 bhyve: tsdb (bhyve) > 0 8782 8303 15 20 0 3231944 2488900 uwait SC 3- 2:04.93 bhyve: tsdb (bhyve) > 0 8782 8303 14 21 0 3231944 2488900 vmidle DC 3- 28:14.79 bhyve: tsdb (bhyve) > 0 8782 8303 22 21 0 3231944 2488900 vmidle DC 3- 24:44.47 bhyve: tsdb (bhyve) > 0 8782 8303 16 21 0 3231944 2488900 vmidle DC 3- 24:48.21 bhyve: tsdb (bhyve) > 0 8782 8303 26 21 0 3231944 2488900 vmidle DC 3- 24:42.32 bhyve: tsdb (bhyve) > 0 9637 9636 4 21 0 21140 6364 select Is 4 0:00.00 sudo -i > 0 9638 9637 14 20 0 14404 2772 wait I 4 0:00.01 -sh (sh) > 0 13555 9638 5 20 0 15200 3276 pause S+ 4 0:00.05 screen -RR -D (screen-5.0.1) > > > > root@bhyve02:/vms # uname -a > FreeBSD bhyve02 14.3-RELEASE FreeBSD 14.3-RELEASE releng/14.3-n271432-8c9ce319fef7 GENERIC amd64 > > root@bhyve02:/vms # nfsstat -m > 10.10.10.10:/mnt/datastore0/bhyve_instances on /vms > nfsv4,minorversion=2,tcp,resvport,nconnect=16,hard,cto,sec=sys,acdirmin=3,acdirmax=60,acregmin=5,acregmax=60,nametimeo=60,negnametim > eo=60,rsize=1048576,wsize=1048576,readdirsize=1048576,readahead=8,wcommitsize=67108864,timeout=120,retrans=2147483647 > 10.10.10.10:/mnt/datastore1/iso on /vms/.iso > nfsv4,minorversion=2,tcp,resvport,nconnect=16,hard,cto,sec=sys,acdirmin=3,acdirmax=60,acregmin=5,acregmax=60,nametimeo=60,negnametim > eo=60,rsize=1048576,wsize=1048576,readdirsize=1048576,readahead=8,wcommitsize=67108864,timeout=120,retrans=2147483647 > 10.10.10.10:/mnt/ds_ssd_vms_03/disks on /vms/.disks > nfsv4,minorversion=2,tcp,resvport,nconnect=16,hard,cto,sec=sys,acdirmin=3,acdirmax=60,acregmin=5,acregmax=60,nametimeo=60,negnametim > eo=60,rsize=1048576,wsize=1048576,readdirsize=1048576,readahead=8,wcommitsize=67108864,timeout=120,retrans=2147483647 >