ZFS related hang with FreeBSD 9.2
Ryan Baldwin
ryan.baldwin at nexusalpha.com
Sat Dec 14 07:13:27 UTC 2013
Yes we are using snapshots. The snapshot-manager processes are
continually creating and destroying snapshots at a rate of 10,000's a
day. We have had such a system running on FreeBSD 9.1 with just one
snapshot-manager process several months on end and we are not aware of
any problems with this system.
I run another type of system currently on FreeBSD 9.1 as a storage
server and it serves terrabytes a day and uses snapshots and clones
extensively and I have not had any problems with this system.
This is the first real problem I have come across. So far the 9.1 based
version of the problem server is still going although it is still early
days.
On 12/13/13 23:21, Rod Taylor wrote:
> Are you using snapshots?
>
> I've found ZFS Snapshots on 9.0, 9.1, and 9.2 regularly crash the
> system. Delete the snapshots and don't create any new ones and
> suddenly it's stable for months.
>
>
>
> On Fri, Dec 13, 2013 at 12:14 AM, Ryan Baldwin
> <ryan.baldwin at nexusalpha.com <mailto:ryan.baldwin at nexusalpha.com>> wrote:
>
> Hi,
>
> We have a server based on FreeBSD 9.2 which hangs at times on a
> daily basis. The longest uptime we have achieved is 5 days
> conversely it has stopped daily several days in a row.
>
> When this occurs it appears there are two proceses stuck in
> 'tx->tx' state. In the top output shown these are snapshot-manager
> processes which create and destroy snapshots generally and
> sometime rollback filesystems to snapshots. When the lockup occurs
> other processes which try to access the file system can seem to
> end up stuck in state 'rrl->r'. The reboot command that was issued
> to try and reboot the server has ended up stuck in this state as
> can be seen.
>
> The server is not under particularly heavy load.
>
> It has remained in this state for hours. The 'deadman handler'?
> does not appear to restart the system. Once this has occurred
> there is no further disk activity.
>
> We did not experience this problem at all previously using 9.1
> although we had less snapshot-manager processes before. We have
> built this server against 9.1 again now but it has only been going
> one day so far.
>
> We can try and reproduce this problem again on 9.2 if by doing so
> we can gather any additional information that could help resolve
> this problem. Please let me know what other information would be
> helpful.
>
> The hardware is a Dell R420 with Perc H310 raid controller in JBOD
> mode with the pool mirrored on two SAS disks.
>
> Thanks
>
> top and procstat output follow:
>
>
> last pid: 76225; load averages: 0.00, 0.00, 0.00 up 0+20:04:22
> 09:56:24
> 46 processes: 1 running, 44 sleeping, 1 stopped
> CPU: % user, % nice, % system, % interrupt, % idle
> Mem: 405M Active, 133M Inact, 1206M Wired, 6084M Free
> ARC: 797M Total, 184M MFU, 488M MRU, 28M Anon, 32M Header, 65M Other
> Swap: 8192M Total, 8192M Free
>
> PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU
> COMMAND
> 1170 root 1 20 0 16244K 5460K kqread 11 1:28 0.00%
> CustomInit
> 1531 root 1 20 0 39648K 6368K tx->tx 7 1:17 0.00%
> snapshot-manager
> 1135 root 1 20 0 16244K 5100K kqread 1 1:10 0.00%
> CustomInit
> 2097 root 2 20 0 2581M 472M STOP 4 0:46 0.00%
> java
> 1183 root 1 20 0 16244K 5504K kqread 0 0:22 0.00%
> CustomInit
> 1145 root 1 20 0 16244K 5464K kqread 9 0:18 0.00%
> CustomInit
> 1444 root 1 20 0 39648K 6352K tx->tx 5 0:13 0.00%
> snapshot-manager
> 1628 root 1 20 0 39648K 6348K rrl->r 10 0:12 0.00%
> snapshot-manager
> 1168 root 1 20 0 16244K 5388K kqread 7 0:07 0.00%
> CustomInit
> 1535 root 1 20 0 22388K 14296K select 4 0:05 0.00%
> openvpn
> 1163 root 1 20 0 16244K 5388K kqread 2 0:04 0.00%
> CustomInit
> 1511 root 1 20 0 18292K 10192K select 0 0:04 0.00%
> openvpn
> 1156 root 1 20 0 16244K 5392K kqread 10 0:04 0.00%
> CustomInit
> 1174 root 1 20 0 16244K 5388K kqread 1 0:04 0.00%
> CustomInit
> 1161 root 1 20 0 16244K 5392K kqread 4 0:03 0.00%
> CustomInit
> 913 root 1 20 0 12076K 1820K select 7 0:03 0.00%
> syslogd
> 2102 root 1 20 0 109M 13616K zfsvfs 3 0:03 0.00%
> data-processor
> 1445 root 1 20 0 22388K 14296K select 5 0:02 0.00%
> openvpn
> 1014 root 1 20 0 22256K 3360K select 4 0:02 0.00%
> ntpd
> 1494 root 1 20 0 18292K 10192K select 2 0:01 0.00%
> openvpn
> 1180 root 1 20 0 18292K 4780K select 0 0:01 0.00%
> openvpn
> 1505 root 1 20 0 18292K 4712K select 2 0:01 0.00%
> openvpn
> 1495 root 1 20 0 18292K 4712K select 5 0:00 0.00%
> openvpn
> 1479 root 1 20 0 18292K 4708K select 1 0:00 0.00%
> openvpn
> 1567 root 1 20 0 18292K 4712K select 3 0:00 0.00%
> openvpn
> 1545 root 1 20 0 18292K 4712K select 1 0:00 0.00%
> openvpn
> 1486 root 1 20 0 18292K 4708K select 0 0:00 0.00%
> openvpn
> 1443 root 1 20 0 18292K 4436K select 9 0:00 0.00%
> openvpn
> 1447 root 1 20 0 18292K 4448K select 4 0:00 0.00%
> openvpn
> 1496 root 1 20 0 18292K 4708K select 3 0:00 0.00%
> openvpn
> 1478 root 1 20 0 18292K 4704K select 5 0:00 0.00%
> openvpn
> 1488 root 1 20 0 18292K 10192K select 5 0:00 0.00%
> openvpn
> 1032 root 1 20 0 14176K 1860K nanslp 1 0:00 0.00%
> cron
> 76069 root 1 20 0 9948K 1680K rrl->r 7 0:00 0.00%
> reboot
> 782 root 1 20 0 10376K 4416K select 5 0:00 0.00%
> devd
> 76196 root 1 20 0 51536K 5828K select 0 0:00 0.00%
> sshd
> 76203 root 1 20 0 51536K 5828K select 2 0:00 0.00%
> sshd
> 76205 root 1 20 0 17564K 3252K ttyin 11 0:00 0.00% csh
> 76198 root 1 20 0 17564K 3252K pause 8 0:00 0.00% csh
> 75718 root 1 20 0 35556K 3956K rrl->r 11 0:00 0.00%
> snapshot-manager-counts
> 75715 root 1 20 0 35556K 3956K rrl->r 10 0:00 0.00%
> snapshot-manager-counts
> 1181 root 1 52 0 12084K 1676K ttyin 1 0:00 0.00%
> getty
>
>
> 0 100000 kernel swapper mi_switch+0x186
> sleepq_timedwait+0x42 _sleep+0x1c9 scheduler+0x359 mi_startup+0x77
> btext+0x2c
> 0 100032 kernel firmware taskq mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100037 kernel ffs_trim taskq mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100038 kernel acpi_task_0 mi_switch+0x186
> sleepq_wait+0x42 msleep_spin+0x194 taskqueue_thread_loop+0x67
> fork_exit+0x11f fork_trampoline+0xe
> 0 100039 kernel acpi_task_1 mi_switch+0x186
> sleepq_wait+0x42 msleep_spin+0x194 taskqueue_thread_loop+0x67
> fork_exit+0x11f fork_trampoline+0xe
> 0 100040 kernel acpi_task_2 mi_switch+0x186
> sleepq_wait+0x42 msleep_spin+0x194 taskqueue_thread_loop+0x67
> fork_exit+0x11f fork_trampoline+0xe
> 0 100041 kernel kqueue taskq mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100044 kernel thread taskq mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100052 kernel bge0 taskq mi_switch+0x186
> sleepq_wait+0x42 msleep_spin+0x194 taskqueue_thread_loop+0x67
> fork_exit+0x11f fork_trampoline+0xe
> 0 100053 kernel bge1 taskq mi_switch+0x186
> sleepq_wait+0x42 msleep_spin+0x194 taskqueue_thread_loop+0x67
> fork_exit+0x11f fork_trampoline+0xe
> 0 100062 kernel mca taskq mi_switch+0x186
> sleepq_wait+0x42 msleep_spin+0x194 taskqueue_thread_loop+0x67
> fork_exit+0x11f fork_trampoline+0xe
> 0 100063 kernel system_taskq_0 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100064 kernel system_taskq_1 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100065 kernel system_taskq_2 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100066 kernel system_taskq_3 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100067 kernel system_taskq_4 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100068 kernel system_taskq_5 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100069 kernel system_taskq_6 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100070 kernel system_taskq_7 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100071 kernel system_taskq_8 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100072 kernel system_taskq_9 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100073 kernel system_taskq_10 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100074 kernel system_taskq_11 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100247 kernel zio_null_issue mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100248 kernel zio_null_intr mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100249 kernel zio_read_issue_0 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100250 kernel zio_read_issue_1 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100251 kernel zio_read_issue_2 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100252 kernel zio_read_issue_3 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100253 kernel zio_read_issue_4 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100254 kernel zio_read_issue_5 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100255 kernel zio_read_issue_6 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100256 kernel zio_read_issue_7 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100257 kernel zio_read_intr_0 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100258 kernel zio_read_intr_1 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100259 kernel zio_read_intr_2 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100260 kernel zio_read_intr_3 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100261 kernel zio_read_intr_4 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100262 kernel zio_read_intr_5 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100263 kernel zio_read_intr_6 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100264 kernel zio_read_intr_7 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100265 kernel zio_read_intr_8 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100266 kernel zio_read_intr_9 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100267 kernel zio_read_intr_10 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100268 kernel zio_read_intr_11 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100269 kernel zio_write_issue_ mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100270 kernel zio_write_issue_ mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100271 kernel zio_write_issue_ mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100272 kernel zio_write_issue_ mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100273 kernel zio_write_issue_ mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100274 kernel zio_write_issue_ mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100275 kernel zio_write_issue_ mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100276 kernel zio_write_issue_ mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100277 kernel zio_write_issue_ mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100278 kernel zio_write_issue_ mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100279 kernel zio_write_issue_ mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100280 kernel zio_write_issue_ mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100281 kernel zio_write_issue_ mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100282 kernel zio_write_issue_ mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100283 kernel zio_write_issue_ mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100284 kernel zio_write_issue_ mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100285 kernel zio_write_issue_ mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100286 kernel zio_write_intr_0 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100287 kernel zio_write_intr_1 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100288 kernel zio_write_intr_2 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100289 kernel zio_write_intr_3 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100290 kernel zio_write_intr_4 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100291 kernel zio_write_intr_5 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100292 kernel zio_write_intr_6 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100293 kernel zio_write_intr_7 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100294 kernel zio_write_intr_h mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100295 kernel zio_write_intr_h mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100296 kernel zio_write_intr_h mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100297 kernel zio_write_intr_h mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100298 kernel zio_write_intr_h mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100299 kernel zio_free_issue_0 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100300 kernel zio_free_issue_1 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100301 kernel zio_free_issue_2 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100302 kernel zio_free_issue_3 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100303 kernel zio_free_issue_4 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100304 kernel zio_free_issue_5 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100305 kernel zio_free_issue_6 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100306 kernel zio_free_issue_7 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100307 kernel zio_free_issue_8 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100308 kernel zio_free_issue_9 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100309 kernel zio_free_issue_1 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100310 kernel zio_free_issue_1 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100311 kernel zio_free_issue_1 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100312 kernel zio_free_issue_1 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100313 kernel zio_free_issue_1 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100314 kernel zio_free_issue_1 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100315 kernel zio_free_issue_1 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100316 kernel zio_free_issue_1 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100317 kernel zio_free_issue_1 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100318 kernel zio_free_issue_1 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100319 kernel zio_free_issue_2 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100320 kernel zio_free_issue_2 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100321 kernel zio_free_issue_2 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100322 kernel zio_free_issue_2 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100323 kernel zio_free_issue_2 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100324 kernel zio_free_issue_2 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100325 kernel zio_free_issue_2 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100326 kernel zio_free_issue_2 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100327 kernel zio_free_issue_2 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100328 kernel zio_free_issue_2 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100329 kernel zio_free_issue_3 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100330 kernel zio_free_issue_3 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100331 kernel zio_free_issue_3 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100332 kernel zio_free_issue_3 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100333 kernel zio_free_issue_3 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100334 kernel zio_free_issue_3 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100335 kernel zio_free_issue_3 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100336 kernel zio_free_issue_3 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100337 kernel zio_free_issue_3 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100338 kernel zio_free_issue_3 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100339 kernel zio_free_issue_4 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100340 kernel zio_free_issue_4 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100341 kernel zio_free_issue_4 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100342 kernel zio_free_issue_4 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100343 kernel zio_free_issue_4 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100344 kernel zio_free_issue_4 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100345 kernel zio_free_issue_4 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100346 kernel zio_free_issue_4 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100347 kernel zio_free_issue_4 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100348 kernel zio_free_issue_4 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100349 kernel zio_free_issue_5 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100350 kernel zio_free_issue_5 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100351 kernel zio_free_issue_5 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100352 kernel zio_free_issue_5 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100353 kernel zio_free_issue_5 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100354 kernel zio_free_issue_5 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100355 kernel zio_free_issue_5 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100356 kernel zio_free_issue_5 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100357 kernel zio_free_issue_5 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100358 kernel zio_free_issue_5 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100359 kernel zio_free_issue_6 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100360 kernel zio_free_issue_6 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100361 kernel zio_free_issue_6 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100362 kernel zio_free_issue_6 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100363 kernel zio_free_issue_6 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100364 kernel zio_free_issue_6 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100365 kernel zio_free_issue_6 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100366 kernel zio_free_issue_6 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100367 kernel zio_free_issue_6 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100368 kernel zio_free_issue_6 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100369 kernel zio_free_issue_7 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100370 kernel zio_free_issue_7 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100371 kernel zio_free_issue_7 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100372 kernel zio_free_issue_7 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100373 kernel zio_free_issue_7 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100374 kernel zio_free_issue_7 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100375 kernel zio_free_issue_7 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100376 kernel zio_free_issue_7 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100377 kernel zio_free_issue_7 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100378 kernel zio_free_issue_7 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100379 kernel zio_free_issue_8 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100380 kernel zio_free_issue_8 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100381 kernel zio_free_issue_8 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100382 kernel zio_free_issue_8 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100383 kernel zio_free_issue_8 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100384 kernel zio_free_issue_8 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100385 kernel zio_free_issue_8 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100386 kernel zio_free_issue_8 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100387 kernel zio_free_issue_8 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100388 kernel zio_free_issue_8 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100389 kernel zio_free_issue_9 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100390 kernel zio_free_issue_9 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100391 kernel zio_free_issue_9 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100392 kernel zio_free_issue_9 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100393 kernel zio_free_issue_9 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100394 kernel zio_free_issue_9 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100395 kernel zio_free_issue_9 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100396 kernel zio_free_issue_9 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100397 kernel zio_free_issue_9 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100398 kernel zio_free_issue_9 mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100399 kernel zio_free_intr mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100400 kernel zio_claim_issue mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100401 kernel zio_claim_intr mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100402 kernel zio_ioctl_issue mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100403 kernel zio_ioctl_intr mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100405 kernel zfs_vn_rele_task mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100418 kernel zil_clean mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100441 kernel zil_clean mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100477 kernel zil_clean mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100478 kernel zil_clean mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100479 kernel zil_clean mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100480 kernel zil_clean mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100481 kernel zil_clean mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100482 kernel zil_clean mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100484 kernel zil_clean mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100486 kernel zil_clean mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100488 kernel zil_clean mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100489 kernel zil_clean mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100490 kernel zil_clean mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100491 kernel zil_clean mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100492 kernel zil_clean mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100493 kernel zil_clean mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100494 kernel zil_clean mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100495 kernel zil_clean mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100496 kernel zil_clean mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100497 kernel zil_clean mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100498 kernel zil_clean mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100499 kernel zil_clean mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100636 kernel zil_clean mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 0 100638 kernel zil_clean mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb
> fork_exit+0x11f fork_trampoline+0xe
> 1 100002 init - mi_switch+0x186
> sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _sleep+0x295
> kern_wait6+0x8c3 kern_wait+0x9c sys_wait4+0x35 amd64_syscall+0x540
> Xfast_syscall+0xf7
> 2 100033 crypto - mi_switch+0x186 sleepq_wait+0x42
> _sleep+0x379 crypto_proc+0x197 fork_exit+0x11f fork_trampoline+0xe
> 3 100034 <tel:3%20100034> crypto returns -
> mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379
> crypto_ret_proc+0x192 fork_exit+0x11f fork_trampoline+0xe
> 4 100061 ctl_thrd - mi_switch+0x186 sleepq_wait+0x42
> _sleep+0x379 ctl_work_thread+0x2fd0 fork_exit+0x11f
> fork_trampoline+0xe
> 5 100075 zfskern arc_reclaim_thre mi_switch+0x186
> sleepq_timedwait+0x42 _cv_timedwait+0x135 arc_reclaim_thread+0x29d
> fork_exit+0x11f fork_trampoline+0xe
> 5 100076 zfskern l2arc_feed_threa mi_switch+0x186
> sleepq_timedwait+0x42 _cv_timedwait+0x135 l2arc_feed_thread+0x1a2
> fork_exit+0x11f fork_trampoline+0xe
> 5 100404 zfskern trim nsgroot mi_switch+0x186
> sleepq_timedwait+0x42 _cv_timedwait+0x135 trim_thread+0x68
> fork_exit+0x11f fork_trampoline+0xe
> 5 100406 zfskern txg_thread_enter mi_switch+0x186
> sleepq_wait+0x42 _cv_wait+0x112 txg_thread_wait+0x79
> txg_quiesce_thread+0xbb fork_exit+0x11f fork_trampoline+0xe
> 5 100407 zfskern txg_thread_enter mi_switch+0x186
> sleepq_wait+0x42 _cv_wait+0x112 rrw_enter_write+0x35
> dsl_sync_task_sync+0xb8 dsl_pool_sync+0x47d spa_sync+0x3ba
> txg_sync_thread+0x139 fork_exit+0x11f fork_trampoline+0xe
> 5 100408 zfskern zvol nsgroot/swa mi_switch+0x186
> sleepq_wait+0x42 _sleep+0x379 zvol_geom_worker+0xfc
> fork_exit+0x11f fork_trampoline+0xe
> 6 100077 sctp_iterator - mi_switch+0x186 sleepq_wait+0x42
> _sleep+0x379 sctp_iterator_thread+0x41 fork_exit+0x11f
> fork_trampoline+0xe
> 7 100078 xpt_thrd - mi_switch+0x186 sleepq_wait+0x42
> _sleep+0x379 xpt_scanner_thread+0xff fork_exit+0x11f
> fork_trampoline+0xe
> 8 100079 ipmi0: kcs - mi_switch+0x186 sleepq_wait+0x42
> _cv_wait+0x112 ipmi_dequeue_request+0x47 kcs_loop+0x3d
> fork_exit+0x11f fork_trampoline+0xe
> 9 100080 enc_daemon0 - mi_switch+0x186 sleepq_wait+0x42
> _sleep+0x379 enc_daemon+0xe4 fork_exit+0x11f fork_trampoline+0xe
> 10 100001 audit - mi_switch+0x186 sleepq_wait+0x42
> _cv_wait+0x112 audit_worker+0x359 fork_exit+0x11f fork_trampoline+0xe
> 11 100003 idle idle: cpu0 <running>
> 11 100004 idle idle: cpu1 <running>
> 11 100005 idle idle: cpu2 <running>
> 11 100006 idle idle: cpu3 <running>
> 11 100007 idle idle: cpu4 <running>
> 11 100008 idle idle: cpu5 <running>
> 11 100009 idle idle: cpu6 <running>
> 11 100010 idle idle: cpu7 <running>
> 11 100011 idle idle: cpu8 <running>
> 11 100012 idle idle: cpu9 <running>
> 11 100013 idle idle: cpu10 mi_switch+0x186
> critical_exit+0xa5 sched_idletd+0x118 fork_exit+0x11f
> fork_trampoline+0xe
> 11 100014 idle idle: cpu11 <running>
> 12 100015 intr swi4: clock mi_switch+0x186
> ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe
> 12 100016 intr swi4: clock mi_switch+0x186
> ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe
> 12 100017 intr swi4: clock mi_switch+0x186
> ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe
> 12 100018 intr swi4: clock mi_switch+0x186
> ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe
> 12 100019 intr swi4: clock mi_switch+0x186
> ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe
> 12 100020 intr swi4: clock mi_switch+0x186
> ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe
> 12 100021 intr swi4: clock mi_switch+0x186
> ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe
> 12 100022 intr swi4: clock mi_switch+0x186
> ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe
> 12 100023 intr swi4: clock mi_switch+0x186
> ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe
> 12 100024 intr swi4: clock mi_switch+0x186
> ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe
> 12 100025 intr swi4: clock mi_switch+0x186
> ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe
> 12 100026 intr swi4: clock mi_switch+0x186
> ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe
> 12 100027 intr swi3: vm
> 12 100028 intr swi1: netisr 0 mi_switch+0x186
> ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe
> 12 100036 intr swi6: task queue mi_switch+0x186
> ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe
> 12 100042 intr swi2: cambio mi_switch+0x186
> ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe
> 12 100043 intr swi5: fast taskq
> 12 100045 intr swi6: Giant task mi_switch+0x186
> ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe
> 12 100046 intr irq264: mfi0 mi_switch+0x186
> ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe
> 12 100047 intr irq23: ehci0 mi_switch+0x186
> ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe
> 12 100054 intr irq22: ehci1 mi_switch+0x186
> ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe
> 12 100059 intr irq267: ahci0
> 12 100060 intr swi0: uart uart
> 13 100029 <tel:13%20100029> geom g_event
> mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379 g_run_events+0x440
> fork_exit+0x11f fork_trampoline+0xe
> 13 100030 <tel:13%20100030> geom g_up
> mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379
> g_io_schedule_up+0xe6 g_up_procbody+0x5c fork_exit+0x11f
> fork_trampoline+0xe
> 13 100031 <tel:13%20100031> geom g_down
> mi_switch+0x186 sleepq_wait+0x42 _sleep+0x379
> g_io_schedule_down+0x25f g_down_procbody+0x5c fork_exit+0x11f
> fork_trampoline+0xe
> 14 100035 yarrow - mi_switch+0x186
> sleepq_timedwait+0x42 _sleep+0x1c9 random_kthread+0x1ea
> fork_exit+0x11f fork_trampoline+0xe
> 15 100048 usb usbus0 mi_switch+0x186
> sleepq_wait+0x42 _cv_wait+0x112 usb_process+0x18b fork_exit+0x11f
> fork_trampoline+0xe
> 15 100049 usb usbus0 mi_switch+0x186
> sleepq_wait+0x42 _cv_wait+0x112 usb_process+0x18b fork_exit+0x11f
> fork_trampoline+0xe
> 15 100050 usb usbus0 mi_switch+0x186
> sleepq_wait+0x42 _cv_wait+0x112 usb_process+0x18b fork_exit+0x11f
> fork_trampoline+0xe
> 15 100051 usb usbus0 mi_switch+0x186
> sleepq_wait+0x42 _cv_wait+0x112 usb_process+0x18b fork_exit+0x11f
> fork_trampoline+0xe
> 15 100055 usb usbus1 mi_switch+0x186
> sleepq_wait+0x42 _cv_wait+0x112 usb_process+0x18b fork_exit+0x11f
> fork_trampoline+0xe
> 15 100056 usb usbus1 mi_switch+0x186
> sleepq_wait+0x42 _cv_wait+0x112 usb_process+0x18b fork_exit+0x11f
> fork_trampoline+0xe
> 15 100057 usb usbus1 mi_switch+0x186
> sleepq_wait+0x42 _cv_wait+0x112 usb_process+0x18b fork_exit+0x11f
> fork_trampoline+0xe
> 15 100058 usb usbus1 mi_switch+0x186
> sleepq_wait+0x42 _cv_wait+0x112 usb_process+0x18b fork_exit+0x11f
> fork_trampoline+0xe
> 16 100081 pagedaemon - mi_switch+0x186
> sleepq_timedwait+0x42 _sleep+0x1c9 vm_pageout+0xb34
> fork_exit+0x11f fork_trampoline+0xe
> 17 100082 vmdaemon - mi_switch+0x186 sleepq_wait+0x42
> _sleep+0x379 vm_daemon+0x58 fork_exit+0x11f fork_trampoline+0xe
> 18 100083 pagezero - mi_switch+0x186
> sleepq_timedwait+0x42 _sleep+0x1c9 vm_pagezero+0x83
> fork_exit+0x11f fork_trampoline+0xe
> 19 100084 bufdaemon - mi_switch+0x186
> sleepq_timedwait+0x42 _sleep+0x1c9 buf_daemon+0x1e1
> fork_exit+0x11f fork_trampoline+0xe
> 20 100085 syncer - mi_switch+0x186 sleepq_wait+0x42
> _cv_wait+0x112 rrw_enter_read+0x4b zfs_sync+0x5e sync_fsync+0x1a2
> VOP_FSYNC_APV+0x68 sync_vnode+0x16b sched_sync+0x1c5
> fork_exit+0x11f fork_trampoline+0xe
> 21 100086 vnlru - mi_switch+0x186 sleepq_wait+0x42
> _sx_slock_hard+0x318 _sx_slock+0x56 zfs_freebsd_reclaim+0x4b
> VOP_RECLAIM_APV+0x68 vgonel+0x134 vnlru_free+0x362
> vnlru_proc+0x61e fork_exit+0x11f fork_trampoline+0xe
> 22 100087 softdepflush - mi_switch+0x186
> sleepq_timedwait+0x42 _sleep+0x1c9 softdep_flush+0x375
> fork_exit+0x11f fork_trampoline+0xe
> 782 100432 devd - mi_switch+0x186
> sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19
> _cv_timedwait_sig+0x135 seltdwait+0x8d kern_select+0x6ef
> sys_select+0x5d amd64_syscall+0x540 Xfast_syscall+0xf7
> 790 100545 pfpurge - mi_switch+0x186
> sleepq_timedwait+0x42 _sleep+0x1c9 pf_purge_thread+0x31
> fork_exit+0x11f fork_trampoline+0xe
> 913 100449 syslogd - mi_switch+0x186
> sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _cv_wait_sig+0x12a
> seltdwait+0xf6 kern_select+0x6ef sys_select+0x5d
> amd64_syscall+0x540 Xfast_syscall+0xf7
> 1014 100581 ntpd - mi_switch+0x186
> sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _cv_wait_sig+0x12a
> seltdwait+0xf6 kern_select+0x6ef sys_select+0x5d
> amd64_syscall+0x540 Xfast_syscall+0xf7
> 1028 100532 sshd - mi_switch+0x186
> sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _cv_wait_sig+0x12a
> seltdwait+0xf6 kern_select+0x6ef sys_select+0x5d
> amd64_syscall+0x540 Xfast_syscall+0xf7
> 1032 100473 cron - mi_switch+0x186
> sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19 _sleep+0x2ca
> kern_nanosleep+0x118 sys_nanosleep+0x6e amd64_syscall+0x540
> Xfast_syscall+0xf7
> 1135 100558 CustomInit - mi_switch+0x186
> sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _sleep+0x295
> kern_kevent+0x369 sys_kevent+0x90 amd64_syscall+0x540
> Xfast_syscall+0xf7
> 1145 100509 CustomInit - mi_switch+0x186
> sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _sleep+0x295
> kern_kevent+0x369 sys_kevent+0x90 amd64_syscall+0x540
> Xfast_syscall+0xf7
> 1156 100508 CustomInit - mi_switch+0x186
> sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _sleep+0x295
> kern_kevent+0x369 sys_kevent+0x90 amd64_syscall+0x540
> Xfast_syscall+0xf7
> 1161 100507 CustomInit - mi_switch+0x186
> sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _sleep+0x295
> kern_kevent+0x369 sys_kevent+0x90 amd64_syscall+0x540
> Xfast_syscall+0xf7
> 1163 100521 CustomInit - mi_switch+0x186
> sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _sleep+0x295
> kern_kevent+0x369 sys_kevent+0x90 amd64_syscall+0x540
> Xfast_syscall+0xf7
> 1168 100574 CustomInit - mi_switch+0x186
> sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _sleep+0x295
> kern_kevent+0x369 sys_kevent+0x90 amd64_syscall+0x540
> Xfast_syscall+0xf7
> 1170 100561 CustomInit - mi_switch+0x186
> sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _sleep+0x295
> kern_kevent+0x369 sys_kevent+0x90 amd64_syscall+0x540
> Xfast_syscall+0xf7
> 1174 100560 CustomInit - mi_switch+0x186
> sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _sleep+0x295
> kern_kevent+0x369 sys_kevent+0x90 amd64_syscall+0x540
> Xfast_syscall+0xf7
> 1180 100420 openvpn - mi_switch+0x186
> sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19
> _cv_timedwait_sig+0x135 seltdwait+0x8d sys_poll+0x478
> amd64_syscall+0x540 Xfast_syscall+0xf7
> 1181 100463 getty - mi_switch+0x186
> sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _cv_wait_sig+0x12a
> tty_wait+0x25 ttydisc_read+0x2dd ttydev_read+0xc4
> devfs_read_f+0x90 dofileread+0xa1 kern_readv+0x6c sys_read+0x64
> amd64_syscall+0x540 Xfast_syscall+0xf7
> 1182 100430 getty - mi_switch+0x186
> sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _cv_wait_sig+0x12a
> tty_wait+0x25 ttydisc_read+0x2dd ttydev_read+0xc4
> devfs_read_f+0x90 dofileread+0xa1 kern_readv+0x6c sys_read+0x64
> amd64_syscall+0x540 Xfast_syscall+0xf7
> 1183 100421 CustomInit - mi_switch+0x186
> sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _sleep+0x295
> kern_kevent+0x369 sys_kevent+0x90 amd64_syscall+0x540
> Xfast_syscall+0xf7
> 1443 100599 openvpn - mi_switch+0x186
> sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19
> _cv_timedwait_sig+0x135 seltdwait+0x8d sys_poll+0x478
> amd64_syscall+0x540 Xfast_syscall+0xf7
> 1444 100527 snapshot-manager - mi_switch+0x186
> sleepq_wait+0x42 _cv_wait+0x112 txg_wait_synced+0x85
> dsl_sync_task+0x139 dsl_destroy_snapshots_nvl+0x71
> dsl_destroy_snapshot+0x4a zfs_ioc_destroy+0x3e zfsdev_ioctl+0x58d
> devfs_ioctl_f+0x7b kern_ioctl+0x106 sys_ioctl+0xfd
> amd64_syscall+0x540 Xfast_syscall+0xf7
> 1445 100514 openvpn - mi_switch+0x186
> sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19
> _cv_timedwait_sig+0x135 seltdwait+0x8d sys_poll+0x478
> amd64_syscall+0x540 Xfast_syscall+0xf7
> 1447 100557 openvpn - mi_switch+0x186
> sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19
> _cv_timedwait_sig+0x135 seltdwait+0x8d sys_poll+0x478
> amd64_syscall+0x540 Xfast_syscall+0xf7
> 1478 100555 openvpn - mi_switch+0x186
> sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19
> _cv_timedwait_sig+0x135 seltdwait+0x8d sys_poll+0x478
> amd64_syscall+0x540 Xfast_syscall+0xf7
> 1479 100551 openvpn - mi_switch+0x186
> sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19
> _cv_timedwait_sig+0x135 seltdwait+0x8d sys_poll+0x478
> amd64_syscall+0x540 Xfast_syscall+0xf7
> 1486 100576 openvpn - mi_switch+0x186
> sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19
> _cv_timedwait_sig+0x135 seltdwait+0x8d sys_poll+0x478
> amd64_syscall+0x540 Xfast_syscall+0xf7
> 1488 100563 openvpn - mi_switch+0x186
> sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19
> _cv_timedwait_sig+0x135 seltdwait+0x8d sys_poll+0x478
> amd64_syscall+0x540 Xfast_syscall+0xf7
> 1494 100575 openvpn - mi_switch+0x186
> sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19
> _cv_timedwait_sig+0x135 seltdwait+0x8d sys_poll+0x478
> amd64_syscall+0x540 Xfast_syscall+0xf7
> 1495 100442 openvpn - mi_switch+0x186
> sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19
> _cv_timedwait_sig+0x135 seltdwait+0x8d sys_poll+0x478
> amd64_syscall+0x540 Xfast_syscall+0xf7
> 1496 100556 openvpn - mi_switch+0x186
> sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19
> _cv_timedwait_sig+0x135 seltdwait+0x8d sys_poll+0x478
> amd64_syscall+0x540 Xfast_syscall+0xf7
> 1505 100431 openvpn - mi_switch+0x186
> sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19
> _cv_timedwait_sig+0x135 seltdwait+0x8d sys_poll+0x478
> amd64_syscall+0x540 Xfast_syscall+0xf7
> 1511 100475 openvpn - mi_switch+0x186
> sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19
> _cv_timedwait_sig+0x135 seltdwait+0x8d sys_poll+0x478
> amd64_syscall+0x540 Xfast_syscall+0xf7
> 1531 100429 snapshot-manager - mi_switch+0x186
> sleepq_wait+0x42 _cv_wait+0x112 txg_wait_synced+0x85
> dsl_sync_task+0x139 zfs_ioc_rollback+0xb8 zfsdev_ioctl+0x58d
> devfs_ioctl_f+0x7b kern_ioctl+0x106 sys_ioctl+0xfd
> amd64_syscall+0x540 Xfast_syscall+0xf7
> 1535 100434 openvpn - mi_switch+0x186
> sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19
> _cv_timedwait_sig+0x135 seltdwait+0x8d sys_poll+0x478
> amd64_syscall+0x540 Xfast_syscall+0xf7
> 1545 100459 openvpn - mi_switch+0x186
> sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19
> _cv_timedwait_sig+0x135 seltdwait+0x8d sys_poll+0x478
> amd64_syscall+0x540 Xfast_syscall+0xf7
> 1567 100539 openvpn - mi_switch+0x186
> sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19
> _cv_timedwait_sig+0x135 seltdwait+0x8d sys_poll+0x478
> amd64_syscall+0x540 Xfast_syscall+0xf7
> 1628 100454 snapshot-manager - mi_switch+0x186
> sleepq_wait+0x42 _cv_wait+0x112 rrw_enter_read+0x4b
> dsl_pool_hold+0x44 dmu_objset_hold+0x2a zfs_ioc_objset_stats+0x23
> zfsdev_ioctl+0x58d devfs_ioctl_f+0x7b kern_ioctl+0x106
> sys_ioctl+0xfd amd64_syscall+0x540 Xfast_syscall+0xf7
> 2097 100710 <tel:2097%20100710> java -
> mi_switch+0x186 thread_suspend_switch+0xcd thread_single+0x1b2
> exit1+0x72 sys_sys_exit+0xe amd64_syscall+0x540 Xfast_syscall+0xf7
> 2097 100792 <tel:2097%20100792> java -
> mi_switch+0x186 sleepq_wait+0x42 __lockmgr_args+0x5cb
> vop_stdlock+0x39 VOP_LOCK1_APV+0x70 _vn_lock+0x47
> zfsctl_freebsd_root_lookup+0xd5 VOP_LOOKUP_APV+0x62 lookup+0x437
> namei+0x4ac kern_statat_vnhook+0xb3 kern_statat+0x15 sys_stat+0x2a
> amd64_syscall+0x540 Xfast_syscall+0xf7
> 2102 100635 <tel:2102%20100635> data-processor dm_siri_to_chiro
> mi_switch+0x186 sleepq_wait+0x42 _sx_slock_hard+0x318
> _sx_slock+0x56 zfs_freebsd_reclaim+0x4b VOP_RECLAIM_APV+0x68
> vgonel+0x134 vnlru_free+0x362 getnewvnode+0x27d
> gfs_file_create+0x4b gfs_dir_create+0x16
> zfsctl_snapdir_lookup+0x474 VOP_LOOKUP_APV+0x62 lookup+0x437
> namei+0x4ac kern_statat_vnhook+0xb3 kern_statat+0x15 sys_stat+0x2a
> 75715 100424 snapshot-manager-counts mi_switch+0x186
> sleepq_wait+0x42 _cv_wait+0x112 rrw_enter_read+0x4b
> dsl_pool_hold+0x44 dmu_objset_hold+0x2a zfs_ioc_objset_stats+0x23
> zfsdev_ioctl+0x58d devfs_ioctl_f+0x7b kern_ioctl+0x106
> sys_ioctl+0xfd amd64_syscall+0x540 Xfast_syscall+0xf7
> 75718 100861 snapshot-manager-counts mi_switch+0x186
> sleepq_wait+0x42 _cv_wait+0x112 rrw_enter_read+0x4b
> dsl_pool_hold+0x44 dmu_objset_hold+0x2a zfs_ioc_objset_stats+0x23
> zfsdev_ioctl+0x58d devfs_ioctl_f+0x7b kern_ioctl+0x106
> sys_ioctl+0xfd amd64_syscall+0x540 Xfast_syscall+0xf7
> 75719 100578 snapshot-manager-counts mi_switch+0x186
> sleepq_wait+0x42 _cv_wait+0x112 rrw_enter_read+0x4b
> dsl_pool_hold+0x44 dmu_objset_hold+0x2a zfs_ioc_objset_stats+0x23
> zfsdev_ioctl+0x58d devfs_ioctl_f+0x7b kern_ioctl+0x106
> sys_ioctl+0xfd amd64_syscall+0x540 Xfast_syscall+0xf7
> 76069 101690 reboot - mi_switch+0x186 sleepq_wait+0x42
> _cv_wait+0x112 rrw_enter_read+0x4b zfs_sync+0x5e sys_sync+0x1e8
> amd64_syscall+0x540 Xfast_syscall+0xf7
> 76196 100847 sshd - mi_switch+0x186
> sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _cv_wait_sig+0x12a
> seltdwait+0xf6 kern_select+0x6ef sys_select+0x5d
> amd64_syscall+0x540 Xfast_syscall+0xf7
> 76198 101378 csh - mi_switch+0x186
> sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _sleep+0x295
> kern_sigsuspend+0xab sys_sigsuspend+0x34 amd64_syscall+0x540
> Xfast_syscall+0xf7
> 76200 100892 top - mi_switch+0x186
> sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _cv_wait_sig+0x12a
> tty_wait+0x25 ttydisc_read+0x2dd ttydev_read+0xc4
> devfs_read_f+0x90 dofileread+0xa1 kern_readv+0x6c sys_read+0x64
> amd64_syscall+0x540 Xfast_syscall+0xf7
> 76203 100447 sshd - <running>
> 76205 101410 csh - mi_switch+0x186
> sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _sleep+0x295
> kern_sigsuspend+0xab sys_sigsuspend+0x34 amd64_syscall+0x540
> Xfast_syscall+0xf7
> 76209 101030 procstat - <running>
>
>
> _______________________________________________
> freebsd-fs at freebsd.org <mailto:freebsd-fs at freebsd.org> mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
> To unsubscribe, send any mail to
> "freebsd-fs-unsubscribe at freebsd.org
> <mailto:freebsd-fs-unsubscribe at freebsd.org>"
>
>
More information about the freebsd-fs
mailing list