ZFS related hang with FreeBSD 9.2
Steven Hartland
killing at multiplay.co.uk
Sat Dec 14 00:55:43 UTC 2013
Are you doing any snapshot sends as well as interacting with
snapshots such as listing files in them via the .zfs?
If so make sure you have the following patch applied as
that can cause a deadlock between these two operations
http://svnweb.freebsd.org/changeset/base/258595
Regards
Steve
----- Original Message -----
From: "Rod Taylor" <rod.taylor at gmail.com>
To: "Ryan Baldwin" <ryan.baldwin at nexusalpha.com>
Cc: <freebsd-fs at freebsd.org>
Sent: Friday, December 13, 2013 11:21 PM
Subject: Re: ZFS related hang with FreeBSD 9.2
> Are you using snapshots?
>
> I've found ZFS Snapshots on 9.0, 9.1, and 9.2 regularly crash the system.
> Delete the snapshots and don't create any new ones and suddenly it's stable
> for months.
>
>
>
> On Fri, Dec 13, 2013 at 12:14 AM, Ryan Baldwin
> <ryan.baldwin at nexusalpha.com>wrote:
>
>> Hi,
>>
>> We have a server based on FreeBSD 9.2 which hangs at times on a daily
>> basis. The longest uptime we have achieved is 5 days conversely it has
>> stopped daily several days in a row.
>>
>> When this occurs it appears there are two proceses stuck in 'tx->tx'
>> state. In the top output shown these are snapshot-manager processes which
>> create and destroy snapshots generally and sometime rollback filesystems to
>> snapshots. When the lockup occurs other processes which try to access the
>> file system can seem to end up stuck in state 'rrl->r'. The reboot command
>> that was issued to try and reboot the server has ended up stuck in this
>> state as can be seen.
>>
>> The server is not under particularly heavy load.
>>
>> It has remained in this state for hours. The 'deadman handler'? does not
>> appear to restart the system. Once this has occurred there is no further
>> disk activity.
>>
>> We did not experience this problem at all previously using 9.1 although we
>> had less snapshot-manager processes before. We have built this server
>> against 9.1 again now but it has only been going one day so far.
>>
>> We can try and reproduce this problem again on 9.2 if by doing so we can
>> gather any additional information that could help resolve this problem.
>> Please let me know what other information would be helpful.
>>
>> The hardware is a Dell R420 with Perc H310 raid controller in JBOD mode
>> with the pool mirrored on two SAS disks.
>>
>> Thanks
>>
>> top and procstat output follow:
>>
>>
>> last pid: 76225; load averages: 0.00, 0.00, 0.00 up 0+20:04:22
>> 09:56:24
>> 46 processes: 1 running, 44 sleeping, 1 stopped
>> CPU: % user, % nice, % system, % interrupt, % idle
>> Mem: 405M Active, 133M Inact, 1206M Wired, 6084M Free
>> ARC: 797M Total, 184M MFU, 488M MRU, 28M Anon, 32M Header, 65M Other
>> Swap: 8192M Total, 8192M Free
>>
>> PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU
>> COMMAND
>> 1170 root 1 20 0 16244K 5460K kqread 11 1:28 0.00%
>> CustomInit
>> 1531 root 1 20 0 39648K 6368K tx->tx 7 1:17 0.00%
>> snapshot-manager
>> 1135 root 1 20 0 16244K 5100K kqread 1 1:10 0.00%
>> CustomInit
>> 2097 root 2 20 0 2581M 472M STOP 4 0:46 0.00% java
>> 1183 root 1 20 0 16244K 5504K kqread 0 0:22 0.00%
>> CustomInit
>> 1145 root 1 20 0 16244K 5464K kqread 9 0:18 0.00%
>> CustomInit
>> 1444 root 1 20 0 39648K 6352K tx->tx 5 0:13 0.00%
>> snapshot-manager
>> 1628 root 1 20 0 39648K 6348K rrl->r 10 0:12 0.00%
>> snapshot-manager
>> 1168 root 1 20 0 16244K 5388K kqread 7 0:07 0.00%
>> CustomInit
>> 1535 root 1 20 0 22388K 14296K select 4 0:05 0.00%
>> openvpn
>> 1163 root 1 20 0 16244K 5388K kqread 2 0:04 0.00%
>> CustomInit
>> 1511 root 1 20 0 18292K 10192K select 0 0:04 0.00%
>> openvpn
>> 1156 root 1 20 0 16244K 5392K kqread 10 0:04 0.00%
>> CustomInit
>> 1174 root 1 20 0 16244K 5388K kqread 1 0:04 0.00%
>> CustomInit
>> 1161 root 1 20 0 16244K 5392K kqread 4 0:03 0.00%
>> CustomInit
>> 913 root 1 20 0 12076K 1820K select 7 0:03 0.00%
>> syslogd
>> 2102 root 1 20 0 109M 13616K zfsvfs 3 0:03 0.00%
>> data-processor
>> 1445 root 1 20 0 22388K 14296K select 5 0:02 0.00%
>> openvpn
>> 1014 root 1 20 0 22256K 3360K select 4 0:02 0.00% ntpd
>> 1494 root 1 20 0 18292K 10192K select 2 0:01 0.00%
>> openvpn
>> 1180 root 1 20 0 18292K 4780K select 0 0:01 0.00%
>> openvpn
>> 1505 root 1 20 0 18292K 4712K select 2 0:01 0.00%
>> openvpn
>> 1495 root 1 20 0 18292K 4712K select 5 0:00 0.00%
>> openvpn
>> 1479 root 1 20 0 18292K 4708K select 1 0:00 0.00%
>> openvpn
>> 1567 root 1 20 0 18292K 4712K select 3 0:00 0.00%
>> openvpn
>> 1545 root 1 20 0 18292K 4712K select 1 0:00 0.00%
>> openvpn
>> 1486 root 1 20 0 18292K 4708K select 0 0:00 0.00%
>> openvpn
>> 1443 root 1 20 0 18292K 4436K select 9 0:00 0.00%
>> openvpn
>> 1447 root 1 20 0 18292K 4448K select 4 0:00 0.00%
>> openvpn
>> 1496 root 1 20 0 18292K 4708K select 3 0:00 0.00%
>> openvpn
>> 1478 root 1 20 0 18292K 4704K select 5 0:00 0.00%
>> openvpn
>> 1488 root 1 20 0 18292K 10192K select 5 0:00 0.00%
>> openvpn
>> 1032 root 1 20 0 14176K 1860K nanslp 1 0:00 0.00% cron
>> 76069 root 1 20 0 9948K 1680K rrl->r 7 0:00 0.00% reboot
>> 782 root 1 20 0 10376K 4416K select 5 0:00 0.00% devd
>> 76196 root 1 20 0 51536K 5828K select 0 0:00 0.00% sshd
>> 76203 root 1 20 0 51536K 5828K select 2 0:00 0.00% sshd
>> 76205 root 1 20 0 17564K 3252K ttyin 11 0:00 0.00% csh
>> 76198 root 1 20 0 17564K 3252K pause 8 0:00 0.00% csh
>> 75718 root 1 20 0 35556K 3956K rrl->r 11 0:00 0.00%
>> snapshot-manager-counts
>> 75715 root 1 20 0 35556K 3956K rrl->r 10 0:00 0.00%
>> snapshot-manager-counts
>> 1181 root 1 52 0 12084K 1676K ttyin 1 0:00 0.00% getty
>>
>>
>> 0 100000 kernel swapper mi_switch+0x186
>> sleepq_timedwait+0x42 _sleep+0x1c9 scheduler+0x359 mi_startup+0x77
>> btext+0x2c
>> 0 100032 kernel firmware taskq mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100037 kernel ffs_trim taskq mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100038 kernel acpi_task_0 mi_switch+0x186
>> sleepq_wait+0x42 msleep_spin+0x194 taskqueue_thread_loop+0x67
>> fork_exit+0x11f fork_trampoline+0xe
>> 0 100039 kernel acpi_task_1 mi_switch+0x186
>> sleepq_wait+0x42 msleep_spin+0x194 taskqueue_thread_loop+0x67
>> fork_exit+0x11f fork_trampoline+0xe
>> 0 100040 kernel acpi_task_2 mi_switch+0x186
>> sleepq_wait+0x42 msleep_spin+0x194 taskqueue_thread_loop+0x67
>> fork_exit+0x11f fork_trampoline+0xe
>> 0 100041 kernel kqueue taskq mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100044 kernel thread taskq mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100052 kernel bge0 taskq mi_switch+0x186
>> sleepq_wait+0x42 msleep_spin+0x194 taskqueue_thread_loop+0x67
>> fork_exit+0x11f fork_trampoline+0xe
>> 0 100053 kernel bge1 taskq mi_switch+0x186
>> sleepq_wait+0x42 msleep_spin+0x194 taskqueue_thread_loop+0x67
>> fork_exit+0x11f fork_trampoline+0xe
>> 0 100062 kernel mca taskq mi_switch+0x186
>> sleepq_wait+0x42 msleep_spin+0x194 taskqueue_thread_loop+0x67
>> fork_exit+0x11f fork_trampoline+0xe
>> 0 100063 kernel system_taskq_0 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100064 kernel system_taskq_1 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100065 kernel system_taskq_2 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100066 kernel system_taskq_3 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100067 kernel system_taskq_4 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100068 kernel system_taskq_5 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100069 kernel system_taskq_6 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100070 kernel system_taskq_7 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100071 kernel system_taskq_8 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100072 kernel system_taskq_9 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100073 kernel system_taskq_10 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100074 kernel system_taskq_11 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100247 kernel zio_null_issue mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100248 kernel zio_null_intr mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100249 kernel zio_read_issue_0 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100250 kernel zio_read_issue_1 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100251 kernel zio_read_issue_2 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100252 kernel zio_read_issue_3 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100253 kernel zio_read_issue_4 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100254 kernel zio_read_issue_5 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100255 kernel zio_read_issue_6 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100256 kernel zio_read_issue_7 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100257 kernel zio_read_intr_0 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100258 kernel zio_read_intr_1 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100259 kernel zio_read_intr_2 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100260 kernel zio_read_intr_3 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100261 kernel zio_read_intr_4 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100262 kernel zio_read_intr_5 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100263 kernel zio_read_intr_6 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100264 kernel zio_read_intr_7 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100265 kernel zio_read_intr_8 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100266 kernel zio_read_intr_9 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100267 kernel zio_read_intr_10 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100268 kernel zio_read_intr_11 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100269 kernel zio_write_issue_ mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100270 kernel zio_write_issue_ mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100271 kernel zio_write_issue_ mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100272 kernel zio_write_issue_ mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100273 kernel zio_write_issue_ mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100274 kernel zio_write_issue_ mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100275 kernel zio_write_issue_ mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100276 kernel zio_write_issue_ mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100277 kernel zio_write_issue_ mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100278 kernel zio_write_issue_ mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100279 kernel zio_write_issue_ mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100280 kernel zio_write_issue_ mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100281 kernel zio_write_issue_ mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100282 kernel zio_write_issue_ mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100283 kernel zio_write_issue_ mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100284 kernel zio_write_issue_ mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100285 kernel zio_write_issue_ mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100286 kernel zio_write_intr_0 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100287 kernel zio_write_intr_1 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100288 kernel zio_write_intr_2 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100289 kernel zio_write_intr_3 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100290 kernel zio_write_intr_4 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100291 kernel zio_write_intr_5 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100292 kernel zio_write_intr_6 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100293 kernel zio_write_intr_7 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100294 kernel zio_write_intr_h mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100295 kernel zio_write_intr_h mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100296 kernel zio_write_intr_h mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100297 kernel zio_write_intr_h mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100298 kernel zio_write_intr_h mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100299 kernel zio_free_issue_0 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100300 kernel zio_free_issue_1 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100301 kernel zio_free_issue_2 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100302 kernel zio_free_issue_3 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100303 kernel zio_free_issue_4 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100304 kernel zio_free_issue_5 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100305 kernel zio_free_issue_6 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100306 kernel zio_free_issue_7 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100307 kernel zio_free_issue_8 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100308 kernel zio_free_issue_9 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100309 kernel zio_free_issue_1 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100310 kernel zio_free_issue_1 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100311 kernel zio_free_issue_1 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100312 kernel zio_free_issue_1 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100313 kernel zio_free_issue_1 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100314 kernel zio_free_issue_1 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100315 kernel zio_free_issue_1 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100316 kernel zio_free_issue_1 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100317 kernel zio_free_issue_1 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100318 kernel zio_free_issue_1 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100319 kernel zio_free_issue_2 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100320 kernel zio_free_issue_2 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100321 kernel zio_free_issue_2 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100322 kernel zio_free_issue_2 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100323 kernel zio_free_issue_2 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100324 kernel zio_free_issue_2 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100325 kernel zio_free_issue_2 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100326 kernel zio_free_issue_2 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100327 kernel zio_free_issue_2 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100328 kernel zio_free_issue_2 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100329 kernel zio_free_issue_3 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100330 kernel zio_free_issue_3 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100331 kernel zio_free_issue_3 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100332 kernel zio_free_issue_3 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100333 kernel zio_free_issue_3 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100334 kernel zio_free_issue_3 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100335 kernel zio_free_issue_3 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100336 kernel zio_free_issue_3 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100337 kernel zio_free_issue_3 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100338 kernel zio_free_issue_3 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100339 kernel zio_free_issue_4 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100340 kernel zio_free_issue_4 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100341 kernel zio_free_issue_4 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100342 kernel zio_free_issue_4 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100343 kernel zio_free_issue_4 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100344 kernel zio_free_issue_4 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100345 kernel zio_free_issue_4 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100346 kernel zio_free_issue_4 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100347 kernel zio_free_issue_4 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100348 kernel zio_free_issue_4 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100349 kernel zio_free_issue_5 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100350 kernel zio_free_issue_5 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100351 kernel zio_free_issue_5 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100352 kernel zio_free_issue_5 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100353 kernel zio_free_issue_5 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100354 kernel zio_free_issue_5 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100355 kernel zio_free_issue_5 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100356 kernel zio_free_issue_5 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100357 kernel zio_free_issue_5 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100358 kernel zio_free_issue_5 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100359 kernel zio_free_issue_6 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100360 kernel zio_free_issue_6 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100361 kernel zio_free_issue_6 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100362 kernel zio_free_issue_6 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100363 kernel zio_free_issue_6 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100364 kernel zio_free_issue_6 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100365 kernel zio_free_issue_6 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100366 kernel zio_free_issue_6 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100367 kernel zio_free_issue_6 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100368 kernel zio_free_issue_6 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100369 kernel zio_free_issue_7 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100370 kernel zio_free_issue_7 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100371 kernel zio_free_issue_7 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100372 kernel zio_free_issue_7 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100373 kernel zio_free_issue_7 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100374 kernel zio_free_issue_7 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100375 kernel zio_free_issue_7 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100376 kernel zio_free_issue_7 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100377 kernel zio_free_issue_7 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100378 kernel zio_free_issue_7 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100379 kernel zio_free_issue_8 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100380 kernel zio_free_issue_8 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100381 kernel zio_free_issue_8 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100382 kernel zio_free_issue_8 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100383 kernel zio_free_issue_8 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100384 kernel zio_free_issue_8 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100385 kernel zio_free_issue_8 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100386 kernel zio_free_issue_8 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100387 kernel zio_free_issue_8 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100388 kernel zio_free_issue_8 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100389 kernel zio_free_issue_9 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100390 kernel zio_free_issue_9 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100391 kernel zio_free_issue_9 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100392 kernel zio_free_issue_9 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100393 kernel zio_free_issue_9 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100394 kernel zio_free_issue_9 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100395 kernel zio_free_issue_9 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100396 kernel zio_free_issue_9 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100397 kernel zio_free_issue_9 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100398 kernel zio_free_issue_9 mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100399 kernel zio_free_intr mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100400 kernel zio_claim_issue mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100401 kernel zio_claim_intr mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100402 kernel zio_ioctl_issue mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100403 kernel zio_ioctl_intr mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100405 kernel zfs_vn_rele_task mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100418 kernel zil_clean mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100441 kernel zil_clean mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100477 kernel zil_clean mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100478 kernel zil_clean mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100479 kernel zil_clean mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100480 kernel zil_clean mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100481 kernel zil_clean mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100482 kernel zil_clean mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100484 kernel zil_clean mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100486 kernel zil_clean mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100488 kernel zil_clean mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100489 kernel zil_clean mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100490 kernel zil_clean mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100491 kernel zil_clean mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100492 kernel zil_clean mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100493 kernel zil_clean mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100494 kernel zil_clean mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100495 kernel zil_clean mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100496 kernel zil_clean mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100497 kernel zil_clean mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100498 kernel zil_clean mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100499 kernel zil_clean mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100636 kernel zil_clean mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 0 100638 kernel zil_clean mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 taskqueue_thread_loop+0xbb fork_exit+0x11f
>> fork_trampoline+0xe
>> 1 100002 init - mi_switch+0x186
>> sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _sleep+0x295
>> kern_wait6+0x8c3 kern_wait+0x9c sys_wait4+0x35 amd64_syscall+0x540
>> Xfast_syscall+0xf7
>> 2 100033 crypto - mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 crypto_proc+0x197 fork_exit+0x11f
>> fork_trampoline+0xe
>> 3 100034 crypto returns - mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 crypto_ret_proc+0x192 fork_exit+0x11f
>> fork_trampoline+0xe
>> 4 100061 ctl_thrd - mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 ctl_work_thread+0x2fd0 fork_exit+0x11f
>> fork_trampoline+0xe
>> 5 100075 zfskern arc_reclaim_thre mi_switch+0x186
>> sleepq_timedwait+0x42 _cv_timedwait+0x135 arc_reclaim_thread+0x29d
>> fork_exit+0x11f fork_trampoline+0xe
>> 5 100076 zfskern l2arc_feed_threa mi_switch+0x186
>> sleepq_timedwait+0x42 _cv_timedwait+0x135 l2arc_feed_thread+0x1a2
>> fork_exit+0x11f fork_trampoline+0xe
>> 5 100404 zfskern trim nsgroot mi_switch+0x186
>> sleepq_timedwait+0x42 _cv_timedwait+0x135 trim_thread+0x68 fork_exit+0x11f
>> fork_trampoline+0xe
>> 5 100406 zfskern txg_thread_enter mi_switch+0x186
>> sleepq_wait+0x42 _cv_wait+0x112 txg_thread_wait+0x79
>> txg_quiesce_thread+0xbb fork_exit+0x11f fork_trampoline+0xe
>> 5 100407 zfskern txg_thread_enter mi_switch+0x186
>> sleepq_wait+0x42 _cv_wait+0x112 rrw_enter_write+0x35
>> dsl_sync_task_sync+0xb8 dsl_pool_sync+0x47d spa_sync+0x3ba
>> txg_sync_thread+0x139 fork_exit+0x11f fork_trampoline+0xe
>> 5 100408 zfskern zvol nsgroot/swa mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 zvol_geom_worker+0xfc fork_exit+0x11f
>> fork_trampoline+0xe
>> 6 100077 sctp_iterator - mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 sctp_iterator_thread+0x41 fork_exit+0x11f
>> fork_trampoline+0xe
>> 7 100078 xpt_thrd - mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 xpt_scanner_thread+0xff fork_exit+0x11f
>> fork_trampoline+0xe
>> 8 100079 ipmi0: kcs - mi_switch+0x186
>> sleepq_wait+0x42 _cv_wait+0x112 ipmi_dequeue_request+0x47 kcs_loop+0x3d
>> fork_exit+0x11f fork_trampoline+0xe
>> 9 100080 enc_daemon0 - mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 enc_daemon+0xe4 fork_exit+0x11f
>> fork_trampoline+0xe
>> 10 100001 audit - mi_switch+0x186
>> sleepq_wait+0x42 _cv_wait+0x112 audit_worker+0x359 fork_exit+0x11f
>> fork_trampoline+0xe
>> 11 100003 idle idle: cpu0 <running>
>> 11 100004 idle idle: cpu1 <running>
>> 11 100005 idle idle: cpu2 <running>
>> 11 100006 idle idle: cpu3 <running>
>> 11 100007 idle idle: cpu4 <running>
>> 11 100008 idle idle: cpu5 <running>
>> 11 100009 idle idle: cpu6 <running>
>> 11 100010 idle idle: cpu7 <running>
>> 11 100011 idle idle: cpu8 <running>
>> 11 100012 idle idle: cpu9 <running>
>> 11 100013 idle idle: cpu10 mi_switch+0x186
>> critical_exit+0xa5 sched_idletd+0x118 fork_exit+0x11f fork_trampoline+0xe
>> 11 100014 idle idle: cpu11 <running>
>> 12 100015 intr swi4: clock mi_switch+0x186
>> ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe
>> 12 100016 intr swi4: clock mi_switch+0x186
>> ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe
>> 12 100017 intr swi4: clock mi_switch+0x186
>> ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe
>> 12 100018 intr swi4: clock mi_switch+0x186
>> ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe
>> 12 100019 intr swi4: clock mi_switch+0x186
>> ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe
>> 12 100020 intr swi4: clock mi_switch+0x186
>> ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe
>> 12 100021 intr swi4: clock mi_switch+0x186
>> ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe
>> 12 100022 intr swi4: clock mi_switch+0x186
>> ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe
>> 12 100023 intr swi4: clock mi_switch+0x186
>> ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe
>> 12 100024 intr swi4: clock mi_switch+0x186
>> ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe
>> 12 100025 intr swi4: clock mi_switch+0x186
>> ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe
>> 12 100026 intr swi4: clock mi_switch+0x186
>> ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe
>> 12 100027 intr swi3: vm
>> 12 100028 intr swi1: netisr 0 mi_switch+0x186
>> ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe
>> 12 100036 intr swi6: task queue mi_switch+0x186
>> ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe
>> 12 100042 intr swi2: cambio mi_switch+0x186
>> ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe
>> 12 100043 intr swi5: fast taskq
>> 12 100045 intr swi6: Giant task mi_switch+0x186
>> ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe
>> 12 100046 intr irq264: mfi0 mi_switch+0x186
>> ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe
>> 12 100047 intr irq23: ehci0 mi_switch+0x186
>> ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe
>> 12 100054 intr irq22: ehci1 mi_switch+0x186
>> ithread_loop+0x21e fork_exit+0x11f fork_trampoline+0xe
>> 12 100059 intr irq267: ahci0
>> 12 100060 intr swi0: uart uart
>> 13 100029 geom g_event mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 g_run_events+0x440 fork_exit+0x11f
>> fork_trampoline+0xe
>> 13 100030 geom g_up mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 g_io_schedule_up+0xe6 g_up_procbody+0x5c
>> fork_exit+0x11f fork_trampoline+0xe
>> 13 100031 geom g_down mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 g_io_schedule_down+0x25f g_down_procbody+0x5c
>> fork_exit+0x11f fork_trampoline+0xe
>> 14 100035 yarrow - mi_switch+0x186
>> sleepq_timedwait+0x42 _sleep+0x1c9 random_kthread+0x1ea fork_exit+0x11f
>> fork_trampoline+0xe
>> 15 100048 usb usbus0 mi_switch+0x186
>> sleepq_wait+0x42 _cv_wait+0x112 usb_process+0x18b fork_exit+0x11f
>> fork_trampoline+0xe
>> 15 100049 usb usbus0 mi_switch+0x186
>> sleepq_wait+0x42 _cv_wait+0x112 usb_process+0x18b fork_exit+0x11f
>> fork_trampoline+0xe
>> 15 100050 usb usbus0 mi_switch+0x186
>> sleepq_wait+0x42 _cv_wait+0x112 usb_process+0x18b fork_exit+0x11f
>> fork_trampoline+0xe
>> 15 100051 usb usbus0 mi_switch+0x186
>> sleepq_wait+0x42 _cv_wait+0x112 usb_process+0x18b fork_exit+0x11f
>> fork_trampoline+0xe
>> 15 100055 usb usbus1 mi_switch+0x186
>> sleepq_wait+0x42 _cv_wait+0x112 usb_process+0x18b fork_exit+0x11f
>> fork_trampoline+0xe
>> 15 100056 usb usbus1 mi_switch+0x186
>> sleepq_wait+0x42 _cv_wait+0x112 usb_process+0x18b fork_exit+0x11f
>> fork_trampoline+0xe
>> 15 100057 usb usbus1 mi_switch+0x186
>> sleepq_wait+0x42 _cv_wait+0x112 usb_process+0x18b fork_exit+0x11f
>> fork_trampoline+0xe
>> 15 100058 usb usbus1 mi_switch+0x186
>> sleepq_wait+0x42 _cv_wait+0x112 usb_process+0x18b fork_exit+0x11f
>> fork_trampoline+0xe
>> 16 100081 pagedaemon - mi_switch+0x186
>> sleepq_timedwait+0x42 _sleep+0x1c9 vm_pageout+0xb34 fork_exit+0x11f
>> fork_trampoline+0xe
>> 17 100082 vmdaemon - mi_switch+0x186
>> sleepq_wait+0x42 _sleep+0x379 vm_daemon+0x58 fork_exit+0x11f
>> fork_trampoline+0xe
>> 18 100083 pagezero - mi_switch+0x186
>> sleepq_timedwait+0x42 _sleep+0x1c9 vm_pagezero+0x83 fork_exit+0x11f
>> fork_trampoline+0xe
>> 19 100084 bufdaemon - mi_switch+0x186
>> sleepq_timedwait+0x42 _sleep+0x1c9 buf_daemon+0x1e1 fork_exit+0x11f
>> fork_trampoline+0xe
>> 20 100085 syncer - mi_switch+0x186
>> sleepq_wait+0x42 _cv_wait+0x112 rrw_enter_read+0x4b zfs_sync+0x5e
>> sync_fsync+0x1a2 VOP_FSYNC_APV+0x68 sync_vnode+0x16b sched_sync+0x1c5
>> fork_exit+0x11f fork_trampoline+0xe
>> 21 100086 vnlru - mi_switch+0x186
>> sleepq_wait+0x42 _sx_slock_hard+0x318 _sx_slock+0x56
>> zfs_freebsd_reclaim+0x4b VOP_RECLAIM_APV+0x68 vgonel+0x134 vnlru_free+0x362
>> vnlru_proc+0x61e fork_exit+0x11f fork_trampoline+0xe
>> 22 100087 softdepflush - mi_switch+0x186
>> sleepq_timedwait+0x42 _sleep+0x1c9 softdep_flush+0x375 fork_exit+0x11f
>> fork_trampoline+0xe
>> 782 100432 devd - mi_switch+0x186
>> sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19
>> _cv_timedwait_sig+0x135 seltdwait+0x8d kern_select+0x6ef sys_select+0x5d
>> amd64_syscall+0x540 Xfast_syscall+0xf7
>> 790 100545 pfpurge - mi_switch+0x186
>> sleepq_timedwait+0x42 _sleep+0x1c9 pf_purge_thread+0x31 fork_exit+0x11f
>> fork_trampoline+0xe
>> 913 100449 syslogd - mi_switch+0x186
>> sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _cv_wait_sig+0x12a
>> seltdwait+0xf6 kern_select+0x6ef sys_select+0x5d amd64_syscall+0x540
>> Xfast_syscall+0xf7
>> 1014 100581 ntpd - mi_switch+0x186
>> sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _cv_wait_sig+0x12a
>> seltdwait+0xf6 kern_select+0x6ef sys_select+0x5d amd64_syscall+0x540
>> Xfast_syscall+0xf7
>> 1028 100532 sshd - mi_switch+0x186
>> sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _cv_wait_sig+0x12a
>> seltdwait+0xf6 kern_select+0x6ef sys_select+0x5d amd64_syscall+0x540
>> Xfast_syscall+0xf7
>> 1032 100473 cron - mi_switch+0x186
>> sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19 _sleep+0x2ca
>> kern_nanosleep+0x118 sys_nanosleep+0x6e amd64_syscall+0x540
>> Xfast_syscall+0xf7
>> 1135 100558 CustomInit - mi_switch+0x186
>> sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _sleep+0x295
>> kern_kevent+0x369 sys_kevent+0x90 amd64_syscall+0x540 Xfast_syscall+0xf7
>> 1145 100509 CustomInit - mi_switch+0x186
>> sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _sleep+0x295
>> kern_kevent+0x369 sys_kevent+0x90 amd64_syscall+0x540 Xfast_syscall+0xf7
>> 1156 100508 CustomInit - mi_switch+0x186
>> sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _sleep+0x295
>> kern_kevent+0x369 sys_kevent+0x90 amd64_syscall+0x540 Xfast_syscall+0xf7
>> 1161 100507 CustomInit - mi_switch+0x186
>> sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _sleep+0x295
>> kern_kevent+0x369 sys_kevent+0x90 amd64_syscall+0x540 Xfast_syscall+0xf7
>> 1163 100521 CustomInit - mi_switch+0x186
>> sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _sleep+0x295
>> kern_kevent+0x369 sys_kevent+0x90 amd64_syscall+0x540 Xfast_syscall+0xf7
>> 1168 100574 CustomInit - mi_switch+0x186
>> sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _sleep+0x295
>> kern_kevent+0x369 sys_kevent+0x90 amd64_syscall+0x540 Xfast_syscall+0xf7
>> 1170 100561 CustomInit - mi_switch+0x186
>> sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _sleep+0x295
>> kern_kevent+0x369 sys_kevent+0x90 amd64_syscall+0x540 Xfast_syscall+0xf7
>> 1174 100560 CustomInit - mi_switch+0x186
>> sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _sleep+0x295
>> kern_kevent+0x369 sys_kevent+0x90 amd64_syscall+0x540 Xfast_syscall+0xf7
>> 1180 100420 openvpn - mi_switch+0x186
>> sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19
>> _cv_timedwait_sig+0x135 seltdwait+0x8d sys_poll+0x478 amd64_syscall+0x540
>> Xfast_syscall+0xf7
>> 1181 100463 getty - mi_switch+0x186
>> sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _cv_wait_sig+0x12a
>> tty_wait+0x25 ttydisc_read+0x2dd ttydev_read+0xc4 devfs_read_f+0x90
>> dofileread+0xa1 kern_readv+0x6c sys_read+0x64 amd64_syscall+0x540
>> Xfast_syscall+0xf7
>> 1182 100430 getty - mi_switch+0x186
>> sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _cv_wait_sig+0x12a
>> tty_wait+0x25 ttydisc_read+0x2dd ttydev_read+0xc4 devfs_read_f+0x90
>> dofileread+0xa1 kern_readv+0x6c sys_read+0x64 amd64_syscall+0x540
>> Xfast_syscall+0xf7
>> 1183 100421 CustomInit - mi_switch+0x186
>> sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _sleep+0x295
>> kern_kevent+0x369 sys_kevent+0x90 amd64_syscall+0x540 Xfast_syscall+0xf7
>> 1443 100599 openvpn - mi_switch+0x186
>> sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19
>> _cv_timedwait_sig+0x135 seltdwait+0x8d sys_poll+0x478 amd64_syscall+0x540
>> Xfast_syscall+0xf7
>> 1444 100527 snapshot-manager - mi_switch+0x186
>> sleepq_wait+0x42 _cv_wait+0x112 txg_wait_synced+0x85 dsl_sync_task+0x139
>> dsl_destroy_snapshots_nvl+0x71 dsl_destroy_snapshot+0x4a
>> zfs_ioc_destroy+0x3e zfsdev_ioctl+0x58d devfs_ioctl_f+0x7b kern_ioctl+0x106
>> sys_ioctl+0xfd amd64_syscall+0x540 Xfast_syscall+0xf7
>> 1445 100514 openvpn - mi_switch+0x186
>> sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19
>> _cv_timedwait_sig+0x135 seltdwait+0x8d sys_poll+0x478 amd64_syscall+0x540
>> Xfast_syscall+0xf7
>> 1447 100557 openvpn - mi_switch+0x186
>> sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19
>> _cv_timedwait_sig+0x135 seltdwait+0x8d sys_poll+0x478 amd64_syscall+0x540
>> Xfast_syscall+0xf7
>> 1478 100555 openvpn - mi_switch+0x186
>> sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19
>> _cv_timedwait_sig+0x135 seltdwait+0x8d sys_poll+0x478 amd64_syscall+0x540
>> Xfast_syscall+0xf7
>> 1479 100551 openvpn - mi_switch+0x186
>> sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19
>> _cv_timedwait_sig+0x135 seltdwait+0x8d sys_poll+0x478 amd64_syscall+0x540
>> Xfast_syscall+0xf7
>> 1486 100576 openvpn - mi_switch+0x186
>> sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19
>> _cv_timedwait_sig+0x135 seltdwait+0x8d sys_poll+0x478 amd64_syscall+0x540
>> Xfast_syscall+0xf7
>> 1488 100563 openvpn - mi_switch+0x186
>> sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19
>> _cv_timedwait_sig+0x135 seltdwait+0x8d sys_poll+0x478 amd64_syscall+0x540
>> Xfast_syscall+0xf7
>> 1494 100575 openvpn - mi_switch+0x186
>> sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19
>> _cv_timedwait_sig+0x135 seltdwait+0x8d sys_poll+0x478 amd64_syscall+0x540
>> Xfast_syscall+0xf7
>> 1495 100442 openvpn - mi_switch+0x186
>> sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19
>> _cv_timedwait_sig+0x135 seltdwait+0x8d sys_poll+0x478 amd64_syscall+0x540
>> Xfast_syscall+0xf7
>> 1496 100556 openvpn - mi_switch+0x186
>> sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19
>> _cv_timedwait_sig+0x135 seltdwait+0x8d sys_poll+0x478 amd64_syscall+0x540
>> Xfast_syscall+0xf7
>> 1505 100431 openvpn - mi_switch+0x186
>> sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19
>> _cv_timedwait_sig+0x135 seltdwait+0x8d sys_poll+0x478 amd64_syscall+0x540
>> Xfast_syscall+0xf7
>> 1511 100475 openvpn - mi_switch+0x186
>> sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19
>> _cv_timedwait_sig+0x135 seltdwait+0x8d sys_poll+0x478 amd64_syscall+0x540
>> Xfast_syscall+0xf7
>> 1531 100429 snapshot-manager - mi_switch+0x186
>> sleepq_wait+0x42 _cv_wait+0x112 txg_wait_synced+0x85 dsl_sync_task+0x139
>> zfs_ioc_rollback+0xb8 zfsdev_ioctl+0x58d devfs_ioctl_f+0x7b
>> kern_ioctl+0x106 sys_ioctl+0xfd amd64_syscall+0x540 Xfast_syscall+0xf7
>> 1535 100434 openvpn - mi_switch+0x186
>> sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19
>> _cv_timedwait_sig+0x135 seltdwait+0x8d sys_poll+0x478 amd64_syscall+0x540
>> Xfast_syscall+0xf7
>> 1545 100459 openvpn - mi_switch+0x186
>> sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19
>> _cv_timedwait_sig+0x135 seltdwait+0x8d sys_poll+0x478 amd64_syscall+0x540
>> Xfast_syscall+0xf7
>> 1567 100539 openvpn - mi_switch+0x186
>> sleepq_catch_signals+0x2e1 sleepq_timedwait_sig+0x19
>> _cv_timedwait_sig+0x135 seltdwait+0x8d sys_poll+0x478 amd64_syscall+0x540
>> Xfast_syscall+0xf7
>> 1628 100454 snapshot-manager - mi_switch+0x186
>> sleepq_wait+0x42 _cv_wait+0x112 rrw_enter_read+0x4b dsl_pool_hold+0x44
>> dmu_objset_hold+0x2a zfs_ioc_objset_stats+0x23 zfsdev_ioctl+0x58d
>> devfs_ioctl_f+0x7b kern_ioctl+0x106 sys_ioctl+0xfd amd64_syscall+0x540
>> Xfast_syscall+0xf7
>> 2097 100710 java - mi_switch+0x186
>> thread_suspend_switch+0xcd thread_single+0x1b2 exit1+0x72 sys_sys_exit+0xe
>> amd64_syscall+0x540 Xfast_syscall+0xf7
>> 2097 100792 java - mi_switch+0x186
>> sleepq_wait+0x42 __lockmgr_args+0x5cb vop_stdlock+0x39 VOP_LOCK1_APV+0x70
>> _vn_lock+0x47 zfsctl_freebsd_root_lookup+0xd5 VOP_LOOKUP_APV+0x62
>> lookup+0x437 namei+0x4ac kern_statat_vnhook+0xb3 kern_statat+0x15
>> sys_stat+0x2a amd64_syscall+0x540 Xfast_syscall+0xf7
>> 2102 100635 data-processor dm_siri_to_chiro mi_switch+0x186
>> sleepq_wait+0x42 _sx_slock_hard+0x318 _sx_slock+0x56
>> zfs_freebsd_reclaim+0x4b VOP_RECLAIM_APV+0x68 vgonel+0x134 vnlru_free+0x362
>> getnewvnode+0x27d gfs_file_create+0x4b gfs_dir_create+0x16
>> zfsctl_snapdir_lookup+0x474 VOP_LOOKUP_APV+0x62 lookup+0x437 namei+0x4ac
>> kern_statat_vnhook+0xb3 kern_statat+0x15 sys_stat+0x2a
>> 75715 100424 snapshot-manager-counts mi_switch+0x186 sleepq_wait+0x42
>> _cv_wait+0x112 rrw_enter_read+0x4b dsl_pool_hold+0x44 dmu_objset_hold+0x2a
>> zfs_ioc_objset_stats+0x23 zfsdev_ioctl+0x58d devfs_ioctl_f+0x7b
>> kern_ioctl+0x106 sys_ioctl+0xfd amd64_syscall+0x540 Xfast_syscall+0xf7
>> 75718 100861 snapshot-manager-counts mi_switch+0x186 sleepq_wait+0x42
>> _cv_wait+0x112 rrw_enter_read+0x4b dsl_pool_hold+0x44 dmu_objset_hold+0x2a
>> zfs_ioc_objset_stats+0x23 zfsdev_ioctl+0x58d devfs_ioctl_f+0x7b
>> kern_ioctl+0x106 sys_ioctl+0xfd amd64_syscall+0x540 Xfast_syscall+0xf7
>> 75719 100578 snapshot-manager-counts mi_switch+0x186 sleepq_wait+0x42
>> _cv_wait+0x112 rrw_enter_read+0x4b dsl_pool_hold+0x44 dmu_objset_hold+0x2a
>> zfs_ioc_objset_stats+0x23 zfsdev_ioctl+0x58d devfs_ioctl_f+0x7b
>> kern_ioctl+0x106 sys_ioctl+0xfd amd64_syscall+0x540 Xfast_syscall+0xf7
>> 76069 101690 reboot - mi_switch+0x186
>> sleepq_wait+0x42 _cv_wait+0x112 rrw_enter_read+0x4b zfs_sync+0x5e
>> sys_sync+0x1e8 amd64_syscall+0x540 Xfast_syscall+0xf7
>> 76196 100847 sshd - mi_switch+0x186
>> sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _cv_wait_sig+0x12a
>> seltdwait+0xf6 kern_select+0x6ef sys_select+0x5d amd64_syscall+0x540
>> Xfast_syscall+0xf7
>> 76198 101378 csh - mi_switch+0x186
>> sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _sleep+0x295
>> kern_sigsuspend+0xab sys_sigsuspend+0x34 amd64_syscall+0x540
>> Xfast_syscall+0xf7
>> 76200 100892 top - mi_switch+0x186
>> sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _cv_wait_sig+0x12a
>> tty_wait+0x25 ttydisc_read+0x2dd ttydev_read+0xc4 devfs_read_f+0x90
>> dofileread+0xa1 kern_readv+0x6c sys_read+0x64 amd64_syscall+0x540
>> Xfast_syscall+0xf7
>> 76203 100447 sshd - <running>
>> 76205 101410 csh - mi_switch+0x186
>> sleepq_catch_signals+0x2e1 sleepq_wait_sig+0x16 _sleep+0x295
>> kern_sigsuspend+0xab sys_sigsuspend+0x34 amd64_syscall+0x540
>> Xfast_syscall+0xf7
>> 76209 101030 procstat - <running>
>>
>>
>> _______________________________________________
>> freebsd-fs at freebsd.org mailing list
>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
>> To unsubscribe, send any mail to "freebsd-fs-unsubscribe at freebsd.org"
>>
> _______________________________________________
> freebsd-fs at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
> To unsubscribe, send any mail to "freebsd-fs-unsubscribe at freebsd.org"
>
================================================
This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it.
In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337
or return the E.mail to postmaster at multiplay.co.uk.
More information about the freebsd-fs
mailing list