[zfs] 10.1-R i386 deadlock
Alex Bakhtin
alex.bakhtin at gmail.com
Fri Jul 31 16:42:00 UTC 2015
Hello,
I'm experiencing a problems with ZFS locks on my i386 buildbox. This is
a VM running under vSphere with 4 vCPU (non-HT) and 4G RAM. I'm
experiencing a deadlock sometime in the normal build process (poudriere)
and right now I found a way to easily reproduce it - make index in
/usr/ports hangs (tried several times - no success).
To troubleshoot this I moved VM from very slow NFS datastore to a fast
SSD based local one (the idea was - if the problem with some timing - it
would be fixed). This doesn't help. Right now I have the latest
10.1-RELEASE:
FreeBSD builder-i386 10.1-RELEASE-p16 FreeBSD 10.1-RELEASE-p16 #0: Tue Jul
28 11:41:12 UTC 2015
root at amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC
i386
I read the recomendations in https://wiki.freebsd.org/AvgZfsDeadlockDebug
and I really see some threads locked in *zfs_freebsd_read* and *arc_read.*
procstat -kk -a output attached.
ufs based root is fine. To recover usually I have to reset VM from vSphere.
I have a recommended ZFS tunables in loader.conf:
======================================
vm.kmem_size="512M"
vm.kmem_size_max="512M"
vfs.zfs.arc_max="40M"
vfs.zfs.vdev.cache.size="5M"
vfs.zfs.prefetch_disable=0
# Decrease ZFS txg timeout value from 30 (default) to 5 seconds. This
# should increase throughput and decrease the "bursty" stalls that
# happen during immense I/O with ZFS.
# http://lists.freebsd.org/pipermail/freebsd-fs/2009-December/007343.html
# http://lists.freebsd.org/pipermail/freebsd-fs/2009-December/007355.html
# default in FreeBSD since ZFS v28
vfs.zfs.txg.timeout="5"
======================================
==============================
bakhtin at builder-i386:~ % zpool status
pool: storage
state: ONLINE
scan: scrub repaired 0 in 0h16m with 0 errors on Fri Jun 12 09:43:00 2015
config:
NAME STATE READ WRITE CKSUM
storage ONLINE 0 0 0
da1 ONLINE 0 0 0
errors: No known data errors
==============================
I have no ideas. Because of this problem I have to reset VM and restart
poudriere build several times to build packages I need. Moving to amd64 is
not an option because I need 32bit host to build packages for armv6 target.
The same setup of amd64 host on the same vSphere is working pretty fine.
--
---
Alex Bakhtin
-------------- next part --------------
PID TID COMM TDNAME KSTACK
0 100000 kernel swapper mi_switch+0x122 sleepq_switch+0x15b sleepq_timedwait+0x3f _sleep+0x282 swapper+0x2c0 begin+0x2c
0 100016 kernel firmware taskq mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100018 kernel acpi_task_0 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f msleep_spin_sbt+0x1d4 taskqueue_thread_loop+0x15c fork_exit+0xa3 fork_trampoline+0x8
0 100019 kernel acpi_task_1 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f msleep_spin_sbt+0x1d4 taskqueue_thread_loop+0x15c fork_exit+0xa3 fork_trampoline+0x8
0 100020 kernel acpi_task_2 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f msleep_spin_sbt+0x1d4 taskqueue_thread_loop+0x15c fork_exit+0xa3 fork_trampoline+0x8
0 100023 kernel thread taskq mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100024 kernel ffs_trim taskq mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100026 kernel kqueue taskq mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100033 kernel vmx0 taskq mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100040 kernel CAM taskq mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100053 kernel system_taskq_0 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100054 kernel system_taskq_1 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100055 kernel system_taskq_2 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100056 kernel system_taskq_3 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100198 kernel zio_null_issue mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100199 kernel zio_null_intr mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100200 kernel zio_read_issue_0 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100201 kernel zio_read_issue_1 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100202 kernel zio_read_issue_2 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100203 kernel zio_read_issue_3 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100204 kernel zio_read_issue_4 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100205 kernel zio_read_issue_5 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100206 kernel zio_read_issue_6 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100207 kernel zio_read_issue_7 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100208 kernel zio_read_intr_0 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100209 kernel zio_read_intr_1 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100210 kernel zio_read_intr_2 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100211 kernel zio_write_issue_ mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100212 kernel zio_write_issue_ mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100213 kernel zio_write_issue_ mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100214 kernel zio_write_issue_ mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100215 kernel zio_write_issue_ mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100216 kernel zio_write_issue_ mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100217 kernel zio_write_issue_ mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100218 kernel zio_write_issue_ mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100219 kernel zio_write_intr_0 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100220 kernel zio_write_intr_1 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100221 kernel zio_write_intr_2 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100222 kernel zio_write_intr_3 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100223 kernel zio_write_intr_4 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100224 kernel zio_write_intr_5 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100225 kernel zio_write_intr_6 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100226 kernel zio_write_intr_7 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100227 kernel zio_write_intr_h mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100228 kernel zio_write_intr_h mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100229 kernel zio_write_intr_h mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100230 kernel zio_write_intr_h mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100231 kernel zio_write_intr_h mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100232 kernel zio_free_issue_0 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100233 kernel zio_free_issue_0 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100234 kernel zio_free_issue_0 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100235 kernel zio_free_issue_0 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100236 kernel zio_free_issue_0 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100237 kernel zio_free_issue_0 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100238 kernel zio_free_issue_0 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100239 kernel zio_free_issue_0 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100240 kernel zio_free_issue_0 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100241 kernel zio_free_issue_0 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100242 kernel zio_free_issue_0 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100243 kernel zio_free_issue_0 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100244 kernel zio_free_issue_1 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100245 kernel zio_free_issue_1 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100246 kernel zio_free_issue_1 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100247 kernel zio_free_issue_1 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100248 kernel zio_free_issue_1 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100249 kernel zio_free_issue_1 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100250 kernel zio_free_issue_1 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100251 kernel zio_free_issue_1 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100252 kernel zio_free_issue_1 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100253 kernel zio_free_issue_1 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100254 kernel zio_free_issue_1 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100255 kernel zio_free_issue_1 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100256 kernel zio_free_issue_2 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100257 kernel zio_free_issue_2 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100258 kernel zio_free_issue_2 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100259 kernel zio_free_issue_2 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100260 kernel zio_free_issue_2 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100261 kernel zio_free_issue_2 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100262 kernel zio_free_issue_2 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100263 kernel zio_free_issue_2 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100264 kernel zio_free_issue_2 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100265 kernel zio_free_issue_2 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100266 kernel zio_free_issue_2 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100267 kernel zio_free_issue_2 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100268 kernel zio_free_issue_3 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100269 kernel zio_free_issue_3 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100270 kernel zio_free_issue_3 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100271 kernel zio_free_issue_3 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100272 kernel zio_free_issue_3 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100273 kernel zio_free_issue_3 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100274 kernel zio_free_issue_3 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100275 kernel zio_free_issue_3 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100276 kernel zio_free_issue_3 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100277 kernel zio_free_issue_3 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100278 kernel zio_free_issue_3 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100279 kernel zio_free_issue_3 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100280 kernel zio_free_issue_4 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100281 kernel zio_free_issue_4 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100282 kernel zio_free_issue_4 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100283 kernel zio_free_issue_4 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100284 kernel zio_free_issue_4 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100285 kernel zio_free_issue_4 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100286 kernel zio_free_issue_4 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100287 kernel zio_free_issue_4 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100288 kernel zio_free_issue_4 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100289 kernel zio_free_issue_4 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100290 kernel zio_free_issue_4 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100291 kernel zio_free_issue_4 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100292 kernel zio_free_issue_5 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100293 kernel zio_free_issue_5 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100294 kernel zio_free_issue_5 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100295 kernel zio_free_issue_5 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100296 kernel zio_free_issue_5 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100297 kernel zio_free_issue_5 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100298 kernel zio_free_issue_5 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100299 kernel zio_free_issue_5 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100300 kernel zio_free_issue_5 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100301 kernel zio_free_issue_5 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100302 kernel zio_free_issue_5 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100303 kernel zio_free_issue_5 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100304 kernel zio_free_issue_6 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100305 kernel zio_free_issue_6 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100306 kernel zio_free_issue_6 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100307 kernel zio_free_issue_6 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100308 kernel zio_free_issue_6 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100309 kernel zio_free_issue_6 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100310 kernel zio_free_issue_6 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100311 kernel zio_free_issue_6 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100312 kernel zio_free_issue_6 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100313 kernel zio_free_issue_6 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100314 kernel zio_free_issue_6 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100315 kernel zio_free_issue_6 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100316 kernel zio_free_issue_7 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100317 kernel zio_free_issue_7 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100318 kernel zio_free_issue_7 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100319 kernel zio_free_issue_7 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100320 kernel zio_free_issue_7 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100321 kernel zio_free_issue_7 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100322 kernel zio_free_issue_7 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100323 kernel zio_free_issue_7 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100324 kernel zio_free_issue_7 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100325 kernel zio_free_issue_7 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100326 kernel zio_free_issue_7 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100327 kernel zio_free_issue_7 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100328 kernel zio_free_intr mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100329 kernel zio_claim_issue mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100330 kernel zio_claim_intr mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100331 kernel zio_ioctl_issue mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100332 kernel zio_ioctl_intr mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100334 kernel metaslab_group_t mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100335 kernel metaslab_group_t mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100336 kernel zfs_vn_rele_task mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100347 kernel zil_clean mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100348 kernel zil_clean mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100349 kernel zil_clean mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100350 kernel zil_clean mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100351 kernel zil_clean mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100352 kernel zil_clean mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100353 kernel zil_clean mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100354 kernel zil_clean mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100355 kernel zil_clean mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100356 kernel zil_clean mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100357 kernel zil_clean mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100358 kernel zil_clean mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100359 kernel zil_clean mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100360 kernel zil_clean mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100361 kernel zil_clean mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100362 kernel zil_clean mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100363 kernel zil_clean mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100364 kernel zil_clean mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100365 kernel zil_clean mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100366 kernel zil_clean mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100367 kernel zil_clean mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
0 100368 kernel zil_clean mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae taskqueue_thread_loop+0x11b fork_exit+0xa3 fork_trampoline+0x8
1 100002 init - mi_switch+0x122 sleepq_switch+0x15b sleepq_catch_signals+0x5be sleepq_wait_sig+0x14 _sleep+0x29b kern_wait6+0x725 sys_wait4+0x94 syscall+0x48b Xint0x80_syscall+0x21
2 100027 cam doneq0 mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae xpt_done_td+0xce fork_exit+0xa3 fork_trampoline+0x8
2 100041 cam scanner mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae xpt_scanner_thread+0xcc fork_exit+0xa3 fork_trampoline+0x8
3 100031 mpt_recovery0 - mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae mpt_recovery_thread+0xd6 fork_exit+0xa3 fork_trampoline+0x8
4 100038 fdc0 - mi_switch+0x122 sleepq_switch+0x15b sleepq_timedwait+0x3f _sleep+0x282 fdc_thread+0x877 fork_exit+0xa3 fork_trampoline+0x8
5 100039 sctp_iterator - mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae sctp_iterator_thread+0x9c fork_exit+0xa3 fork_trampoline+0x8
6 100042 pagedaemon - mi_switch+0x122 sleepq_switch+0x15b sleepq_timedwait+0x3f _sleep+0x282 vm_pageout+0x2ab fork_exit+0xa3 fork_trampoline+0x8
7 100043 vmdaemon - mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae vm_daemon+0xcf fork_exit+0xa3 fork_trampoline+0x8
8 100044 pagezero - mi_switch+0x122 sleepq_switch+0x15b sleepq_timedwait+0x3f _sleep+0x282 vm_pagezero+0xd2 fork_exit+0xa3 fork_trampoline+0x8
9 100045 bufdaemon - mi_switch+0x122 sleepq_switch+0x15b sleepq_timedwait+0x3f _sleep+0x282 buf_daemon+0xac fork_exit+0xa3 fork_trampoline+0x8
9 100343 bufdaemon / worker mi_switch+0x122 sleepq_switch+0x15b sleepq_timedwait+0x3f _sleep+0x282 softdep_flush+0x1fa fork_exit+0xa3 fork_trampoline+0x8
10 100001 audit - mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _cv_wait+0x182 audit_worker+0xa4 fork_exit+0xa3 fork_trampoline+0x8
11 100003 idle idle: cpu0 <running>
11 100004 idle idle: cpu1 mi_switch+0x122 sched_idletd+0x3cf fork_exit+0xa3 fork_trampoline+0x8
11 100005 idle idle: cpu2 <running>
11 100006 idle idle: cpu3 <running>
12 100007 intr swi3: vm
12 100008 intr swi4: clock mi_switch+0x122 ithread_loop+0x1b1 fork_exit+0xa3 fork_trampoline+0x8
12 100009 intr swi4: clock
12 100010 intr swi4: clock
12 100011 intr swi4: clock
12 100012 intr swi1: netisr 0 mi_switch+0x122 ithread_loop+0x1b1 fork_exit+0xa3 fork_trampoline+0x8
12 100021 intr swi6: task queue mi_switch+0x122 ithread_loop+0x1b1 fork_exit+0xa3 fork_trampoline+0x8
12 100022 intr swi6: Giant task mi_switch+0x122 ithread_loop+0x1b1 fork_exit+0xa3 fork_trampoline+0x8
12 100025 intr swi5: fast taskq
12 100028 intr irq14: ata0
12 100029 intr irq15: ata1 mi_switch+0x122 ithread_loop+0x1b1 fork_exit+0xa3 fork_trampoline+0x8
12 100030 intr irq17: mpt0 mi_switch+0x122 ithread_loop+0x1b1 fork_exit+0xa3 fork_trampoline+0x8
12 100032 intr irq256: vmx0 mi_switch+0x122 ithread_loop+0x1b1 fork_exit+0xa3 fork_trampoline+0x8
12 100034 intr irq1: atkbd0 mi_switch+0x122 ithread_loop+0x1b1 fork_exit+0xa3 fork_trampoline+0x8
12 100035 intr irq12: psm0
12 100036 intr irq7: ppc0
12 100037 intr swi0: uart uart
13 100013 geom g_event mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae g_run_events+0x62 fork_exit+0xa3 fork_trampoline+0x8
13 100014 geom g_up mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae g_io_schedule_up+0xd5 g_up_procbody+0x6d fork_exit+0xa3 fork_trampoline+0x8
13 100015 geom g_down mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sleep+0x2ae g_io_schedule_down+0x5c g_down_procbody+0x6d fork_exit+0xa3 fork_trampoline+0x8
14 100017 rand_harvestq - mi_switch+0x122 sleepq_switch+0x15b sleepq_timedwait+0x3f msleep_spin_sbt+0x1c0 random_kthread+0x2a2 fork_exit+0xa3 fork_trampoline+0x8
15 100046 vnlru - mi_switch+0x122 sleepq_switch+0x15b sleepq_timedwait+0x3f _sleep+0x282 vnlru_proc+0xcf fork_exit+0xa3 fork_trampoline+0x8
16 100047 syncer - mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _cv_wait+0x182 vmem_xalloc+0x16f vmem_alloc+0x5d kmem_malloc+0x3d page_alloc+0x28 keg_alloc_slab+0xe4 keg_fetch_slab+0x16e zone_fetch_slab+0x80 zone_import+0x39 uma_zalloc_arg+0x35a malloc+0x112 softdep_disk_io_initiation+0x978 ffs_geom_strategy+0xe0 ufs_strategy+0xb4 VOP_STRATEGY_APV+0xa1
35 100057 zfskern arc_reclaim_thre mi_switch+0x122 sleepq_switch+0x15b sleepq_timedwait+0x3f _cv_timedwait_sbt+0x1a7 arc_reclaim_thread+0x3b2 fork_exit+0xa3 fork_trampoline+0x8
35 100058 zfskern l2arc_feed_threa mi_switch+0x122 sleepq_switch+0x15b sleepq_timedwait+0x3f _cv_timedwait_sbt+0x1a7 l2arc_feed_thread+0x272 fork_exit+0xa3 fork_trampoline+0x8
35 100333 zfskern trim storage mi_switch+0x122 sleepq_switch+0x15b sleepq_timedwait+0x3f _cv_timedwait_sbt+0x1a7 trim_thread+0xc9 fork_exit+0xa3 fork_trampoline+0x8
35 100339 zfskern txg_thread_enter mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _cv_wait+0x182 txg_quiesce_thread+0x43d fork_exit+0xa3 fork_trampoline+0x8
35 100340 zfskern txg_thread_enter mi_switch+0x122 sleepq_switch+0x15b sleepq_timedwait+0x3f _cv_timedwait_sbt+0x1a7 txg_sync_thread+0x28b fork_exit+0xa3 fork_trampoline+0x8
303 100374 dhclient - mi_switch+0x122 sleepq_switch+0x15b sleepq_catch_signals+0x5be sleepq_wait_sig+0x14 _cv_wait_sig+0x17c seltdwait+0xcf sys_poll+0x472 syscall+0x48b Xint0x80_syscall+0x21
339 100371 dhclient - mi_switch+0x122 sleepq_switch+0x15b sleepq_catch_signals+0x5be sleepq_timedwait_sig+0x14 _cv_timedwait_sig_sbt+0x1a7 seltdwait+0xc1 sys_poll+0x472 syscall+0x48b Xint0x80_syscall+0x21
340 100373 devd - mi_switch+0x122 sleepq_switch+0x15b sleepq_catch_signals+0x5be sleepq_timedwait_sig+0x14 _cv_timedwait_sig_sbt+0x1a7 seltdwait+0xc1 kern_select+0x8c2 sys_select+0x69 syscall+0x48b Xint0x80_syscall+0x21
419 100380 syslogd - mi_switch+0x122 sleepq_switch+0x15b sleepq_catch_signals+0x5be sleepq_wait_sig+0x14 _cv_wait_sig+0x17c seltdwait+0xcf kern_select+0x8c2 sys_select+0x69 syscall+0x48b Xint0x80_syscall+0x21
544 100381 vmtoolsd - mi_switch+0x122 sleepq_switch+0x15b sleepq_catch_signals+0x5be sleepq_timedwait_sig+0x14 _cv_timedwait_sig_sbt+0x1a7 seltdwait+0xc1 sys_poll+0x472 syscall+0x48b Xint0x80_syscall+0x21
575 100375 ntpd - mi_switch+0x122 sleepq_switch+0x15b sleepq_catch_signals+0x5be sleepq_wait_sig+0x14 _cv_wait_sig+0x17c seltdwait+0xcf kern_select+0x8c2 sys_select+0x69 syscall+0x48b Xint0x80_syscall+0x21
616 100379 sshd - mi_switch+0x122 sleepq_switch+0x15b sleepq_catch_signals+0x5be sleepq_wait_sig+0x14 _cv_wait_sig+0x17c seltdwait+0xcf kern_select+0x8c2 sys_select+0x69 syscall+0x48b Xint0x80_syscall+0x21
619 100370 sendmail - mi_switch+0x122 sleepq_switch+0x15b sleepq_catch_signals+0x5be sleepq_timedwait_sig+0x14 _cv_timedwait_sig_sbt+0x1a7 seltdwait+0xc1 kern_select+0x8c2 sys_select+0x69 syscall+0x48b Xint0x80_syscall+0x21
622 100385 sendmail - mi_switch+0x122 sleepq_switch+0x15b sleepq_catch_signals+0x5be sleepq_wait_sig+0x14 _sleep+0x29b kern_sigsuspend+0x137 sys_sigsuspend+0x58 syscall+0x48b Xint0x80_syscall+0x21
626 100377 cron - mi_switch+0x122 sleepq_switch+0x15b sleepq_catch_signals+0x5be sleepq_timedwait_sig+0x14 _sleep+0x26b kern_nanosleep+0x14b sys_nanosleep+0x69 syscall+0x48b Xint0x80_syscall+0x21
668 100048 getty - mi_switch+0x122 sleepq_switch+0x15b sleepq_catch_signals+0x5be sleepq_wait_sig+0x14 _cv_wait_sig+0x17c tty_wait+0x1f ttydisc_read+0x31c ttydev_read+0x8f devfs_read_f+0xbf dofileread+0x9e kern_readv+0x96 sys_read+0x5c syscall+0x48b Xint0x80_syscall+0x21
669 100394 getty - mi_switch+0x122 sleepq_switch+0x15b sleepq_catch_signals+0x5be sleepq_wait_sig+0x14 _cv_wait_sig+0x17c tty_wait+0x1f ttydisc_read+0x31c ttydev_read+0x8f devfs_read_f+0xbf dofileread+0x9e kern_readv+0x96 sys_read+0x5c syscall+0x48b Xint0x80_syscall+0x21
670 100395 getty - mi_switch+0x122 sleepq_switch+0x15b sleepq_catch_signals+0x5be sleepq_wait_sig+0x14 _cv_wait_sig+0x17c tty_wait+0x1f ttydisc_read+0x31c ttydev_read+0x8f devfs_read_f+0xbf dofileread+0x9e kern_readv+0x96 sys_read+0x5c syscall+0x48b Xint0x80_syscall+0x21
671 100396 getty - mi_switch+0x122 sleepq_switch+0x15b sleepq_catch_signals+0x5be sleepq_wait_sig+0x14 _cv_wait_sig+0x17c tty_wait+0x1f ttydisc_read+0x31c ttydev_read+0x8f devfs_read_f+0xbf dofileread+0x9e kern_readv+0x96 sys_read+0x5c syscall+0x48b Xint0x80_syscall+0x21
672 100397 getty - mi_switch+0x122 sleepq_switch+0x15b sleepq_catch_signals+0x5be sleepq_wait_sig+0x14 _cv_wait_sig+0x17c tty_wait+0x1f ttydisc_read+0x31c ttydev_read+0x8f devfs_read_f+0xbf dofileread+0x9e kern_readv+0x96 sys_read+0x5c syscall+0x48b Xint0x80_syscall+0x21
673 100398 getty - mi_switch+0x122 sleepq_switch+0x15b sleepq_catch_signals+0x5be sleepq_wait_sig+0x14 _cv_wait_sig+0x17c tty_wait+0x1f ttydisc_read+0x31c ttydev_read+0x8f devfs_read_f+0xbf dofileread+0x9e kern_readv+0x96 sys_read+0x5c syscall+0x48b Xint0x80_syscall+0x21
674 100399 getty - mi_switch+0x122 sleepq_switch+0x15b sleepq_catch_signals+0x5be sleepq_wait_sig+0x14 _cv_wait_sig+0x17c tty_wait+0x1f ttydisc_read+0x31c ttydev_read+0x8f devfs_read_f+0xbf dofileread+0x9e kern_readv+0x96 sys_read+0x5c syscall+0x48b Xint0x80_syscall+0x21
675 100400 getty - mi_switch+0x122 sleepq_switch+0x15b sleepq_catch_signals+0x5be sleepq_wait_sig+0x14 _cv_wait_sig+0x17c tty_wait+0x1f ttydisc_read+0x31c ttydev_read+0x8f devfs_read_f+0xbf dofileread+0x9e kern_readv+0x96 sys_read+0x5c syscall+0x48b Xint0x80_syscall+0x21
676 100401 sshd - mi_switch+0x122 sleepq_switch+0x15b sleepq_catch_signals+0x5be sleepq_wait_sig+0x14 _cv_wait_sig+0x17c seltdwait+0xcf sys_poll+0x472 syscall+0x48b Xint0x80_syscall+0x21
678 100388 sshd - mi_switch+0x122 sleepq_switch+0x15b sleepq_catch_signals+0x5be sleepq_wait_sig+0x14 _cv_wait_sig+0x17c seltdwait+0xcf kern_select+0x8c2 sys_select+0x69 syscall+0x48b Xint0x80_syscall+0x21
679 100402 tcsh - mi_switch+0x122 sleepq_switch+0x15b sleepq_catch_signals+0x5be sleepq_wait_sig+0x14 _sleep+0x29b kern_sigsuspend+0x137 sys_sigsuspend+0x58 syscall+0x48b Xint0x80_syscall+0x21
681 100382 screen - mi_switch+0x122 sleepq_switch+0x15b sleepq_catch_signals+0x5be sleepq_wait_sig+0x14 _sleep+0x29b kern_sigsuspend+0x137 sys_sigsuspend+0x58 syscall+0x48b Xint0x80_syscall+0x21
682 100392 screen - mi_switch+0x122 sleepq_switch+0x15b sleepq_catch_signals+0x5be sleepq_wait_sig+0x14 _cv_wait_sig+0x17c seltdwait+0xcf kern_select+0x8c2 sys_select+0x69 syscall+0x48b Xint0x80_syscall+0x21
683 100403 tcsh - mi_switch+0x122 sleepq_switch+0x15b sleepq_catch_signals+0x5be sleepq_wait_sig+0x14 _sleep+0x29b kern_sigsuspend+0x137 sys_sigsuspend+0x58 syscall+0x48b Xint0x80_syscall+0x21
24036 100410 sudo - mi_switch+0x122 sleepq_switch+0x15b sleepq_catch_signals+0x5be sleepq_wait_sig+0x14 _cv_wait_sig+0x17c seltdwait+0xcf sys_poll+0x472 syscall+0x48b Xint0x80_syscall+0x21
24037 100412 make - mi_switch+0x122 sleepq_switch+0x15b sleepq_catch_signals+0x5be sleepq_wait_sig+0x14 _sleep+0x29b kern_wait6+0x725 sys_wait4+0x94 syscall+0x48b Xint0x80_syscall+0x21
24086 100428 make - mi_switch+0x122 sleepq_switch+0x15b sleepq_catch_signals+0x5be sleepq_wait_sig+0x14 _sleep+0x29b kern_wait6+0x725 sys_wait4+0x94 syscall+0x48b Xint0x80_syscall+0x21
24089 100404 sh - mi_switch+0x122 sleepq_switch+0x15b sleepq_catch_signals+0x5be sleepq_wait_sig+0x14 _sleep+0x29b kern_wait6+0x725 sys_wait4+0x94 syscall+0x48b Xint0x80_syscall+0x21
24091 100426 sh - mi_switch+0x122 sleepq_switch+0x15b sleepq_catch_signals+0x5be sleepq_wait_sig+0x14 _sleep+0x29b kern_wait6+0x725 sys_wait4+0x94 syscall+0x48b Xint0x80_syscall+0x21
24092 100420 make - mi_switch+0x122 sleepq_switch+0x15b sleepq_catch_signals+0x5be sleepq_timedwait_sig+0x14 _cv_timedwait_sig_sbt+0x1a7 seltdwait+0xc1 sys_poll+0x472 syscall+0x48b Xint0x80_syscall+0x21
25199 100430 tcsh - mi_switch+0x122 sleepq_switch+0x15b sleepq_catch_signals+0x5be sleepq_wait_sig+0x14 _sleep+0x29b kern_sigsuspend+0x137 sys_sigsuspend+0x58 syscall+0x48b Xint0x80_syscall+0x21
27240 100456 top - mi_switch+0x122 sleepq_switch+0x15b sleepq_catch_signals+0x5be sleepq_timedwait_sig+0x14 _cv_timedwait_sig_sbt+0x1a7 seltdwait+0xc1 kern_select+0x8c2 sys_select+0x69 syscall+0x48b Xint0x80_syscall+0x21
27452 100465 tcsh - mi_switch+0x122 sleepq_switch+0x15b sleepq_catch_signals+0x5be sleepq_wait_sig+0x14 _sleep+0x29b kern_sigsuspend+0x137 sys_sigsuspend+0x58 syscall+0x48b Xint0x80_syscall+0x21
29719 100468 sudo - mi_switch+0x122 sleepq_switch+0x15b sleepq_catch_signals+0x5be sleepq_wait_sig+0x14 _cv_wait_sig+0x17c seltdwait+0xcf sys_poll+0x472 syscall+0x48b Xint0x80_syscall+0x21
29723 100476 gstat - mi_switch+0x122 sleepq_switch+0x15b sleepq_catch_signals+0x5be sleepq_timedwait_sig+0x14 _sleep+0x26b kern_nanosleep+0x14b sys_nanosleep+0x69 syscall+0x48b Xint0x80_syscall+0x21
37306 100453 sh - mi_switch+0x122 sleepq_switch+0x15b sleepq_catch_signals+0x5be sleepq_wait_sig+0x14 _sleep+0x29b kern_wait6+0x725 sys_wait4+0x94 syscall+0x48b Xint0x80_syscall+0x21
37308 100464 make - mi_switch+0x122 sleepq_switch+0x15b sleepq_catch_signals+0x5be sleepq_wait_sig+0x14 _sleep+0x29b kern_wait6+0x725 sys_wait4+0x94 syscall+0x48b Xint0x80_syscall+0x21
37321 100419 make - mi_switch+0x122 sleepq_switch+0x15b sleepq_catch_signals+0x5be sleepq_wait_sig+0x14 _sleep+0x29b kern_wait6+0x725 sys_wait4+0x94 syscall+0x48b Xint0x80_syscall+0x21
38075 100345 sh - mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _cv_wait+0x182 vmem_xalloc+0x16f vmem_alloc+0x5d kmem_malloc+0x3d page_alloc+0x28 keg_alloc_slab+0xe4 keg_fetch_slab+0x16e zone_fetch_slab+0x80 zone_import+0x39 uma_zalloc_arg+0x35a malloc+0x112 zfs_kmem_alloc+0x20 zio_buf_alloc+0x54 arc_get_data_buf+0x452 arc_read+0x5bc
65755 100438 sh - mi_switch+0x122 sleepq_switch+0x15b sleepq_catch_signals+0x5be sleepq_wait_sig+0x14 _sleep+0x29b kern_wait6+0x725 sys_wait4+0x94 syscall+0x48b Xint0x80_syscall+0x21
65756 100427 make - mi_switch+0x122 sleepq_switch+0x15b sleepq_catch_signals+0x5be sleepq_wait_sig+0x14 _sleep+0x29b kern_wait6+0x725 sys_wait4+0x94 syscall+0x48b Xint0x80_syscall+0x21
65767 100429 make - mi_switch+0x122 sleepq_switch+0x15b sleepq_catch_signals+0x5be sleepq_wait_sig+0x14 _sleep+0x29b kern_wait6+0x725 sys_wait4+0x94 syscall+0x48b Xint0x80_syscall+0x21
65880 100466 sh - mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _cv_wait+0x182 vmem_xalloc+0x16f vmem_alloc+0x5d kmem_malloc+0x3d page_alloc+0x28 keg_alloc_slab+0xe4 keg_fetch_slab+0x16e zone_fetch_slab+0x80 zone_import+0x39 uma_zalloc_arg+0x35a malloc+0x112 zfs_kmem_alloc+0x20 zio_read_bp_init+0xaa zio_execute+0x139 arc_read+0xa8b
76566 100422 sh - mi_switch+0x122 sleepq_switch+0x15b sleepq_catch_signals+0x5be sleepq_wait_sig+0x14 _sleep+0x29b kern_wait6+0x725 sys_wait4+0x94 syscall+0x48b Xint0x80_syscall+0x21
76568 100455 make - mi_switch+0x122 sleepq_switch+0x15b sleepq_catch_signals+0x5be sleepq_wait_sig+0x14 _sleep+0x29b kern_wait6+0x725 sys_wait4+0x94 syscall+0x48b Xint0x80_syscall+0x21
76581 100376 make - mi_switch+0x122 sleepq_switch+0x15b sleepq_catch_signals+0x5be sleepq_wait_sig+0x14 _sleep+0x29b kern_wait6+0x725 sys_wait4+0x94 syscall+0x48b Xint0x80_syscall+0x21
76765 100437 sh - mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _cv_wait+0x182 vmem_xalloc+0x16f vmem_alloc+0x5d kmem_malloc+0x3d uma_large_malloc+0x42 malloc+0x3e zfs_kmem_alloc+0x20 zio_buf_alloc+0x54 arc_get_data_buf+0x452 arc_read+0x5bc dbuf_read+0x7e2 dmu_buf_hold+0x68 zap_lockdir+0x52 zap_lookup_norm+0x49 zap_lookup+0x6c
99282 100477 sh - mi_switch+0x122 sleepq_switch+0x15b sleepq_catch_signals+0x5be sleepq_wait_sig+0x14 _sleep+0x29b kern_wait6+0x725 sys_wait4+0x94 syscall+0x48b Xint0x80_syscall+0x21
99283 100472 make - mi_switch+0x122 sleepq_switch+0x15b sleepq_catch_signals+0x5be sleepq_wait_sig+0x14 _sleep+0x29b kern_wait6+0x725 sys_wait4+0x94 syscall+0x48b Xint0x80_syscall+0x21
99299 100461 make - mi_switch+0x122 sleepq_switch+0x15b sleepq_wait+0x3f _sx_xlock_hard+0x533 _sx_xlock+0x79 buf_hash_find+0xc2 arc_read+0x7b dbuf_read+0x7e2 dmu_buf_hold+0x68 zap_get_leaf_byblk+0x6a fzap_cursor_retrieve+0x184 zap_cursor_retrieve+0x24b zfs_freebsd_readdir+0x3b4 VOP_READDIR_APV+0x95 kern_getdirentries+0x1f8 sys_getdirentries+0x42 syscall+0x48b Xint0x80_syscall+0x21
99887 100442 tcsh - mi_switch+0x122 sleepq_switch+0x15b sleepq_catch_signals+0x5be sleepq_wait_sig+0x14 _sleep+0x29b kern_sigsuspend+0x137 sys_sigsuspend+0x58 syscall+0x48b Xint0x80_syscall+0x21
99892 100369 sudo - mi_switch+0x122 sleepq_switch+0x15b sleepq_catch_signals+0x5be sleepq_wait_sig+0x14 _cv_wait_sig+0x17c seltdwait+0xcf sys_poll+0x472 syscall+0x48b Xint0x80_syscall+0x21
99893 100389 procstat - <running>
More information about the freebsd-fs
mailing list