kern/175950: Possible deadlock in zfs after long uptime
Pascal Guitierrez
pascal.guitierrez at gmail.com
Fri Feb 8 06:10:00 UTC 2013
>Number: 175950
>Category: kern
>Synopsis: Possible deadlock in zfs after long uptime
>Confidential: no
>Severity: non-critical
>Priority: low
>Responsible: freebsd-bugs
>State: open
>Quarter:
>Keywords:
>Date-Required:
>Class: sw-bug
>Submitter-Id: current-users
>Arrival-Date: Fri Feb 08 06:10:00 UTC 2013
>Closed-Date:
>Last-Modified:
>Originator: Pascal Guitierrez
>Release: 8.3-RELEASEp3
>Organization:
>Environment:
FreeBSD melstg03 8.3-RELEASE-p3 FreeBSD 8.3-RELEASE-p3 #0: Tue Jun 12 00:39:29 UTC 2012 root at amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64
>Description:
We experienced a lockup on two identically configured servers (hardware, firmware, bios, drivers + freebsd release)...
LSI-9211 + 9208e
Driver (mpslsi) + firmware @ v14
Root filesystem is UFS on gmirror.
I/O freezes when attempting to "ls -R" under the ZFS pool.
On this server, uptime was 193 days - with both servers exhibiting the same issue one day apart (almost exactly 24 hours).
The other server's uptime before deadlocking was around 30 days less than this.
We only seem to see this issue on systems that have been under load for a while.
I've read https://wiki.freebsd.org/AvgZfsDeadlockDebug and so have included procstat -kk -a output below.
Actually - I took a procstat when first logged in, then another during the "ls" command that was blocked.
Let me know if you need the second procstat, which shows ls being blocked as well?
PID TID COMM TDNAME KSTACK
0 100000 kernel swapper mi_switch+0x176 sleepq_timedwait+0x42 _sleep+0x301 scheduler+0x357 mi_startup+0x77 btext+0x2c
0 100024 kernel firmware taskq mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100026 kernel acpi_task_0 mi_switch+0x176 sleepq_wait+0x42 msleep_spin+0x1a2 taskqueue_thread_loop+0x71 fork_exit+0x11f fork_trampoline+0xe
0 100027 kernel acpi_task_1 mi_switch+0x176 sleepq_wait+0x42 msleep_spin+0x1a2 taskqueue_thread_loop+0x71 fork_exit+0x11f fork_trampoline+0xe
0 100028 kernel acpi_task_2 mi_switch+0x176 sleepq_wait+0x42 msleep_spin+0x1a2 taskqueue_thread_loop+0x71 fork_exit+0x11f fork_trampoline+0xe
0 100030 kernel kqueue taskq mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100032 kernel thread taskq mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100035 kernel ffs_trim taskq mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100038 kernel igb0 que mi_switch+0x176 sleepq_wait+0x42 msleep_spin+0x1a2 taskqueue_thread_loop+0x71 fork_exit+0x11f fork_trampoline+0xe
0 100040 kernel igb0 que mi_switch+0x176 sleepq_wait+0x42 msleep_spin+0x1a2 taskqueue_thread_loop+0x71 fork_exit+0x11f fork_trampoline+0xe
0 100042 kernel igb0 que mi_switch+0x176 sleepq_wait+0x42 msleep_spin+0x1a2 taskqueue_thread_loop+0x71 fork_exit+0x11f fork_trampoline+0xe
0 100044 kernel igb0 que mi_switch+0x176 sleepq_wait+0x42 msleep_spin+0x1a2 taskqueue_thread_loop+0x71 fork_exit+0x11f fork_trampoline+0xe
0 100046 kernel igb0 que mi_switch+0x176 sleepq_wait+0x42 msleep_spin+0x1a2 taskqueue_thread_loop+0x71 fork_exit+0x11f fork_trampoline+0xe
0 100048 kernel igb0 que mi_switch+0x176 sleepq_wait+0x42 msleep_spin+0x1a2 taskqueue_thread_loop+0x71 fork_exit+0x11f fork_trampoline+0xe
0 100050 kernel igb0 que mi_switch+0x176 sleepq_wait+0x42 msleep_spin+0x1a2 taskqueue_thread_loop+0x71 fork_exit+0x11f fork_trampoline+0xe
0 100052 kernel igb0 que mi_switch+0x176 sleepq_wait+0x42 msleep_spin+0x1a2 taskqueue_thread_loop+0x71 fork_exit+0x11f fork_trampoline+0xe
0 100055 kernel igb1 que mi_switch+0x176 sleepq_wait+0x42 msleep_spin+0x1a2 taskqueue_thread_loop+0x71 fork_exit+0x11f fork_trampoline+0xe
0 100057 kernel igb1 que mi_switch+0x176 sleepq_wait+0x42 msleep_spin+0x1a2 taskqueue_thread_loop+0x71 fork_exit+0x11f fork_trampoline+0xe
0 100059 kernel igb1 que mi_switch+0x176 sleepq_wait+0x42 msleep_spin+0x1a2 taskqueue_thread_loop+0x71 fork_exit+0x11f fork_trampoline+0xe
0 100061 kernel igb1 que mi_switch+0x176 sleepq_wait+0x42 msleep_spin+0x1a2 taskqueue_thread_loop+0x71 fork_exit+0x11f fork_trampoline+0xe
0 100063 kernel igb1 que mi_switch+0x176 sleepq_wait+0x42 msleep_spin+0x1a2 taskqueue_thread_loop+0x71 fork_exit+0x11f fork_trampoline+0xe
0 100065 kernel igb1 que mi_switch+0x176 sleepq_wait+0x42 msleep_spin+0x1a2 taskqueue_thread_loop+0x71 fork_exit+0x11f fork_trampoline+0xe
0 100067 kernel igb1 que mi_switch+0x176 sleepq_wait+0x42 msleep_spin+0x1a2 taskqueue_thread_loop+0x71 fork_exit+0x11f fork_trampoline+0xe
0 100069 kernel igb1 que mi_switch+0x176 sleepq_wait+0x42 msleep_spin+0x1a2 taskqueue_thread_loop+0x71 fork_exit+0x11f fork_trampoline+0xe
0 100071 kernel mpslsi0 taskq mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100074 kernel mpslsi1 taskq mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100078 kernel ix0 que mi_switch+0x176 sleepq_wait+0x42 msleep_spin+0x1a2 taskqueue_thread_loop+0x71 fork_exit+0x11f fork_trampoline+0xe
0 100080 kernel ix0 que mi_switch+0x176 sleepq_wait+0x42 msleep_spin+0x1a2 taskqueue_thread_loop+0x71 fork_exit+0x11f fork_trampoline+0xe
0 100082 kernel ix0 que mi_switch+0x176 sleepq_wait+0x42 msleep_spin+0x1a2 taskqueue_thread_loop+0x71 fork_exit+0x11f fork_trampoline+0xe
0 100084 kernel ix0 que mi_switch+0x176 sleepq_wait+0x42 msleep_spin+0x1a2 taskqueue_thread_loop+0x71 fork_exit+0x11f fork_trampoline+0xe
0 100086 kernel ix0 que mi_switch+0x176 sleepq_wait+0x42 msleep_spin+0x1a2 taskqueue_thread_loop+0x71 fork_exit+0x11f fork_trampoline+0xe
0 100088 kernel ix0 que mi_switch+0x176 sleepq_wait+0x42 msleep_spin+0x1a2 taskqueue_thread_loop+0x71 fork_exit+0x11f fork_trampoline+0xe
0 100090 kernel ix0 que mi_switch+0x176 sleepq_wait+0x42 msleep_spin+0x1a2 taskqueue_thread_loop+0x71 fork_exit+0x11f fork_trampoline+0xe
0 100092 kernel ix0 que mi_switch+0x176 sleepq_wait+0x42 msleep_spin+0x1a2 taskqueue_thread_loop+0x71 fork_exit+0x11f fork_trampoline+0xe
0 100094 kernel ix0 linkq mi_switch+0x176 sleepq_wait+0x42 msleep_spin+0x1a2 taskqueue_thread_loop+0x71 fork_exit+0x11f fork_trampoline+0xe
0 100096 kernel ix1 que mi_switch+0x176 sleepq_wait+0x42 msleep_spin+0x1a2 taskqueue_thread_loop+0x71 fork_exit+0x11f fork_trampoline+0xe
0 100098 kernel ix1 que mi_switch+0x176 sleepq_wait+0x42 msleep_spin+0x1a2 taskqueue_thread_loop+0x71 fork_exit+0x11f fork_trampoline+0xe
0 100100 kernel ix1 que mi_switch+0x176 sleepq_wait+0x42 msleep_spin+0x1a2 taskqueue_thread_loop+0x71 fork_exit+0x11f fork_trampoline+0xe
0 100102 kernel ix1 que mi_switch+0x176 sleepq_wait+0x42 msleep_spin+0x1a2 taskqueue_thread_loop+0x71 fork_exit+0x11f fork_trampoline+0xe
0 100104 kernel ix1 que mi_switch+0x176 sleepq_wait+0x42 msleep_spin+0x1a2 taskqueue_thread_loop+0x71 fork_exit+0x11f fork_trampoline+0xe
0 100106 kernel ix1 que mi_switch+0x176 sleepq_wait+0x42 msleep_spin+0x1a2 taskqueue_thread_loop+0x71 fork_exit+0x11f fork_trampoline+0xe
0 100108 kernel ix1 que mi_switch+0x176 sleepq_wait+0x42 msleep_spin+0x1a2 taskqueue_thread_loop+0x71 fork_exit+0x11f fork_trampoline+0xe
0 100110 kernel ix1 que mi_switch+0x176 sleepq_wait+0x42 msleep_spin+0x1a2 taskqueue_thread_loop+0x71 fork_exit+0x11f fork_trampoline+0xe
0 100112 kernel ix1 linkq mi_switch+0x176 sleepq_wait+0x42 msleep_spin+0x1a2 taskqueue_thread_loop+0x71 fork_exit+0x11f fork_trampoline+0xe
0 100174 kernel system_taskq_0 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100175 kernel system_taskq_1 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100176 kernel system_taskq_2 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100177 kernel system_taskq_3 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100178 kernel system_taskq_4 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100179 kernel system_taskq_5 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100180 kernel system_taskq_6 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100181 kernel system_taskq_7 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100366 kernel zio_null_issue mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100367 kernel zio_null_intr mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100368 kernel zio_read_issue_0 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100369 kernel zio_read_issue_1 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100370 kernel zio_read_issue_2 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100371 kernel zio_read_issue_3 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100372 kernel zio_read_issue_4 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100373 kernel zio_read_issue_5 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100374 kernel zio_read_issue_6 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100375 kernel zio_read_issue_7 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100376 kernel zio_read_intr_0 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100377 kernel zio_read_intr_1 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100378 kernel zio_read_intr_2 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100379 kernel zio_read_intr_3 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100380 kernel zio_read_intr_4 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100381 kernel zio_read_intr_5 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100382 kernel zio_read_intr_6 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100383 kernel zio_read_intr_7 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100384 kernel zio_write_issue_ mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100385 kernel zio_write_issue_ mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100386 kernel zio_write_issue_ mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100387 kernel zio_write_issue_ mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100388 kernel zio_write_issue_ mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100389 kernel zio_write_issue_ mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100390 kernel zio_write_issue_ mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100391 kernel zio_write_issue_ mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100392 kernel zio_write_issue_ mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100393 kernel zio_write_issue_ mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100394 kernel zio_write_issue_ mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100395 kernel zio_write_issue_ mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100396 kernel zio_write_issue_ mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100397 kernel zio_write_intr_0 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100398 kernel zio_write_intr_1 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100399 kernel zio_write_intr_2 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100400 kernel zio_write_intr_3 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100401 kernel zio_write_intr_4 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100402 kernel zio_write_intr_5 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100403 kernel zio_write_intr_6 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100404 kernel zio_write_intr_7 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100405 kernel zio_write_intr_h mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100406 kernel zio_write_intr_h mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100407 kernel zio_write_intr_h mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100408 kernel zio_write_intr_h mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100409 kernel zio_write_intr_h mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100410 kernel zio_free_issue_0 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100411 kernel zio_free_issue_1 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100412 kernel zio_free_issue_2 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100413 kernel zio_free_issue_3 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100414 kernel zio_free_issue_4 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100415 kernel zio_free_issue_5 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100416 kernel zio_free_issue_6 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100417 kernel zio_free_issue_7 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100418 kernel zio_free_issue_8 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100419 kernel zio_free_issue_9 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100420 kernel zio_free_issue_1 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100421 kernel zio_free_issue_1 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100422 kernel zio_free_issue_1 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100423 kernel zio_free_issue_1 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100424 kernel zio_free_issue_1 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100425 kernel zio_free_issue_1 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100426 kernel zio_free_issue_1 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100427 kernel zio_free_issue_1 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100428 kernel zio_free_issue_1 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100429 kernel zio_free_issue_1 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100430 kernel zio_free_issue_2 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100431 kernel zio_free_issue_2 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100432 kernel zio_free_issue_2 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100433 kernel zio_free_issue_2 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100434 kernel zio_free_issue_2 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100435 kernel zio_free_issue_2 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100436 kernel zio_free_issue_2 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100437 kernel zio_free_issue_2 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100438 kernel zio_free_issue_2 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100439 kernel zio_free_issue_2 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100440 kernel zio_free_issue_3 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100441 kernel zio_free_issue_3 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100442 kernel zio_free_issue_3 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100443 kernel zio_free_issue_3 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100444 kernel zio_free_issue_3 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100445 kernel zio_free_issue_3 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100446 kernel zio_free_issue_3 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100447 kernel zio_free_issue_3 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100448 kernel zio_free_issue_3 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100449 kernel zio_free_issue_3 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100450 kernel zio_free_issue_4 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100451 kernel zio_free_issue_4 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100452 kernel zio_free_issue_4 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100453 kernel zio_free_issue_4 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100454 kernel zio_free_issue_4 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100455 kernel zio_free_issue_4 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100456 kernel zio_free_issue_4 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100457 kernel zio_free_issue_4 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100458 kernel zio_free_issue_4 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100459 kernel zio_free_issue_4 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100460 kernel zio_free_issue_5 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100461 kernel zio_free_issue_5 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100462 kernel zio_free_issue_5 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100463 kernel zio_free_issue_5 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100464 kernel zio_free_issue_5 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100465 kernel zio_free_issue_5 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100466 kernel zio_free_issue_5 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100467 kernel zio_free_issue_5 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100468 kernel zio_free_issue_5 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100469 kernel zio_free_issue_5 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100470 kernel zio_free_issue_6 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100471 kernel zio_free_issue_6 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100472 kernel zio_free_issue_6 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100473 kernel zio_free_issue_6 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100474 kernel zio_free_issue_6 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100475 kernel zio_free_issue_6 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100476 kernel zio_free_issue_6 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100477 kernel zio_free_issue_6 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100478 kernel zio_free_issue_6 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100479 kernel zio_free_issue_6 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100480 kernel zio_free_issue_7 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100481 kernel zio_free_issue_7 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100482 kernel zio_free_issue_7 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100483 kernel zio_free_issue_7 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100484 kernel zio_free_issue_7 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100485 kernel zio_free_issue_7 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100486 kernel zio_free_issue_7 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100487 kernel zio_free_issue_7 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100488 kernel zio_free_issue_7 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100489 kernel zio_free_issue_7 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100490 kernel zio_free_issue_8 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100491 kernel zio_free_issue_8 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100492 kernel zio_free_issue_8 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100493 kernel zio_free_issue_8 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100494 kernel zio_free_issue_8 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100495 kernel zio_free_issue_8 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100496 kernel zio_free_issue_8 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100497 kernel zio_free_issue_8 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100498 kernel zio_free_issue_8 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100499 kernel zio_free_issue_8 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100500 kernel zio_free_issue_9 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100501 kernel zio_free_issue_9 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100502 kernel zio_free_issue_9 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100503 kernel zio_free_issue_9 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100504 kernel zio_free_issue_9 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100505 kernel zio_free_issue_9 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100506 kernel zio_free_issue_9 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100507 kernel zio_free_issue_9 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100508 kernel zio_free_issue_9 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100509 kernel zio_free_issue_9 mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100510 kernel zio_free_intr mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100511 kernel zio_claim_issue mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100512 kernel zio_claim_intr mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100513 kernel zio_ioctl_issue mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100514 kernel zio_ioctl_intr mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100515 kernel zfs_vn_rele_task mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100518 kernel zil_clean mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100519 kernel zil_clean mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100520 kernel zil_clean mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100521 kernel zil_clean mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100522 kernel zil_clean mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100523 kernel zil_clean mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100524 kernel zil_clean mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100525 kernel zil_clean mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
0 100526 kernel zil_clean mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 taskqueue_thread_loop+0xbe fork_exit+0x11f fork_trampoline+0xe
1 100002 init - mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_wait_sig+0x16 _sleep+0x269 kern_wait+0x6fd wait4+0x35 amd64_syscall+0x1f4 Xfast_syscall+0xfc
2 100021 g_event - mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 g_run_events+0x430 fork_exit+0x11f fork_trampoline+0xe
3 100022 g_up - mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 g_io_schedule_up+0xd8 g_up_procbody+0x58 fork_exit+0x11f fork_trampoline+0xe
4 100023 g_down - mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 g_io_schedule_down+0x20e g_down_procbody+0x58 fork_exit+0x11f fork_trampoline+0xe
5 100072 mps_scan0 - mi_switch+0x176 sleepq_timedwait+0x42 _sleep+0x301 mpssas_scanner_thread+0x5e fork_exit+0x11f fork_trampoline+0xe
6 100075 mps_scan1 - mi_switch+0x176 sleepq_timedwait+0x42 _sleep+0x301 mpssas_scanner_thread+0x5e fork_exit+0x11f fork_trampoline+0xe
7 100151 sctp_iterator - mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 sctp_iterator_thread+0x41 fork_exit+0x11f fork_trampoline+0xe
8 100152 xpt_thrd - mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 xpt_scanner_thread+0xfa fork_exit+0x11f fork_trampoline+0xe
9 100153 pagedaemon - mi_switch+0x176 sleepq_timedwait+0x42 _sleep+0x301 vm_pageout+0x90e fork_exit+0x11f fork_trampoline+0xe
10 100001 audit - mi_switch+0x176 sleepq_wait+0x42 _cv_wait+0x129 audit_worker+0x384 fork_exit+0x11f fork_trampoline+0xe
11 100003 idle idle: cpu7 <running>
11 100004 idle idle: cpu6 <running>
11 100005 idle idle: cpu5 <running>
11 100006 idle idle: cpu4 mi_switch+0x176 sched_preempt+0xb5 ipi_bitmap_handler+0x70 Xipi_intr_bitmap_handler+0x9c acpi_cpu_idle+0x20a sched_idletd+0x11f fork_exit+0x11f fork_trampoline+0xe
11 100007 idle idle: cpu3 <running>
11 100008 idle idle: cpu2 <running>
11 100009 idle idle: cpu1 <running>
11 100010 idle idle: cpu0 <running>
12 100011 intr swi1: netisr 0 mi_switch+0x176 ithread_loop+0x1d1 fork_exit+0x11f fork_trampoline+0xe
12 100012 intr swi4: clock mi_switch+0x176 ithread_loop+0x1d1 fork_exit+0x11f fork_trampoline+0xe
12 100013 intr swi4: clock mi_switch+0x176 ithread_loop+0x1d1 fork_exit+0x11f fork_trampoline+0xe
12 100014 intr swi4: clock mi_switch+0x176 ithread_loop+0x1d1 fork_exit+0x11f fork_trampoline+0xe
12 100015 intr swi4: clock mi_switch+0x176 ithread_loop+0x1d1 fork_exit+0x11f fork_trampoline+0xe
12 100016 intr swi4: clock mi_switch+0x176 ithread_loop+0x1d1 fork_exit+0x11f fork_trampoline+0xe
12 100017 intr swi4: clock mi_switch+0x176 ithread_loop+0x1d1 fork_exit+0x11f fork_trampoline+0xe
12 100018 intr swi4: clock mi_switch+0x176 ithread_loop+0x1d1 fork_exit+0x11f fork_trampoline+0xe
12 100019 intr swi4: clock mi_switch+0x176 ithread_loop+0x1d1 fork_exit+0x11f fork_trampoline+0xe
12 100020 intr swi3: vm
12 100029 intr swi2: cambio mi_switch+0x176 ithread_loop+0x1d1 fork_exit+0x11f fork_trampoline+0xe
12 100031 intr swi5: +
12 100033 intr swi6: Giant task mi_switch+0x176 ithread_loop+0x1d1 fork_exit+0x11f fork_trampoline+0xe
12 100034 intr swi6: task queue mi_switch+0x176 ithread_loop+0x1d1 fork_exit+0x11f fork_trampoline+0xe
12 100036 intr irq20: acpi0
12 100037 intr irq256: igb0:que mi_switch+0x176 ithread_loop+0x1d1 fork_exit+0x11f fork_trampoline+0xe
12 100039 intr irq257: igb0:que mi_switch+0x176 ithread_loop+0x1d1 fork_exit+0x11f fork_trampoline+0xe
12 100041 intr irq258: igb0:que mi_switch+0x176 ithread_loop+0x1d1 fork_exit+0x11f fork_trampoline+0xe
12 100043 intr irq259: igb0:que mi_switch+0x176 ithread_loop+0x1d1 fork_exit+0x11f fork_trampoline+0xe
12 100045 intr irq260: igb0:que mi_switch+0x176 ithread_loop+0x1d1 fork_exit+0x11f fork_trampoline+0xe
12 100047 intr irq261: igb0:que mi_switch+0x176 ithread_loop+0x1d1 fork_exit+0x11f fork_trampoline+0xe
12 100049 intr irq262: igb0:que mi_switch+0x176 ithread_loop+0x1d1 fork_exit+0x11f fork_trampoline+0xe
12 100051 intr irq263: igb0:que mi_switch+0x176 ithread_loop+0x1d1 fork_exit+0x11f fork_trampoline+0xe
12 100053 intr irq264: igb0:lin mi_switch+0x176 ithread_loop+0x1d1 fork_exit+0x11f fork_trampoline+0xe
12 100054 intr irq265: igb1:que
12 100056 intr irq266: igb1:que
12 100058 intr irq267: igb1:que
12 100060 intr irq268: igb1:que
12 100062 intr irq269: igb1:que
12 100064 intr irq270: igb1:que
12 100066 intr irq271: igb1:que
12 100068 intr irq272: igb1:que
12 100070 intr irq273: igb1:lin
12 100073 intr irq274: mpslsi0 mi_switch+0x176 ithread_loop+0x1d1 fork_exit+0x11f fork_trampoline+0xe
12 100076 intr irq275: mpslsi1 mi_switch+0x176 ithread_loop+0x1d1 fork_exit+0x11f fork_trampoline+0xe
12 100077 intr irq276: ix0:que mi_switch+0x176 ithread_loop+0x1d1 fork_exit+0x11f fork_trampoline+0xe
12 100079 intr irq277: ix0:que mi_switch+0x176 ithread_loop+0x1d1 fork_exit+0x11f fork_trampoline+0xe
12 100081 intr irq278: ix0:que mi_switch+0x176 ithread_loop+0x1d1 fork_exit+0x11f fork_trampoline+0xe
12 100083 intr irq279: ix0:que mi_switch+0x176 ithread_loop+0x1d1 fork_exit+0x11f fork_trampoline+0xe
12 100085 intr irq280: ix0:que mi_switch+0x176 ithread_loop+0x1d1 fork_exit+0x11f fork_trampoline+0xe
12 100087 intr irq281: ix0:que mi_switch+0x176 ithread_loop+0x1d1 fork_exit+0x11f fork_trampoline+0xe
12 100089 intr irq282: ix0:que mi_switch+0x176 ithread_loop+0x1d1 fork_exit+0x11f fork_trampoline+0xe
12 100091 intr irq283: ix0:que mi_switch+0x176 ithread_loop+0x1d1 fork_exit+0x11f fork_trampoline+0xe
12 100093 intr irq284: ix0:link mi_switch+0x176 ithread_loop+0x1d1 fork_exit+0x11f fork_trampoline+0xe
12 100095 intr irq285: ix1:que
12 100097 intr irq286: ix1:que
12 100099 intr irq287: ix1:que
12 100101 intr irq288: ix1:que
12 100103 intr irq289: ix1:que
12 100105 intr irq290: ix1:que
12 100107 intr irq291: ix1:que
12 100109 intr irq292: ix1:que
12 100111 intr irq293: ix1:link
12 100113 intr irq16: uhci0
12 100118 intr irq21: uhci1
12 100123 intr irq19: uhci2 uhc mi_switch+0x176 ithread_loop+0x1d1 fork_exit+0x11f fork_trampoline+0xe
12 100128 intr irq18: ehci0 uhc mi_switch+0x176 ithread_loop+0x1d1 fork_exit+0x11f fork_trampoline+0xe
12 100133 intr irq23: uhci3 ehc mi_switch+0x176 ithread_loop+0x1d1 fork_exit+0x11f fork_trampoline+0xe
12 100150 intr irq1: atkbd0
13 100025 yarrow - mi_switch+0x176 sleepq_timedwait+0x42 _sleep+0x301 random_kthread+0x1e2 fork_exit+0x11f fork_trampoline+0xe
14 100114 usb usbus0 mi_switch+0x176 sleepq_wait+0x42 _cv_wait+0x129 usb_process+0x18b fork_exit+0x11f fork_trampoline+0xe
14 100115 usb usbus0 mi_switch+0x176 sleepq_wait+0x42 _cv_wait+0x129 usb_process+0x18b fork_exit+0x11f fork_trampoline+0xe
14 100116 usb usbus0 mi_switch+0x176 sleepq_wait+0x42 _cv_wait+0x129 usb_process+0x18b fork_exit+0x11f fork_trampoline+0xe
14 100117 usb usbus0 mi_switch+0x176 sleepq_wait+0x42 _cv_wait+0x129 usb_process+0x18b fork_exit+0x11f fork_trampoline+0xe
14 100119 usb usbus1 mi_switch+0x176 sleepq_wait+0x42 _cv_wait+0x129 usb_process+0x18b fork_exit+0x11f fork_trampoline+0xe
14 100120 usb usbus1 mi_switch+0x176 sleepq_wait+0x42 _cv_wait+0x129 usb_process+0x18b fork_exit+0x11f fork_trampoline+0xe
14 100121 usb usbus1 mi_switch+0x176 sleepq_wait+0x42 _cv_wait+0x129 usb_process+0x18b fork_exit+0x11f fork_trampoline+0xe
14 100122 usb usbus1 mi_switch+0x176 sleepq_wait+0x42 _cv_wait+0x129 usb_process+0x18b fork_exit+0x11f fork_trampoline+0xe
14 100124 usb usbus2 mi_switch+0x176 sleepq_wait+0x42 _cv_wait+0x129 usb_process+0x18b fork_exit+0x11f fork_trampoline+0xe
14 100125 usb usbus2 mi_switch+0x176 sleepq_wait+0x42 _cv_wait+0x129 usb_process+0x18b fork_exit+0x11f fork_trampoline+0xe
14 100126 usb usbus2 mi_switch+0x176 sleepq_wait+0x42 _cv_wait+0x129 usb_process+0x18b fork_exit+0x11f fork_trampoline+0xe
14 100127 usb usbus2 mi_switch+0x176 sleepq_wait+0x42 _cv_wait+0x129 usb_process+0x18b fork_exit+0x11f fork_trampoline+0xe
14 100129 usb usbus3 mi_switch+0x176 sleepq_wait+0x42 _cv_wait+0x129 usb_process+0x18b fork_exit+0x11f fork_trampoline+0xe
14 100130 usb usbus3 mi_switch+0x176 sleepq_wait+0x42 _cv_wait+0x129 usb_process+0x18b fork_exit+0x11f fork_trampoline+0xe
14 100131 usb usbus3 mi_switch+0x176 sleepq_wait+0x42 _cv_wait+0x129 usb_process+0x18b fork_exit+0x11f fork_trampoline+0xe
14 100132 usb usbus3 mi_switch+0x176 sleepq_wait+0x42 _cv_wait+0x129 usb_process+0x18b fork_exit+0x11f fork_trampoline+0xe
14 100134 usb usbus4 mi_switch+0x176 sleepq_wait+0x42 _cv_wait+0x129 usb_process+0x18b fork_exit+0x11f fork_trampoline+0xe
14 100135 usb usbus4 mi_switch+0x176 sleepq_wait+0x42 _cv_wait+0x129 usb_process+0x18b fork_exit+0x11f fork_trampoline+0xe
14 100136 usb usbus4 mi_switch+0x176 sleepq_wait+0x42 _cv_wait+0x129 usb_process+0x18b fork_exit+0x11f fork_trampoline+0xe
14 100137 usb usbus4 mi_switch+0x176 sleepq_wait+0x42 _cv_wait+0x129 usb_process+0x18b fork_exit+0x11f fork_trampoline+0xe
14 100138 usb usbus5 mi_switch+0x176 sleepq_wait+0x42 _cv_wait+0x129 usb_process+0x18b fork_exit+0x11f fork_trampoline+0xe
14 100139 usb usbus5 mi_switch+0x176 sleepq_wait+0x42 _cv_wait+0x129 usb_process+0x18b fork_exit+0x11f fork_trampoline+0xe
14 100140 usb usbus5 mi_switch+0x176 sleepq_wait+0x42 _cv_wait+0x129 usb_process+0x18b fork_exit+0x11f fork_trampoline+0xe
14 100141 usb usbus5 mi_switch+0x176 sleepq_wait+0x42 _cv_wait+0x129 usb_process+0x18b fork_exit+0x11f fork_trampoline+0xe
14 100142 usb usbus6 mi_switch+0x176 sleepq_wait+0x42 _cv_wait+0x129 usb_process+0x18b fork_exit+0x11f fork_trampoline+0xe
14 100143 usb usbus6 mi_switch+0x176 sleepq_wait+0x42 _cv_wait+0x129 usb_process+0x18b fork_exit+0x11f fork_trampoline+0xe
14 100144 usb usbus6 mi_switch+0x176 sleepq_wait+0x42 _cv_wait+0x129 usb_process+0x18b fork_exit+0x11f fork_trampoline+0xe
14 100145 usb usbus6 mi_switch+0x176 sleepq_wait+0x42 _cv_wait+0x129 usb_process+0x18b fork_exit+0x11f fork_trampoline+0xe
14 100146 usb usbus7 mi_switch+0x176 sleepq_wait+0x42 _cv_wait+0x129 usb_process+0x18b fork_exit+0x11f fork_trampoline+0xe
14 100147 usb usbus7 mi_switch+0x176 sleepq_wait+0x42 _cv_wait+0x129 usb_process+0x18b fork_exit+0x11f fork_trampoline+0xe
14 100148 usb usbus7 mi_switch+0x176 sleepq_wait+0x42 _cv_wait+0x129 usb_process+0x18b fork_exit+0x11f fork_trampoline+0xe
14 100149 usb usbus7 mi_switch+0x176 sleepq_wait+0x42 _cv_wait+0x129 usb_process+0x18b fork_exit+0x11f fork_trampoline+0xe
15 100154 vmdaemon - mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 vm_daemon+0x81 fork_exit+0x11f fork_trampoline+0xe
16 100155 pagezero - mi_switch+0x176 sleepq_timedwait+0x42 _sleep+0x301 vm_pagezero+0x83 fork_exit+0x11f fork_trampoline+0xe
17 100156 bufdaemon - mi_switch+0x176 sleepq_timedwait+0x42 _sleep+0x301 buf_daemon+0x182 fork_exit+0x11f fork_trampoline+0xe
18 100157 syncer - mi_switch+0x176 sleepq_timedwait+0x42 _cv_timedwait+0x134 sched_sync+0x531 fork_exit+0x11f fork_trampoline+0xe
19 100158 vnlru - mi_switch+0x176 sleepq_timedwait+0x42 _sleep+0x301 vnlru_proc+0x626 fork_exit+0x11f fork_trampoline+0xe
20 100159 softdepflush - mi_switch+0x176 sleepq_timedwait+0x42 _sleep+0x301 softdep_flush+0x334 fork_exit+0x11f fork_trampoline+0xe
21 100160 g_mirror gm0 - mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 g_mirror_worker+0xa8b fork_exit+0x11f fork_trampoline+0xe
41 100168 zfskern arc_reclaim_thre mi_switch+0x176 sleepq_wait+0x42 _sx_xlock_hard+0x305 _sx_xlock+0x4e arc_buf_remove_ref+0x8a dbuf_rele_and_unlock+0x132 dbuf_evict+0x11 dbuf_do_evict+0x53 arc_do_user_evicts+0xb4 arc_reclaim_thread+0x264 fork_exit+0x11f fork_trampoline+0xe
41 100182 zfskern l2arc_feed_threa mi_switch+0x176 sleepq_timedwait+0x42 _cv_timedwait+0x134 l2arc_feed_thread+0x1a8 fork_exit+0x11f fork_trampoline+0xe
41 100516 zfskern txg_thread_enter mi_switch+0x176 sleepq_wait+0x42 _cv_wait+0x129 txg_thread_wait+0x79 txg_quiesce_thread+0xb5 fork_exit+0x11f fork_trampoline+0xe
41 100517 zfskern txg_thread_enter mi_switch+0x176 sleepq_wait+0x42 _sx_xlock_hard+0x305 _sx_xlock+0x4e arc_release+0x8e arc_release_bp+0x9 dbuf_release_bp+0x6e dbuf_write+0x1ec dbuf_sync_list+0xdf dbuf_sync_list+0x106 dnode_sync+0x52d dmu_objset_sync+0x10e dsl_pool_sync+0xc1 spa_sync+0x336 txg_sync_thread+0x139 fork_exit+0x11f fork_trampoline+0xe
1290 100190 moused - mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_wait_sig+0x16 _cv_wait_sig+0x128 seltdwait+0x110 kern_select+0x6ef select+0x5d amd64_syscall+0x1f4 Xfast_syscall+0xfc
1307 100164 devd - mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_wait_sig+0x16 _cv_wait_sig+0x128 seltdwait+0x110 kern_select+0x6ef select+0x5d amd64_syscall+0x1f4 Xfast_syscall+0xfc
1430 100169 syslogd - mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_wait_sig+0x16 _cv_wait_sig+0x128 seltdwait+0x110 kern_select+0x6ef select+0x5d amd64_syscall+0x1f4 Xfast_syscall+0xfc
1441 100166 rpcbind - mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 seltdwait+0x98 poll+0x478 amd64_syscall+0x1f4 Xfast_syscall+0xfc
1548 100165 sendmail - mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 seltdwait+0x98 kern_select+0x6ef select+0x5d amd64_syscall+0x1f4 Xfast_syscall+0xfc
1551 100347 sendmail - mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_wait_sig+0x16 _sleep+0x269 kern_sigsuspend+0xbc sigsuspend+0x34 amd64_syscall+0x1f4 Xfast_syscall+0xfc
1555 100171 cron - mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _sleep+0x1b1 kern_nanosleep+0x118 nanosleep+0x6e amd64_syscall+0x1f4 Xfast_syscall+0xfc
1595 100167 getty - mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_wait_sig+0x16 _cv_wait_sig+0x128 tty_wait+0x25 ttydisc_read+0x2b1 ttydev_read+0x10f devfs_read_f+0x88 dofileread+0xa1 kern_readv+0x60 read+0x55 amd64_syscall+0x1f4 Xfast_syscall+0xfc
1596 100351 getty - mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_wait_sig+0x16 _cv_wait_sig+0x128 tty_wait+0x25 ttydisc_read+0x2b1 ttydev_read+0x10f devfs_read_f+0x88 dofileread+0xa1 kern_readv+0x60 read+0x55 amd64_syscall+0x1f4 Xfast_syscall+0xfc
1597 100352 getty - mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_wait_sig+0x16 _cv_wait_sig+0x128 tty_wait+0x25 ttydisc_read+0x2b1 ttydev_read+0x10f devfs_read_f+0x88 dofileread+0xa1 kern_readv+0x60 read+0x55 amd64_syscall+0x1f4 Xfast_syscall+0xfc
1598 100353 getty - mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_wait_sig+0x16 _cv_wait_sig+0x128 tty_wait+0x25 ttydisc_read+0x2b1 ttydev_read+0x10f devfs_read_f+0x88 dofileread+0xa1 kern_readv+0x60 read+0x55 amd64_syscall+0x1f4 Xfast_syscall+0xfc
1599 100354 getty - mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_wait_sig+0x16 _cv_wait_sig+0x128 tty_wait+0x25 ttydisc_read+0x2b1 ttydev_read+0x10f devfs_read_f+0x88 dofileread+0xa1 kern_readv+0x60 read+0x55 amd64_syscall+0x1f4 Xfast_syscall+0xfc
1600 100355 getty - mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_wait_sig+0x16 _cv_wait_sig+0x128 tty_wait+0x25 ttydisc_read+0x2b1 ttydev_read+0x10f devfs_read_f+0x88 dofileread+0xa1 kern_readv+0x60 read+0x55 amd64_syscall+0x1f4 Xfast_syscall+0xfc
1962 100221 nfsd - mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_wait_sig+0x16 _cv_wait_sig+0x128 seltdwait+0x110 kern_select+0x6ef select+0x5d amd64_syscall+0x1f4 Xfast_syscall+0xfc
1963 100222 nfsd nfsd: service mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 svc_run_internal+0x930 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100223 nfsd nfsd: service mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 svc_run_internal+0x930 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100224 nfsd nfsd: service mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 svc_run_internal+0x930 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100225 nfsd nfsd: service mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 svc_run_internal+0x930 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100226 nfsd nfsd: service mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 svc_run_internal+0x930 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100227 nfsd nfsd: service mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 svc_run_internal+0x930 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100228 nfsd nfsd: service mi_switch+0x176 sleepq_wait+0x42 _sx_xlock_hard+0x305 _sx_xlock+0x4e zfs_zget+0x21a zfs_fhtovp+0x12d nfsrv_fhtovp+0x16e nfsrv3_access+0x1d3 nfssvc_program+0x1a6 svc_run_internal+0x6e9 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100229 nfsd nfsd: service mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 svc_run_internal+0x930 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100230 nfsd nfsd: service mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 svc_run_internal+0x930 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100231 nfsd nfsd: service mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 svc_run_internal+0x930 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100232 nfsd nfsd: service mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 svc_run_internal+0x930 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100233 nfsd nfsd: service mi_switch+0x176 sleepq_wait+0x42 _sx_xlock_hard+0x305 _sx_xlock+0x4e buf_hash_find+0x52 arc_read_nolock+0x54 arc_read+0x93 dbuf_read+0x452 dmu_buf_hold_array_by_dnode+0x16b dmu_buf_hold_array+0x67 dmu_read_uio+0x3f zfs_freebsd_read+0x3e4 nfsrv_read+0x96b nfssvc_program+0x1a6 svc_run_internal+0x6e9 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100234 nfsd nfsd: service mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 svc_run_internal+0x930 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100235 nfsd nfsd: service mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 svc_run_internal+0x930 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100236 nfsd nfsd: service mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 svc_run_internal+0x930 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100237 nfsd nfsd: service mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 svc_run_internal+0x930 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100238 nfsd nfsd: service mi_switch+0x176 sleepq_wait+0x42 _sx_xlock_hard+0x305 _sx_xlock+0x4e buf_hash_find+0x52 arc_read_nolock+0x54 arc_read+0x93 dbuf_prefetch+0x12c dmu_zfetch_dofetch+0x10b dmu_zfetch+0xaf8 dbuf_read+0x4a7 dmu_buf_hold_array_by_dnode+0x16b dmu_buf_hold_array+0x67 dmu_read_uio+0x3f zfs_freebsd_read+0x3e4 nfsrv_read+0x96b nfssvc_program+0x1a6 svc_run_internal+0x6e9
1963 100239 nfsd nfsd: service mi_switch+0x176 sleepq_wait+0x42 _sx_xlock_hard+0x305 _sx_xlock+0x4e arc_buf_add_ref+0x87 dbuf_hold_impl+0x6d dbuf_hold_level+0x1a dmu_tx_check_ioerr+0x52 dmu_tx_count_write+0x29c dmu_tx_hold_write+0x4a zfs_freebsd_write+0x372 VOP_WRITE_APV+0xb2 nfsrv_write+0x969 nfssvc_program+0x1a6 svc_run_internal+0x6e9 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100240 nfsd nfsd: service mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 svc_run_internal+0x930 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100241 nfsd nfsd: service mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 svc_run_internal+0x930 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100242 nfsd nfsd: service mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 svc_run_internal+0x930 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100243 nfsd nfsd: service mi_switch+0x176 sleepq_wait+0x42 _sx_xlock_hard+0x305 _sx_xlock+0x4e zfs_zget+0x21a zfs_fhtovp+0x12d nfsrv_fhtovp+0x16e nfsrv_getattr+0xf9 nfssvc_program+0x1a6 svc_run_internal+0x6e9 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100244 nfsd nfsd: service mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 svc_run_internal+0x930 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100245 nfsd nfsd: service mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 svc_run_internal+0x930 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100246 nfsd nfsd: service mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 svc_run_internal+0x930 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100247 nfsd nfsd: service mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 svc_run_internal+0x930 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100248 nfsd nfsd: service mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 svc_run_internal+0x930 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100249 nfsd nfsd: service mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 svc_run_internal+0x930 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100250 nfsd nfsd: service mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 svc_run_internal+0x930 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100251 nfsd nfsd: service mi_switch+0x176 sleepq_wait+0x42 _sx_xlock_hard+0x305 _sx_xlock+0x4e arc_buf_add_ref+0x87 dbuf_hold_impl+0x6d dbuf_hold+0x1b dmu_buf_hold+0x9b zap_lockdir+0x58 zap_lookup_norm+0x45 zap_lookup+0x2e zfs_dirent_lock+0x4ff zfs_dirlook+0x69 zfs_lookup+0x26b zfs_freebsd_lookup+0x81 vfs_cache_lookup+0xf0 VOP_LOOKUP_APV+0x40 lookup+0x452
1963 100252 nfsd nfsd: service mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 svc_run_internal+0x930 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100253 nfsd nfsd: service mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 svc_run_internal+0x930 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100254 nfsd nfsd: service mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 svc_run_internal+0x930 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100255 nfsd nfsd: service mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 svc_run_internal+0x930 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100256 nfsd nfsd: service mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 svc_run_internal+0x930 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100257 nfsd nfsd: service mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 svc_run_internal+0x930 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100258 nfsd nfsd: service mi_switch+0x176 sleepq_wait+0x42 __lockmgr_args+0x49b vop_stdlock+0x39 VOP_LOCK1_APV+0x46 _vn_lock+0x47 zfs_vnode_lock+0x1d zfs_fhtovp+0x32b nfsrv_fhtovp+0x16e nfsrv_read+0x2bc nfssvc_program+0x1a6 svc_run_internal+0x6e9 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100259 nfsd nfsd: service mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 svc_run_internal+0x930 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100260 nfsd nfsd: service mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 svc_run_internal+0x930 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100261 nfsd nfsd: service mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 svc_run_internal+0x930 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100262 nfsd nfsd: service mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 svc_run_internal+0x930 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100263 nfsd nfsd: service mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 svc_run_internal+0x930 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100264 nfsd nfsd: service mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 svc_run_internal+0x930 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100265 nfsd nfsd: service mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 svc_run_internal+0x930 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100266 nfsd nfsd: service mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 svc_run_internal+0x930 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100267 nfsd nfsd: service mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 svc_run_internal+0x930 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100268 nfsd nfsd: service mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 svc_run_internal+0x930 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100269 nfsd nfsd: service mi_switch+0x176 sleepq_wait+0x42 __lockmgr_args+0x49b vop_stdlock+0x39 VOP_LOCK1_APV+0x46 _vn_lock+0x47 zfs_vnode_lock+0x1d zfs_fhtovp+0x32b nfsrv_fhtovp+0x16e nfsrv_getattr+0xf9 nfssvc_program+0x1a6 svc_run_internal+0x6e9 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100270 nfsd nfsd: service mi_switch+0x176 sleepq_wait+0x42 _sx_xlock_hard+0x305 _sx_xlock+0x4e arc_buf_add_ref+0x87 dbuf_hold_impl+0x6d dbuf_hold+0x1b dmu_buf_hold_array_by_dnode+0x147 dmu_buf_hold_array+0x67 dmu_read_uio+0x3f zfs_freebsd_read+0x3e4 nfsrv_read+0x96b nfssvc_program+0x1a6 svc_run_internal+0x6e9 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100271 nfsd nfsd: service mi_switch+0x176 sleepq_wait+0x42 _sx_xlock_hard+0x305 _sx_xlock+0x4e zfs_zget+0x21a zfs_fhtovp+0x12d nfsrv_fhtovp+0x16e nfsrv_getattr+0xf9 nfssvc_program+0x1a6 svc_run_internal+0x6e9 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100272 nfsd nfsd: service mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 svc_run_internal+0x930 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100273 nfsd nfsd: service mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 svc_run_internal+0x930 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100274 nfsd nfsd: service mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 svc_run_internal+0x930 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100275 nfsd nfsd: service mi_switch+0x176 sleepq_wait+0x42 __lockmgr_args+0x49b vop_stdlock+0x39 VOP_LOCK1_APV+0x46 _vn_lock+0x47 zfs_vnode_lock+0x1d zfs_fhtovp+0x32b nfsrv_fhtovp+0x16e nfsrv_getattr+0xf9 nfssvc_program+0x1a6 svc_run_internal+0x6e9 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100276 nfsd nfsd: service mi_switch+0x176 sleepq_wait+0x42 __lockmgr_args+0x49b vop_stdlock+0x39 VOP_LOCK1_APV+0x46 _vn_lock+0x47 zfs_vnode_lock+0x1d zfs_fhtovp+0x32b nfsrv_fhtovp+0x16e nfsrv_read+0x2bc nfssvc_program+0x1a6 svc_run_internal+0x6e9 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100277 nfsd nfsd: service mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 svc_run_internal+0x930 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100278 nfsd nfsd: service mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 svc_run_internal+0x930 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100279 nfsd nfsd: service mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 svc_run_internal+0x930 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100280 nfsd nfsd: service mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 svc_run_internal+0x930 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100281 nfsd nfsd: service mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 svc_run_internal+0x930 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100282 nfsd nfsd: service mi_switch+0x176 sleepq_wait+0x42 __lockmgr_args+0x49b vop_stdlock+0x39 VOP_LOCK1_APV+0x46 _vn_lock+0x47 zfs_vnode_lock+0x1d zfs_fhtovp+0x32b nfsrv_fhtovp+0x16e nfsrv_read+0x2bc nfssvc_program+0x1a6 svc_run_internal+0x6e9 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100283 nfsd nfsd: service mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 svc_run_internal+0x930 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100284 nfsd nfsd: service mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 svc_run_internal+0x930 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100285 nfsd nfsd: service mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 svc_run_internal+0x930 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100286 nfsd nfsd: service mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 svc_run_internal+0x930 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100287 nfsd nfsd: service mi_switch+0x176 sleepq_wait+0x42 _sx_xlock_hard+0x305 _sx_xlock+0x4e zfs_zget+0x21a zfs_fhtovp+0x12d nfsrv_fhtovp+0x16e nfsrv_getattr+0xf9 nfssvc_program+0x1a6 svc_run_internal+0x6e9 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100288 nfsd nfsd: service mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 svc_run_internal+0x930 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100289 nfsd nfsd: service mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 svc_run_internal+0x930 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100290 nfsd nfsd: service mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 svc_run_internal+0x930 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100291 nfsd nfsd: service mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 svc_run_internal+0x930 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100292 nfsd nfsd: service mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 svc_run_internal+0x930 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100293 nfsd nfsd: service mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 svc_run_internal+0x930 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100294 nfsd nfsd: service mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 svc_run_internal+0x930 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100295 nfsd nfsd: service mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 svc_run_internal+0x930 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100296 nfsd nfsd: service mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 svc_run_internal+0x930 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100297 nfsd nfsd: service mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 svc_run_internal+0x930 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100298 nfsd nfsd: service mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 svc_run_internal+0x930 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100299 nfsd nfsd: service mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 svc_run_internal+0x930 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100300 nfsd nfsd: service mi_switch+0x176 sleepq_wait+0x42 __lockmgr_args+0x49b vop_stdlock+0x39 VOP_LOCK1_APV+0x46 _vn_lock+0x47 zfs_vnode_lock+0x1d zfs_fhtovp+0x32b nfsrv_fhtovp+0x16e nfsrv_read+0x2bc nfssvc_program+0x1a6 svc_run_internal+0x6e9 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100301 nfsd nfsd: service mi_switch+0x176 sleepq_wait+0x42 _sx_xlock_hard+0x305 _sx_xlock+0x4e arc_buf_add_ref+0x87 dbuf_hold_impl+0x6d dbuf_hold+0x1b dmu_buf_hold+0x9b zap_lockdir+0x58 zap_lookup_norm+0x45 zap_lookup+0x2e zfs_dirent_lock+0x4ff zfs_dirlook+0x69 zfs_lookup+0x26b zfs_freebsd_lookup+0x81 vfs_cache_lookup+0xf0 VOP_LOOKUP_APV+0x40 lookup+0x452
1963 100302 nfsd nfsd: service mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 svc_run_internal+0x930 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100303 nfsd nfsd: service mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 svc_run_internal+0x930 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100304 nfsd nfsd: service mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 svc_run_internal+0x930 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100305 nfsd nfsd: service mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 svc_run_internal+0x930 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100306 nfsd nfsd: service mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 svc_run_internal+0x930 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100307 nfsd nfsd: service mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 svc_run_internal+0x930 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100308 nfsd nfsd: service mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 arc_lowmem+0x77 kmem_malloc+0xc1 uma_large_malloc+0x4a malloc+0xd7 arc_get_data_buf+0xb5 arc_read_nolock+0x1ec arc_read+0x93 dbuf_read+0x452 dmu_tx_check_ioerr+0x9a dmu_tx_count_write+0x29c dmu_tx_hold_write+0x4a zfs_freebsd_write+0x372 VOP_WRITE_APV+0xb2 nfsrv_write+0x969 nfssvc_program+0x1a6
1963 100309 nfsd nfsd: service mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 svc_run_internal+0x930 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100310 nfsd nfsd: service mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 svc_run_internal+0x930 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100311 nfsd nfsd: service mi_switch+0x176 sleepq_wait+0x42 __lockmgr_args+0x49b vop_stdlock+0x39 VOP_LOCK1_APV+0x46 _vn_lock+0x47 zfs_vnode_lock+0x1d zfs_fhtovp+0x32b nfsrv_fhtovp+0x16e nfsrv_read+0x2bc nfssvc_program+0x1a6 svc_run_internal+0x6e9 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100312 nfsd nfsd: service mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 svc_run_internal+0x930 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100313 nfsd nfsd: service mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 svc_run_internal+0x930 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100314 nfsd nfsd: service mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 svc_run_internal+0x930 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100315 nfsd nfsd: service mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 svc_run_internal+0x930 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100316 nfsd nfsd: service mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 svc_run_internal+0x930 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100317 nfsd nfsd: service mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 svc_run_internal+0x930 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100318 nfsd nfsd: service mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 svc_run_internal+0x930 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100319 nfsd nfsd: service mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 svc_run_internal+0x930 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100320 nfsd nfsd: service mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 svc_run_internal+0x930 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100321 nfsd nfsd: service mi_switch+0x176 sleepq_wait+0x42 _sx_xlock_hard+0x305 _sx_xlock+0x4e buf_hash_find+0x52 arc_read_nolock+0x54 arc_read+0x93 dbuf_prefetch+0x12c dmu_zfetch_dofetch+0x10b dmu_zfetch+0xaf8 dbuf_read+0x4a7 dmu_buf_hold_array_by_dnode+0x16b dmu_buf_hold_array+0x67 dmu_read_uio+0x3f zfs_freebsd_read+0x3e4 nfsrv_read+0x96b nfssvc_program+0x1a6 svc_run_internal+0x6e9
1963 100322 nfsd nfsd: service mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 svc_run_internal+0x930 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100323 nfsd nfsd: service mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 svc_run_internal+0x930 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100324 nfsd nfsd: service mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 svc_run_internal+0x930 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100325 nfsd nfsd: service mi_switch+0x176 sleepq_wait+0x42 __lockmgr_args+0x49b vop_stdlock+0x39 VOP_LOCK1_APV+0x46 _vn_lock+0x47 zfs_vnode_lock+0x1d zfs_fhtovp+0x32b nfsrv_fhtovp+0x16e nfsrv_read+0x2bc nfssvc_program+0x1a6 svc_run_internal+0x6e9 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100326 nfsd nfsd: service mi_switch+0x176 sleepq_wait+0x42 _sx_xlock_hard+0x305 _sx_xlock+0x4e buf_hash_find+0x52 arc_read_nolock+0x54 arc_read+0x93 dbuf_prefetch+0x12c dmu_zfetch_dofetch+0x10b dmu_zfetch+0xaf8 dbuf_read+0x675 dnode_hold_impl+0xf2 dmu_bonus_hold+0x27 zfs_zget+0x237 zfs_fhtovp+0x12d nfsrv_fhtovp+0x16e nfsrv3_access+0x1d3 nfssvc_program+0x1a6
1963 100327 nfsd nfsd: service mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 svc_run_internal+0x930 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100328 nfsd nfsd: service mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 svc_run_internal+0x930 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100329 nfsd nfsd: service mi_switch+0x176 sleepq_wait+0x42 __lockmgr_args+0x49b vop_stdlock+0x39 VOP_LOCK1_APV+0x46 _vn_lock+0x47 zfs_vnode_lock+0x1d zfs_fhtovp+0x32b nfsrv_fhtovp+0x16e nfsrv_read+0x2bc nfssvc_program+0x1a6 svc_run_internal+0x6e9 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100330 nfsd nfsd: service mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 svc_run_internal+0x930 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100331 nfsd nfsd: service mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 svc_run_internal+0x930 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100332 nfsd nfsd: service mi_switch+0x176 sleepq_wait+0x42 _sx_xlock_hard+0x305 _sx_xlock+0x4e zfs_zget+0x21a zfs_fhtovp+0x12d nfsrv_fhtovp+0x16e nfsrv3_access+0x1d3 nfssvc_program+0x1a6 svc_run_internal+0x6e9 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100333 nfsd nfsd: service mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 svc_run_internal+0x930 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100334 nfsd nfsd: service mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 svc_run_internal+0x930 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100335 nfsd nfsd: service mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 svc_run_internal+0x930 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100336 nfsd nfsd: service mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 svc_run_internal+0x930 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100344 nfsd nfsd: master mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 svc_run_internal+0x930 svc_run+0x8f nfssvc_nfsd+0x12f nfssvc_nfsserver+0x5b nfssvc+0x48 amd64_syscall+0x1f4 Xfast_syscall+0xfc
1963 100534 nfsd nfsd: service mi_switch+0x176 sleepq_wait+0x42 _sx_xlock_hard+0x305 _sx_xlock+0x4e arc_buf_add_ref+0x87 dbuf_hold_impl+0x6d dbuf_hold+0x1b dmu_buf_hold+0x9b zap_lockdir+0x58 zap_lookup_norm+0x45 zap_lookup+0x2e zfs_dirent_lock+0x4ff zfs_dirlook+0x69 zfs_lookup+0x26b zfs_freebsd_lookup+0x81 vfs_cache_lookup+0xf0 VOP_LOOKUP_APV+0x40 lookup+0x452
1963 100535 nfsd nfsd: service mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 svc_run_internal+0x930 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100536 nfsd nfsd: service mi_switch+0x176 sleepq_wait+0x42 _sx_xlock_hard+0x305 _sx_xlock+0x4e zfs_zget+0x21a zfs_fhtovp+0x12d nfsrv_fhtovp+0x16e nfsrv_getattr+0xf9 nfssvc_program+0x1a6 svc_run_internal+0x6e9 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100537 nfsd nfsd: service mi_switch+0x176 sleepq_wait+0x42 __lockmgr_args+0x49b vop_stdlock+0x39 VOP_LOCK1_APV+0x46 _vn_lock+0x47 zfs_vnode_lock+0x1d zfs_fhtovp+0x32b nfsrv_fhtovp+0x16e nfsrv_read+0x2bc nfssvc_program+0x1a6 svc_run_internal+0x6e9 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100538 nfsd nfsd: service mi_switch+0x176 sleepq_wait+0x42 __lockmgr_args+0x49b vop_stdlock+0x39 VOP_LOCK1_APV+0x46 _vn_lock+0x47 zfs_vnode_lock+0x1d zfs_fhtovp+0x32b nfsrv_fhtovp+0x16e nfsrv_read+0x2bc nfssvc_program+0x1a6 svc_run_internal+0x6e9 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100539 nfsd nfsd: service mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 svc_run_internal+0x930 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100540 nfsd nfsd: service mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 svc_run_internal+0x930 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100541 nfsd nfsd: service mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 svc_run_internal+0x930 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100542 nfsd nfsd: service mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 svc_run_internal+0x930 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100543 nfsd nfsd: service mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 svc_run_internal+0x930 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100544 nfsd nfsd: service mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 svc_run_internal+0x930 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
1963 100545 nfsd nfsd: service mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 svc_run_internal+0x930 svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe
3505 100173 cron - mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_wait_sig+0x16 _sleep+0x269 pipe_read+0x4a6 dofileread+0xa1 kern_readv+0x60 read+0x55 amd64_syscall+0x1f4 Xfast_syscall+0xfc
3506 100529 cron - mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_wait_sig+0x16 _sleep+0x269 pipe_read+0x4a6 dofileread+0xa1 kern_readv+0x60 read+0x55 amd64_syscall+0x1f4 Xfast_syscall+0xfc
3510 101664 bash - mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_wait_sig+0x16 _sleep+0x269 kern_wait+0x6fd wait4+0x35 amd64_syscall+0x1f4 Xfast_syscall+0xfc
3511 101353 bash - mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_wait_sig+0x16 _sleep+0x269 kern_wait+0x6fd wait4+0x35 amd64_syscall+0x1f4 Xfast_syscall+0xfc
3517 101816 zfs initial thread mi_switch+0x176 sleepq_wait+0x42 _cv_wait+0x129 txg_wait_synced+0x85 dsl_sync_task_group_wait+0x128 dmu_objset_snapshot+0x302 zfs_ioc_snapshot+0x1a8 zfsdev_ioctl+0xe6 devfs_ioctl_f+0x7b kern_ioctl+0x102 ioctl+0xfd amd64_syscall+0x1f4 Xfast_syscall+0xfc
3518 101606 zfs initial thread mi_switch+0x176 sleepq_wait+0x42 _cv_wait+0x129 txg_wait_synced+0x85 dsl_sync_task_group_wait+0x128 dmu_objset_snapshot+0x302 zfs_ioc_snapshot+0x1a8 zfsdev_ioctl+0xe6 devfs_ioctl_f+0x7b kern_ioctl+0x102 ioctl+0xfd amd64_syscall+0x1f4 Xfast_syscall+0xfc
3791 100597 ntpd - mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_wait_sig+0x16 _cv_wait_sig+0x128 seltdwait+0x110 kern_select+0x6ef select+0x5d amd64_syscall+0x1f4 Xfast_syscall+0xfc
3859 100678 cron - mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_wait_sig+0x16 _sleep+0x269 pipe_read+0x4a6 dofileread+0xa1 kern_readv+0x60 read+0x55 amd64_syscall+0x1f4 Xfast_syscall+0xfc
3861 101527 bash - mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_wait_sig+0x16 _sleep+0x269 kern_wait+0x6fd wait4+0x35 amd64_syscall+0x1f4 Xfast_syscall+0xfc
3863 101689 zfs initial thread mi_switch+0x176 sleepq_wait+0x42 _cv_wait+0x129 txg_wait_synced+0x85 dsl_sync_task_group_wait+0x128 dmu_objset_snapshot+0x302 zfs_ioc_snapshot+0x1a8 zfsdev_ioctl+0xe6 devfs_ioctl_f+0x7b kern_ioctl+0x102 ioctl+0xfd amd64_syscall+0x1f4 Xfast_syscall+0xfc
4060 100943 sshd - mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_wait_sig+0x16 _sleep+0x269 soreceive_generic+0xf9a dofileread+0xa1 kern_readv+0x60 read+0x55 amd64_syscall+0x1f4 Xfast_syscall+0xfc
4064 101040 sshd - mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_wait_sig+0x16 _cv_wait_sig+0x128 seltdwait+0x110 kern_select+0x6ef select+0x5d amd64_syscall+0x1f4 Xfast_syscall+0xfc
4065 100737 bash - mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_wait_sig+0x16 _cv_wait_sig+0x128 tty_wait+0x25 ttydisc_read+0x2b1 ttydev_read+0x10f devfs_read_f+0x88 dofileread+0xa1 kern_readv+0x60 read+0x55 amd64_syscall+0x1f4 Xfast_syscall+0xfc
4100 101039 sshd - mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_wait_sig+0x16 _sleep+0x269 soreceive_generic+0xf9a dofileread+0xa1 kern_readv+0x60 read+0x55 amd64_syscall+0x1f4 Xfast_syscall+0xfc
4111 101336 sshd - mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_wait_sig+0x16 _cv_wait_sig+0x128 seltdwait+0x110 kern_select+0x6ef select+0x5d amd64_syscall+0x1f4 Xfast_syscall+0xfc
4112 100909 sudo - mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_wait_sig+0x16 _cv_wait_sig+0x128 seltdwait+0x110 kern_select+0x6ef select+0x5d amd64_syscall+0x1f4 Xfast_syscall+0xfc
4113 101706 procstat - <running>
5456 101080 getty - mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_wait_sig+0x16 _cv_wait_sig+0x128 tty_wait+0x25 ttydisc_read+0x2b1 ttydev_read+0x10f devfs_read_f+0x88 dofileread+0xa1 kern_readv+0x60 read+0x55 amd64_syscall+0x1f4 Xfast_syscall+0xfc
6299 100216 mountd - mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_wait_sig+0x16 _cv_wait_sig+0x128 seltdwait+0x110 kern_select+0x6ef select+0x5d amd64_syscall+0x1f4 Xfast_syscall+0xfc
6676 101985 tmux - mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _cv_timedwait_sig+0x134 seltdwait+0x98 poll+0x478 amd64_syscall+0x1f4 Xfast_syscall+0xfc
6677 100822 bash - mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_wait_sig+0x16 _cv_wait_sig+0x128 tty_wait+0x25 ttydisc_read+0x2b1 ttydev_read+0x10f devfs_read_f+0x88 dofileread+0xa1 kern_readv+0x60 read+0x55 amd64_syscall+0x1f4 Xfast_syscall+0xfc
12082 100565 sshd - mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_wait_sig+0x16 _cv_wait_sig+0x128 seltdwait+0x110 kern_select+0x6ef select+0x5d amd64_syscall+0x1f4 Xfast_syscall+0xfc
12460 100688 zabbix_agentd - mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_wait_sig+0x16 _sleep+0x269 kern_wait+0x6fd wait4+0x35 amd64_syscall+0x1f4 Xfast_syscall+0xfc
12461 100578 zabbix_agentd - mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _sleep+0x1b1 kern_nanosleep+0x118 nanosleep+0x6e amd64_syscall+0x1f4 Xfast_syscall+0xfc
12462 100343 zabbix_agentd - mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_wait_sig+0x16 _sleep+0x269 kern_accept+0x1de accept+0x75 amd64_syscall+0x1f4 Xfast_syscall+0xfc
12463 100661 zabbix_agentd - mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_wait_sig+0x16 _cv_wait_sig+0x128 seltdwait+0x110 kern_select+0x6ef select+0x5d amd64_syscall+0x1f4 Xfast_syscall+0xfc
12464 100205 zabbix_agentd - mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_wait_sig+0x16 _sleep+0x269 kern_accept+0x1de accept+0x75 amd64_syscall+0x1f4 Xfast_syscall+0xfc
12465 100679 zabbix_agentd - mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_timedwait_sig+0x19 _sleep+0x1b1 kern_nanosleep+0x118 nanosleep+0x6e amd64_syscall+0x1f4 Xfast_syscall+0xfc
76407 100733 getty - mi_switch+0x176 sleepq_catch_signals+0x31c sleepq_wait_sig+0x16 _cv_wait_sig+0x128 tty_wait+0x25 ttydisc_read+0x2b1 ttydev_read+0x10f devfs_read_f+0x88 dofileread+0xa1 kern_readv+0x60 read+0x55 amd64_syscall+0x1f4 Xfast_syscall+0xfc
>How-To-Repeat:
Heavy NFS + CIFS traffic over ZFS - leave it long enough and it'll deadlock eventually.
>Fix:
>Release-Note:
>Audit-Trail:
>Unformatted:
More information about the freebsd-bugs
mailing list