kern/134584: [panic] spin lock held too long

Sun Jul 26 06:50:03 UTC 2009

The following reply was made to PR kern/134584; it has been noted by GNATS.

From: "barbara" <barbara.xxx1975 at libero.it>
To: "bug-followup" <bug-followup at FreeBSD.org>
Cc: "FreeBSD-stable" <FreeBSD-stable at FreeBSD.org>
Subject: Re: kern/134584: [panic] spin lock held too long
Date: Sun, 26 Jul 2009 08:46:56 +0200

 It happened again, on shutdown.
 As the previous time, it happened after a high (for a desktop) uptime and=
 , if it could matter, after running net-p2p/transmission-gtk2 for several=
  hours.
 I don't know if it's related, but often quitting transmission, doesn't te=
 rminate the process. Sometimes it end after several minutes the gui exite=
 d, sometimes it's still running after hours.
 I've noticed it as the destination folder is on a manually mounted device=
  and I can't umount it as fstat reports the device used by a transmission=
  process.
 So I often have to kill it.
 This happened both the time I had this kind of panic.

 Barbara

 Script started on Sun Jul 26 08:22:33 2009
 You have mail.
 satanasso# kgdb kernel.debug /var/crash/vmcore.17
 GNU gdb 6.1.1 [FreeBSD]
 Copyright 2004 Free Software Foundation, Inc.
 GDB is free software, covered by the GNU General Public License, and you =
 are
 welcome to change it and/or distribute copies of it under certain conditi=
 ons.
 Type "show copying" to see the conditions.
 There is absolutely no warranty for GDB.  Type "show warranty" for detail=
 s.
 This GDB was configured as "i386-marcel-freebsd"...

 Unread portion of the kernel message buffer:
 <118>Stopping devd.
 <118>Writing entropy file:
 <118>.
 <118>Terminated
 <118>.
 <118>Jul 25 17:45:18 satanasso syslogd: exiting on signal 15
 Waiting (max 60 seconds) for system process `vnlru' to stop...done
 Waiting (max 60 seconds) for system process `bufdaemon' to stop...done
 Waiting (max 60 seconds) for system process `syncer' to stop...
 Syncing disks, vnodes remaining...7 6 5 3 1 2 2 0 0 done
 All buffers synced.
 Uptime: 22h41m12s
 Rebooting...
 cpu_reset: Stopping other CPUs
 spin lock 0xc08ae540 (sched lock 1) held by 0xc588f480 (tid 100006) too l=
 ong
 panic: spin lock held too long
 cpuid =3D 0
 Uptime: 22h41m14s
 Physical memory: 2031 MB
 Dumping 171 MB: (CTRL-C to abort)  156 140 124 108 92 76 60 44 28 12

 Reading symbols from /boot/kernel/linux.ko...Reading symbols from /boot/k=
 ernel/linux.ko.symbols...done.
 done.
 Loaded symbols for /boot/kernel/linux.ko
 Reading symbols from /boot/modules/nvidia.ko...done.
 Loaded symbols for /boot/modules/nvidia.ko
 Reading symbols from /boot/kernel/acpi.ko...Reading symbols from /boot/ke=
 rnel/acpi.ko.symbols...done.
 done.
 Loaded symbols for /boot/kernel/acpi.ko
 Reading symbols from /boot/kernel/linprocfs.ko...Reading symbols from /bo=
 ot/kernel/linprocfs.ko.symbols...done.
 done.
 Loaded symbols for /boot/kernel/linprocfs.ko
 Reading symbols from /boot/kernel/logo_saver.ko...Reading symbols from /b=
 oot/kernel/logo_saver.ko.symbols...done.
 done.
 Loaded symbols for /boot/kernel/logo_saver.ko
 Reading symbols from /boot/kernel/ntfs.ko...Reading symbols from /boot/ke=
 rnel/ntfs.ko.symbols...done.
 done.
 Loaded symbols for /boot/kernel/ntfs.ko
 #0  doadump () at pcpu.h:196
 196		__asm __volatile("movl %%fs:0,%0" : "=3Dr" (td));
 (kgdb) bt
 #0  doadump () at pcpu.h:196
 #1  0xc059b097 in boot (howto=3D260) at /usr/src/sys/kern/kern_shutdown.c=
 :418
 #2  0xc059b36c in panic (fmt=3DVariable "fmt" is not available.
 ) at /usr/src/sys/kern/kern_shutdown.c:574
 #3  0xc058bd1f in _mtx_lock_spin_failed (m=3D0x0) at /usr/src/sys/kern/ke=
 rn_mutex.c:449
 #4  0xc058c425 in _thread_lock_flags (td=3D0xc588f480, opts=3D0, file=3D0=
 xc08200c8 "/usr/src/sys/kern/kern_intr.c", line=3D801) at /usr/src/sys/ke=
 rn/kern_mutex.c:537
 #5  0xc0576787 in intr_event_schedule_thread (ie=3DVariable "ie" is not a=
 vailable.
 ) at /usr/src/sys/kern/kern_intr.c:801
 #6  0xc0577aa5 in swi_sched (cookie=3D0xc5880e00, flags=3DVariable "flags=
 " is not available.
 ) at /usr/src/sys/kern/kern_intr.c:1028
 #7  0xc055a6df in hardclock (usermode=3D0, pc=3D3229503748) at /usr/src/s=
 ys/kern/kern_clock.c:339
 #8  0xc07ce23a in lapic_handle_timer (frame=3D0xc54aaba0) at /usr/src/sys=
 /i386/i386/local_apic.c:705
 #9  0xc07c6c9f in Xtimerint () at apic_vector.s:108
 #10 0xc07e5104 in DELAY (n=3D1000000) at cpufunc.h:352
 #11 0xc07e2c03 in cpu_reset () at /usr/src/sys/i386/i386/vm_machdep.c:594=

 #12 0xc059a753 in shutdown_reset (junk=3D0x0, howto=3D0) at /usr/src/sys/=
 kern/kern_shutdown.c:493
 #13 0xc059b117 in boot (howto=3D0) at /usr/src/sys/kern/kern_shutdown.c:4=
 21
 #14 0xc059b247 in reboot (td=3D0xc588fd80, uap=3D0xc54aacfc) at /usr/src/=
 sys/kern/kern_shutdown.c:172
 #15 0xc07e07d5 in syscall (frame=3D0xc54aad38) at /usr/src/sys/i386/i386/=
 trap.c:1089
 #16 0xc07c6720 in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception=
 .s:262
 #17 0x00000033 in ?? ()
 Previous frame inner to this frame (corrupt stack?)
 (kgdb) thread apply all bt

 Thread 52 (Thread 100051):
 #0  sched_switch (td=3D0xc5c91d80, newtd=3DVariable "newtd" is not availa=
 ble.
 ) at /usr/src/sys/kern/sched_ule.c:1944
 #1  0xc05a3033 in mi_switch (flags=3DVariable "flags" is not available.
 ) at /usr/src/sys/kern/kern_synch.c:444
 #2  0xc05cf859 in sleepq_switch (wchan=3DVariable "wchan" is not availabl=
 e.
 ) at /usr/src/sys/kern/subr_sleepqueue.c:497
 #3  0xc05d03d7 in sleepq_timedwait (wchan=3D0xc08bd300) at /usr/src/sys/k=
 ern/subr_sleepqueue.c:615
 #4  0xc05a3483 in _sleep (ident=3D0xc08bd300, lock=3D0xc08bd2c0, priority=
 =3D68, wmesg=3D0xc0833855 "sdflush", timo=3D1000) at /usr/src/sys/kern/ke=
 rn_synch.c:226
 #5  0xc076407e in softdep_flush () at /usr/src/sys/ufs/ffs/ffs_softdep.c:=
 773
 #6  0xc0574587 in fork_exit (callout=3D0xc0763cb0 <softdep_flush>, arg=3D=
 0x0, frame=3D0xc5617d38) at /usr/src/sys/kern/kern_fork.c:811
 #7  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.=
 s:271

 Thread 51 (Thread 100050):
 #0  sched_switch (td=3D0xc5a616c0, newtd=3DVariable "newtd" is not availa=
 ble.
 ) at /usr/src/sys/kern/sched_ule.c:1944
 #1  0xc05a3033 in mi_switch (flags=3DVariable "flags" is not available.
 ) at /usr/src/sys/kern/kern_synch.c:444
 #2  0xc05cf859 in sleepq_switch (wchan=3DVariable "wchan" is not availabl=
 e.
 ) at /usr/src/sys/kern/subr_sleepqueue.c:497
 #3  0xc05cfec6 in sleepq_wait (wchan=3D0xc5abe36c) at /usr/src/sys/kern/s=
 ubr_sleepqueue.c:580
 #4  0xc05a34a5 in _sleep (ident=3D0xc5abe36c, lock=3D0xc5abe350, priority=
 =3D104, wmesg=3D0xc0820618 "ktsusp", timo=3D0) at /usr/src/sys/kern/kern_=
 synch.c:230
 #5  0xc057e632 in kthread_suspend_check (p=3D0xc5abe2c0) at /usr/src/sys/=
 kern/kern_kthread.c:197
 #6  0xc0620b1e in vnlru_proc () at /usr/src/sys/kern/vfs_subr.c:724
 #7  0xc0574587 in fork_exit (callout=3D0xc0620ad0 <vnlru_proc>, arg=3D0x0=
 , frame=3D0xc5614d38) at /usr/src/sys/kern/kern_fork.c:811
 #8  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.=
 s:271

 Thread 50 (Thread 100049):
 #0  sched_switch (td=3D0xc5a61900, newtd=3DVariable "newtd" is not availa=
 ble.
 ) at /usr/src/sys/kern/sched_ule.c:1944
 #1  0xc05a3033 in mi_switch (flags=3DVariable "flags" is not available.
 ) at /usr/src/sys/kern/kern_synch.c:444
 #2  0xc05cf859 in sleepq_switch (wchan=3DVariable "wchan" is not availabl=
 e.
 ) at /usr/src/sys/kern/subr_sleepqueue.c:497
 #3  0xc05cfec6 in sleepq_wait (wchan=3D0xc5abe62c) at /usr/src/sys/kern/s=
 ubr_sleepqueue.c:580
 #4  0xc05a34a5 in _sleep (ident=3D0xc5abe62c, lock=3D0xc5abe610, priority=
 =3D104, wmesg=3D0xc0820618 "ktsusp", timo=3D0) at /usr/src/sys/kern/kern_=
 synch.c:230
 #5  0xc057e632 in kthread_suspend_check (p=3D0xc5abe580) at /usr/src/sys/=
 kern/kern_kthread.c:197
 #6  0xc061f9c2 in sched_sync () at /usr/src/sys/kern/vfs_subr.c:1719
 #7  0xc0574587 in fork_exit (callout=3D0xc061f8a0 <sched_sync>, arg=3D0x0=
 , frame=3D0xc5611d38) at /usr/src/sys/kern/kern_fork.c:811
 #8  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.=
 s:271

 Thread 49 (Thread 100048):
 #0  sched_switch (td=3D0xc5a61b40, newtd=3DVariable "newtd" is not availa=
 ble.
 ) at /usr/src/sys/kern/sched_ule.c:1944
 #1  0xc05a3033 in mi_switch (flags=3DVariable "flags" is not available.
 ) at /usr/src/sys/kern/kern_synch.c:444
 #2  0xc05cf859 in sleepq_switch (wchan=3DVariable "wchan" is not availabl=
 e.
 ) at /usr/src/sys/kern/subr_sleepqueue.c:497
 #3  0xc05cfec6 in sleepq_wait (wchan=3D0xc5abe8ec) at /usr/src/sys/kern/s=
 ubr_sleepqueue.c:580
 #4  0xc05a34a5 in _sleep (ident=3D0xc5abe8ec, lock=3D0xc5abe8d0, priority=
 =3D104, wmesg=3D0xc0820618 "ktsusp", timo=3D0) at /usr/src/sys/kern/kern_=
 synch.c:230
 #5  0xc057e632 in kthread_suspend_check (p=3D0xc5abe840) at /usr/src/sys/=
 kern/kern_kthread.c:197
 #6  0xc0609c4b in buf_daemon () at /usr/src/sys/kern/vfs_bio.c:2155
 #7  0xc0574587 in fork_exit (callout=3D0xc0609b70 <buf_daemon>, arg=3D0x0=
 , frame=3D0xc560ed38) at /usr/src/sys/kern/kern_fork.c:811
 #8  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.=
 s:271

 Thread 48 (Thread 100047):
 #0  sched_switch (td=3D0xc5a61d80, newtd=3DVariable "newtd" is not availa=
 ble.
 ) at /usr/src/sys/kern/sched_ule.c:1944
 #1  0xc05a3033 in mi_switch (flags=3DVariable "flags" is not available.
 ) at /usr/src/sys/kern/kern_synch.c:444
 #2  0xc05cf859 in sleepq_switch (wchan=3DVariable "wchan" is not availabl=
 e.
 ) at /usr/src/sys/kern/subr_sleepqueue.c:497
 #3  0xc05d03d7 in sleepq_timedwait (wchan=3D0xc08bdf14) at /usr/src/sys/k=
 ern/subr_sleepqueue.c:615
 #4  0xc05a3483 in _sleep (ident=3D0xc08bdf14, lock=3D0xc08bdad0, priority=
 =3D0, wmesg=3D0xc0836ae9 "pgzero", timo=3D300000) at /usr/src/sys/kern/ke=
 rn_synch.c:226
 #5  0xc07a155c in vm_pagezero (arg=3D0x0) at /usr/src/sys/vm/vm_zeroidle.=
 c:136
 #6  0xc0574587 in fork_exit (callout=3D0xc07a1480 <vm_pagezero>, arg=3D0x=
 0, frame=3D0xc560bd38) at /usr/src/sys/kern/kern_fork.c:811
 #7  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.=
 s:271

 Thread 47 (Thread 100046):
 #0  sched_switch (td=3D0xc5aa1000, newtd=3DVariable "newtd" is not availa=
 ble.
 ) at /usr/src/sys/kern/sched_ule.c:1944
 #1  0xc05a3033 in mi_switch (flags=3DVariable "flags" is not available.
 ) at /usr/src/sys/kern/kern_synch.c:444
 #2  0xc05cf859 in sleepq_switch (wchan=3DVariable "wchan" is not availabl=
 e.
 ) at /usr/src/sys/kern/subr_sleepqueue.c:497
 ---Type <return> to continue, or q <return> to quit---
 #3  0xc05cfec6 in sleepq_wait (wchan=3D0xc08bdb44) at /usr/src/sys/kern/s=
 ubr_sleepqueue.c:580
 #4  0xc05a34a5 in _sleep (ident=3D0xc08bdb44, lock=3D0xc08bdb48, priority=
 =3D104, wmesg=3D0xc0826d0c "psleep", timo=3D0) at /usr/src/sys/kern/kern_=
 synch.c:230
 #5  0xc079c4e8 in vm_daemon () at /usr/src/sys/vm/vm_pageout.c:1535
 #6  0xc0574587 in fork_exit (callout=3D0xc079c470 <vm_daemon>, arg=3D0x0,=
  frame=3D0xc5608d38) at /usr/src/sys/kern/kern_fork.c:811
 #7  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.=
 s:271

 Thread 46 (Thread 100045):
 #0  sched_switch (td=3D0xc5aa1240, newtd=3DVariable "newtd" is not availa=
 ble.
 ) at /usr/src/sys/kern/sched_ule.c:1944
 #1  0xc05a3033 in mi_switch (flags=3DVariable "flags" is not available.
 ) at /usr/src/sys/kern/kern_synch.c:444
 #2  0xc05cf859 in sleepq_switch (wchan=3DVariable "wchan" is not availabl=
 e.
 ) at /usr/src/sys/kern/subr_sleepqueue.c:497
 #3  0xc05d03d7 in sleepq_timedwait (wchan=3D0xc08bdb0c) at /usr/src/sys/k=
 ern/subr_sleepqueue.c:615
 #4  0xc05a3483 in _sleep (ident=3D0xc08bdb0c, lock=3D0xc08bdad0, priority=
 =3D68, wmesg=3D0xc0826d0c "psleep", timo=3D5000) at /usr/src/sys/kern/ker=
 n_synch.c:226
 #5  0xc079ce6b in vm_pageout () at /usr/src/sys/vm/vm_pageout.c:1475
 #6  0xc0574587 in fork_exit (callout=3D0xc079cb80 <vm_pageout>, arg=3D0x0=
 , frame=3D0xc5605d38) at /usr/src/sys/kern/kern_fork.c:811
 #7  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.=
 s:271

 Thread 45 (Thread 100044):
 #0  sched_switch (td=3D0xc5aa1480, newtd=3DVariable "newtd" is not availa=
 ble.
 ) at /usr/src/sys/kern/sched_ule.c:1944
 #1  0xc05a3033 in mi_switch (flags=3DVariable "flags" is not available.
 ) at /usr/src/sys/kern/kern_synch.c:444
 #2  0xc05cf859 in sleepq_switch (wchan=3DVariable "wchan" is not availabl=
 e.
 ) at /usr/src/sys/kern/subr_sleepqueue.c:497
 #3  0xc05cfec6 in sleepq_wait (wchan=3D0xc08b376c) at /usr/src/sys/kern/s=
 ubr_sleepqueue.c:580
 #4  0xc05a34a5 in _sleep (ident=3D0xc08b376c, lock=3D0xc08b3674, priority=
 =3D0, wmesg=3D0xc082a92a "waiting_for_work", timo=3D0) at /usr/src/sys/ke=
 rn/kern_synch.c:230
 #5  0xc06693df in sctp_iterator_thread (v=3D0x0) at /usr/src/sys/netinet/=
 sctp_bsd_addr.c:95
 #6  0xc0574587 in fork_exit (callout=3D0xc0669360 <sctp_iterator_thread>,=
  arg=3D0x0, frame=3D0xc55d2d38) at /usr/src/sys/kern/kern_fork.c:811
 #7  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.=
 s:271

 Thread 44 (Thread 100043):
 #0  fork_trampoline () at /usr/src/sys/i386/i386/exception.s:268
 Current language:  auto; currently asm

 Thread 43 (Thread 100042):
 #0  sched_switch (td=3D0xc5aa1900, newtd=3DVariable "newtd" is not availa=
 ble.
 ) at /usr/src/sys/kern/sched_ule.c:1944
 #1  0xc05a3033 in mi_switch (flags=3DVariable "flags" is not available.
 ) at /usr/src/sys/kern/kern_synch.c:444
 #2  0xc0577dbb in ithread_loop (arg=3D0xc5ab2c20) at /usr/src/sys/kern/ke=
 rn_intr.c:1225
 #3  0xc0574587 in fork_exit (callout=3D0xc0577ab0 <ithread_loop>, arg=3D0=
 xc5ab2c20, frame=3D0xc55c9d38) at /usr/src/sys/kern/kern_fork.c:811
 #4  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.=
 s:271
 Current language:  auto; currently c

 Thread 42 (Thread 100041):
 #0  fork_trampoline () at /usr/src/sys/i386/i386/exception.s:268
 Current language:  auto; currently asm

 Thread 41 (Thread 100040):
 #0  sched_switch (td=3D0xc5aa1d80, newtd=3DVariable "newtd" is not availa=
 ble.
 ) at /usr/src/sys/kern/sched_ule.c:1944
 #1  0xc05a3033 in mi_switch (flags=3DVariable "flags" is not available.
 ) at /usr/src/sys/kern/kern_synch.c:444
 #2  0xc05cf859 in sleepq_switch (wchan=3DVariable "wchan" is not availabl=
 e.
 ) at /usr/src/sys/kern/subr_sleepqueue.c:497
 #3  0xc05d03d7 in sleepq_timedwait (wchan=3D0xc5a83c3c) at /usr/src/sys/k=
 ern/subr_sleepqueue.c:615
 #4  0xc05a3483 in _sleep (ident=3D0xc5a83c3c, lock=3D0xc5a83cf0, priority=
 =3D76, wmesg=3D0xc081c912 "-", timo=3D1000) at /usr/src/sys/kern/kern_syn=
 ch.c:226
 #5  0xc07b319d in fdc_thread (arg=3Ddwarf2_read_address: Corrupted DWARF =
 expression.
 ) at /usr/src/sys/dev/fdc/fdc.c:812
 #6  0xc0574587 in fork_exit (callout=3D0xc07b2db0 <fdc_thread>, arg=3D0xc=
 5a83c00, frame=3D0xc55c3d38) at /usr/src/sys/kern/kern_fork.c:811
 #7  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.=
 s:271
 Current language:  auto; currently c

 Thread 40 (Thread 100039):
 #0  sched_switch (td=3D0xc5aa2000, newtd=3DVariable "newtd" is not availa=
 ble.
 ) at /usr/src/sys/kern/sched_ule.c:1944
 #1  0xc05a3033 in mi_switch (flags=3DVariable "flags" is not available.
 ) at /usr/src/sys/kern/kern_synch.c:444
 #2  0xc0577dbb in ithread_loop (arg=3D0xc5a8daa0) at /usr/src/sys/kern/ke=
 rn_intr.c:1225
 #3  0xc0574587 in fork_exit (callout=3D0xc0577ab0 <ithread_loop>, arg=3D0=
 xc5a8daa0, frame=3D0xc55b6d38) at /usr/src/sys/kern/kern_fork.c:811
 #4  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.=
 s:271

 Thread 39 (Thread 100038):
 #0  sched_switch (td=3D0xc59b2d80, newtd=3DVariable "newtd" is not availa=
 ble.
 ) at /usr/src/sys/kern/sched_ule.c:1944
 ---Type <return> to continue, or q <return> to quit---
 #1  0xc05a3033 in mi_switch (flags=3DVariable "flags" is not available.
 ) at /usr/src/sys/kern/kern_synch.c:444
 #2  0xc0577dbb in ithread_loop (arg=3D0xc5a8d1e0) at /usr/src/sys/kern/ke=
 rn_intr.c:1225
 #3  0xc0574587 in fork_exit (callout=3D0xc0577ab0 <ithread_loop>, arg=3D0=
 xc5a8d1e0, frame=3D0xc559ed38) at /usr/src/sys/kern/kern_fork.c:811
 #4  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.=
 s:271

 Thread 38 (Thread 100037):
 #0  sched_switch (td=3D0xc5a60000, newtd=3DVariable "newtd" is not availa=
 ble.
 ) at /usr/src/sys/kern/sched_ule.c:1944
 #1  0xc05a3033 in mi_switch (flags=3DVariable "flags" is not available.
 ) at /usr/src/sys/kern/kern_synch.c:444
 #2  0xc05cf859 in sleepq_switch (wchan=3DVariable "wchan" is not availabl=
 e.
 ) at /usr/src/sys/kern/subr_sleepqueue.c:497
 #3  0xc05d03d7 in sleepq_timedwait (wchan=3D0xc5999210) at /usr/src/sys/k=
 ern/subr_sleepqueue.c:615
 #4  0xc05a3483 in _sleep (ident=3D0xc5999210, lock=3D0x0, priority=3D92, =
 wmesg=3D0xc081bb4e "usbevt", timo=3D60000) at /usr/src/sys/kern/kern_sync=
 h.c:226
 #5  0xc051cad5 in usb_event_thread (arg=3D0xc5a8f3c0) at /usr/src/sys/dev=
 /usb/usb.c:445
 #6  0xc0574587 in fork_exit (callout=3D0xc051c9e0 <usb_event_thread>, arg=
 =3D0xc5a8f3c0, frame=3D0xc5597d38) at /usr/src/sys/kern/kern_fork.c:811
 #7  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.=
 s:271

 Thread 37 (Thread 100036):
 #0  sched_switch (td=3D0xc5a60240, newtd=3DVariable "newtd" is not availa=
 ble.
 ) at /usr/src/sys/kern/sched_ule.c:1944
 #1  0xc05a3033 in mi_switch (flags=3DVariable "flags" is not available.
 ) at /usr/src/sys/kern/kern_synch.c:444
 #2  0xc05cf859 in sleepq_switch (wchan=3DVariable "wchan" is not availabl=
 e.
 ) at /usr/src/sys/kern/subr_sleepqueue.c:497
 #3  0xc05d03d7 in sleepq_timedwait (wchan=3D0xc5a7e210) at /usr/src/sys/k=
 ern/subr_sleepqueue.c:615
 #4  0xc05a3483 in _sleep (ident=3D0xc5a7e210, lock=3D0x0, priority=3D92, =
 wmesg=3D0xc081bb4e "usbevt", timo=3D60000) at /usr/src/sys/kern/kern_sync=
 h.c:226
 #5  0xc051cad5 in usb_event_thread (arg=3D0xc5a57900) at /usr/src/sys/dev=
 /usb/usb.c:445
 #6  0xc0574587 in fork_exit (callout=3D0xc051c9e0 <usb_event_thread>, arg=
 =3D0xc5a57900, frame=3D0xc5584d38) at /usr/src/sys/kern/kern_fork.c:811
 #7  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.=
 s:271

 Thread 36 (Thread 100035):
 #0  fork_trampoline () at /usr/src/sys/i386/i386/exception.s:268
 Current language:  auto; currently asm

 Thread 35 (Thread 100034):
 #0  sched_switch (td=3D0xc5a606c0, newtd=3DVariable "newtd" is not availa=
 ble.
 ) at /usr/src/sys/kern/sched_ule.c:1944
 #1  0xc05a3033 in mi_switch (flags=3DVariable "flags" is not available.
 ) at /usr/src/sys/kern/kern_synch.c:444
 #2  0xc05cf859 in sleepq_switch (wchan=3DVariable "wchan" is not availabl=
 e.
 ) at /usr/src/sys/kern/subr_sleepqueue.c:497
 #3  0xc05d03d7 in sleepq_timedwait (wchan=3D0xc5a65210) at /usr/src/sys/k=
 ern/subr_sleepqueue.c:615
 #4  0xc05a3483 in _sleep (ident=3D0xc5a65210, lock=3D0x0, priority=3D92, =
 wmesg=3D0xc081bb4e "usbevt", timo=3D60000) at /usr/src/sys/kern/kern_sync=
 h.c:226
 #5  0xc051cad5 in usb_event_thread (arg=3D0xc5a62240) at /usr/src/sys/dev=
 /usb/usb.c:445
 #6  0xc0574587 in fork_exit (callout=3D0xc051c9e0 <usb_event_thread>, arg=
 =3D0xc5a62240, frame=3D0xc557dd38) at /usr/src/sys/kern/kern_fork.c:811
 #7  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.=
 s:271
 Current language:  auto; currently c

 Thread 34 (Thread 100033):
 #0  sched_switch (td=3D0xc5a60900, newtd=3DVariable "newtd" is not availa=
 ble.
 ) at /usr/src/sys/kern/sched_ule.c:1944
 #1  0xc05a3033 in mi_switch (flags=3DVariable "flags" is not available.
 ) at /usr/src/sys/kern/kern_synch.c:444
 #2  0xc05cf859 in sleepq_switch (wchan=3DVariable "wchan" is not availabl=
 e.
 ) at /usr/src/sys/kern/subr_sleepqueue.c:497
 #3  0xc05d03d7 in sleepq_timedwait (wchan=3D0xc5a6d210) at /usr/src/sys/k=
 ern/subr_sleepqueue.c:615
 #4  0xc05a3483 in _sleep (ident=3D0xc5a6d210, lock=3D0x0, priority=3D92, =
 wmesg=3D0xc081bb4e "usbevt", timo=3D60000) at /usr/src/sys/kern/kern_sync=
 h.c:226
 #5  0xc051cad5 in usb_event_thread (arg=3D0xc5a62a80) at /usr/src/sys/dev=
 /usb/usb.c:445
 #6  0xc0574587 in fork_exit (callout=3D0xc051c9e0 <usb_event_thread>, arg=
 =3D0xc5a62a80, frame=3D0xc5579d38) at /usr/src/sys/kern/kern_fork.c:811
 #7  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.=
 s:271

 Thread 33 (Thread 100032):
 #0  fork_trampoline () at /usr/src/sys/i386/i386/exception.s:268
 Current language:  auto; currently asm

 Thread 32 (Thread 100031):
 #0  sched_switch (td=3D0xc5a60d80, newtd=3DVariable "newtd" is not availa=
 ble.
 ) at /usr/src/sys/kern/sched_ule.c:1944
 #1  0xc05a3033 in mi_switch (flags=3DVariable "flags" is not available.
 ) at /usr/src/sys/kern/kern_synch.c:444
 #2  0xc05cf859 in sleepq_switch (wchan=3DVariable "wchan" is not availabl=
 e.
 ) at /usr/src/sys/kern/subr_sleepqueue.c:497
 #3  0xc05cfec6 in sleepq_wait (wchan=3D0xc08a1b54) at /usr/src/sys/kern/s=
 ubr_sleepqueue.c:580
 #4  0xc05a34a5 in _sleep (ident=3D0xc08a1b54, lock=3D0x0, priority=3D92, =
 wmesg=3D0xc081bb40 "usbtsk", timo=3D0) at /usr/src/sys/kern/kern_synch.c:=
 230
 #5  0xc051c97e in usb_task_thread (arg=3D0xc08a1b54) at /usr/src/sys/dev/=
 usb/usb.c:475
 ---Type <return> to continue, or q <return> to quit---
 #6  0xc0574587 in fork_exit (callout=3D0xc051c900 <usb_task_thread>, arg=3D=
 0xc08a1b54, frame=3D0xc5572d38) at /usr/src/sys/kern/kern_fork.c:811
 #7  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.=
 s:271
 Current language:  auto; currently c

 Thread 31 (Thread 100030):
 #0  sched_switch (td=3D0xc5a61000, newtd=3DVariable "newtd" is not availa=
 ble.
 ) at /usr/src/sys/kern/sched_ule.c:1944
 #1  0xc05a3033 in mi_switch (flags=3DVariable "flags" is not available.
 ) at /usr/src/sys/kern/kern_synch.c:444
 #2  0xc05cf859 in sleepq_switch (wchan=3DVariable "wchan" is not availabl=
 e.
 ) at /usr/src/sys/kern/subr_sleepqueue.c:497
 #3  0xc05cfec6 in sleepq_wait (wchan=3D0xc08a1b40) at /usr/src/sys/kern/s=
 ubr_sleepqueue.c:580
 #4  0xc05a34a5 in _sleep (ident=3D0xc08a1b40, lock=3D0x0, priority=3D92, =
 wmesg=3D0xc081bb40 "usbtsk", timo=3D0) at /usr/src/sys/kern/kern_synch.c:=
 230
 #5  0xc051c97e in usb_task_thread (arg=3D0xc08a1b40) at /usr/src/sys/dev/=
 usb/usb.c:475
 #6  0xc0574587 in fork_exit (callout=3D0xc051c900 <usb_task_thread>, arg=3D=
 0xc08a1b40, frame=3D0xc556fd38) at /usr/src/sys/kern/kern_fork.c:811
 #7  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.=
 s:271

 Thread 30 (Thread 100029):
 #0  sched_switch (td=3D0xc5a61240, newtd=3DVariable "newtd" is not availa=
 ble.
 ) at /usr/src/sys/kern/sched_ule.c:1944
 #1  0xc05a3033 in mi_switch (flags=3DVariable "flags" is not available.
 ) at /usr/src/sys/kern/kern_synch.c:444
 #2  0xc05cf859 in sleepq_switch (wchan=3DVariable "wchan" is not availabl=
 e.
 ) at /usr/src/sys/kern/subr_sleepqueue.c:497
 #3  0xc05d03d7 in sleepq_timedwait (wchan=3D0xc5a4c210) at /usr/src/sys/k=
 ern/subr_sleepqueue.c:615
 #4  0xc05a3483 in _sleep (ident=3D0xc5a4c210, lock=3D0x0, priority=3D92, =
 wmesg=3D0xc081bb4e "usbevt", timo=3D60000) at /usr/src/sys/kern/kern_sync=
 h.c:226
 #5  0xc051cad5 in usb_event_thread (arg=3D0xc5a63500) at /usr/src/sys/dev=
 /usb/usb.c:445
 #6  0xc0574587 in fork_exit (callout=3D0xc051c9e0 <usb_event_thread>, arg=
 =3D0xc5a63500, frame=3D0xc556cd38) at /usr/src/sys/kern/kern_fork.c:811
 #7  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.=
 s:271

 Thread 29 (Thread 100028):
 #0  sched_switch (td=3D0xc5a61480, newtd=3DVariable "newtd" is not availa=
 ble.
 ) at /usr/src/sys/kern/sched_ule.c:1944
 #1  0xc05a3033 in mi_switch (flags=3DVariable "flags" is not available.
 ) at /usr/src/sys/kern/kern_synch.c:444
 #2  0xc0577dbb in ithread_loop (arg=3D0xc5a5b9d0) at /usr/src/sys/kern/ke=
 rn_intr.c:1225
 #3  0xc0574587 in fork_exit (callout=3D0xc0577ab0 <ithread_loop>, arg=3D0=
 xc5a5b9d0, frame=3D0xc5568d38) at /usr/src/sys/kern/kern_fork.c:811
 #4  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.=
 s:271

 Thread 28 (Thread 100027):
 #0  sched_switch (td=3D0xc58da6c0, newtd=3DVariable "newtd" is not availa=
 ble.
 ) at /usr/src/sys/kern/sched_ule.c:1944
 #1  0xc05a3033 in mi_switch (flags=3DVariable "flags" is not available.
 ) at /usr/src/sys/kern/kern_synch.c:444
 #2  0xc0577dbb in ithread_loop (arg=3D0xc5a5b220) at /usr/src/sys/kern/ke=
 rn_intr.c:1225
 #3  0xc0574587 in fork_exit (callout=3D0xc0577ab0 <ithread_loop>, arg=3D0=
 xc5a5b220, frame=3D0xc5565d38) at /usr/src/sys/kern/kern_fork.c:811
 #4  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.=
 s:271

 Thread 27 (Thread 100026):
 #0  sched_switch (td=3D0xc58da900, newtd=3DVariable "newtd" is not availa=
 ble.
 ) at /usr/src/sys/kern/sched_ule.c:1944
 #1  0xc05a3033 in mi_switch (flags=3DVariable "flags" is not available.
 ) at /usr/src/sys/kern/kern_synch.c:444
 #2  0xc0577dbb in ithread_loop (arg=3D0xc5a5b110) at /usr/src/sys/kern/ke=
 rn_intr.c:1225
 #3  0xc0574587 in fork_exit (callout=3D0xc0577ab0 <ithread_loop>, arg=3D0=
 xc5a5b110, frame=3D0xc554fd38) at /usr/src/sys/kern/kern_fork.c:811
 #4  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.=
 s:271

 Thread 26 (Thread 100025):
 #0  sched_switch (td=3D0xc58dab40, newtd=3DVariable "newtd" is not availa=
 ble.
 ) at /usr/src/sys/kern/sched_ule.c:1944
 #1  0xc05a3033 in mi_switch (flags=3DVariable "flags" is not available.
 ) at /usr/src/sys/kern/kern_synch.c:444
 #2  0xc0577dbb in ithread_loop (arg=3D0xc5a5b020) at /usr/src/sys/kern/ke=
 rn_intr.c:1225
 #3  0xc0574587 in fork_exit (callout=3D0xc0577ab0 <ithread_loop>, arg=3D0=
 xc5a5b020, frame=3D0xc5513d38) at /usr/src/sys/kern/kern_fork.c:811
 #4  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.=
 s:271

 Thread 25 (Thread 100024):
 #0  fork_trampoline () at /usr/src/sys/i386/i386/exception.s:268
 Current language:  auto; currently asm

 Thread 24 (Thread 100023):
 #0  sched_switch (td=3D0xc59b2000, newtd=3DVariable "newtd" is not availa=
 ble.
 ) at /usr/src/sys/kern/sched_ule.c:1944
 #1  0xc05a3033 in mi_switch (flags=3DVariable "flags" is not available.
 ) at /usr/src/sys/kern/kern_synch.c:444
 #2  0xc0577dbb in ithread_loop (arg=3D0xc59fa980) at /usr/src/sys/kern/ke=
 rn_intr.c:1225
 ---Type <return> to continue, or q <return> to quit---
 #3  0xc0574587 in fork_exit (callout=3D0xc0577ab0 <ithread_loop>, arg=3D0=
 xc59fa980, frame=3D0xc54fad38) at /usr/src/sys/kern/kern_fork.c:811
 #4  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.=
 s:271
 Current language:  auto; currently c

 Thread 23 (Thread 100022):
 #0  fork_trampoline () at /usr/src/sys/i386/i386/exception.s:268
 Current language:  auto; currently asm

 Thread 22 (Thread 100021):
 #0  sched_switch (td=3D0xc59b2480, newtd=3DVariable "newtd" is not availa=
 ble.
 ) at /usr/src/sys/kern/sched_ule.c:1944
 #1  0xc05a3033 in mi_switch (flags=3DVariable "flags" is not available.
 ) at /usr/src/sys/kern/kern_synch.c:444
 #2  0xc05cf859 in sleepq_switch (wchan=3DVariable "wchan" is not availabl=
 e.
 ) at /usr/src/sys/kern/subr_sleepqueue.c:497
 #3  0xc05cfec6 in sleepq_wait (wchan=3D0xc588bd80) at /usr/src/sys/kern/s=
 ubr_sleepqueue.c:580
 #4  0xc05a34a5 in _sleep (ident=3D0xc588bd80, lock=3D0xc588bd9c, priority=
 =3D0, wmesg=3D0xc081c912 "-", timo=3D0) at /usr/src/sys/kern/kern_synch.c=
 :230
 #5  0xc05d1708 in taskqueue_thread_loop (arg=3D0xc08a279c) at /usr/src/sy=
 s/kern/subr_taskqueue.c:95
 #6  0xc0574587 in fork_exit (callout=3D0xc05d1600 <taskqueue_thread_loop>=
 , arg=3D0xc08a279c, frame=3D0xc54e6d38) at /usr/src/sys/kern/kern_fork.c:=
 811
 #7  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.=
 s:271
 Current language:  auto; currently c

 Thread 21 (Thread 100020):
 #0  sched_switch (td=3D0xc59b26c0, newtd=3DVariable "newtd" is not availa=
 ble.
 ) at /usr/src/sys/kern/sched_ule.c:1944
 #1  0xc05a3033 in mi_switch (flags=3DVariable "flags" is not available.
 ) at /usr/src/sys/kern/kern_synch.c:444
 #2  0xc0577dbb in ithread_loop (arg=3D0xc59a3290) at /usr/src/sys/kern/ke=
 rn_intr.c:1225
 #3  0xc0574587 in fork_exit (callout=3D0xc0577ab0 <ithread_loop>, arg=3D0=
 xc59a3290, frame=3D0xc54e3d38) at /usr/src/sys/kern/kern_fork.c:811
 #4  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.=
 s:271

 Thread 20 (Thread 100019):
 #0  sched_switch (td=3D0xc59b2900, newtd=3DVariable "newtd" is not availa=
 ble.
 ) at /usr/src/sys/kern/sched_ule.c:1944
 #1  0xc05a3033 in mi_switch (flags=3DVariable "flags" is not available.
 ) at /usr/src/sys/kern/kern_synch.c:444
 #2  0xc0577dbb in ithread_loop (arg=3D0xc59a32a0) at /usr/src/sys/kern/ke=
 rn_intr.c:1225
 #3  0xc0574587 in fork_exit (callout=3D0xc0577ab0 <ithread_loop>, arg=3D0=
 xc59a32a0, frame=3D0xc54e0d38) at /usr/src/sys/kern/kern_fork.c:811
 #4  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.=
 s:271

 Thread 19 (Thread 100018):
 #0  sched_switch (td=3D0xc59b2b40, newtd=3DVariable "newtd" is not availa=
 ble.
 ) at /usr/src/sys/kern/sched_ule.c:1944
 #1  0xc05a3033 in mi_switch (flags=3DVariable "flags" is not available.
 ) at /usr/src/sys/kern/kern_synch.c:444
 #2  0xc05cf859 in sleepq_switch (wchan=3DVariable "wchan" is not availabl=
 e.
 ) at /usr/src/sys/kern/subr_sleepqueue.c:497
 #3  0xc05cfec6 in sleepq_wait (wchan=3D0xc58ef000) at /usr/src/sys/kern/s=
 ubr_sleepqueue.c:580
 #4  0xc05a34a5 in _sleep (ident=3D0xc58ef000, lock=3D0xc58ef01c, priority=
 =3D0, wmesg=3D0xc081c912 "-", timo=3D0) at /usr/src/sys/kern/kern_synch.c=
 :230
 #5  0xc05d1708 in taskqueue_thread_loop (arg=3D0xc1159f24) at /usr/src/sy=
 s/kern/subr_taskqueue.c:95
 #6  0xc0574587 in fork_exit (callout=3D0xc05d1600 <taskqueue_thread_loop>=
 , arg=3D0xc1159f24, frame=3D0xc54ddd38) at /usr/src/sys/kern/kern_fork.c:=
 811
 #7  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.=
 s:271

 Thread 18 (Thread 100017):
 #0  sched_switch (td=3D0xc5891240, newtd=3DVariable "newtd" is not availa=
 ble.
 ) at /usr/src/sys/kern/sched_ule.c:1944
 #1  0xc05a3033 in mi_switch (flags=3DVariable "flags" is not available.
 ) at /usr/src/sys/kern/kern_synch.c:444
 #2  0xc05cf859 in sleepq_switch (wchan=3DVariable "wchan" is not availabl=
 e.
 ) at /usr/src/sys/kern/subr_sleepqueue.c:497
 #3  0xc05cfec6 in sleepq_wait (wchan=3D0xc58ef000) at /usr/src/sys/kern/s=
 ubr_sleepqueue.c:580
 #4  0xc05a34a5 in _sleep (ident=3D0xc58ef000, lock=3D0xc58ef01c, priority=
 =3D0, wmesg=3D0xc081c912 "-", timo=3D0) at /usr/src/sys/kern/kern_synch.c=
 :230
 #5  0xc05d1708 in taskqueue_thread_loop (arg=3D0xc1159f24) at /usr/src/sy=
 s/kern/subr_taskqueue.c:95
 #6  0xc0574587 in fork_exit (callout=3D0xc05d1600 <taskqueue_thread_loop>=
 , arg=3D0xc1159f24, frame=3D0xc54dad38) at /usr/src/sys/kern/kern_fork.c:=
 811
 #7  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.=
 s:271

 Thread 17 (Thread 100016):
 #0  sched_switch (td=3D0xc5891480, newtd=3DVariable "newtd" is not availa=
 ble.
 ) at /usr/src/sys/kern/sched_ule.c:1944
 #1  0xc05a3033 in mi_switch (flags=3DVariable "flags" is not available.
 ) at /usr/src/sys/kern/kern_synch.c:444
 #2  0xc05cf859 in sleepq_switch (wchan=3DVariable "wchan" is not availabl=
 e.
 ) at /usr/src/sys/kern/subr_sleepqueue.c:497
 #3  0xc05cfec6 in sleepq_wait (wchan=3D0xc58ef000) at /usr/src/sys/kern/s=
 ubr_sleepqueue.c:580
 #4  0xc05a34a5 in _sleep (ident=3D0xc58ef000, lock=3D0xc58ef01c, priority=
 =3D0, wmesg=3D0xc081c912 "-", timo=3D0) at /usr/src/sys/kern/kern_synch.c=
 :230
 #5  0xc05d1708 in taskqueue_thread_loop (arg=3D0xc1159f24) at /usr/src/sy=
 s/kern/subr_taskqueue.c:95
 #6  0xc0574587 in fork_exit (callout=3D0xc05d1600 <taskqueue_thread_loop>=
 , arg=3D0xc1159f24, frame=3D0xc54d7d38) at /usr/src/sys/kern/kern_fork.c:=
 811
 ---Type <return> to continue, or q <return> to quit---
 #7  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.=
 s:271

 Thread 16 (Thread 100015):
 #0  sched_switch (td=3D0xc58916c0, newtd=3DVariable "newtd" is not availa=
 ble.
 ) at /usr/src/sys/kern/sched_ule.c:1944
 #1  0xc05a3033 in mi_switch (flags=3DVariable "flags" is not available.
 ) at /usr/src/sys/kern/kern_synch.c:444
 #2  0xc05cf859 in sleepq_switch (wchan=3DVariable "wchan" is not availabl=
 e.
 ) at /usr/src/sys/kern/subr_sleepqueue.c:497
 #3  0xc05cfec6 in sleepq_wait (wchan=3D0xc58ef080) at /usr/src/sys/kern/s=
 ubr_sleepqueue.c:580
 #4  0xc05a34a5 in _sleep (ident=3D0xc58ef080, lock=3D0xc58ef09c, priority=
 =3D0, wmesg=3D0xc081c912 "-", timo=3D0) at /usr/src/sys/kern/kern_synch.c=
 :230
 #5  0xc05d1708 in taskqueue_thread_loop (arg=3D0xc08b061c) at /usr/src/sy=
 s/kern/subr_taskqueue.c:95
 #6  0xc0574587 in fork_exit (callout=3D0xc05d1600 <taskqueue_thread_loop>=
 , arg=3D0xc08b061c, frame=3D0xc54d4d38) at /usr/src/sys/kern/kern_fork.c:=
 811
 #7  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.=
 s:271

 Thread 15 (Thread 100014):
 #0  fork_trampoline () at /usr/src/sys/i386/i386/exception.s:268
 Current language:  auto; currently asm

 Thread 14 (Thread 100013):
 #0  sched_switch (td=3D0xc5891b40, newtd=3DVariable "newtd" is not availa=
 ble.
 ) at /usr/src/sys/kern/sched_ule.c:1944
 #1  0xc05a3033 in mi_switch (flags=3DVariable "flags" is not available.
 ) at /usr/src/sys/kern/kern_synch.c:444
 #2  0xc0577dbb in ithread_loop (arg=3D0xc59a32e0) at /usr/src/sys/kern/ke=
 rn_intr.c:1225
 #3  0xc0574587 in fork_exit (callout=3D0xc0577ab0 <ithread_loop>, arg=3D0=
 xc59a32e0, frame=3D0xc54ced38) at /usr/src/sys/kern/kern_fork.c:811
 #4  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.=
 s:271
 Current language:  auto; currently c

 Thread 13 (Thread 100012):
 #0  sched_switch (td=3D0xc5891d80, newtd=3DVariable "newtd" is not availa=
 ble.
 ) at /usr/src/sys/kern/sched_ule.c:1944
 #1  0xc05a3033 in mi_switch (flags=3DVariable "flags" is not available.
 ) at /usr/src/sys/kern/kern_synch.c:444
 #2  0xc05cf859 in sleepq_switch (wchan=3DVariable "wchan" is not availabl=
 e.
 ) at /usr/src/sys/kern/subr_sleepqueue.c:497
 #3  0xc05cfec6 in sleepq_wait (wchan=3D0xc088d214) at /usr/src/sys/kern/s=
 ubr_sleepqueue.c:580
 #4  0xc05a34a5 in _sleep (ident=3D0xc088d214, lock=3D0xc088d22c, priority=
 =3D76, wmesg=3D0xc0803242 "ccb_scanq", timo=3D0) at /usr/src/sys/kern/ker=
 n_synch.c:230
 #5  0xc0446491 in xpt_scanner_thread (dummy=3D0x0) at /usr/src/sys/cam/ca=
 m_xpt.c:1399
 #6  0xc0574587 in fork_exit (callout=3D0xc0446450 <xpt_scanner_thread>, a=
 rg=3D0x0, frame=3D0xc54cbd38) at /usr/src/sys/kern/kern_fork.c:811
 #7  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.=
 s:271

 Thread 12 (Thread 100011):
 #0  sched_switch (td=3D0xc58da000, newtd=3DVariable "newtd" is not availa=
 ble.
 ) at /usr/src/sys/kern/sched_ule.c:1944
 #1  0xc05a3033 in mi_switch (flags=3DVariable "flags" is not available.
 ) at /usr/src/sys/kern/kern_synch.c:444
 #2  0xc05cf859 in sleepq_switch (wchan=3DVariable "wchan" is not availabl=
 e.
 ) at /usr/src/sys/kern/subr_sleepqueue.c:497
 #3  0xc05d03d7 in sleepq_timedwait (wchan=3D0xc08a44b4) at /usr/src/sys/k=
 ern/subr_sleepqueue.c:615
 #4  0xc05a3483 in _sleep (ident=3D0xc08a44b4, lock=3D0x0, priority=3D0, w=
 mesg=3D0xc081c912 "-", timo=3D100) at /usr/src/sys/kern/kern_synch.c:226
 #5  0xc05a35a0 in pause (wmesg=3D0xc081c912 "-", timo=3D100) at /usr/src/=
 sys/kern/kern_synch.c:334
 #6  0xc04c3e72 in random_kthread (arg=3D0x0) at /usr/src/sys/dev/random/r=
 andomdev_soft.c:289
 #7  0xc0574587 in fork_exit (callout=3D0xc04c3c40 <random_kthread>, arg=3D=
 0x0, frame=3D0xc54c8d38) at /usr/src/sys/kern/kern_fork.c:811
 #8  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.=
 s:271

 Thread 11 (Thread 100010):
 #0  sched_switch (td=3D0xc58da240, newtd=3DVariable "newtd" is not availa=
 ble.
 ) at /usr/src/sys/kern/sched_ule.c:1944
 #1  0xc05a3033 in mi_switch (flags=3DVariable "flags" is not available.
 ) at /usr/src/sys/kern/kern_synch.c:444
 #2  0xc05cf859 in sleepq_switch (wchan=3DVariable "wchan" is not availabl=
 e.
 ) at /usr/src/sys/kern/subr_sleepqueue.c:497
 #3  0xc05d03d7 in sleepq_timedwait (wchan=3D0xc08a1f0c) at /usr/src/sys/k=
 ern/subr_sleepqueue.c:615
 #4  0xc05a3483 in _sleep (ident=3D0xc08a1f0c, lock=3D0xc08a1e28, priority=
 =3D588, wmesg=3D0xc081c912 "-", timo=3D100) at /usr/src/sys/kern/kern_syn=
 ch.c:226
 #5  0xc053f22b in g_io_schedule_down (tp=3D0xc58da240) at /usr/src/sys/ge=
 om/geom_io.c:487
 #6  0xc053f94c in g_down_procbody () at /usr/src/sys/geom/geom_kern.c:118=

 #7  0xc0574587 in fork_exit (callout=3D0xc053f8e0 <g_down_procbody>, arg=3D=
 0x0, frame=3D0xc54c5d38) at /usr/src/sys/kern/kern_fork.c:811
 #8  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.=
 s:271

 Thread 10 (Thread 100009):
 #0  sched_switch (td=3D0xc58da480, newtd=3DVariable "newtd" is not availa=
 ble.
 ) at /usr/src/sys/kern/sched_ule.c:1944
 #1  0xc05a3033 in mi_switch (flags=3DVariable "flags" is not available.
 ) at /usr/src/sys/kern/kern_synch.c:444
 #2  0xc05cf859 in sleepq_switch (wchan=3DVariable "wchan" is not availabl=
 e.
 ) at /usr/src/sys/kern/subr_sleepqueue.c:497
 ---Type <return> to continue, or q <return> to quit---
 #3  0xc05d03d7 in sleepq_timedwait (wchan=3D0xc08a1f08) at /usr/src/sys/k=
 ern/subr_sleepqueue.c:615
 #4  0xc05a3483 in _sleep (ident=3D0xc08a1f08, lock=3D0xc08a1e68, priority=
 =3D588, wmesg=3D0xc081c912 "-", timo=3D100) at /usr/src/sys/kern/kern_syn=
 ch.c:226
 #5  0xc053f532 in g_io_schedule_up (tp=3D0xc58da480) at /usr/src/sys/geom=
 /geom_io.c:592
 #6  0xc053f82c in g_up_procbody () at /usr/src/sys/geom/geom_kern.c:95
 #7  0xc0574587 in fork_exit (callout=3D0xc053f7c0 <g_up_procbody>, arg=3D=
 0x0, frame=3D0xc54c2d38) at /usr/src/sys/kern/kern_fork.c:811
 #8  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.=
 s:271

 Thread 9 (Thread 100008):
 #0  sched_switch (td=3D0xc588f000, newtd=3DVariable "newtd" is not availa=
 ble.
 ) at /usr/src/sys/kern/sched_ule.c:1944
 #1  0xc05a3033 in mi_switch (flags=3DVariable "flags" is not available.
 ) at /usr/src/sys/kern/kern_synch.c:444
 #2  0xc05cf859 in sleepq_switch (wchan=3DVariable "wchan" is not availabl=
 e.
 ) at /usr/src/sys/kern/subr_sleepqueue.c:497
 #3  0xc05d03d7 in sleepq_timedwait (wchan=3D0xc08a1f00) at /usr/src/sys/k=
 ern/subr_sleepqueue.c:615
 #4  0xc05a3483 in _sleep (ident=3D0xc08a1f00, lock=3D0x0, priority=3D76, =
 wmesg=3D0xc081c912 "-", timo=3D100) at /usr/src/sys/kern/kern_synch.c:226=

 #5  0xc053f8da in g_event_procbody () at /usr/src/sys/geom/geom_kern.c:14=
 2
 #6  0xc0574587 in fork_exit (callout=3D0xc053f830 <g_event_procbody>, arg=
 =3D0x0, frame=3D0xc54bfd38) at /usr/src/sys/kern/kern_fork.c:811
 #7  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.=
 s:271

 Thread 8 (Thread 100007):
 #0  fork_trampoline () at /usr/src/sys/i386/i386/exception.s:268
 Current language:  auto; currently asm

 Thread 7 (Thread 100006):
 #0  sched_switch (td=3D0xc588f480, newtd=3DVariable "newtd" is not availa=
 ble.
 ) at /usr/src/sys/kern/sched_ule.c:1944
 #1  0xc05a3033 in mi_switch (flags=3DVariable "flags" is not available.
 ) at /usr/src/sys/kern/kern_synch.c:444
 #2  0xc07d4682 in ipi_bitmap_handler (frame=3D
       {tf_fs =3D 8, tf_es =3D 40, tf_ds =3D 40, tf_edi =3D -644953128, tf=
 _esi =3D -980880256, tf_ebp =3D -984900488, tf_isp =3D -984900532, tf_ebx=
  =3D -1064681904, tf_edx =3D -980877952, tf_ecx =3D -980877952, tf_eax =3D=
  4, tf_trapno =3D 0, tf_err =3D 0, tf_eip =3D -1067925921, tf_cs =3D 32, =
 tf_eflags =3D 582, tf_esp =3D -1066191232, tf_ss =3D 0}) at /usr/src/sys/=
 i386/i386/mp_machdep.c:1187
 #3  0xc07c6dae in Xipi_intr_bitmap_handler () at apic_vector.s:284
 #4  0xc058be5f in _mtx_lock_sleep (m=3D0xc08a3e50, tid=3D3314087040, opts=
 =3D0, file=3D0x0, line=3D0) at /usr/src/sys/kern/kern_mutex.c:349
 #5  0xc05ad91b in softclock (dummy=3D0x0) at /usr/src/sys/kern/kern_timeo=
 ut.c:241
 #6  0xc0577c5b in ithread_loop (arg=3D0xc588c390) at /usr/src/sys/kern/ke=
 rn_intr.c:1126
 #7  0xc0574587 in fork_exit (callout=3D0xc0577ab0 <ithread_loop>, arg=3D0=
 xc588c390, frame=3D0xc54b9d38) at /usr/src/sys/kern/kern_fork.c:811
 #8  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.=
 s:271
 Current language:  auto; currently c

 Thread 6 (Thread 100005):
 #0  sched_switch (td=3D0xc588f6c0, newtd=3DVariable "newtd" is not availa=
 ble.
 ) at /usr/src/sys/kern/sched_ule.c:1944
 #1  0xc05a3033 in mi_switch (flags=3DVariable "flags" is not available.
 ) at /usr/src/sys/kern/kern_synch.c:444
 #2  0xc0577dbb in ithread_loop (arg=3D0xc588c3a0) at /usr/src/sys/kern/ke=
 rn_intr.c:1225
 #3  0xc0574587 in fork_exit (callout=3D0xc0577ab0 <ithread_loop>, arg=3D0=
 xc588c3a0, frame=3D0xc54b6d38) at /usr/src/sys/kern/kern_fork.c:811
 #4  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.=
 s:271

 Thread 5 (Thread 100004):
 #0  sched_switch (td=3D0xc588f900, newtd=3DVariable "newtd" is not availa=
 ble.
 ) at /usr/src/sys/kern/sched_ule.c:1944
 #1  0xc05a3033 in mi_switch (flags=3DVariable "flags" is not available.
 ) at /usr/src/sys/kern/kern_synch.c:444
 #2  0xc05bcedd in critical_exit () at kern_switch.c:185
 #3  0xc07ce2f5 in lapic_handle_timer (frame=3D0xc54b1c48) at /usr/src/sys=
 /i386/i386/local_apic.c:724
 #4  0xc07c6c9f in Xtimerint () at apic_vector.s:108
 #5  0xc1149b15 in acpi_cpu_c1 () at /usr/src/sys/modules/acpi/acpi/../../=
 ../i386/acpica/acpi_machdep.c:550
 #6  0xc1142a1d in acpi_cpu_idle () at /usr/src/sys/modules/acpi/acpi/../.=
 ./../dev/acpica/acpi_cpu.c:912
 #7  0xc07d0a58 in cpu_idle () at /usr/src/sys/i386/i386/machdep.c:1183
 #8  0xc05bdd6b in sched_idletd (dummy=3D0x0) at /usr/src/sys/kern/sched_u=
 le.c:2681
 #9  0xc0574587 in fork_exit (callout=3D0xc05bdaa0 <sched_idletd>, arg=3D0=
 x0, frame=3D0xc54b1d38) at /usr/src/sys/kern/kern_fork.c:811
 #10 0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.=
 s:271

 Thread 4 (Thread 100003):
 #0  sched_switch (td=3D0xc588fb40, newtd=3DVariable "newtd" is not availa=
 ble.
 ) at /usr/src/sys/kern/sched_ule.c:1944
 #1  0xc05a3033 in mi_switch (flags=3DVariable "flags" is not available.
 ) at /usr/src/sys/kern/kern_synch.c:444
 #2  0xc07d4682 in ipi_bitmap_handler (frame=3D
       {tf_fs =3D 8, tf_es =3D 40, tf_ds =3D 40, tf_edi =3D 1, tf_esi =3D =
 15, tf_ebp =3D -984945472, tf_isp =3D -984945496, tf_ebx =3D -980878528, =
 tf_edx =3D 256, tf_ecx =3D 16, tf_eax =3D 582, tf_trapno =3D 0, tf_err =3D=
  0, tf_e---Type <return> to continue, or q <return> to quit---
 ip =3D -1065551975, tf_cs =3D 32, tf_eflags =3D 582, tf_esp =3D -10646632=
 96, tf_ss =3D -984945420}) at /usr/src/sys/i386/i386/mp_machdep.c:1187
 #3  0xc07c6dae in Xipi_intr_bitmap_handler () at apic_vector.s:284
 #4  0xc07cf799 in spinlock_exit () at cpufunc.h:365
 #5  0xc05bdcd6 in sched_idletd (dummy=3D0x0) at /usr/src/sys/kern/sched_u=
 le.c:806
 #6  0xc0574587 in fork_exit (callout=3D0xc05bdaa0 <sched_idletd>, arg=3D0=
 x0, frame=3D0xc54aed38) at /usr/src/sys/kern/kern_fork.c:811
 #7  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.=
 s:271

 Thread 3 (Thread 100002):
 #0  doadump () at pcpu.h:196
 #1  0xc059b097 in boot (howto=3D260) at /usr/src/sys/kern/kern_shutdown.c=
 :418
 #2  0xc059b36c in panic (fmt=3DVariable "fmt" is not available.
 ) at /usr/src/sys/kern/kern_shutdown.c:574
 #3  0xc058bd1f in _mtx_lock_spin_failed (m=3D0x0) at /usr/src/sys/kern/ke=
 rn_mutex.c:449
 #4  0xc058c425 in _thread_lock_flags (td=3D0xc588f480, opts=3D0, file=3D0=
 xc08200c8 "/usr/src/sys/kern/kern_intr.c", line=3D801) at /usr/src/sys/ke=
 rn/kern_mutex.c:537
 #5  0xc0576787 in intr_event_schedule_thread (ie=3DVariable "ie" is not a=
 vailable.
 ) at /usr/src/sys/kern/kern_intr.c:801
 #6  0xc0577aa5 in swi_sched (cookie=3D0xc5880e00, flags=3DVariable "flags=
 " is not available.
 ) at /usr/src/sys/kern/kern_intr.c:1028
 #7  0xc055a6df in hardclock (usermode=3D0, pc=3D3229503748) at /usr/src/s=
 ys/kern/kern_clock.c:339
 #8  0xc07ce23a in lapic_handle_timer (frame=3D0xc54aaba0) at /usr/src/sys=
 /i386/i386/local_apic.c:705
 #9  0xc07c6c9f in Xtimerint () at apic_vector.s:108
 #10 0xc07e5104 in DELAY (n=3D1000000) at cpufunc.h:352
 #11 0xc07e2c03 in cpu_reset () at /usr/src/sys/i386/i386/vm_machdep.c:594=

 #12 0xc059a753 in shutdown_reset (junk=3D0x0, howto=3D0) at /usr/src/sys/=
 kern/kern_shutdown.c:493
 #13 0xc059b117 in boot (howto=3D0) at /usr/src/sys/kern/kern_shutdown.c:4=
 21
 #14 0xc059b247 in reboot (td=3D0xc588fd80, uap=3D0xc54aacfc) at /usr/src/=
 sys/kern/kern_shutdown.c:172
 #15 0xc07e07d5 in syscall (frame=3D0xc54aad38) at /usr/src/sys/i386/i386/=
 trap.c:1089
 #16 0xc07c6720 in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception=
 .s:262
 #17 0x00000033 in ?? ()
 Previous frame inner to this frame (corrupt stack?)

 Thread 2 (Thread 100001):
 #0  sched_switch (td=3D0xc5891000, newtd=3DVariable "newtd" is not availa=
 ble.
 ) at /usr/src/sys/kern/sched_ule.c:1944
 #1  0xc05a3033 in mi_switch (flags=3DVariable "flags" is not available.
 ) at /usr/src/sys/kern/kern_synch.c:444
 #2  0xc05cf859 in sleepq_switch (wchan=3DVariable "wchan" is not availabl=
 e.
 ) at /usr/src/sys/kern/subr_sleepqueue.c:497
 #3  0xc05cfec6 in sleepq_wait (wchan=3D0xc08bcd48) at /usr/src/sys/kern/s=
 ubr_sleepqueue.c:580
 #4  0xc055aaaf in _cv_wait (cvp=3D0xc08bcd48, lock=3D0xc08bcd28) at /usr/=
 src/sys/kern/kern_condvar.c:137
 #5  0xc0749e2f in audit_worker (arg=3D0x0) at /usr/src/sys/security/audit=
 /audit_worker.c:410
 #6  0xc0574587 in fork_exit (callout=3D0xc0749dc0 <audit_worker>, arg=3D0=
 x0, frame=3D0xc54a7d38) at /usr/src/sys/kern/kern_fork.c:811
 #7  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.=
 s:271

 Thread 1 (Thread 100000):
 #0  sched_switch (td=3D0xc08a2280, newtd=3DVariable "newtd" is not availa=
 ble.
 ) at /usr/src/sys/kern/sched_ule.c:1944
 #1  0xc05a3033 in mi_switch (flags=3DVariable "flags" is not available.
 ) at /usr/src/sys/kern/kern_synch.c:444
 #2  0xc05cf859 in sleepq_switch (wchan=3DVariable "wchan" is not availabl=
 e.
 ) at /usr/src/sys/kern/subr_sleepqueue.c:497
 #3  0xc05d03d7 in sleepq_timedwait (wchan=3D0xc08a1fc0) at /usr/src/sys/k=
 ern/subr_sleepqueue.c:615
 #4  0xc05a3483 in _sleep (ident=3D0xc08a1fc0, lock=3D0x0, priority=3D68, =
 wmesg=3D0xc0823aef "sched", timo=3D10000) at /usr/src/sys/kern/kern_synch=
 .c:226
 #5  0xc078df39 in scheduler (dummy=3D0x0) at /usr/src/sys/vm/vm_glue.c:73=
 3
 #6  0xc05577b6 in mi_startup () at /usr/src/sys/kern/init_main.c:251
 #7  0xc0440685 in begin () at /usr/src/sys/i386/i386/locore.s:328
 196		__asm __volatile("movl %%fs:0,%0" : "=3Dr" (td));
 (kgdb) quit