kern/134584: [panic] spin lock held too long

barbara barbara.xxx1975 at libero.it
Sun Jul 26 06:46:58 UTC 2009


It happened again, on shutdown.
As the previous time, it happened after a high (for a desktop) uptime and, if it could matter, after running net-p2p/transmission-gtk2 for several hours.
I don't know if it's related, but often quitting transmission, doesn't terminate the process. Sometimes it end after several minutes the gui exited, sometimes it's still running after hours.
I've noticed it as the destination folder is on a manually mounted device and I can't umount it as fstat reports the device used by a transmission process.
So I often have to kill it.
This happened both the time I had this kind of panic.

Barbara


Script started on Sun Jul 26 08:22:33 2009
You have mail.
satanasso# kgdb kernel.debug /var/crash/vmcore.17
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-marcel-freebsd"...

Unread portion of the kernel message buffer:
<118>Stopping devd.
<118>Writing entropy file:
<118>.
<118>Terminated
<118>.
<118>Jul 25 17:45:18 satanasso syslogd: exiting on signal 15
Waiting (max 60 seconds) for system process `vnlru' to stop...done
Waiting (max 60 seconds) for system process `bufdaemon' to stop...done
Waiting (max 60 seconds) for system process `syncer' to stop...
Syncing disks, vnodes remaining...7 6 5 3 1 2 2 0 0 done
All buffers synced.
Uptime: 22h41m12s
Rebooting...
cpu_reset: Stopping other CPUs
spin lock 0xc08ae540 (sched lock 1) held by 0xc588f480 (tid 100006) too long
panic: spin lock held too long
cpuid = 0
Uptime: 22h41m14s
Physical memory: 2031 MB
Dumping 171 MB: (CTRL-C to abort)  156 140 124 108 92 76 60 44 28 12

Reading symbols from /boot/kernel/linux.ko...Reading symbols from /boot/kernel/linux.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/linux.ko
Reading symbols from /boot/modules/nvidia.ko...done.
Loaded symbols for /boot/modules/nvidia.ko
Reading symbols from /boot/kernel/acpi.ko...Reading symbols from /boot/kernel/acpi.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/acpi.ko
Reading symbols from /boot/kernel/linprocfs.ko...Reading symbols from /boot/kernel/linprocfs.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/linprocfs.ko
Reading symbols from /boot/kernel/logo_saver.ko...Reading symbols from /boot/kernel/logo_saver.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/logo_saver.ko
Reading symbols from /boot/kernel/ntfs.ko...Reading symbols from /boot/kernel/ntfs.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/ntfs.ko
#0  doadump () at pcpu.h:196
196		__asm __volatile("movl %%fs:0,%0" : "=r" (td));
(kgdb) bt
#0  doadump () at pcpu.h:196
#1  0xc059b097 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:418
#2  0xc059b36c in panic (fmt=Variable "fmt" is not available.
) at /usr/src/sys/kern/kern_shutdown.c:574
#3  0xc058bd1f in _mtx_lock_spin_failed (m=0x0) at /usr/src/sys/kern/kern_mutex.c:449
#4  0xc058c425 in _thread_lock_flags (td=0xc588f480, opts=0, file=0xc08200c8 "/usr/src/sys/kern/kern_intr.c", line=801) at /usr/src/sys/kern/kern_mutex.c:537
#5  0xc0576787 in intr_event_schedule_thread (ie=Variable "ie" is not available.
) at /usr/src/sys/kern/kern_intr.c:801
#6  0xc0577aa5 in swi_sched (cookie=0xc5880e00, flags=Variable "flags" is not available.
) at /usr/src/sys/kern/kern_intr.c:1028
#7  0xc055a6df in hardclock (usermode=0, pc=3229503748) at /usr/src/sys/kern/kern_clock.c:339
#8  0xc07ce23a in lapic_handle_timer (frame=0xc54aaba0) at /usr/src/sys/i386/i386/local_apic.c:705
#9  0xc07c6c9f in Xtimerint () at apic_vector.s:108
#10 0xc07e5104 in DELAY (n=1000000) at cpufunc.h:352
#11 0xc07e2c03 in cpu_reset () at /usr/src/sys/i386/i386/vm_machdep.c:594
#12 0xc059a753 in shutdown_reset (junk=0x0, howto=0) at /usr/src/sys/kern/kern_shutdown.c:493
#13 0xc059b117 in boot (howto=0) at /usr/src/sys/kern/kern_shutdown.c:421
#14 0xc059b247 in reboot (td=0xc588fd80, uap=0xc54aacfc) at /usr/src/sys/kern/kern_shutdown.c:172
#15 0xc07e07d5 in syscall (frame=0xc54aad38) at /usr/src/sys/i386/i386/trap.c:1089
#16 0xc07c6720 in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:262
#17 0x00000033 in ?? ()
Previous frame inner to this frame (corrupt stack?)
(kgdb) thread apply all bt

Thread 52 (Thread 100051):
#0  sched_switch (td=0xc5c91d80, newtd=Variable "newtd" is not available.
) at /usr/src/sys/kern/sched_ule.c:1944
#1  0xc05a3033 in mi_switch (flags=Variable "flags" is not available.
) at /usr/src/sys/kern/kern_synch.c:444
#2  0xc05cf859 in sleepq_switch (wchan=Variable "wchan" is not available.
) at /usr/src/sys/kern/subr_sleepqueue.c:497
#3  0xc05d03d7 in sleepq_timedwait (wchan=0xc08bd300) at /usr/src/sys/kern/subr_sleepqueue.c:615
#4  0xc05a3483 in _sleep (ident=0xc08bd300, lock=0xc08bd2c0, priority=68, wmesg=0xc0833855 "sdflush", timo=1000) at /usr/src/sys/kern/kern_synch.c:226
#5  0xc076407e in softdep_flush () at /usr/src/sys/ufs/ffs/ffs_softdep.c:773
#6  0xc0574587 in fork_exit (callout=0xc0763cb0 <softdep_flush>, arg=0x0, frame=0xc5617d38) at /usr/src/sys/kern/kern_fork.c:811
#7  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:271

Thread 51 (Thread 100050):
#0  sched_switch (td=0xc5a616c0, newtd=Variable "newtd" is not available.
) at /usr/src/sys/kern/sched_ule.c:1944
#1  0xc05a3033 in mi_switch (flags=Variable "flags" is not available.
) at /usr/src/sys/kern/kern_synch.c:444
#2  0xc05cf859 in sleepq_switch (wchan=Variable "wchan" is not available.
) at /usr/src/sys/kern/subr_sleepqueue.c:497
#3  0xc05cfec6 in sleepq_wait (wchan=0xc5abe36c) at /usr/src/sys/kern/subr_sleepqueue.c:580
#4  0xc05a34a5 in _sleep (ident=0xc5abe36c, lock=0xc5abe350, priority=104, wmesg=0xc0820618 "ktsusp", timo=0) at /usr/src/sys/kern/kern_synch.c:230
#5  0xc057e632 in kthread_suspend_check (p=0xc5abe2c0) at /usr/src/sys/kern/kern_kthread.c:197
#6  0xc0620b1e in vnlru_proc () at /usr/src/sys/kern/vfs_subr.c:724
#7  0xc0574587 in fork_exit (callout=0xc0620ad0 <vnlru_proc>, arg=0x0, frame=0xc5614d38) at /usr/src/sys/kern/kern_fork.c:811
#8  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:271

Thread 50 (Thread 100049):
#0  sched_switch (td=0xc5a61900, newtd=Variable "newtd" is not available.
) at /usr/src/sys/kern/sched_ule.c:1944
#1  0xc05a3033 in mi_switch (flags=Variable "flags" is not available.
) at /usr/src/sys/kern/kern_synch.c:444
#2  0xc05cf859 in sleepq_switch (wchan=Variable "wchan" is not available.
) at /usr/src/sys/kern/subr_sleepqueue.c:497
#3  0xc05cfec6 in sleepq_wait (wchan=0xc5abe62c) at /usr/src/sys/kern/subr_sleepqueue.c:580
#4  0xc05a34a5 in _sleep (ident=0xc5abe62c, lock=0xc5abe610, priority=104, wmesg=0xc0820618 "ktsusp", timo=0) at /usr/src/sys/kern/kern_synch.c:230
#5  0xc057e632 in kthread_suspend_check (p=0xc5abe580) at /usr/src/sys/kern/kern_kthread.c:197
#6  0xc061f9c2 in sched_sync () at /usr/src/sys/kern/vfs_subr.c:1719
#7  0xc0574587 in fork_exit (callout=0xc061f8a0 <sched_sync>, arg=0x0, frame=0xc5611d38) at /usr/src/sys/kern/kern_fork.c:811
#8  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:271

Thread 49 (Thread 100048):
#0  sched_switch (td=0xc5a61b40, newtd=Variable "newtd" is not available.
) at /usr/src/sys/kern/sched_ule.c:1944
#1  0xc05a3033 in mi_switch (flags=Variable "flags" is not available.
) at /usr/src/sys/kern/kern_synch.c:444
#2  0xc05cf859 in sleepq_switch (wchan=Variable "wchan" is not available.
) at /usr/src/sys/kern/subr_sleepqueue.c:497
#3  0xc05cfec6 in sleepq_wait (wchan=0xc5abe8ec) at /usr/src/sys/kern/subr_sleepqueue.c:580
#4  0xc05a34a5 in _sleep (ident=0xc5abe8ec, lock=0xc5abe8d0, priority=104, wmesg=0xc0820618 "ktsusp", timo=0) at /usr/src/sys/kern/kern_synch.c:230
#5  0xc057e632 in kthread_suspend_check (p=0xc5abe840) at /usr/src/sys/kern/kern_kthread.c:197
#6  0xc0609c4b in buf_daemon () at /usr/src/sys/kern/vfs_bio.c:2155
#7  0xc0574587 in fork_exit (callout=0xc0609b70 <buf_daemon>, arg=0x0, frame=0xc560ed38) at /usr/src/sys/kern/kern_fork.c:811
#8  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:271

Thread 48 (Thread 100047):
#0  sched_switch (td=0xc5a61d80, newtd=Variable "newtd" is not available.
) at /usr/src/sys/kern/sched_ule.c:1944
#1  0xc05a3033 in mi_switch (flags=Variable "flags" is not available.
) at /usr/src/sys/kern/kern_synch.c:444
#2  0xc05cf859 in sleepq_switch (wchan=Variable "wchan" is not available.
) at /usr/src/sys/kern/subr_sleepqueue.c:497
#3  0xc05d03d7 in sleepq_timedwait (wchan=0xc08bdf14) at /usr/src/sys/kern/subr_sleepqueue.c:615
#4  0xc05a3483 in _sleep (ident=0xc08bdf14, lock=0xc08bdad0, priority=0, wmesg=0xc0836ae9 "pgzero", timo=300000) at /usr/src/sys/kern/kern_synch.c:226
#5  0xc07a155c in vm_pagezero (arg=0x0) at /usr/src/sys/vm/vm_zeroidle.c:136
#6  0xc0574587 in fork_exit (callout=0xc07a1480 <vm_pagezero>, arg=0x0, frame=0xc560bd38) at /usr/src/sys/kern/kern_fork.c:811
#7  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:271

Thread 47 (Thread 100046):
#0  sched_switch (td=0xc5aa1000, newtd=Variable "newtd" is not available.
) at /usr/src/sys/kern/sched_ule.c:1944
#1  0xc05a3033 in mi_switch (flags=Variable "flags" is not available.
) at /usr/src/sys/kern/kern_synch.c:444
#2  0xc05cf859 in sleepq_switch (wchan=Variable "wchan" is not available.
) at /usr/src/sys/kern/subr_sleepqueue.c:497
---Type <return> to continue, or q <return> to quit---
#3  0xc05cfec6 in sleepq_wait (wchan=0xc08bdb44) at /usr/src/sys/kern/subr_sleepqueue.c:580
#4  0xc05a34a5 in _sleep (ident=0xc08bdb44, lock=0xc08bdb48, priority=104, wmesg=0xc0826d0c "psleep", timo=0) at /usr/src/sys/kern/kern_synch.c:230
#5  0xc079c4e8 in vm_daemon () at /usr/src/sys/vm/vm_pageout.c:1535
#6  0xc0574587 in fork_exit (callout=0xc079c470 <vm_daemon>, arg=0x0, frame=0xc5608d38) at /usr/src/sys/kern/kern_fork.c:811
#7  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:271

Thread 46 (Thread 100045):
#0  sched_switch (td=0xc5aa1240, newtd=Variable "newtd" is not available.
) at /usr/src/sys/kern/sched_ule.c:1944
#1  0xc05a3033 in mi_switch (flags=Variable "flags" is not available.
) at /usr/src/sys/kern/kern_synch.c:444
#2  0xc05cf859 in sleepq_switch (wchan=Variable "wchan" is not available.
) at /usr/src/sys/kern/subr_sleepqueue.c:497
#3  0xc05d03d7 in sleepq_timedwait (wchan=0xc08bdb0c) at /usr/src/sys/kern/subr_sleepqueue.c:615
#4  0xc05a3483 in _sleep (ident=0xc08bdb0c, lock=0xc08bdad0, priority=68, wmesg=0xc0826d0c "psleep", timo=5000) at /usr/src/sys/kern/kern_synch.c:226
#5  0xc079ce6b in vm_pageout () at /usr/src/sys/vm/vm_pageout.c:1475
#6  0xc0574587 in fork_exit (callout=0xc079cb80 <vm_pageout>, arg=0x0, frame=0xc5605d38) at /usr/src/sys/kern/kern_fork.c:811
#7  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:271

Thread 45 (Thread 100044):
#0  sched_switch (td=0xc5aa1480, newtd=Variable "newtd" is not available.
) at /usr/src/sys/kern/sched_ule.c:1944
#1  0xc05a3033 in mi_switch (flags=Variable "flags" is not available.
) at /usr/src/sys/kern/kern_synch.c:444
#2  0xc05cf859 in sleepq_switch (wchan=Variable "wchan" is not available.
) at /usr/src/sys/kern/subr_sleepqueue.c:497
#3  0xc05cfec6 in sleepq_wait (wchan=0xc08b376c) at /usr/src/sys/kern/subr_sleepqueue.c:580
#4  0xc05a34a5 in _sleep (ident=0xc08b376c, lock=0xc08b3674, priority=0, wmesg=0xc082a92a "waiting_for_work", timo=0) at /usr/src/sys/kern/kern_synch.c:230
#5  0xc06693df in sctp_iterator_thread (v=0x0) at /usr/src/sys/netinet/sctp_bsd_addr.c:95
#6  0xc0574587 in fork_exit (callout=0xc0669360 <sctp_iterator_thread>, arg=0x0, frame=0xc55d2d38) at /usr/src/sys/kern/kern_fork.c:811
#7  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:271

Thread 44 (Thread 100043):
#0  fork_trampoline () at /usr/src/sys/i386/i386/exception.s:268
Current language:  auto; currently asm

Thread 43 (Thread 100042):
#0  sched_switch (td=0xc5aa1900, newtd=Variable "newtd" is not available.
) at /usr/src/sys/kern/sched_ule.c:1944
#1  0xc05a3033 in mi_switch (flags=Variable "flags" is not available.
) at /usr/src/sys/kern/kern_synch.c:444
#2  0xc0577dbb in ithread_loop (arg=0xc5ab2c20) at /usr/src/sys/kern/kern_intr.c:1225
#3  0xc0574587 in fork_exit (callout=0xc0577ab0 <ithread_loop>, arg=0xc5ab2c20, frame=0xc55c9d38) at /usr/src/sys/kern/kern_fork.c:811
#4  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:271
Current language:  auto; currently c

Thread 42 (Thread 100041):
#0  fork_trampoline () at /usr/src/sys/i386/i386/exception.s:268
Current language:  auto; currently asm

Thread 41 (Thread 100040):
#0  sched_switch (td=0xc5aa1d80, newtd=Variable "newtd" is not available.
) at /usr/src/sys/kern/sched_ule.c:1944
#1  0xc05a3033 in mi_switch (flags=Variable "flags" is not available.
) at /usr/src/sys/kern/kern_synch.c:444
#2  0xc05cf859 in sleepq_switch (wchan=Variable "wchan" is not available.
) at /usr/src/sys/kern/subr_sleepqueue.c:497
#3  0xc05d03d7 in sleepq_timedwait (wchan=0xc5a83c3c) at /usr/src/sys/kern/subr_sleepqueue.c:615
#4  0xc05a3483 in _sleep (ident=0xc5a83c3c, lock=0xc5a83cf0, priority=76, wmesg=0xc081c912 "-", timo=1000) at /usr/src/sys/kern/kern_synch.c:226
#5  0xc07b319d in fdc_thread (arg=dwarf2_read_address: Corrupted DWARF expression.
) at /usr/src/sys/dev/fdc/fdc.c:812
#6  0xc0574587 in fork_exit (callout=0xc07b2db0 <fdc_thread>, arg=0xc5a83c00, frame=0xc55c3d38) at /usr/src/sys/kern/kern_fork.c:811
#7  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:271
Current language:  auto; currently c

Thread 40 (Thread 100039):
#0  sched_switch (td=0xc5aa2000, newtd=Variable "newtd" is not available.
) at /usr/src/sys/kern/sched_ule.c:1944
#1  0xc05a3033 in mi_switch (flags=Variable "flags" is not available.
) at /usr/src/sys/kern/kern_synch.c:444
#2  0xc0577dbb in ithread_loop (arg=0xc5a8daa0) at /usr/src/sys/kern/kern_intr.c:1225
#3  0xc0574587 in fork_exit (callout=0xc0577ab0 <ithread_loop>, arg=0xc5a8daa0, frame=0xc55b6d38) at /usr/src/sys/kern/kern_fork.c:811
#4  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:271

Thread 39 (Thread 100038):
#0  sched_switch (td=0xc59b2d80, newtd=Variable "newtd" is not available.
) at /usr/src/sys/kern/sched_ule.c:1944
---Type <return> to continue, or q <return> to quit---
#1  0xc05a3033 in mi_switch (flags=Variable "flags" is not available.
) at /usr/src/sys/kern/kern_synch.c:444
#2  0xc0577dbb in ithread_loop (arg=0xc5a8d1e0) at /usr/src/sys/kern/kern_intr.c:1225
#3  0xc0574587 in fork_exit (callout=0xc0577ab0 <ithread_loop>, arg=0xc5a8d1e0, frame=0xc559ed38) at /usr/src/sys/kern/kern_fork.c:811
#4  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:271

Thread 38 (Thread 100037):
#0  sched_switch (td=0xc5a60000, newtd=Variable "newtd" is not available.
) at /usr/src/sys/kern/sched_ule.c:1944
#1  0xc05a3033 in mi_switch (flags=Variable "flags" is not available.
) at /usr/src/sys/kern/kern_synch.c:444
#2  0xc05cf859 in sleepq_switch (wchan=Variable "wchan" is not available.
) at /usr/src/sys/kern/subr_sleepqueue.c:497
#3  0xc05d03d7 in sleepq_timedwait (wchan=0xc5999210) at /usr/src/sys/kern/subr_sleepqueue.c:615
#4  0xc05a3483 in _sleep (ident=0xc5999210, lock=0x0, priority=92, wmesg=0xc081bb4e "usbevt", timo=60000) at /usr/src/sys/kern/kern_synch.c:226
#5  0xc051cad5 in usb_event_thread (arg=0xc5a8f3c0) at /usr/src/sys/dev/usb/usb.c:445
#6  0xc0574587 in fork_exit (callout=0xc051c9e0 <usb_event_thread>, arg=0xc5a8f3c0, frame=0xc5597d38) at /usr/src/sys/kern/kern_fork.c:811
#7  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:271

Thread 37 (Thread 100036):
#0  sched_switch (td=0xc5a60240, newtd=Variable "newtd" is not available.
) at /usr/src/sys/kern/sched_ule.c:1944
#1  0xc05a3033 in mi_switch (flags=Variable "flags" is not available.
) at /usr/src/sys/kern/kern_synch.c:444
#2  0xc05cf859 in sleepq_switch (wchan=Variable "wchan" is not available.
) at /usr/src/sys/kern/subr_sleepqueue.c:497
#3  0xc05d03d7 in sleepq_timedwait (wchan=0xc5a7e210) at /usr/src/sys/kern/subr_sleepqueue.c:615
#4  0xc05a3483 in _sleep (ident=0xc5a7e210, lock=0x0, priority=92, wmesg=0xc081bb4e "usbevt", timo=60000) at /usr/src/sys/kern/kern_synch.c:226
#5  0xc051cad5 in usb_event_thread (arg=0xc5a57900) at /usr/src/sys/dev/usb/usb.c:445
#6  0xc0574587 in fork_exit (callout=0xc051c9e0 <usb_event_thread>, arg=0xc5a57900, frame=0xc5584d38) at /usr/src/sys/kern/kern_fork.c:811
#7  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:271

Thread 36 (Thread 100035):
#0  fork_trampoline () at /usr/src/sys/i386/i386/exception.s:268
Current language:  auto; currently asm

Thread 35 (Thread 100034):
#0  sched_switch (td=0xc5a606c0, newtd=Variable "newtd" is not available.
) at /usr/src/sys/kern/sched_ule.c:1944
#1  0xc05a3033 in mi_switch (flags=Variable "flags" is not available.
) at /usr/src/sys/kern/kern_synch.c:444
#2  0xc05cf859 in sleepq_switch (wchan=Variable "wchan" is not available.
) at /usr/src/sys/kern/subr_sleepqueue.c:497
#3  0xc05d03d7 in sleepq_timedwait (wchan=0xc5a65210) at /usr/src/sys/kern/subr_sleepqueue.c:615
#4  0xc05a3483 in _sleep (ident=0xc5a65210, lock=0x0, priority=92, wmesg=0xc081bb4e "usbevt", timo=60000) at /usr/src/sys/kern/kern_synch.c:226
#5  0xc051cad5 in usb_event_thread (arg=0xc5a62240) at /usr/src/sys/dev/usb/usb.c:445
#6  0xc0574587 in fork_exit (callout=0xc051c9e0 <usb_event_thread>, arg=0xc5a62240, frame=0xc557dd38) at /usr/src/sys/kern/kern_fork.c:811
#7  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:271
Current language:  auto; currently c

Thread 34 (Thread 100033):
#0  sched_switch (td=0xc5a60900, newtd=Variable "newtd" is not available.
) at /usr/src/sys/kern/sched_ule.c:1944
#1  0xc05a3033 in mi_switch (flags=Variable "flags" is not available.
) at /usr/src/sys/kern/kern_synch.c:444
#2  0xc05cf859 in sleepq_switch (wchan=Variable "wchan" is not available.
) at /usr/src/sys/kern/subr_sleepqueue.c:497
#3  0xc05d03d7 in sleepq_timedwait (wchan=0xc5a6d210) at /usr/src/sys/kern/subr_sleepqueue.c:615
#4  0xc05a3483 in _sleep (ident=0xc5a6d210, lock=0x0, priority=92, wmesg=0xc081bb4e "usbevt", timo=60000) at /usr/src/sys/kern/kern_synch.c:226
#5  0xc051cad5 in usb_event_thread (arg=0xc5a62a80) at /usr/src/sys/dev/usb/usb.c:445
#6  0xc0574587 in fork_exit (callout=0xc051c9e0 <usb_event_thread>, arg=0xc5a62a80, frame=0xc5579d38) at /usr/src/sys/kern/kern_fork.c:811
#7  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:271

Thread 33 (Thread 100032):
#0  fork_trampoline () at /usr/src/sys/i386/i386/exception.s:268
Current language:  auto; currently asm

Thread 32 (Thread 100031):
#0  sched_switch (td=0xc5a60d80, newtd=Variable "newtd" is not available.
) at /usr/src/sys/kern/sched_ule.c:1944
#1  0xc05a3033 in mi_switch (flags=Variable "flags" is not available.
) at /usr/src/sys/kern/kern_synch.c:444
#2  0xc05cf859 in sleepq_switch (wchan=Variable "wchan" is not available.
) at /usr/src/sys/kern/subr_sleepqueue.c:497
#3  0xc05cfec6 in sleepq_wait (wchan=0xc08a1b54) at /usr/src/sys/kern/subr_sleepqueue.c:580
#4  0xc05a34a5 in _sleep (ident=0xc08a1b54, lock=0x0, priority=92, wmesg=0xc081bb40 "usbtsk", timo=0) at /usr/src/sys/kern/kern_synch.c:230
#5  0xc051c97e in usb_task_thread (arg=0xc08a1b54) at /usr/src/sys/dev/usb/usb.c:475
---Type <return> to continue, or q <return> to quit---
#6  0xc0574587 in fork_exit (callout=0xc051c900 <usb_task_thread>, arg=0xc08a1b54, frame=0xc5572d38) at /usr/src/sys/kern/kern_fork.c:811
#7  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:271
Current language:  auto; currently c

Thread 31 (Thread 100030):
#0  sched_switch (td=0xc5a61000, newtd=Variable "newtd" is not available.
) at /usr/src/sys/kern/sched_ule.c:1944
#1  0xc05a3033 in mi_switch (flags=Variable "flags" is not available.
) at /usr/src/sys/kern/kern_synch.c:444
#2  0xc05cf859 in sleepq_switch (wchan=Variable "wchan" is not available.
) at /usr/src/sys/kern/subr_sleepqueue.c:497
#3  0xc05cfec6 in sleepq_wait (wchan=0xc08a1b40) at /usr/src/sys/kern/subr_sleepqueue.c:580
#4  0xc05a34a5 in _sleep (ident=0xc08a1b40, lock=0x0, priority=92, wmesg=0xc081bb40 "usbtsk", timo=0) at /usr/src/sys/kern/kern_synch.c:230
#5  0xc051c97e in usb_task_thread (arg=0xc08a1b40) at /usr/src/sys/dev/usb/usb.c:475
#6  0xc0574587 in fork_exit (callout=0xc051c900 <usb_task_thread>, arg=0xc08a1b40, frame=0xc556fd38) at /usr/src/sys/kern/kern_fork.c:811
#7  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:271

Thread 30 (Thread 100029):
#0  sched_switch (td=0xc5a61240, newtd=Variable "newtd" is not available.
) at /usr/src/sys/kern/sched_ule.c:1944
#1  0xc05a3033 in mi_switch (flags=Variable "flags" is not available.
) at /usr/src/sys/kern/kern_synch.c:444
#2  0xc05cf859 in sleepq_switch (wchan=Variable "wchan" is not available.
) at /usr/src/sys/kern/subr_sleepqueue.c:497
#3  0xc05d03d7 in sleepq_timedwait (wchan=0xc5a4c210) at /usr/src/sys/kern/subr_sleepqueue.c:615
#4  0xc05a3483 in _sleep (ident=0xc5a4c210, lock=0x0, priority=92, wmesg=0xc081bb4e "usbevt", timo=60000) at /usr/src/sys/kern/kern_synch.c:226
#5  0xc051cad5 in usb_event_thread (arg=0xc5a63500) at /usr/src/sys/dev/usb/usb.c:445
#6  0xc0574587 in fork_exit (callout=0xc051c9e0 <usb_event_thread>, arg=0xc5a63500, frame=0xc556cd38) at /usr/src/sys/kern/kern_fork.c:811
#7  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:271

Thread 29 (Thread 100028):
#0  sched_switch (td=0xc5a61480, newtd=Variable "newtd" is not available.
) at /usr/src/sys/kern/sched_ule.c:1944
#1  0xc05a3033 in mi_switch (flags=Variable "flags" is not available.
) at /usr/src/sys/kern/kern_synch.c:444
#2  0xc0577dbb in ithread_loop (arg=0xc5a5b9d0) at /usr/src/sys/kern/kern_intr.c:1225
#3  0xc0574587 in fork_exit (callout=0xc0577ab0 <ithread_loop>, arg=0xc5a5b9d0, frame=0xc5568d38) at /usr/src/sys/kern/kern_fork.c:811
#4  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:271

Thread 28 (Thread 100027):
#0  sched_switch (td=0xc58da6c0, newtd=Variable "newtd" is not available.
) at /usr/src/sys/kern/sched_ule.c:1944
#1  0xc05a3033 in mi_switch (flags=Variable "flags" is not available.
) at /usr/src/sys/kern/kern_synch.c:444
#2  0xc0577dbb in ithread_loop (arg=0xc5a5b220) at /usr/src/sys/kern/kern_intr.c:1225
#3  0xc0574587 in fork_exit (callout=0xc0577ab0 <ithread_loop>, arg=0xc5a5b220, frame=0xc5565d38) at /usr/src/sys/kern/kern_fork.c:811
#4  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:271

Thread 27 (Thread 100026):
#0  sched_switch (td=0xc58da900, newtd=Variable "newtd" is not available.
) at /usr/src/sys/kern/sched_ule.c:1944
#1  0xc05a3033 in mi_switch (flags=Variable "flags" is not available.
) at /usr/src/sys/kern/kern_synch.c:444
#2  0xc0577dbb in ithread_loop (arg=0xc5a5b110) at /usr/src/sys/kern/kern_intr.c:1225
#3  0xc0574587 in fork_exit (callout=0xc0577ab0 <ithread_loop>, arg=0xc5a5b110, frame=0xc554fd38) at /usr/src/sys/kern/kern_fork.c:811
#4  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:271

Thread 26 (Thread 100025):
#0  sched_switch (td=0xc58dab40, newtd=Variable "newtd" is not available.
) at /usr/src/sys/kern/sched_ule.c:1944
#1  0xc05a3033 in mi_switch (flags=Variable "flags" is not available.
) at /usr/src/sys/kern/kern_synch.c:444
#2  0xc0577dbb in ithread_loop (arg=0xc5a5b020) at /usr/src/sys/kern/kern_intr.c:1225
#3  0xc0574587 in fork_exit (callout=0xc0577ab0 <ithread_loop>, arg=0xc5a5b020, frame=0xc5513d38) at /usr/src/sys/kern/kern_fork.c:811
#4  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:271

Thread 25 (Thread 100024):
#0  fork_trampoline () at /usr/src/sys/i386/i386/exception.s:268
Current language:  auto; currently asm

Thread 24 (Thread 100023):
#0  sched_switch (td=0xc59b2000, newtd=Variable "newtd" is not available.
) at /usr/src/sys/kern/sched_ule.c:1944
#1  0xc05a3033 in mi_switch (flags=Variable "flags" is not available.
) at /usr/src/sys/kern/kern_synch.c:444
#2  0xc0577dbb in ithread_loop (arg=0xc59fa980) at /usr/src/sys/kern/kern_intr.c:1225
---Type <return> to continue, or q <return> to quit---
#3  0xc0574587 in fork_exit (callout=0xc0577ab0 <ithread_loop>, arg=0xc59fa980, frame=0xc54fad38) at /usr/src/sys/kern/kern_fork.c:811
#4  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:271
Current language:  auto; currently c

Thread 23 (Thread 100022):
#0  fork_trampoline () at /usr/src/sys/i386/i386/exception.s:268
Current language:  auto; currently asm

Thread 22 (Thread 100021):
#0  sched_switch (td=0xc59b2480, newtd=Variable "newtd" is not available.
) at /usr/src/sys/kern/sched_ule.c:1944
#1  0xc05a3033 in mi_switch (flags=Variable "flags" is not available.
) at /usr/src/sys/kern/kern_synch.c:444
#2  0xc05cf859 in sleepq_switch (wchan=Variable "wchan" is not available.
) at /usr/src/sys/kern/subr_sleepqueue.c:497
#3  0xc05cfec6 in sleepq_wait (wchan=0xc588bd80) at /usr/src/sys/kern/subr_sleepqueue.c:580
#4  0xc05a34a5 in _sleep (ident=0xc588bd80, lock=0xc588bd9c, priority=0, wmesg=0xc081c912 "-", timo=0) at /usr/src/sys/kern/kern_synch.c:230
#5  0xc05d1708 in taskqueue_thread_loop (arg=0xc08a279c) at /usr/src/sys/kern/subr_taskqueue.c:95
#6  0xc0574587 in fork_exit (callout=0xc05d1600 <taskqueue_thread_loop>, arg=0xc08a279c, frame=0xc54e6d38) at /usr/src/sys/kern/kern_fork.c:811
#7  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:271
Current language:  auto; currently c

Thread 21 (Thread 100020):
#0  sched_switch (td=0xc59b26c0, newtd=Variable "newtd" is not available.
) at /usr/src/sys/kern/sched_ule.c:1944
#1  0xc05a3033 in mi_switch (flags=Variable "flags" is not available.
) at /usr/src/sys/kern/kern_synch.c:444
#2  0xc0577dbb in ithread_loop (arg=0xc59a3290) at /usr/src/sys/kern/kern_intr.c:1225
#3  0xc0574587 in fork_exit (callout=0xc0577ab0 <ithread_loop>, arg=0xc59a3290, frame=0xc54e3d38) at /usr/src/sys/kern/kern_fork.c:811
#4  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:271

Thread 20 (Thread 100019):
#0  sched_switch (td=0xc59b2900, newtd=Variable "newtd" is not available.
) at /usr/src/sys/kern/sched_ule.c:1944
#1  0xc05a3033 in mi_switch (flags=Variable "flags" is not available.
) at /usr/src/sys/kern/kern_synch.c:444
#2  0xc0577dbb in ithread_loop (arg=0xc59a32a0) at /usr/src/sys/kern/kern_intr.c:1225
#3  0xc0574587 in fork_exit (callout=0xc0577ab0 <ithread_loop>, arg=0xc59a32a0, frame=0xc54e0d38) at /usr/src/sys/kern/kern_fork.c:811
#4  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:271

Thread 19 (Thread 100018):
#0  sched_switch (td=0xc59b2b40, newtd=Variable "newtd" is not available.
) at /usr/src/sys/kern/sched_ule.c:1944
#1  0xc05a3033 in mi_switch (flags=Variable "flags" is not available.
) at /usr/src/sys/kern/kern_synch.c:444
#2  0xc05cf859 in sleepq_switch (wchan=Variable "wchan" is not available.
) at /usr/src/sys/kern/subr_sleepqueue.c:497
#3  0xc05cfec6 in sleepq_wait (wchan=0xc58ef000) at /usr/src/sys/kern/subr_sleepqueue.c:580
#4  0xc05a34a5 in _sleep (ident=0xc58ef000, lock=0xc58ef01c, priority=0, wmesg=0xc081c912 "-", timo=0) at /usr/src/sys/kern/kern_synch.c:230
#5  0xc05d1708 in taskqueue_thread_loop (arg=0xc1159f24) at /usr/src/sys/kern/subr_taskqueue.c:95
#6  0xc0574587 in fork_exit (callout=0xc05d1600 <taskqueue_thread_loop>, arg=0xc1159f24, frame=0xc54ddd38) at /usr/src/sys/kern/kern_fork.c:811
#7  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:271

Thread 18 (Thread 100017):
#0  sched_switch (td=0xc5891240, newtd=Variable "newtd" is not available.
) at /usr/src/sys/kern/sched_ule.c:1944
#1  0xc05a3033 in mi_switch (flags=Variable "flags" is not available.
) at /usr/src/sys/kern/kern_synch.c:444
#2  0xc05cf859 in sleepq_switch (wchan=Variable "wchan" is not available.
) at /usr/src/sys/kern/subr_sleepqueue.c:497
#3  0xc05cfec6 in sleepq_wait (wchan=0xc58ef000) at /usr/src/sys/kern/subr_sleepqueue.c:580
#4  0xc05a34a5 in _sleep (ident=0xc58ef000, lock=0xc58ef01c, priority=0, wmesg=0xc081c912 "-", timo=0) at /usr/src/sys/kern/kern_synch.c:230
#5  0xc05d1708 in taskqueue_thread_loop (arg=0xc1159f24) at /usr/src/sys/kern/subr_taskqueue.c:95
#6  0xc0574587 in fork_exit (callout=0xc05d1600 <taskqueue_thread_loop>, arg=0xc1159f24, frame=0xc54dad38) at /usr/src/sys/kern/kern_fork.c:811
#7  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:271

Thread 17 (Thread 100016):
#0  sched_switch (td=0xc5891480, newtd=Variable "newtd" is not available.
) at /usr/src/sys/kern/sched_ule.c:1944
#1  0xc05a3033 in mi_switch (flags=Variable "flags" is not available.
) at /usr/src/sys/kern/kern_synch.c:444
#2  0xc05cf859 in sleepq_switch (wchan=Variable "wchan" is not available.
) at /usr/src/sys/kern/subr_sleepqueue.c:497
#3  0xc05cfec6 in sleepq_wait (wchan=0xc58ef000) at /usr/src/sys/kern/subr_sleepqueue.c:580
#4  0xc05a34a5 in _sleep (ident=0xc58ef000, lock=0xc58ef01c, priority=0, wmesg=0xc081c912 "-", timo=0) at /usr/src/sys/kern/kern_synch.c:230
#5  0xc05d1708 in taskqueue_thread_loop (arg=0xc1159f24) at /usr/src/sys/kern/subr_taskqueue.c:95
#6  0xc0574587 in fork_exit (callout=0xc05d1600 <taskqueue_thread_loop>, arg=0xc1159f24, frame=0xc54d7d38) at /usr/src/sys/kern/kern_fork.c:811
---Type <return> to continue, or q <return> to quit---
#7  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:271

Thread 16 (Thread 100015):
#0  sched_switch (td=0xc58916c0, newtd=Variable "newtd" is not available.
) at /usr/src/sys/kern/sched_ule.c:1944
#1  0xc05a3033 in mi_switch (flags=Variable "flags" is not available.
) at /usr/src/sys/kern/kern_synch.c:444
#2  0xc05cf859 in sleepq_switch (wchan=Variable "wchan" is not available.
) at /usr/src/sys/kern/subr_sleepqueue.c:497
#3  0xc05cfec6 in sleepq_wait (wchan=0xc58ef080) at /usr/src/sys/kern/subr_sleepqueue.c:580
#4  0xc05a34a5 in _sleep (ident=0xc58ef080, lock=0xc58ef09c, priority=0, wmesg=0xc081c912 "-", timo=0) at /usr/src/sys/kern/kern_synch.c:230
#5  0xc05d1708 in taskqueue_thread_loop (arg=0xc08b061c) at /usr/src/sys/kern/subr_taskqueue.c:95
#6  0xc0574587 in fork_exit (callout=0xc05d1600 <taskqueue_thread_loop>, arg=0xc08b061c, frame=0xc54d4d38) at /usr/src/sys/kern/kern_fork.c:811
#7  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:271

Thread 15 (Thread 100014):
#0  fork_trampoline () at /usr/src/sys/i386/i386/exception.s:268
Current language:  auto; currently asm

Thread 14 (Thread 100013):
#0  sched_switch (td=0xc5891b40, newtd=Variable "newtd" is not available.
) at /usr/src/sys/kern/sched_ule.c:1944
#1  0xc05a3033 in mi_switch (flags=Variable "flags" is not available.
) at /usr/src/sys/kern/kern_synch.c:444
#2  0xc0577dbb in ithread_loop (arg=0xc59a32e0) at /usr/src/sys/kern/kern_intr.c:1225
#3  0xc0574587 in fork_exit (callout=0xc0577ab0 <ithread_loop>, arg=0xc59a32e0, frame=0xc54ced38) at /usr/src/sys/kern/kern_fork.c:811
#4  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:271
Current language:  auto; currently c

Thread 13 (Thread 100012):
#0  sched_switch (td=0xc5891d80, newtd=Variable "newtd" is not available.
) at /usr/src/sys/kern/sched_ule.c:1944
#1  0xc05a3033 in mi_switch (flags=Variable "flags" is not available.
) at /usr/src/sys/kern/kern_synch.c:444
#2  0xc05cf859 in sleepq_switch (wchan=Variable "wchan" is not available.
) at /usr/src/sys/kern/subr_sleepqueue.c:497
#3  0xc05cfec6 in sleepq_wait (wchan=0xc088d214) at /usr/src/sys/kern/subr_sleepqueue.c:580
#4  0xc05a34a5 in _sleep (ident=0xc088d214, lock=0xc088d22c, priority=76, wmesg=0xc0803242 "ccb_scanq", timo=0) at /usr/src/sys/kern/kern_synch.c:230
#5  0xc0446491 in xpt_scanner_thread (dummy=0x0) at /usr/src/sys/cam/cam_xpt.c:1399
#6  0xc0574587 in fork_exit (callout=0xc0446450 <xpt_scanner_thread>, arg=0x0, frame=0xc54cbd38) at /usr/src/sys/kern/kern_fork.c:811
#7  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:271

Thread 12 (Thread 100011):
#0  sched_switch (td=0xc58da000, newtd=Variable "newtd" is not available.
) at /usr/src/sys/kern/sched_ule.c:1944
#1  0xc05a3033 in mi_switch (flags=Variable "flags" is not available.
) at /usr/src/sys/kern/kern_synch.c:444
#2  0xc05cf859 in sleepq_switch (wchan=Variable "wchan" is not available.
) at /usr/src/sys/kern/subr_sleepqueue.c:497
#3  0xc05d03d7 in sleepq_timedwait (wchan=0xc08a44b4) at /usr/src/sys/kern/subr_sleepqueue.c:615
#4  0xc05a3483 in _sleep (ident=0xc08a44b4, lock=0x0, priority=0, wmesg=0xc081c912 "-", timo=100) at /usr/src/sys/kern/kern_synch.c:226
#5  0xc05a35a0 in pause (wmesg=0xc081c912 "-", timo=100) at /usr/src/sys/kern/kern_synch.c:334
#6  0xc04c3e72 in random_kthread (arg=0x0) at /usr/src/sys/dev/random/randomdev_soft.c:289
#7  0xc0574587 in fork_exit (callout=0xc04c3c40 <random_kthread>, arg=0x0, frame=0xc54c8d38) at /usr/src/sys/kern/kern_fork.c:811
#8  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:271

Thread 11 (Thread 100010):
#0  sched_switch (td=0xc58da240, newtd=Variable "newtd" is not available.
) at /usr/src/sys/kern/sched_ule.c:1944
#1  0xc05a3033 in mi_switch (flags=Variable "flags" is not available.
) at /usr/src/sys/kern/kern_synch.c:444
#2  0xc05cf859 in sleepq_switch (wchan=Variable "wchan" is not available.
) at /usr/src/sys/kern/subr_sleepqueue.c:497
#3  0xc05d03d7 in sleepq_timedwait (wchan=0xc08a1f0c) at /usr/src/sys/kern/subr_sleepqueue.c:615
#4  0xc05a3483 in _sleep (ident=0xc08a1f0c, lock=0xc08a1e28, priority=588, wmesg=0xc081c912 "-", timo=100) at /usr/src/sys/kern/kern_synch.c:226
#5  0xc053f22b in g_io_schedule_down (tp=0xc58da240) at /usr/src/sys/geom/geom_io.c:487
#6  0xc053f94c in g_down_procbody () at /usr/src/sys/geom/geom_kern.c:118
#7  0xc0574587 in fork_exit (callout=0xc053f8e0 <g_down_procbody>, arg=0x0, frame=0xc54c5d38) at /usr/src/sys/kern/kern_fork.c:811
#8  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:271

Thread 10 (Thread 100009):
#0  sched_switch (td=0xc58da480, newtd=Variable "newtd" is not available.
) at /usr/src/sys/kern/sched_ule.c:1944
#1  0xc05a3033 in mi_switch (flags=Variable "flags" is not available.
) at /usr/src/sys/kern/kern_synch.c:444
#2  0xc05cf859 in sleepq_switch (wchan=Variable "wchan" is not available.
) at /usr/src/sys/kern/subr_sleepqueue.c:497
---Type <return> to continue, or q <return> to quit---
#3  0xc05d03d7 in sleepq_timedwait (wchan=0xc08a1f08) at /usr/src/sys/kern/subr_sleepqueue.c:615
#4  0xc05a3483 in _sleep (ident=0xc08a1f08, lock=0xc08a1e68, priority=588, wmesg=0xc081c912 "-", timo=100) at /usr/src/sys/kern/kern_synch.c:226
#5  0xc053f532 in g_io_schedule_up (tp=0xc58da480) at /usr/src/sys/geom/geom_io.c:592
#6  0xc053f82c in g_up_procbody () at /usr/src/sys/geom/geom_kern.c:95
#7  0xc0574587 in fork_exit (callout=0xc053f7c0 <g_up_procbody>, arg=0x0, frame=0xc54c2d38) at /usr/src/sys/kern/kern_fork.c:811
#8  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:271

Thread 9 (Thread 100008):
#0  sched_switch (td=0xc588f000, newtd=Variable "newtd" is not available.
) at /usr/src/sys/kern/sched_ule.c:1944
#1  0xc05a3033 in mi_switch (flags=Variable "flags" is not available.
) at /usr/src/sys/kern/kern_synch.c:444
#2  0xc05cf859 in sleepq_switch (wchan=Variable "wchan" is not available.
) at /usr/src/sys/kern/subr_sleepqueue.c:497
#3  0xc05d03d7 in sleepq_timedwait (wchan=0xc08a1f00) at /usr/src/sys/kern/subr_sleepqueue.c:615
#4  0xc05a3483 in _sleep (ident=0xc08a1f00, lock=0x0, priority=76, wmesg=0xc081c912 "-", timo=100) at /usr/src/sys/kern/kern_synch.c:226
#5  0xc053f8da in g_event_procbody () at /usr/src/sys/geom/geom_kern.c:142
#6  0xc0574587 in fork_exit (callout=0xc053f830 <g_event_procbody>, arg=0x0, frame=0xc54bfd38) at /usr/src/sys/kern/kern_fork.c:811
#7  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:271

Thread 8 (Thread 100007):
#0  fork_trampoline () at /usr/src/sys/i386/i386/exception.s:268
Current language:  auto; currently asm

Thread 7 (Thread 100006):
#0  sched_switch (td=0xc588f480, newtd=Variable "newtd" is not available.
) at /usr/src/sys/kern/sched_ule.c:1944
#1  0xc05a3033 in mi_switch (flags=Variable "flags" is not available.
) at /usr/src/sys/kern/kern_synch.c:444
#2  0xc07d4682 in ipi_bitmap_handler (frame=
      {tf_fs = 8, tf_es = 40, tf_ds = 40, tf_edi = -644953128, tf_esi = -980880256, tf_ebp = -984900488, tf_isp = -984900532, tf_ebx = -1064681904, tf_edx = -980877952, tf_ecx = -980877952, tf_eax = 4, tf_trapno = 0, tf_err = 0, tf_eip = -1067925921, tf_cs = 32, tf_eflags = 582, tf_esp = -1066191232, tf_ss = 0}) at /usr/src/sys/i386/i386/mp_machdep.c:1187
#3  0xc07c6dae in Xipi_intr_bitmap_handler () at apic_vector.s:284
#4  0xc058be5f in _mtx_lock_sleep (m=0xc08a3e50, tid=3314087040, opts=0, file=0x0, line=0) at /usr/src/sys/kern/kern_mutex.c:349
#5  0xc05ad91b in softclock (dummy=0x0) at /usr/src/sys/kern/kern_timeout.c:241
#6  0xc0577c5b in ithread_loop (arg=0xc588c390) at /usr/src/sys/kern/kern_intr.c:1126
#7  0xc0574587 in fork_exit (callout=0xc0577ab0 <ithread_loop>, arg=0xc588c390, frame=0xc54b9d38) at /usr/src/sys/kern/kern_fork.c:811
#8  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:271
Current language:  auto; currently c

Thread 6 (Thread 100005):
#0  sched_switch (td=0xc588f6c0, newtd=Variable "newtd" is not available.
) at /usr/src/sys/kern/sched_ule.c:1944
#1  0xc05a3033 in mi_switch (flags=Variable "flags" is not available.
) at /usr/src/sys/kern/kern_synch.c:444
#2  0xc0577dbb in ithread_loop (arg=0xc588c3a0) at /usr/src/sys/kern/kern_intr.c:1225
#3  0xc0574587 in fork_exit (callout=0xc0577ab0 <ithread_loop>, arg=0xc588c3a0, frame=0xc54b6d38) at /usr/src/sys/kern/kern_fork.c:811
#4  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:271

Thread 5 (Thread 100004):
#0  sched_switch (td=0xc588f900, newtd=Variable "newtd" is not available.
) at /usr/src/sys/kern/sched_ule.c:1944
#1  0xc05a3033 in mi_switch (flags=Variable "flags" is not available.
) at /usr/src/sys/kern/kern_synch.c:444
#2  0xc05bcedd in critical_exit () at kern_switch.c:185
#3  0xc07ce2f5 in lapic_handle_timer (frame=0xc54b1c48) at /usr/src/sys/i386/i386/local_apic.c:724
#4  0xc07c6c9f in Xtimerint () at apic_vector.s:108
#5  0xc1149b15 in acpi_cpu_c1 () at /usr/src/sys/modules/acpi/acpi/../../../i386/acpica/acpi_machdep.c:550
#6  0xc1142a1d in acpi_cpu_idle () at /usr/src/sys/modules/acpi/acpi/../../../dev/acpica/acpi_cpu.c:912
#7  0xc07d0a58 in cpu_idle () at /usr/src/sys/i386/i386/machdep.c:1183
#8  0xc05bdd6b in sched_idletd (dummy=0x0) at /usr/src/sys/kern/sched_ule.c:2681
#9  0xc0574587 in fork_exit (callout=0xc05bdaa0 <sched_idletd>, arg=0x0, frame=0xc54b1d38) at /usr/src/sys/kern/kern_fork.c:811
#10 0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:271

Thread 4 (Thread 100003):
#0  sched_switch (td=0xc588fb40, newtd=Variable "newtd" is not available.
) at /usr/src/sys/kern/sched_ule.c:1944
#1  0xc05a3033 in mi_switch (flags=Variable "flags" is not available.
) at /usr/src/sys/kern/kern_synch.c:444
#2  0xc07d4682 in ipi_bitmap_handler (frame=
      {tf_fs = 8, tf_es = 40, tf_ds = 40, tf_edi = 1, tf_esi = 15, tf_ebp = -984945472, tf_isp = -984945496, tf_ebx = -980878528, tf_edx = 256, tf_ecx = 16, tf_eax = 582, tf_trapno = 0, tf_err = 0, tf_e---Type <return> to continue, or q <return> to quit---
ip = -1065551975, tf_cs = 32, tf_eflags = 582, tf_esp = -1064663296, tf_ss = -984945420}) at /usr/src/sys/i386/i386/mp_machdep.c:1187
#3  0xc07c6dae in Xipi_intr_bitmap_handler () at apic_vector.s:284
#4  0xc07cf799 in spinlock_exit () at cpufunc.h:365
#5  0xc05bdcd6 in sched_idletd (dummy=0x0) at /usr/src/sys/kern/sched_ule.c:806
#6  0xc0574587 in fork_exit (callout=0xc05bdaa0 <sched_idletd>, arg=0x0, frame=0xc54aed38) at /usr/src/sys/kern/kern_fork.c:811
#7  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:271

Thread 3 (Thread 100002):
#0  doadump () at pcpu.h:196
#1  0xc059b097 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:418
#2  0xc059b36c in panic (fmt=Variable "fmt" is not available.
) at /usr/src/sys/kern/kern_shutdown.c:574
#3  0xc058bd1f in _mtx_lock_spin_failed (m=0x0) at /usr/src/sys/kern/kern_mutex.c:449
#4  0xc058c425 in _thread_lock_flags (td=0xc588f480, opts=0, file=0xc08200c8 "/usr/src/sys/kern/kern_intr.c", line=801) at /usr/src/sys/kern/kern_mutex.c:537
#5  0xc0576787 in intr_event_schedule_thread (ie=Variable "ie" is not available.
) at /usr/src/sys/kern/kern_intr.c:801
#6  0xc0577aa5 in swi_sched (cookie=0xc5880e00, flags=Variable "flags" is not available.
) at /usr/src/sys/kern/kern_intr.c:1028
#7  0xc055a6df in hardclock (usermode=0, pc=3229503748) at /usr/src/sys/kern/kern_clock.c:339
#8  0xc07ce23a in lapic_handle_timer (frame=0xc54aaba0) at /usr/src/sys/i386/i386/local_apic.c:705
#9  0xc07c6c9f in Xtimerint () at apic_vector.s:108
#10 0xc07e5104 in DELAY (n=1000000) at cpufunc.h:352
#11 0xc07e2c03 in cpu_reset () at /usr/src/sys/i386/i386/vm_machdep.c:594
#12 0xc059a753 in shutdown_reset (junk=0x0, howto=0) at /usr/src/sys/kern/kern_shutdown.c:493
#13 0xc059b117 in boot (howto=0) at /usr/src/sys/kern/kern_shutdown.c:421
#14 0xc059b247 in reboot (td=0xc588fd80, uap=0xc54aacfc) at /usr/src/sys/kern/kern_shutdown.c:172
#15 0xc07e07d5 in syscall (frame=0xc54aad38) at /usr/src/sys/i386/i386/trap.c:1089
#16 0xc07c6720 in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:262
#17 0x00000033 in ?? ()
Previous frame inner to this frame (corrupt stack?)

Thread 2 (Thread 100001):
#0  sched_switch (td=0xc5891000, newtd=Variable "newtd" is not available.
) at /usr/src/sys/kern/sched_ule.c:1944
#1  0xc05a3033 in mi_switch (flags=Variable "flags" is not available.
) at /usr/src/sys/kern/kern_synch.c:444
#2  0xc05cf859 in sleepq_switch (wchan=Variable "wchan" is not available.
) at /usr/src/sys/kern/subr_sleepqueue.c:497
#3  0xc05cfec6 in sleepq_wait (wchan=0xc08bcd48) at /usr/src/sys/kern/subr_sleepqueue.c:580
#4  0xc055aaaf in _cv_wait (cvp=0xc08bcd48, lock=0xc08bcd28) at /usr/src/sys/kern/kern_condvar.c:137
#5  0xc0749e2f in audit_worker (arg=0x0) at /usr/src/sys/security/audit/audit_worker.c:410
#6  0xc0574587 in fork_exit (callout=0xc0749dc0 <audit_worker>, arg=0x0, frame=0xc54a7d38) at /usr/src/sys/kern/kern_fork.c:811
#7  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:271

Thread 1 (Thread 100000):
#0  sched_switch (td=0xc08a2280, newtd=Variable "newtd" is not available.
) at /usr/src/sys/kern/sched_ule.c:1944
#1  0xc05a3033 in mi_switch (flags=Variable "flags" is not available.
) at /usr/src/sys/kern/kern_synch.c:444
#2  0xc05cf859 in sleepq_switch (wchan=Variable "wchan" is not available.
) at /usr/src/sys/kern/subr_sleepqueue.c:497
#3  0xc05d03d7 in sleepq_timedwait (wchan=0xc08a1fc0) at /usr/src/sys/kern/subr_sleepqueue.c:615
#4  0xc05a3483 in _sleep (ident=0xc08a1fc0, lock=0x0, priority=68, wmesg=0xc0823aef "sched", timo=10000) at /usr/src/sys/kern/kern_synch.c:226
#5  0xc078df39 in scheduler (dummy=0x0) at /usr/src/sys/vm/vm_glue.c:733
#6  0xc05577b6 in mi_startup () at /usr/src/sys/kern/init_main.c:251
#7  0xc0440685 in begin () at /usr/src/sys/i386/i386/locore.s:328
196		__asm __volatile("movl %%fs:0,%0" : "=r" (td));
(kgdb) quit




More information about the freebsd-stable mailing list