misc/116932: System locking up while carrying out remote ssh
session on -current
Joao Pedras
jpedras at webvolution.net
Thu Oct 4 12:20:02 PDT 2007
>Number: 116932
>Category: misc
>Synopsis: System locking up while carrying out remote ssh session on -current
>Confidential: no
>Severity: serious
>Priority: medium
>Responsible: freebsd-bugs
>State: open
>Quarter:
>Keywords:
>Date-Required:
>Class: sw-bug
>Submitter-Id: current-users
>Arrival-Date: Thu Oct 04 19:20:01 GMT 2007
>Closed-Date:
>Last-Modified:
>Originator: Joao Pedras
>Release: 7-CURRENT
>Organization:
>Environment:
NOTE: Tried with -CURRENT from October 4th and problem still persists.
Copyright (c) 1992-2007 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 7.0-CURRENT #2: Fri Sep 28 11:45:50 PDT 2007
root at daemons2.somedomain.com:/usr/obj/usr/src/sys/GENERIC
WARNING: WITNESS option enabled, expect reduced performance.
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Dual-Core AMD Opteron(tm) Processor 2210 (1800.09-MHz K8-class CPU)
Origin = "AuthenticAMD" Id = 0x40f12 Stepping = 2
Features=0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT>
Features2=0x2001<SSE3,CX16>
AMD Features=0xea500800<SYSCALL,NX,MMX+,FFXSR,RDTSCP,LM,3DNow!+,3DNow!>
AMD Features2=0x1f<LAHF,CMP,SVM,ExtAPIC,CR8>
Cores per package: 2
usable memory = 4281348096 (4083 MB)
avail memory = 4123906048 (3932 MB)
ACPI APIC Table: <082907 APIC1947>
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
cpu0 (BSP): APIC ID: 0
cpu1 (AP): APIC ID: 1
cpu2 (AP): APIC ID: 2
cpu3 (AP): APIC ID: 3
ioapic0 <Version 1.1> irqs 0-23 on motherboard
kbd1 at kbdmux0
ath_hal: 0.9.20.3 (AR5210, AR5211, AR5212, RF5111, RF5112, RF2413, RF5413)
acpi0: <082907 XSDT1947> on motherboard
acpi0: [ITHREAD]
acpi_hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 25000000 Hz quality 900
acpi0: Power Button (fixed)
acpi0: reservation of fec00000, 1400000 (3) failed
acpi0: reservation of 0, a0000 (3) failed
acpi0: reservation of 100000, d7f00000 (3) failed
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x2008-0x200b on acpi0
cpu0: <ACPI CPU> on acpi0
powernow0: <PowerNow! K8> on cpu0
cpu1: <ACPI CPU> on acpi0
powernow1: <PowerNow! K8> on cpu1
cpu2: <ACPI CPU> on acpi0
powernow2: <PowerNow! K8> on cpu2
cpu3: <ACPI CPU> on acpi0
powernow3: <PowerNow! K8> on cpu3
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pci0: <memory, RAM> at device 0.0 (no driver attached)
isab0: <PCI-ISA bridge> port 0x2f00-0x2f7f at device 1.0 on pci0
isa0: <ISA bus> on isab0
pci0: <serial bus, SMBus> at device 1.1 (no driver attached)
ohci0: <OHCI (generic) USB controller> mem 0xfdefb000-0xfdefbfff irq 21 at device 2.0 on pci0
ohci0: [GIANT-LOCKED]
ohci0: [ITHREAD]
usb0: OHCI version 1.0, legacy support
usb0: SMM does not respond, resetting
usb0: <OHCI (generic) USB controller> on ohci0
usb0: USB revision 1.0
uhub0: <nVidia OHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb0
uhub0: 10 ports with 10 removable, self powered
ehci0: <EHCI (generic) USB 2.0 controller> mem 0xfdefac00-0xfdefacff irq 22 at device 2.1 on pci0
ehci0: [GIANT-LOCKED]
ehci0: [ITHREAD]
usb1: EHCI version 1.0
usb1: companion controller, 10 ports each: usb0
usb1: <EHCI (generic) USB 2.0 controller> on ehci0
usb1: USB revision 2.0
uhub1: <nVidia EHCI root hub, class 9/0, rev 2.00/1.00, addr 1> on usb1
uhub1: 10 ports with 10 removable, self powered
atapci0: <nVidia nForce MCP55 UDMA133 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xffa0-0xffaf at device 4.0 on pci0
ata0: <ATA channel 0> on atapci0
ata0: [ITHREAD]
ata1: <ATA channel 1> on atapci0
ata1: [ITHREAD]
atapci1: <nVidia nForce MCP55 SATA300 controller> port 0xc480-0xc487,0xc400-0xc403,0xc080-0xc087,0xc000-0xc003,0xbc00-0xbc0f mem 0xfdef9000-0xfdef9fff irq 23 at device 5.0 on pci0
atapci1: [ITHREAD]
ata2: <ATA channel 0> on atapci1
ata2: [ITHREAD]
ata3: <ATA channel 1> on atapci1
ata3: [ITHREAD]
atapci2: <nVidia nForce MCP55 SATA300 controller> port 0xb880-0xb887,0xb800-0xb803,0xb480-0xb487,0xb400-0xb403,0xb080-0xb08f mem 0xfdef8000-0xfdef8fff irq 20 at device 5.1 on pci0
atapci2: [ITHREAD]
ata4: <ATA channel 0> on atapci2
ata4: [ITHREAD]
ata5: <ATA channel 1> on atapci2
ata5: [ITHREAD]
atapci3: <nVidia nForce MCP55 SATA300 controller> port 0xb000-0xb007,0xac00-0xac03,0xa880-0xa887,0xa800-0xa803,0xa480-0xa48f mem 0xfdef7000-0xfdef7fff irq 21 at device 5.2 on pci0
atapci3: [ITHREAD]
ata6: <ATA channel 0> on atapci3
ata6: [ITHREAD]
ata7: <ATA channel 1> on atapci3
ata7: [ITHREAD]
pcib1: <ACPI PCI-PCI bridge> at device 6.0 on pci0
pci1: <ACPI PCI bus> on pcib1
vgapci0: <VGA-compatible display> port 0xd000-0xd0ff mem 0xf0000000-0xf7ffffff,0xfdff0000-0xfdffffff irq 16 at device 7.0 on pci1
nfe0: <NVIDIA nForce MCP55 Networking Adapter> port 0xa400-0xa407 mem 0xfdef6000-0xfdef6fff,0xfdefa800-0xfdefa8ff,0xfdefa400-0xfdefa40f irq 22 at device 8.0 on pci0
miibus0: <MII bus> on nfe0
e1000phy0: <Marvell 88E1149 Gigabit PHY> PHY 16 on miibus0
e1000phy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseTX-FDX, auto
nfe0: Ethernet address: 00:e0:81:48:f9:64
nfe0: [FILTER]
nfe0: [FILTER]
nfe0: [FILTER]
nfe0: [FILTER]
nfe0: [FILTER]
nfe0: [FILTER]
nfe0: [FILTER]
nfe0: [FILTER]
nfe1: <NVIDIA nForce MCP55 Networking Adapter> port 0xa080-0xa087 mem 0xfdef5000-0xfdef5fff,0xfdefa000-0xfdefa0ff,0xfdef4c00-0xfdef4c0f irq 23 at device 9.0 on pci0
miibus1: <MII bus> on nfe1
e1000phy1: <Marvell 88E1149 Gigabit PHY> PHY 17 on miibus1
e1000phy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseTX-FDX, auto
nfe1: Ethernet address: 00:e0:81:48:f9:65
nfe1: [FILTER]
nfe1: [FILTER]
nfe1: [FILTER]
nfe1: [FILTER]
nfe1: [FILTER]
nfe1: [FILTER]
nfe1: [FILTER]
nfe1: [FILTER]
pcib2: <ACPI PCI-PCI bridge> at device 10.0 on pci0
pci2: <ACPI PCI bus> on pcib2
mpt0: <LSILogic SAS/SATA Adapter> port 0xe000-0xe0ff mem 0xfebfc000-0xfebfffff,0xfebe0000-0xfebeffff irq 17 at device 0.0 on pci2
mpt0: [ITHREAD]
mpt0: MPI Version=1.5.13.0
mpt0: mpt_cam_event: 0x16
mpt0: Unhandled Event Notify Frame. Event 0x16 (ACK not required).
mpt0: mpt_cam_event: 0x16
mpt0: Unhandled Event Notify Frame. Event 0x16 (ACK not required).
pcib3: <ACPI PCI-PCI bridge> at device 13.0 on pci0
pci3: <ACPI PCI bus> on pcib3
pcib4: <PCI-PCI bridge> at device 0.0 on pci3
pci4: <PCI bus> on pcib4
pcib5: <PCI-PCI bridge> at device 0.1 on pci3
pci5: <PCI bus> on pcib5
pcib6: <ACPI PCI-PCI bridge> at device 14.0 on pci0
pci6: <ACPI PCI bus> on pcib6
pcib7: <ACPI PCI-PCI bridge> at device 15.0 on pci0
pci7: <ACPI PCI bus> on pcib7
acpi_button0: <Power Button> on acpi0
sio0: configured irq 4 not in bitmap of probed irqs 0
sio0: port may not be enabled
sio0: configured irq 4 not in bitmap of probed irqs 0
sio0: port may not be enabled
sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
sio0: type 16550A, console
sio0: [FILTER]
sio1: configured irq 3 not in bitmap of probed irqs 0
sio1: port may not be enabled
sio1: configured irq 3 not in bitmap of probed irqs 0
sio1: port may not be enabled
sio1: <16550A-compatible COM port> port 0x2f8-0x2ff irq 3 drq 0 on acpi0
sio1: type 16550A
sio1: [FILTER]
fdc0: <floppy drive controller (FDE)> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0
fdc0: [FILTER]
fd0: <1440-KB 3.5" drive> on fdc0 drive 0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
ipmi0: <IPMI System Interface> on isa0
ipmi0: KCS mode found at io 0xca8 alignment 0x4 on isa
orm0: <ISA Option ROMs> at iomem 0xc0000-0xc8fff,0xc9000-0xca7ff on isa0
ppc0: cannot reserve I/O port range
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
Timecounters tick every 1.000 msec
acd0: DVDROM <DVD-ROM 16X/5.AV> at ata0-master UDMA100
ad4: 238475MB <Hitachi HDT725025VLA380 V5DOA52A> at ata2-master SATA300
ipmi0: IPMI device rev. 0, firmware rev. 0.2, version 2.0
ipmi0: Number of channels 4
ipmi0: Attached watchdog
SMP: AP CPU #1 Launched!
SMP: AP CPU #2 Launched!
SMP: AP CPU #3 Launched!
WARNING: WITNESS option enabled, expect reduced performance.
GEOM_LABEL: Label for provider acd0 is iso9660/OpenBSD4.1 CD 3 www.OpenBSD.org.
Trying to mount root from ufs:/dev/ad4s1a
WARNING: /var was not properly dismounted
nfe0: watchdog timeout
nfe0: watchdog timeout
nfe0: link state changed to DOWN
nfe0: link state changed to UP
>Description:
My apologies for the simplistic way of reproducing the issue but it is indeed the only way I know how to cause it. Please feel free to suggest/request other scenarios so that this issue can be resolved.
While watching a 'make buildworld' over an ssh session, established over an ipsec link between two remote locations, the system will lock up. The same task with a session established over the local network does not cause the lock up.
I don't discard the possibility that the system is simply too slow but there are no replies to pings, console interaction or anything else. It is possible though to break into the debugger with CTRL+ALT+ESC.
The system where this occurs is on a RFC 1918 network and is accessible over the vpn link. The system is *not* the router, just a plain client.
Software:
The OS is plain -CURRENT, with no ports added, one of the nfe interfaces obtains config through DHCP and the other is not being used. I basically installed the OS to test the hardware from a snapshot CD, sup'ed it and started noticing this issue from the start.
I have tried with GENERIC 4BSD and also with ULE.
Before the lockup I see a few 'kernel: rtfree: 0xPPPPPPP has 1 refs' messages, I also have tried without IPv6. Judging by some recent post on freebsd-current these warnings seem to be associated with IPv6.
The problem occurs with all of the above variations (4BSD, ULE, no IPv6). The rest of kernel config is GENERIC.
Hardware:
This problem happens consistently on a couple CURRENTs running on similar hardware (Tyan mobo S2932).
One of them boots off the one SATA hd is has, which is connected to the onboard controller.
The other one boots off a RAID1 array from the onboard LSI SAS controller. This second system also has a RAID5 array in a additional LSI 300-8XLP.
Both machines have the additional IPMI card ( M3291) installed. The problem still occurs *without* the ipmi cards.
Tried also with an old faithful fxp() card and no luck.
from debugger after the lock up:
[root at daemons2 ~]#
[root at daemons2 ~]#
[root at daemons2 ~]# rtfree: 0xffffff00044771e0 has 1 refs
rtfree: 0xffffff00044771e0 has 1 refs
rtfree: 0xffffff00044771e0 has 1 refs
rtfree: 0xffffff00044771e0 has 1 refs
rtfree: 0xffffff00044771e0 has 1 refs
rtfree: 0xffffff00044771e0 has 1 refs
rtfree: 0xffffff00044771e0 has 1 refs
rtfree: 0xffffff00044771e0 has 1 refs
rtfree: 0xffffff00044771e0 has 1 refs
rtfree: 0xffffff00044771e0 has 1 refs
rtfree: 0xffffff00044771e0 has 1 refs
rtfree: 0xffffff00044771e0 has 1 refs
rtfree: 0xffffff00044771e0 has 1 refs
rtfree: 0xffffff00044771e0 has 1 refs
rtfree: 0xffffff00044771e0 has 1 refs
rtfree: 0xffffff00044771e0 has 1 refs
rtfree: 0xffffff00044771e0 has 1 refs
rtfree: 0xffffff00044771e0 has 1 refs
rtfree: 0xffffff00044771e0 has 1 refs
rtfree: 0xffffff00044771e0 has 1 refs
rtfree: 0xffffff00044771e0 has 1 refs
rtfree: 0xffffff00044771e0 has 1 refs
rtfree: 0xffffff00044771e0 has 1 refs
rtfree: 0xffffff00044771e0 has 1 refs
rtfree: 0xffffff00044771e0 has 1 refs
rtfree: 0xffffff00044771e0 has 1 refs
rtfree: 0xffffff00044771e0 has 1 refs
rtfree: 0xffffff00044771e0 has 1 refs
rtfree: 0xffffff00044771e0 has 1 refs
rtfree: 0xffffff00044771e0 has 1 refs
rtfree: 0xffffff00044771e0 has 1 refs
rtfree: 0xffffff00044771e0 has 1 refs
KDB: enter: manual escape to debugger <<<< LOCKED UP BEFORE THIS; THIS CTRL+ALT+ESC
[thread pid 19 tid 100010 ]
Stopped at kdb_enter+0x31: leave
db> ps
pid ppid pgrp uid state wmesg wchan cmd
21498 21496 778 0 S+ piperd 0xffffff000c3c82e8 sed
21497 21496 778 0 S+ vmo_de 0xffffffff80a1de48 sh
21496 21492 778 0 S+ wait 0xffffff0008f97000 sh
21492 21139 778 0 S+ piperd 0xffffff00044ca2e8 make
21139 21132 778 0 S+ wait 0xffffff0004375000 sh
21132 21131 778 0 S+ select 0xffffffff80aa6db0 make
21131 4753 778 0 S+ wait 0xffffff0004098460 sh
4753 4751 778 0 S+ pipewr 0xffffff0004369ba0 make
4751 4660 778 0 S+ wait 0xffffff0004518000 sh
4660 778 778 0 S+ ttywri 0xffffff0004058450 make
778 775 778 0 S+ wait 0xffffff00041e1460 sh
775 772 775 0 Ss+ wait 0xffffff0004432000 bash
772 705 772 0 Ls *rtentry 0xffffff0003be5640 sshd
771 765 771 0 S+ ttyin 0xffffff0003bc6c10 bash
766 1 766 0 Ss+ ttyin 0xffffff0003bce810 getty
765 1 765 0 Ss+ wait 0xffffff00042258c0 login
764 1 764 0 Ss+ ttyin 0xffffff0003bf3c10 getty
763 1 763 0 Ss+ ttyin 0xffffff0003bf4010 getty
721 1 721 0 Ss nanslp 0xffffffff80a1e1c8 cron
--More-- 715 1 715 25 Ss pause 0xffffff00044310c0 sendmail
711 1 711 0 Ss select 0xffffffff80aa6db0 sendmail
705 1 705 0 Ss select 0xffffffff80aa6db0 sshd
575 1 575 0 Ss select 0xffffffff80aa6db0 syslogd
529 1 529 0 Ss select 0xffffffff80aa6db0 devd
324 1 324 65 Ss select 0xffffffff80aa6db0 dhclient
296 1 52 0 S+ select 0xffffffff80aa6db0 dhclient
162 1 162 0 Ss pause 0xffffff00041b70c0 adjkerntz
51 0 0 0 SL - 0xffffffff80a1de48 [schedcpu]
50 0 0 0 SL sdflush 0xffffffff80ab6dc8 [softdepflush]
49 0 0 0 SL vlruwt 0xffffff0003be1000 [vnlru]
48 0 0 0 SL syncer 0xffffffff80a1de40 [syncer]
47 0 0 0 SL psleep 0xffffffff80aa763c [bufdaemon]
46 0 0 0 SL pgzero 0xffffffff80ab8844 [pagezero]
45 0 0 0 SL psleep 0xffffffff80ab7b88 [vmdaemon]
44 0 0 0 SL psleep 0xffffffff80ab7b4c [pagedaemon]
43 0 0 0 SL ipmireq 0xffffff0003be8b48 [ipmi0: kcs]
42 0 0 0 SL waiting_ 0xffffffff80aaad48 [sctp_iterator]
41 0 0 0 WL [irq1: atkbd0]
40 0 0 0 SL - 0xffffff0003bde448 [fdc0]
--More-- 39 0 0 0 WL [swi0: sio]
38 0 0 0 SL idle 0xffffffff80e50000 [mpt_recovery0]
37 0 0 0 WL [irq17: mpt0]
36 0 0 0 SL - 0xffffff0003ba6680 [nfe1 taskq]
35 0 0 0 SL - 0xffffff0003a00b80 [nfe0 taskq]
34 0 0 0 WL [irq20: atapci2]
33 0 0 0 WL [irq23: atapci1]
32 0 0 0 WL [irq15: ata1]
31 0 0 0 WL [irq14: ata0]
30 0 0 0 SL usbevt 0xffffff0001244420 [usb1]
29 0 0 0 WL [irq22: ehci0]
28 0 0 0 SL usbtsk 0xffffffff80a19808 [usbtask-dr]
27 0 0 0 SL usbtsk 0xffffffff80a197e0 [usbtask-hc]
26 0 0 0 SL usbevt 0xffffffff80e44420 [usb0]
25 0 0 0 WL [irq21: ohci0+]
24 0 0 0 WL [irq9: acpi0]
23 0 0 0 SL - 0xffffff0001203880 [thread taskq]
22 0 0 0 LL *rtentry 0xffffff000104e640 [swi5: +]
21 0 0 0 WL [swi2: cambio]
9 0 0 0 SL ccb_scan 0xffffffff809e8540 [xpt_thrd]
--More-- 8 0 0 0 SL - 0xffffff0001203c00 [acpi_task_2]
7 0 0 0 SL - 0xffffff0001203c00 [acpi_task_1]
6 0 0 0 SL - 0xffffff0001203c00 [acpi_task_0]
5 0 0 0 SL - 0xffffff0001235200 [kqueue taskq]
20 0 0 0 WL [swi6: task queue]
19 0 0 0 RL CPU 0 [swi6: Giant taskq]
18 0 0 0 SL - 0xffffffff80a1de48 [yarrow]
4 0 0 0 SL - 0xffffffff80a1a7b8 [g_down]
3 0 0 0 SL - 0xffffffff80a1a7b0 [g_up]
2 0 0 0 SL - 0xffffffff80a1a7a0 [g_event]
17 0 0 0 WL [swi3: vm]
16 0 0 0 LL *inp 0xffffff00085726e0 [swi4: clock sio]
15 0 0 0 WL [swi1: net]
14 0 0 0 RL [idle: cpu0]
13 0 0 0 RL CPU 1 [idle: cpu1]
12 0 0 0 RL CPU 2 [idle: cpu2]
11 0 0 0 RL CPU 3 [idle: cpu3]
1 0 1 0 SLs wait 0xffffff00010e48c0 [init]
10 0 0 0 SL audit_wo 0xffffffff80ab62a0 [audit]
0 0 0 0 WLs [swapper]
--More-- b> trace
Tracing pid 19 tid 100010 td 0xffffff00010e8340
kdb_enter() at kdb_enter+0x31
scgetc() at scgetc+0x461
sckbdevent() at sckbdevent+0xa4
kbdmux_intr() at kbdmux_intr+0x43
kbdmux_kbd_intr() at kbdmux_kbd_intr+0x20
taskqueue_run() at taskqueue_run+0x94
ithread_loop() at ithread_loop+0xe0
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac266d30, rbp = 0 ---
db> alltrace
Tracing command sed pid 21498 tid 100148 td 0xffffff000423e9c0
sched_switch() at sched_switch+0x176
mi_switch() at mi_switch+0x231
sleepq_switch() at sleepq_switch+0xc7
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x2e3
pipe_read() at pipe_read+0x2bc
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x4c
read() at read+0x54
syscall() at syscall+0x1ce
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (3, FreeBSD ELF64, read), rip = 0x80071743c, rsp = 0x7fffffffe808, rbp = 0x7fffffffea40 ---
Tracing command sh pid 21497 tid 100064 td 0xffffff000417a000
sched_switch() at sched_switch+0x176
mi_switch() at mi_switch+0x231
sleepq_switch() at sleepq_switch+0xc7
--More-- leepq_timedwait() at sleepq_timedwait+0x3c
_sleep() at _sleep+0x314
vm_object_deallocate() at vm_object_deallocate+0x4d1
vm_map_delete() at vm_map_delete+0x19d
vm_map_remove() at vm_map_remove+0x59
exec_new_vmspace() at exec_new_vmspace+0x20b
exec_elf64_imgact() at exec_elf64_imgact+0x1bd
kern_execve() at kern_execve+0x31a
execve() at execve+0x3d
syscall() at syscall+0x1ce
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (59, FreeBSD ELF64, execve), rip = 0x80092aa6c, rsp = 0x7fffffffe5a8, rbp = 0x800b09588 ---
Tracing command sh pid 21496 tid 100180 td 0xffffff0008fbc340
sched_switch() at sched_switch+0x176
mi_switch() at mi_switch+0x231
sleepq_switch() at sleepq_switch+0xc7
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x2e3
--More-- ern_wait() at kern_wait+0x3cb
wait4() at wait4+0x35
syscall() at syscall+0x1ce
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (7, FreeBSD ELF64, wait4), rip = 0x80092aaac, rsp = 0x7fffffffe7c8, rbp = 0x30a ---
Tracing command make pid 21492 tid 100181 td 0xffffff0008fbc000
sched_switch() at sched_switch+0x176
mi_switch() at mi_switch+0x231
sleepq_switch() at sleepq_switch+0xc7
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x2e3
pipe_read() at pipe_read+0x2bc
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x4c
read() at read+0x54
syscall() at syscall+0x1ce
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (3, FreeBSD ELF64, read), rip = 0x43bd9c, rsp = 0x7fffffffd378, rbp = 0x7fffffffd390 ---
--More--
Tracing command sh pid 21139 tid 100120 td 0xffffff0004374000
sched_switch() at sched_switch+0x176
mi_switch() at mi_switch+0x231
sleepq_switch() at sleepq_switch+0xc7
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x2e3
kern_wait() at kern_wait+0x3cb
wait4() at wait4+0x35
syscall() at syscall+0x1ce
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (7, FreeBSD ELF64, wait4), rip = 0x80092aaac, rsp = 0x7fffffffe5b8, rbp = 0x30a ---
Tracing command make pid 21132 tid 100170 td 0xffffff0008ee39c0
sched_switch() at sched_switch+0x176
mi_switch() at mi_switch+0x231
sleepq_switch() at sleepq_switch+0xc7
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19
--More-- cv_timedwait_sig() at _cv_timedwait_sig+0x16a
kern_select() at kern_select+0x7df
select() at select+0x56
syscall() at syscall+0x1ce
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (93, FreeBSD ELF64, select), rip = 0x43bd1c, rsp = 0x7fffffffdc68, rbp = 0x80066a580 ---
Tracing command sh pid 21131 tid 100052 td 0xffffff0003be4000
sched_switch() at sched_switch+0x176
mi_switch() at mi_switch+0x231
sleepq_switch() at sleepq_switch+0xc7
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x2e3
kern_wait() at kern_wait+0x3cb
wait4() at wait4+0x35
syscall() at syscall+0x1ce
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (7, FreeBSD ELF64, wait4), rip = 0x80092aaac, rsp = 0x7fffffffe808, rbp = 0x30a ---
--More-- racing command make pid 4753 tid 100066 td 0xffffff0004179680
sched_switch() at sched_switch+0x176
mi_switch() at mi_switch+0x231
sleepq_switch() at sleepq_switch+0xc7
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x2e3
pipe_write() at pipe_write+0x476
dofilewrite() at dofilewrite+0x85
kern_writev() at kern_writev+0x4c
write() at write+0x54
syscall() at syscall+0x1ce
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (4, FreeBSD ELF64, write), rip = 0x43bd7c, rsp = 0x7fffffffdc88, rbp = 0x209 ---
Tracing command sh pid 4751 tid 100129 td 0xffffff00044339c0
sched_switch() at sched_switch+0x176
mi_switch() at mi_switch+0x231
sleepq_switch() at sleepq_switch+0xc7
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
--More-- leepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x2e3
kern_wait() at kern_wait+0x3cb
wait4() at wait4+0x35
syscall() at syscall+0x1ce
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (7, FreeBSD ELF64, wait4), rip = 0x80092aaac, rsp = 0x7fffffffe7f8, rbp = 0x30a ---
Tracing command make pid 4660 tid 100127 td 0xffffff0004435340
sched_switch() at sched_switch+0x176
mi_switch() at mi_switch+0x231
sleepq_switch() at sleepq_switch+0xc7
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19
_sleep() at _sleep+0x20a
ttysleep() at ttysleep+0x25
ttwrite() at ttwrite+0x2fd
giant_write() at giant_write+0x51
devfs_write_f() at devfs_write_f+0x9c
dofilewrite() at dofilewrite+0x85
--More-- ern_writev() at kern_writev+0x4c
write() at write+0x54
syscall() at syscall+0x1ce
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (4, FreeBSD ELF64, write), rip = 0x43bd7c, rsp = 0x7fffffffd428, rbp = 0x60 ---
Tracing command sh pid 778 tid 100094 td 0xffffff0004212680
sched_switch() at sched_switch+0x176
mi_switch() at mi_switch+0x231
sleepq_switch() at sleepq_switch+0xc7
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x2e3
kern_wait() at kern_wait+0x3cb
wait4() at wait4+0x35
syscall() at syscall+0x1ce
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (7, FreeBSD ELF64, wait4), rip = 0x80092aaac, rsp = 0x7fffffffe8d8, rbp = 0x30a ---
Tracing command bash pid 775 tid 100117 td 0xffffff00041dc9c0
--More-- ched_switch() at sched_switch+0x176
mi_switch() at mi_switch+0x231
sleepq_switch() at sleepq_switch+0xc7
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x2e3
kern_wait() at kern_wait+0x3cb
wait4() at wait4+0x35
syscall() at syscall+0x1ce
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (7, FreeBSD ELF64, wait4), rip = 0x800b7aaac, rsp = 0x7fffffffe9c8, rbp = 0x800e6c140 ---
Tracing command sshd pid 772 tid 100054 td 0xffffff00033c7680
sched_switch() at sched_switch+0x176
mi_switch() at mi_switch+0x231
turnstile_wait() at turnstile_wait+0x24d
_mtx_lock_sleep() at _mtx_lock_sleep+0xc2
_mtx_lock_flags() at _mtx_lock_flags+0xe1
rt_check() at rt_check+0x10e
arpresolve() at arpresolve+0x64
--More-- ther_output() at ether_output+0x308
ip_output() at ip_output+0xabc
tcp_output() at tcp_output+0xad6
tcp_usr_send() at tcp_usr_send+0x2e9
sosend_generic() at sosend_generic+0x347
soo_write() at soo_write+0x38
dofilewrite() at dofilewrite+0x85
kern_writev() at kern_writev+0x4c
write() at write+0x54
syscall() at syscall+0x1ce
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (4, FreeBSD ELF64, write), rip = 0x8016f541c, rsp = 0x7fffffffe4f8, rbp = 0x5337c0 ---
Tracing command bash pid 771 tid 100062 td 0xffffff000417a680
sched_switch() at sched_switch+0x176
mi_switch() at mi_switch+0x231
sleepq_switch() at sleepq_switch+0xc7
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x2e3
--More-- tysleep() at ttysleep+0x25
ttread() at ttread+0x306
giant_read() at giant_read+0x51
devfs_read_f() at devfs_read_f+0x72
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x4c
read() at read+0x54
syscall() at syscall+0x1ce
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (3, FreeBSD ELF64, read), rip = 0x800bf243c, rsp = 0x7fffffffe898, rbp = 0x7fffffffe8b7 ---
Tracing command getty pid 766 tid 100065 td 0xffffff00041799c0
sched_switch() at sched_switch+0x176
mi_switch() at mi_switch+0x231
sleepq_switch() at sleepq_switch+0xc7
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x2e3
ttysleep() at ttysleep+0x25
ttread() at ttread+0x306
--More-- iant_read() at giant_read+0x51
devfs_read_f() at devfs_read_f+0x72
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x4c
read() at read+0x54
syscall() at syscall+0x1ce
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (3, FreeBSD ELF64, read), rip = 0x80082343c, rsp = 0x7fffffffed98, rbp = 0 ---
Tracing command login pid 765 tid 100107 td 0xffffff0004295340
sched_switch() at sched_switch+0x176
mi_switch() at mi_switch+0x231
sleepq_switch() at sleepq_switch+0xc7
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x2e3
kern_wait() at kern_wait+0x3cb
wait4() at wait4+0x35
syscall() at syscall+0x1ce
Xfast_syscall() at Xfast_syscall+0xab
--More-- -- syscall (7, FreeBSD ELF64, wait4), rip = 0x8009c8aac, rsp = 0x7fffffffed08, rbp = 0 ---
Tracing command getty pid 764 tid 100095 td 0xffffff0004212340
sched_switch() at sched_switch+0x176
mi_switch() at mi_switch+0x231
sleepq_switch() at sleepq_switch+0xc7
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x2e3
ttysleep() at ttysleep+0x25
ttread() at ttread+0x306
giant_read() at giant_read+0x51
devfs_read_f() at devfs_read_f+0x72
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x4c
read() at read+0x54
syscall() at syscall+0x1ce
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (3, FreeBSD ELF64, read), rip = 0x80082343c, rsp = 0x7fffffffed98, rbp = 0 ---
--More-- racing command getty pid 763 tid 100118 td 0xffffff00041dc680
sched_switch() at sched_switch+0x176
mi_switch() at mi_switch+0x231
sleepq_switch() at sleepq_switch+0xc7
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x2e3
ttysleep() at ttysleep+0x25
ttread() at ttread+0x306
giant_read() at giant_read+0x51
devfs_read_f() at devfs_read_f+0x72
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x4c
read() at read+0x54
syscall() at syscall+0x1ce
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (3, FreeBSD ELF64, read), rip = 0x80082343c, rsp = 0x7fffffffed98, rbp = 0 ---
Tracing command cron pid 721 tid 100115 td 0xffffff0004262340
sched_switch() at sched_switch+0x176
--More-- i_switch() at mi_switch+0x231
sleepq_switch() at sleepq_switch+0xc7
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19
_sleep() at _sleep+0x20a
kern_nanosleep() at kern_nanosleep+0x118
nanosleep() at nanosleep+0x6e
syscall() at syscall+0x1ce
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (240, FreeBSD ELF64, nanosleep), rip = 0x80091303c, rsp = 0x7fffffffec28, rbp = 0x3c ---
Tracing command sendmail pid 715 tid 100114 td 0xffffff0004262680
sched_switch() at sched_switch+0x176
mi_switch() at mi_switch+0x231
sleepq_switch() at sleepq_switch+0xc7
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x2e3
kern_sigsuspend() at kern_sigsuspend+0xca
sigsuspend() at sigsuspend+0x34
--More-- yscall() at syscall+0x1ce
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (341, FreeBSD ELF64, sigsuspend), rip = 0x800d0f97c, rsp = 0x7fffffffda48, rbp = 0x1 ---
Tracing command sendmail pid 711 tid 100055 td 0xffffff00033c7340
sched_switch() at sched_switch+0x176
mi_switch() at mi_switch+0x231
sleepq_switch() at sleepq_switch+0xc7
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19
_cv_timedwait_sig() at _cv_timedwait_sig+0x16a
kern_select() at kern_select+0x7df
select() at select+0x56
syscall() at syscall+0x1ce
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (93, FreeBSD ELF64, select), rip = 0x800d9c3bc, rsp = 0x7fffffffcf48, rbp = 0x7fffffffd7f0 ---
Tracing command sshd pid 705 tid 100100 td 0xffffff00041e2000
sched_switch() at sched_switch+0x176
mi_switch() at mi_switch+0x231
--More-- leepq_switch() at sleepq_switch+0xc7
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_cv_wait_sig() at _cv_wait_sig+0x15f
kern_select() at kern_select+0x80b
select() at select+0x56
syscall() at syscall+0x1ce
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (93, FreeBSD ELF64, select), rip = 0x8016f53bc, rsp = 0x7fffffffe618, rbp = 0x2 ---
Tracing command syslogd pid 575 tid 100060 td 0xffffff000417b000
sched_switch() at sched_switch+0x176
mi_switch() at mi_switch+0x231
sleepq_switch() at sleepq_switch+0xc7
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_cv_wait_sig() at _cv_wait_sig+0x15f
kern_select() at kern_select+0x80b
select() at select+0x56
syscall() at syscall+0x1ce
--More-- fast_syscall() at Xfast_syscall+0xab
--- syscall (93, FreeBSD ELF64, select), rip = 0x8008263bc, rsp = 0x7fffffffdde8, rbp = 0x8 ---
Tracing command devd pid 529 tid 100076 td 0xffffff00041b5680
sched_switch() at sched_switch+0x176
mi_switch() at mi_switch+0x231
sleepq_switch() at sleepq_switch+0xc7
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_cv_wait_sig() at _cv_wait_sig+0x15f
kern_select() at kern_select+0x80b
select() at select+0x56
syscall() at syscall+0x1ce
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (93, FreeBSD ELF64, select), rip = 0x4388dc, rsp = 0x7fffffffe928, rbp = 0x7fffffffed50 ---
Tracing command dhclient pid 324 tid 100061 td 0xffffff000417a9c0
sched_switch() at sched_switch+0x176
mi_switch() at mi_switch+0x231
sleepq_switch() at sleepq_switch+0xc7
--More-- leepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19
_cv_timedwait_sig() at _cv_timedwait_sig+0x16a
poll() at poll+0x1e3
syscall() at syscall+0x1ce
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (209, FreeBSD ELF64, poll), rip = 0x8006d120c, rsp = 0x7fffffffed98, rbp = 0x144b118 ---
Tracing command dhclient pid 296 tid 100069 td 0xffffff00041759c0
sched_switch() at sched_switch+0x176
mi_switch() at mi_switch+0x231
sleepq_switch() at sleepq_switch+0xc7
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_cv_wait_sig() at _cv_wait_sig+0x15f
poll() at poll+0x567
syscall() at syscall+0x1ce
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (209, FreeBSD ELF64, poll), rip = 0x8006d120c, rsp = 0x7fffffffed98, rbp = 0x5 ---
--More-- racing command adjkerntz pid 162 tid 100075 td 0xffffff00041b59c0
sched_switch() at sched_switch+0x176
mi_switch() at mi_switch+0x231
sleepq_switch() at sleepq_switch+0xc7
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x2e3
kern_sigsuspend() at kern_sigsuspend+0xca
sigsuspend() at sigsuspend+0x34
syscall() at syscall+0x1ce
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (341, FreeBSD ELF64, sigsuspend), rip = 0x80068497c, rsp = 0x7fffffffed08, rbp = 0x7fffffffed80 ---
Tracing command schedcpu pid 51 tid 100058 td 0xffffff00040a0340
sched_switch() at sched_switch+0x176
mi_switch() at mi_switch+0x231
sleepq_switch() at sleepq_switch+0xc7
sleepq_timedwait() at sleepq_timedwait+0x3c
_sleep() at _sleep+0x314
schedcpu_thread() at schedcpu_thread+0x21e
--More-- ork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffaef6ed30, rbp = 0 ---
Tracing command softdepflush pid 50 tid 100059 td 0xffffff00040a0000
sched_switch() at sched_switch+0x176
mi_switch() at mi_switch+0x231
sleepq_switch() at sleepq_switch+0xc7
sleepq_timedwait() at sleepq_timedwait+0x3c
_sleep() at _sleep+0x314
softdep_flush() at softdep_flush+0x2d7
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffaef69d30, rbp = 0 ---
Tracing command vnlru pid 49 tid 100039 td 0xffffff00033c6000
sched_switch() at sched_switch+0x176
mi_switch() at mi_switch+0x231
sleepq_switch() at sleepq_switch+0xc7
sleepq_timedwait() at sleepq_timedwait+0x3c
--More-- sleep() at _sleep+0x314
vnlru_proc() at vnlru_proc+0x696
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffae340d30, rbp = 0 ---
Tracing command syncer pid 48 tid 100040 td 0xffffff00033c29c0
sched_switch() at sched_switch+0x176
mi_switch() at mi_switch+0x231
sleepq_switch() at sleepq_switch+0xc7
sleepq_wait() at sleepq_wait+0x44
_sleep() at _sleep+0x327
sched_sync() at sched_sync+0x53d
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffae33bd30, rbp = 0 ---
Tracing command bufdaemon pid 47 tid 100041 td 0xffffff00033c2680
sched_switch() at sched_switch+0x176
mi_switch() at mi_switch+0x231
--More-- leepq_switch() at sleepq_switch+0xc7
sleepq_timedwait() at sleepq_timedwait+0x3c
_sleep() at _sleep+0x314
buf_daemon() at buf_daemon+0x208
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffae336d30, rbp = 0 ---
Tracing command pagezero pid 46 tid 100042 td 0xffffff00033c2340
sched_switch() at sched_switch+0x176
mi_switch() at mi_switch+0x231
sleepq_switch() at sleepq_switch+0xc7
sleepq_timedwait() at sleepq_timedwait+0x3c
_sleep() at _sleep+0x314
vm_pagezero() at vm_pagezero+0x73
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffae331d30, rbp = 0 ---
Tracing command vmdaemon pid 45 tid 100043 td 0xffffff00033c2000
--More-- ched_switch() at sched_switch+0x176
mi_switch() at mi_switch+0x231
sleepq_switch() at sleepq_switch+0xc7
sleepq_wait() at sleepq_wait+0x44
_sleep() at _sleep+0x327
vm_daemon() at vm_daemon+0x4d
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffae32cd30, rbp = 0 ---
Tracing command pagedaemon pid 44 tid 100044 td 0xffffff00012529c0
sched_switch() at sched_switch+0x176
mi_switch() at mi_switch+0x231
sleepq_switch() at sleepq_switch+0xc7
sleepq_timedwait() at sleepq_timedwait+0x3c
_sleep() at _sleep+0x314
vm_pageout() at vm_pageout+0xacd
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac2fdd30, rbp = 0 ---
--More--
Tracing command ipmi0: kcs pid 43 tid 100045 td 0xffffff0003be69c0
sched_switch() at sched_switch+0x176
mi_switch() at mi_switch+0x231
sleepq_switch() at sleepq_switch+0xc7
sleepq_wait() at sleepq_wait+0x44
_cv_wait() at _cv_wait+0x14e
ipmi_dequeue_request() at ipmi_dequeue_request+0x67
kcs_loop() at kcs_loop+0x3d
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffaef19d30, rbp = 0 ---
Tracing command sctp_iterator pid 42 tid 100046 td 0xffffff0003be6680
sched_switch() at sched_switch+0x176
mi_switch() at mi_switch+0x231
sleepq_switch() at sleepq_switch+0xc7
sleepq_wait() at sleepq_wait+0x44
_sleep() at _sleep+0x327
sctp_iterator_thread() at sctp_iterator_thread+0x47
--More-- ork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffaef3cd30, rbp = 0 ---
Tracing command irq1: atkbd0 pid 41 tid 100047 td 0xffffff0003be6340
sched_switch() at sched_switch+0x176
mi_switch() at mi_switch+0x231
ithread_loop() at ithread_loop+0x39a
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffaef0fd30, rbp = 0 ---
Tracing command fdc0 pid 40 tid 100048 td 0xffffff0003be6000
sched_switch() at sched_switch+0x176
mi_switch() at mi_switch+0x231
sleepq_switch() at sleepq_switch+0xc7
sleepq_timedwait() at sleepq_timedwait+0x3c
_sleep() at _sleep+0x314
fdc_thread() at fdc_thread+0x91b
fork_exit() at fork_exit+0x12a
--More-- ork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffaef0ad30, rbp = 0 ---
Tracing command swi0: sio pid 39 tid 100030 td 0xffffff0001251680
sched_switch() at sched_switch+0x176
mi_switch() at mi_switch+0x231
ithread_loop() at ithread_loop+0x39a
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac2e4d30, rbp = 0 ---
Tracing command mpt_recovery0 pid 38 tid 100031 td 0xffffff0001251340
sched_switch() at sched_switch+0x176
mi_switch() at mi_switch+0x231
sleepq_switch() at sleepq_switch+0xc7
sleepq_wait() at sleepq_wait+0x44
_sleep() at _sleep+0x327
mpt_recovery_thread() at mpt_recovery_thread+0x259
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--More-- -- trap 0, rip = 0, rsp = 0xffffffffac2dfd30, rbp = 0 ---
Tracing command irq17: mpt0 pid 37 tid 100032 td 0xffffff0001251000
sched_switch() at sched_switch+0x176
mi_switch() at mi_switch+0x231
ithread_loop() at ithread_loop+0x39a
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac2dad30, rbp = 0 ---
Tracing command nfe1 taskq pid 36 tid 100033 td 0xffffff00012399c0
sched_switch() at sched_switch+0x176
mi_switch() at mi_switch+0x231
sleepq_switch() at sleepq_switch+0xc7
sleepq_wait() at sleepq_wait+0x44
msleep_spin() at msleep_spin+0x1fb
taskqueue_thread_loop() at taskqueue_thread_loop+0x45
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac2d5d30, rbp = 0 ---
--More--
Tracing command nfe0 taskq pid 35 tid 100034 td 0xffffff0001239680
sched_switch() at sched_switch+0x176
mi_switch() at mi_switch+0x231
sleepq_switch() at sleepq_switch+0xc7
sleepq_wait() at sleepq_wait+0x44
msleep_spin() at msleep_spin+0x1fb
taskqueue_thread_loop() at taskqueue_thread_loop+0x45
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac2d0d30, rbp = 0 ---
Tracing command irq20: atapci2 pid 34 tid 100035 td 0xffffff00033c7000
fork_trampoline() at fork_trampoline
Tracing command irq23: atapci1 pid 33 tid 100036 td 0xffffff00033c69c0
sched_switch() at sched_switch+0x176
mi_switch() at mi_switch+0x231
ithread_loop() at ithread_loop+0x39a
fork_exit() at fork_exit+0x12a
--More-- ork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffae34fd30, rbp = 0 ---
Tracing command irq15: ata1 pid 32 tid 100037 td 0xffffff00033c6680
fork_trampoline() at fork_trampoline
Tracing command irq14: ata0 pid 31 tid 100038 td 0xffffff00033c6340
sched_switch() at sched_switch+0x176
mi_switch() at mi_switch+0x231
ithread_loop() at ithread_loop+0x39a
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffae345d30, rbp = 0 ---
Tracing command usb1 pid 30 tid 100022 td 0xffffff0001238340
sched_switch() at sched_switch+0x176
mi_switch() at mi_switch+0x231
sleepq_switch() at sleepq_switch+0xc7
sleepq_timedwait() at sleepq_timedwait+0x3c
_sleep() at _sleep+0x314
--More-- sb_event_thread() at usb_event_thread+0xaa
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac2b7d30, rbp = 0 ---
Tracing command irq22: ehci0 pid 29 tid 100023 td 0xffffff0001238000
fork_trampoline() at fork_trampoline
Tracing command usbtask-dr pid 28 tid 100024 td 0xffffff00011119c0
sched_switch() at sched_switch+0x176
mi_switch() at mi_switch+0x231
sleepq_switch() at sleepq_switch+0xc7
sleepq_wait() at sleepq_wait+0x44
_sleep() at _sleep+0x327
usb_task_thread() at usb_task_thread+0x94
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac2add30, rbp = 0 ---
Tracing command usbtask-hc pid 27 tid 100025 td 0xffffff0001111680
--More-- ched_switch() at sched_switch+0x176
mi_switch() at mi_switch+0x231
sleepq_switch() at sleepq_switch+0xc7
sleepq_wait() at sleepq_wait+0x44
_sleep() at _sleep+0x327
usb_task_thread() at usb_task_thread+0x94
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac2a8d30, rbp = 0 ---
Tracing command usb0 pid 26 tid 100026 td 0xffffff0001252680
sched_switch() at sched_switch+0x176
mi_switch() at mi_switch+0x231
sleepq_switch() at sleepq_switch+0xc7
sleepq_timedwait() at sleepq_timedwait+0x3c
_sleep() at _sleep+0x314
usb_event_thread() at usb_event_thread+0xaa
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac2f8d30, rbp = 0 ---
--More--
Tracing command irq21: ohci0+ pid 25 tid 100027 td 0xffffff0001252340
sched_switch() at sched_switch+0x176
mi_switch() at mi_switch+0x231
ithread_loop() at ithread_loop+0x39a
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac2f3d30, rbp = 0 ---
Tracing command irq9: acpi0 pid 24 tid 100028 td 0xffffff0001252000
fork_trampoline() at fork_trampoline
Tracing command thread taskq pid 23 tid 100029 td 0xffffff00012519c0
sched_switch() at sched_switch+0x176
mi_switch() at mi_switch+0x231
sleepq_switch() at sleepq_switch+0xc7
sleepq_wait() at sleepq_wait+0x44
_sleep() at _sleep+0x327
taskqueue_thread_loop() at taskqueue_thread_loop+0x71
fork_exit() at fork_exit+0x12a
--More-- ork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac2e9d30, rbp = 0 ---
Tracing command swi5: + pid 22 tid 100015 td 0xffffff0001110340
sched_switch() at sched_switch+0x176
mi_switch() at mi_switch+0x231
turnstile_wait() at turnstile_wait+0x24d
_mtx_lock_sleep() at _mtx_lock_sleep+0xc2
_mtx_lock_flags() at _mtx_lock_flags+0xe1
rt_setgate() at rt_setgate+0x26e
rtredirect() at rtredirect+0x250
icmp_input() at icmp_input+0x537
ip_input() at ip_input+0xc0
ether_demux() at ether_demux+0x1d9
ether_input() at ether_input+0x1d6
nfe_int_task() at nfe_int_task+0x949
taskqueue_run() at taskqueue_run+0x94
ithread_loop() at ithread_loop+0xe0
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--More-- -- trap 0, rip = 0, rsp = 0xffffffffac28fd30, rbp = 0 ---
Tracing command swi2: cambio pid 21 tid 100016 td 0xffffff0001110000
sched_switch() at sched_switch+0x176
mi_switch() at mi_switch+0x231
ithread_loop() at ithread_loop+0x39a
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac28ad30, rbp = 0 ---
Tracing command xpt_thrd pid 9 tid 100017 td 0xffffff00010eb9c0
sched_switch() at sched_switch+0x176
mi_switch() at mi_switch+0x231
sleepq_switch() at sleepq_switch+0xc7
sleepq_wait() at sleepq_wait+0x44
_sleep() at _sleep+0x327
xpt_scanner_thread() at xpt_scanner_thread+0x3a
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac284d30, rbp = 0 ---
--More--
Tracing command acpi_task_2 pid 8 tid 100018 td 0xffffff0001239340
sched_switch() at sched_switch+0x176
mi_switch() at mi_switch+0x231
sleepq_switch() at sleepq_switch+0xc7
sleepq_wait() at sleepq_wait+0x44
_sleep() at _sleep+0x327
taskqueue_thread_loop() at taskqueue_thread_loop+0x71
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac2cbd30, rbp = 0 ---
Tracing command acpi_task_1 pid 7 tid 100019 td 0xffffff0001239000
sched_switch() at sched_switch+0x176
mi_switch() at mi_switch+0x231
sleepq_switch() at sleepq_switch+0xc7
sleepq_wait() at sleepq_wait+0x44
_sleep() at _sleep+0x327
taskqueue_thread_loop() at taskqueue_thread_loop+0x71
fork_exit() at fork_exit+0x12a
--More-- ork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac2c6d30, rbp = 0 ---
Tracing command acpi_task_0 pid 6 tid 100020 td 0xffffff00012389c0
sched_switch() at sched_switch+0x176
mi_switch() at mi_switch+0x231
sleepq_switch() at sleepq_switch+0xc7
sleepq_wait() at sleepq_wait+0x44
_sleep() at _sleep+0x327
taskqueue_thread_loop() at taskqueue_thread_loop+0x71
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac2c1d30, rbp = 0 ---
Tracing command kqueue taskq pid 5 tid 100021 td 0xffffff0001238680
sched_switch() at sched_switch+0x176
mi_switch() at mi_switch+0x231
sleepq_switch() at sleepq_switch+0xc7
sleepq_wait() at sleepq_wait+0x44
_sleep() at _sleep+0x327
--More-- askqueue_thread_loop() at taskqueue_thread_loop+0x71
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac2bcd30, rbp = 0 ---
Tracing command swi6: task queue pid 20 tid 100009 td 0xffffff00010e8680
sched_switch() at sched_switch+0x176
mi_switch() at mi_switch+0x231
ithread_loop() at ithread_loop+0x39a
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac26bd30, rbp = 0 ---
Tracing command swi6: Giant taskq pid 19 tid 100010 td 0xffffff00010e8340
kdb_enter() at kdb_enter+0x31
scgetc() at scgetc+0x461
sckbdevent() at sckbdevent+0xa4
kbdmux_intr() at kbdmux_intr+0x43
kbdmux_kbd_intr() at kbdmux_kbd_intr+0x20
taskqueue_run() at taskqueue_run+0x94
--More-- thread_loop() at ithread_loop+0xe0
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac266d30, rbp = 0 ---
Tracing command yarrow pid 18 tid 100011 td 0xffffff0001111340
sched_switch() at sched_switch+0x176
mi_switch() at mi_switch+0x231
sleepq_switch() at sleepq_switch+0xc7
sleepq_timedwait() at sleepq_timedwait+0x3c
_sleep() at _sleep+0x314
random_kthread() at random_kthread+0x191
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac2a3d30, rbp = 0 ---
Tracing command g_down pid 4 tid 100012 td 0xffffff0001111000
sched_switch() at sched_switch+0x176
mi_switch() at mi_switch+0x231
sleepq_switch() at sleepq_switch+0xc7
--More-- leepq_timedwait() at sleepq_timedwait+0x3c
_sleep() at _sleep+0x314
g_io_schedule_down() at g_io_schedule_down+0x20f
g_down_procbody() at g_down_procbody+0x78
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac29ed30, rbp = 0 ---
Tracing command g_up pid 3 tid 100013 td 0xffffff00011109c0
sched_switch() at sched_switch+0x176
mi_switch() at mi_switch+0x231
sleepq_switch() at sleepq_switch+0xc7
sleepq_timedwait() at sleepq_timedwait+0x3c
_sleep() at _sleep+0x314
g_io_schedule_up() at g_io_schedule_up+0x154
g_up_procbody() at g_up_procbody+0x78
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac299d30, rbp = 0 ---
--More-- racing command g_event pid 2 tid 100014 td 0xffffff0001110680
sched_switch() at sched_switch+0x176
mi_switch() at mi_switch+0x231
sleepq_switch() at sleepq_switch+0xc7
sleepq_timedwait() at sleepq_timedwait+0x3c
_sleep() at _sleep+0x314
g_event_procbody() at g_event_procbody+0xaa
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac294d30, rbp = 0 ---
Tracing command swi3: vm pid 17 tid 100004 td 0xffffff00010e5000
fork_trampoline() at fork_trampoline
Tracing command swi4: clock sio pid 16 tid 100005 td 0xffffff00010eb680
sched_switch() at sched_switch+0x176
mi_switch() at mi_switch+0x231
turnstile_wait() at turnstile_wait+0x24d
_mtx_lock_sleep() at _mtx_lock_sleep+0xc2
_mtx_lock_flags() at _mtx_lock_flags+0xe1
--More-- cp_timer_rexmt() at tcp_timer_rexmt+0x5c
softclock() at softclock+0x28d
ithread_loop() at ithread_loop+0xe0
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac27fd30, rbp = 0 ---
Tracing command swi1: net pid 15 tid 100006 td 0xffffff00010eb340
sched_switch() at sched_switch+0x176
mi_switch() at mi_switch+0x231
ithread_loop() at ithread_loop+0x39a
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac27ad30, rbp = 0 ---
Tracing command idle: cpu0 pid 14 tid 100007 td 0xffffff00010eb000
sched_switch() at sched_switch+0x176
mi_switch() at mi_switch+0x231
maybe_preempt() at maybe_preempt+0xf3
sched_add() at sched_add+0x1ce
--More-- ntr_event_schedule_thread() at intr_event_schedule_thread+0xb5
intr_execute_handlers() at intr_execute_handlers+0xe1
Xapic_isr2() at Xapic_isr2+0x7f
--- interrupt, rip = 0xffffffff806cab26, rsp = 0xffffffffac275bc0, rbp = 0xffffffffac275bd0 ---
acpi_cpu_c1() at acpi_cpu_c1+0x6
acpi_cpu_idle() at acpi_cpu_idle+0x1a4
sched_idletd() at sched_idletd+0x35
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac275d30, rbp = 0 ---
Tracing command idle: cpu1 pid 13 tid 100008 td 0xffffff00010e89c0
cpustop_handler() at cpustop_handler+0x40
ipi_nmi_handler() at ipi_nmi_handler+0x30
trap() at trap+0x378
nmi_calltrap() at nmi_calltrap+0x8
--- trap 0x13, rip = 0xffffffff806cab26, rsp = 0xffffffffab790ff0, rbp = 0xffffffffac270bd0 ---
acpi_cpu_c1() at acpi_cpu_c1+0x6
acpi_cpu_idle() at acpi_cpu_idle+0x1a4
sched_idletd() at sched_idletd+0x35
--More-- ork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac270d30, rbp = 0 ---
Tracing command idle: cpu2 pid 12 tid 100000 td 0xffffff00010e8000
cpustop_handler() at cpustop_handler+0x40
ipi_nmi_handler() at ipi_nmi_handler+0x30
trap() at trap+0x378
nmi_calltrap() at nmi_calltrap+0x8
--- trap 0x13, rip = 0xffffffff806cab26, rsp = 0xffffffffab795ff0, rbp = 0xffffffffac261bd0 ---
acpi_cpu_c1() at acpi_cpu_c1+0x6
acpi_cpu_idle() at acpi_cpu_idle+0x1a4
sched_idletd() at sched_idletd+0x35
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac261d30, rbp = 0 ---
Tracing command idle: cpu3 pid 11 tid 100001 td 0xffffff00010e59c0
cpustop_handler() at cpustop_handler+0x40
ipi_nmi_handler() at ipi_nmi_handler+0x30
--More-- rap() at trap+0x378
nmi_calltrap() at nmi_calltrap+0x8
--- trap 0x13, rip = 0xffffffff806cab26, rsp = 0xffffffffab79aff0, rbp = 0xffffffffac25cbd0 ---
acpi_cpu_c1() at acpi_cpu_c1+0x6
acpi_cpu_idle() at acpi_cpu_idle+0x1a4
sched_idletd() at sched_idletd+0x35
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac25cd30, rbp = 0 ---
Tracing command init pid 1 tid 100002 td 0xffffff00010e5680
sched_switch() at sched_switch+0x176
mi_switch() at mi_switch+0x231
sleepq_switch() at sleepq_switch+0xc7
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x2e3
kern_wait() at kern_wait+0x3cb
wait4() at wait4+0x35
syscall() at syscall+0x1ce
--More-- fast_syscall() at Xfast_syscall+0xab
--- syscall (7, FreeBSD ELF64, wait4), rip = 0x40bf5c, rsp = 0x7fffffffe8e8, rbp = 0x7fffffffed70 ---
Tracing command audit pid 10 tid 100003 td 0xffffff00010e5340
sched_switch() at sched_switch+0x176
mi_switch() at mi_switch+0x231
sleepq_switch() at sleepq_switch+0xc7
sleepq_wait() at sleepq_wait+0x44
_cv_wait() at _cv_wait+0x14e
audit_worker() at audit_worker+0x86
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac252d30, rbp = 0 ---
Tracing command swapper pid 0 tid 0 td 0xffffffff80a1ad20
sched_switch() at sched_switch+0x176
mi_switch() at mi_switch+0x231
scheduler() at scheduler+0x3c1
mi_startup() at mi_startup+0x59
btext() at btext+0x2c
--More-- b> panic
panic: from debugger
cpuid = 0
Uptime: 8m24s
Physical memory: 4083 MB
Dumping 303 MB: 288 272 256 240 224 208 192 176 160 144 128 112 96 80 64 48 32 16
Dump complete
Automatic reboot in 15 seconds - press a key on the console to abort
--> Press a key on the console to reboot,
--> or switch off the system now.
~
>How-To-Repeat:
I am not sure if this issue can be easily reproduced on a different setting.
>Fix:
None.
>Release-Note:
>Audit-Trail:
>Unformatted:
More information about the freebsd-bugs
mailing list