kernel core dump hangs with the SMP kernel on dual cpu board (FreeBSD 6.2)

Sat Jul 7 05:01:05 UTC 2007

Hello,

I am attaching more debugs. After core dump hangs, I am able to get 

into ddb. But the trace for the thread where core dump is running

does not look right.

I was able to isolate the dumper function - ad_dump

where it seems to be hanging.

Hopefully these debugs will give more clues. Please advise on what 

more to look for. I am new to SMP and still coming up to speed.

The following debugs are taken after core dump hangs:

   a) show pcpu

   b) bt 100054 -- 100054 is the thread ID, which does the core dump

   c) alltrace  (in attached file "alltrace.txt")

   d) step and next command in debug

a) ---

db> show pcpu

cpuid        = 1

curthread    = 0xc9450d80: pid 16 "swi6: Giant taskq"

curpcb       = 0xe7fe9d90

fpcurthread  = none

idlethread   = 0xc944f600: pid 10 "idle: cpu1"

APIC ID      = 1

currentldt   = 0x50

spin locks held:

db> show pcpu 0

cpuid        = 0

curthread    = 0xc96eb000: pid 894 "disablemeta"

curpcb       = 0xe9b4fd90

fpcurthread  = none

idlethread   = 0xc944f780: pid 11 "idle: cpu0"

APIC ID      = 0

currentldt   = 0x50

spin locks held:

db> 

b) ---

db> bt 100054

Tracing pid 894 tid 100054 td 0xc96eb000

sched_switch(3225858983,10,0,3382180240,3377804288,...) at
sched_switch+331

_end(53,128,606517178,0,0,...) at 3920951848

_end(3377796632,3379230208,3379230476,3377796608,3379229696,...) at
3377796608

_end(3228796064,3228796064,3228796064,3228796064,3228796064,...) at
3379230208

null_method(0,0,0,0,0,...) at null_method

db>

c) ---

Please see attached file "alltrace.txt" for details.

d) trace ---

db> trace

Tracing pid 943 tid 100080 td 0xc998d000

kdb_enter(3228283314) at kdb_enter+43

panic(3390017776,3237235458,3956644984,21554,3389785424,...) at
panic+295

mtpioctl(3389997312,3237235458,3389811456,3,3382235136,...) at
mtpioctl+193

devfs_ioctl_f(3379694976,3237235458,3389811456,3376546688,3382235136) at
devfs_5

ioctl(3382235136,3956645124) at ioctl+874

syscall(59,59,59,3216893976,134516360,...) at syscall+559

Xint0x80_syscall() at Xint0x80_syscall+31

--- syscall (54, FreeBSD ELF32, ioctl), eip = 672367343, esp =
3216893900, ebp -

db> s

[thread pid 943 tid 100080 ]

Stopped at      kdb_enter+44:   leave

db> s

[thread pid 943 tid 100080 ]

Stopped at      kdb_enter+45:   ret

db> s

[thread pid 943 tid 100080 ]

Stopped at      panic+295:      addl    $4,%esp

db> s

[thread pid 943 tid 100080 ]

Stopped at      panic+298:      pushl   $561

db> s

[thread pid 943 tid 100080 ]

Stopped at      panic+303:      pushl   $3228283350

db> s

[thread pid 943 tid 100080 ]

Stopped at      panic+308:      pushl   $0

db> s

[thread pid 943 tid 100080 ]

Stopped at      panic+310:      pushl   $3228776128

db> s

[thread pid 943 tid 100080 ]

Stopped at      panic+315:      call    _mtx_lock_spin_flags

db> s

[thread pid 943 tid 100080 ]

Stopped at      _mtx_lock_spin_flags:   pushl   %ebp

db> s

[thread pid 943 tid 100080 ]

Stopped at      _mtx_lock_spin_flags+1: movl    %esp,%ebp

db> s

[thread pid 943 tid 100080 ]

Stopped at      _mtx_lock_spin_flags+3: pushl   %edi

db> s

[thread pid 943 tid 100080 ]

Stopped at      _mtx_lock_spin_flags+4: pushl   %esi

db> s

[thread pid 943 tid 100080 ]

Stopped at      _mtx_lock_spin_flags+5: pushl   %ebx

db> s

[thread pid 943 tid 100080 ]

Stopped at      _mtx_lock_spin_flags+6: movl    8(%ebp),%esi

db> s

[thread pid 943 tid 100080 ]

Stopped at      _mtx_lock_spin_flags+9: movl    20(%ebp),%edi

db> s

[thread pid 943 tid 100080 ]

Stopped at      _mtx_lock_spin_flags+12:        pushl   %edi

db> s

[thread pid 943 tid 100080 ]

Stopped at      _mtx_lock_spin_flags+13:        pushl   16(%ebp)

db> s

[thread pid 943 tid 100080 ]

Stopped at      _mtx_lock_spin_flags+16:        movl    12(%ebp),%eax

db> s

[thread pid 943 tid 100080 ]

Stopped at      _mtx_lock_spin_flags+19:        orl     $9,%eax

db> s

[thread pid 943 tid 100080 ]

Stopped at      _mtx_lock_spin_flags+22:        pushl   %eax

db> s

[thread pid 943 tid 100080 ]

Stopped at      _mtx_lock_spin_flags+23:        pushl   %esi

db> s

[thread pid 943 tid 100080 ]

Stopped at      _mtx_lock_spin_flags+24:        call
witness_checkorder

db> n

After 22 instructions (0 loads, 0 stores),

[thread pid 943 tid 100080 ]

Stopped at      witness_checkorder+1259:        ret

db> n

KDB: stack backtrace:

hangs here ...

Thanks,

Yong Rao

Force10 Networks Inc.

408 571 3617

-----Original Message-----
From: Yong Rao 
Sent: Wednesday, June 20, 2007 4:54 PM
To: freebsd-smp at freebsd.org
Cc: Jagjit Choudhary
Subject: kernel core dump hangs with the SMP kernel (FreeBSD 6.2)

Hello,

I am able to isolate the problem to kernel configurations which

have SMP enabled when used with 2 cpus. With ONE cpu the core dump works
ok.

I built the kernel with GENERIC, and deliberately crash the kernel (for
testing purpose). The core dump works fine.

Only added the "options SMP" and crashed the kernel, then prior to any
pages being dumped out, it hangs there.

Has someone successfully core dumped on a system using SMP kernel with
multiple CPUs?

I tried on two different boxes (different motherboards, CPUs and hard
disks). Both got failed.

I tried to enable the DDB, but don't know what to look for when it goes

into ddb. Appreciate any pointers.

a) The CPU information is 

CPU: Dual Core AMD Opteron(tm) Processor 280 (2405.47-MHz 686-class CPU)

  Origin = "AuthenticAMD"  Id = 0x20f12  Stepping = 2

Features=0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE
,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,S

SE,SSE2,HTT>

  Features2=0x1<SSE3>

  AMD Features=0xe2500800<SYSCALL,NX,MMX+,FFXSR,LM,3DNow+,3DNow>

  AMD Features2=0x3<LAHF,CMP>

  Cores per package: 2

b) We also tried on another mother board, which has 2 CPUs. The CPU
information is below.

CPU: Intel(R) Xeon(TM) CPU 2.80GHz (2800.11-MHz 686-class CPU)

  Origin = "GenuineIntel"  Id = 0xf29  Stepping = 9

Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE
,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>

  Features2=0x4400<CNTX-ID,<b14>>

real memory  = 2147418112 (2047 MB)

avail memory = 2096300032 (1999 MB)

ACPI APIC Table: <A M I  OEMAPIC >

FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs

 cpu0 (BSP): APIC ID:  0

 cpu1 (AP): APIC ID:  6

c) The following are the prints when the dump hung.

mem dump: start address = 0x4352, len=0x30

Fatal trap 12: page fault while in kernel mode

cpuid = 1; apic id = 01

fault virtual address   = 0x4352

fault code              = supervisor read, page not present

instruction pointer     = 0x20:0xc9e9fc92

stack pointer           = 0x28:0xebdbdbdc

frame pointer           = 0x28:0xebdbdbf8

code segment            = base 0x0, limit 0xfffff, type 0x1b

                        = DPL 0, pres 1, def32 1, gran 1

processor eflags        = interrupt enabled, resume, IOPL = 0

current process         = 74231 (pnicdbg)

trap number             = 12

panic: page fault

cpuid = 1

Uptime: 1d18h27m42s

Dumping 4030 MB (2 chunks)

  chunk 0: 1MB (154 pages) ... ok

  chunk 1: 4031MB (1031776 pages)   (stopped and hung here)

Thanks,

Yong

-------------- next part --------------
db> alltrace

Tracing command disablemeta pid 894 tid 100054 td 0xc96eb000

sched_switch(3225858983,10,0,3382180240,3377804288,...) at sched_switch+331

_end(53,128,606517178,0,0,...) at 3920951848

_end(3377796632,3379230208,3379230476,3377796608,3379229696,...) at 3377796608

_end(3228796064,3228796064,3228796064,3228796064,3228796064,...) at 3379230208

null_method(0,0,0,0,0,...) at null_method

Tracing command bash pid 884 tid 100058 td 0xc9991c00

sched_switch(3382254592,0,1) at sched_switch+331

mi_switch(1,0,0,3956742568,3226748917,...) at mi_switch+426

sleepq_switch(3377983504) at sleepq_switch+113

sleepq_wait_sig(3377983504) at sleepq_wait_sig+29

msleep(3377983504,0,345,3228302894,0) at msleep+510

ttysleep(3377983488,3377983504,345,3228302894,0,...) at ttysleep+33

ttread(3377983488,3956743356,0) at ttread+1167

ttyread(3377918464,3956743356,0,3228776192,0,...) at ttyread+47

giant_read(3377918464,3956743356,0) at giant_read+45

devfs_read_f(3379716600,3956743356,3387476352,0,3382254592) at devfs_read_f+98

dofileread(3382254592,0,3379716600,3956743356,4294967295,...) at dofileread+133

kern_readv(3382254592,0,3956743356,3217025183,1,...) at kern_readv+54

read(3382254592,3956743428) at read+69

syscall(59,59,59,0,3217025183,...) at syscall+559

Xint0x80_syscall() at Xint0x80_syscall+31

--- syscall (3, FreeBSD ELF32, read), eip = 135257039, esp = 3217025148, ebp = -

Tracing command bash pid 878 tid 100071 td 0xc96ea000

sched_switch(3379470336,0,1) at sched_switch+331

mi_switch(1,0,0,3920829452,3226748917,...) at mi_switch+426

sleepq_switch(3382776344) at sleepq_switch+113

sleepq_wait_sig(3382776344) at sleepq_wait_sig+29

msleep(3382776344,3382776448,348,3228297560,0) at msleep+510

kern_wait(3379470336,4294967295,3920829572,6,0) at kern_wait+1699

wait4(3379470336,3920829700) at wait4+42

syscall(59,59,59,894,0,...) at syscall+559

Xint0x80_syscall() at Xint0x80_syscall+31

--- syscall (7, FreeBSD ELF32, wait4), eip = 135042415, esp = 3217025340, ebp =-

Tracing command login pid 866 tid 100083 td 0xc9992780

sched_switch(3382257536,0,1) at sched_switch+331

mi_switch(1,0,0,3956829196,3226748917,...) at mi_switch+426

sleepq_switch(3385359504) at sleepq_switch+113

sleepq_wait_sig(3385359504) at sleepq_wait_sig+29

msleep(3385359504,3385359608,348,3228297560,0) at msleep+510

kern_wait(3382257536,884,3956829316,0,0) at kern_wait+1699

wait4(3382257536,3956829444) at wait4+42

syscall(59,59,59,582,1,...) at syscall+559

Xint0x80_syscall() at Xint0x80_syscall+31

--- syscall (7, FreeBSD ELF32, wait4), eip = 672093903, esp = 3217026396, ebp =-

Tracing command getty pid 865 tid 100090 td 0xc9c8c000

sched_switch(3385376768,0,1) at sched_switch+331

mi_switch(1,0,0,3957242260,3226748917,...) at mi_switch+426

sleepq_switch(3377985552) at sleepq_switch+113

sleepq_wait_sig(3377985552) at sleepq_wait_sig+29

msleep(3377985552,0,345,3228302894,0) at msleep+510

ttysleep(3377985536,3377985552,345,3228302894,0,...) at ttysleep+33

ttread(3377985536,3957243068,0) at ttread+1167

ttyread(3378041344,3957243068,0,3957242900,3226488553,...) at ttyread+47

scread(3378041344,3957243068,0,3228776192,0,...) at scread+34

giant_read(3378041344,3957243068,0) at giant_read+45

devfs_read_f(3379723360,3957243068,3376727936,0,3385376768) at devfs_read_f+98

dofileread(3385376768,0,3379723360,3957243068,4294967295,...) at dofileread+133

kern_readv(3385376768,0,3957243068,3217026591,1,...) at kern_readv+54

read(3385376768,3957243140) at read+69

syscall(59,59,59,0,134538624,...) at syscall+559

Xint0x80_syscall() at Xint0x80_syscall+31

--- syscall (3, FreeBSD ELF32, read), eip = 672428879, esp = 3217026556, ebp = -

Tracing command getty pid 864 tid 100068 td 0xc96ea480

sched_switch(3379471488,0,1) at sched_switch+331

mi_switch(1,0,0,3920865684,3226748917,...) at mi_switch+426

sleepq_switch(3377979408) at sleepq_switch+113

sleepq_wait_sig(3377979408) at sleepq_wait_sig+29

msleep(3377979408,0,345,3228302894,0) at msleep+510

ttysleep(3377979392,3377979408,345,3228302894,0,...) at ttysleep+33

ttread(3377979392,3920866492,0) at ttread+1167

ttyread(3378041088,3920866492,0,3920866324,3226488553,...) at ttyread+47

scread(3378041088,3920866492,0,3228776192,0,...) at scread+34

giant_read(3378041088,3920866492,0) at giant_read+45

devfs_read_f(3382342424,3920866492,3376727936,0,3379471488) at devfs_read_f+98

dofileread(3379471488,0,3382342424,3920866492,4294967295,...) at dofileread+133

kern_readv(3379471488,0,3920866492,3217026591,1,...) at kern_readv+54

read(3379471488,3920866564) at read+69

syscall(59,59,59,0,134538624,...) at syscall+559

Xint0x80_syscall() at Xint0x80_syscall+31

--- syscall (3, FreeBSD ELF32, read), eip = 672428879, esp = 3217026556, ebp = -

Tracing command getty pid 863 tid 100042 td 0xc9566480

sched_switch(3377882240,0,1) at sched_switch+331

mi_switch(1,0,0,3920550292,3226748917,...) at mi_switch+426

sleepq_switch(3378064400) at sleepq_switch+113

sleepq_wait_sig(3378064400) at sleepq_wait_sig+29

msleep(3378064400,0,345,3228302894,0) at msleep+510

ttysleep(3378064384,3378064400,345,3228302894,0,...) at ttysleep+33

ttread(3378064384,3920551100,0) at ttread+1167

ttyread(3378040832,3920551100,0,3920550932,3226488553,...) at ttyread+47

scread(3378040832,3920551100,0,3228776192,0,...) at scread+34

giant_read(3378040832,3920551100,0) at giant_read+45

devfs_read_f(3382344728,3920551100,3376727936,0,3377882240) at devfs_read_f+98

dofileread(3377882240,0,3382344728,3920551100,4294967295,...) at dofileread+133

kern_readv(3377882240,0,3920551100,3217026591,1,...) at kern_readv+54

read(3377882240,3920551172) at read+69

syscall(59,59,59,0,134538624,...) at syscall+559

Xint0x80_syscall() at Xint0x80_syscall+31

--- syscall (3, FreeBSD ELF32, read), eip = 672428879, esp = 3217026556, ebp = -

Tracing command getty pid 862 tid 100072 td 0xc9a12900

sched_switch(3382782208,0,1) at sched_switch+331

mi_switch(1,0,0,3957016980,3226748917,...) at mi_switch+426

sleepq_switch(3378063376) at sleepq_switch+113

sleepq_wait_sig(3378063376) at sleepq_wait_sig+29

msleep(3378063376,0,345,3228302894,0) at msleep+510

ttysleep(3378063360,3378063376,345,3228302894,0,...) at ttysleep+33

ttread(3378063360,3957017788,0) at ttread+1167

ttyread(3377924352,3957017788,0,3957017620,3226488553,...) at ttyread+47

scread(3377924352,3957017788,0,3228776192,0,...) at scread+34

giant_read(3377924352,3957017788,0) at giant_read+45

devfs_read_f(3382342640,3957017788,3376727936,0,3382782208) at devfs_read_f+98

dofileread(3382782208,0,3382342640,3957017788,4294967295,...) at dofileread+133

kern_readv(3382782208,0,3957017788,3217026591,1,...) at kern_readv+54

read(3382782208,3957017860) at read+69

syscall(59,59,59,0,134538624,...) at syscall+559

Xint0x80_syscall() at Xint0x80_syscall+31

--- syscall (3, FreeBSD ELF32, read), eip = 672428879, esp = 3217026556, ebp = -

Tracing command getty pid 861 tid 100079 td 0xc9992d80

sched_switch(3382259072,0,1) at sched_switch+331

mi_switch(1,0,0,3956877716,3226748917,...) at mi_switch+426

sleepq_switch(3378059280) at sleepq_switch+113

sleepq_wait_sig(3378059280) at sleepq_wait_sig+29

msleep(3378059280,0,345,3228302894,0) at msleep+510

ttysleep(3378059264,3378059280,345,3228302894,0,...) at ttysleep+33

ttread(3378059264,3956878524,0) at ttread+1167

ttyread(3377924608,3956878524,0,3956878356,3226488553,...) at ttyread+47

scread(3377924608,3956878524,0,3228776192,0,...) at scread+34

giant_read(3377924608,3956878524,0) at giant_read+45

devfs_read_f(3379719696,3956878524,3376727936,0,3382259072) at devfs_read_f+98

dofileread(3382259072,0,3379719696,3956878524,4294967295,...) at dofileread+133

kern_readv(3382259072,0,3956878524,3217026591,1,...) at kern_readv+54

read(3382259072,3956878596) at read+69

syscall(59,59,59,0,134538624,...) at syscall+559

Xint0x80_syscall() at Xint0x80_syscall+31

--- syscall (3, FreeBSD ELF32, read), eip = 672428879, esp = 3217026556, ebp = -

Tracing command getty pid 860 tid 100057 td 0xc9991d80

sched_switch(3382254976,0,1) at sched_switch+331

mi_switch(1,0,0,3956754836,3226748917,...) at mi_switch+426

sleepq_switch(3378058256) at sleepq_switch+113

sleepq_wait_sig(3378058256) at sleepq_wait_sig+29

msleep(3378058256,0,345,3228302894,0) at msleep+510

ttysleep(3378058240,3378058256,345,3228302894,0,...) at ttysleep+33

ttread(3378058240,3956755644,0) at ttread+1167

ttyread(3377924864,3956755644,0,3956755476,3226488553,...) at ttyread+47

scread(3377924864,3956755644,0,3228776192,0,...) at scread+34

giant_read(3377924864,3956755644,0) at giant_read+45

devfs_read_f(3379722928,3956755644,3376727936,0,3382254976) at devfs_read_f+98

dofileread(3382254976,0,3379722928,3956755644,4294967295,...) at dofileread+133

kern_readv(3382254976,0,3956755644,3217026591,1,...) at kern_readv+54

read(3382254976,3956755716) at read+69

syscall(59,59,59,0,134538624,...) at syscall+559

Xint0x80_syscall() at Xint0x80_syscall+31

--- syscall (3, FreeBSD ELF32, read), eip = 672428879, esp = 3217026556, ebp = -

Tracing command getty pid 859 tid 100088 td 0xc9c8c300

sched_switch(3385377536,0,1) at sched_switch+331

mi_switch(1,0,0,3957266836,3226748917,...) at mi_switch+426

sleepq_switch(3378062352) at sleepq_switch+113

sleepq_wait_sig(3378062352) at sleepq_wait_sig+29

msleep(3378062352,0,345,3228302894,0) at msleep+510

ttysleep(3378062336,3378062352,345,3228302894,0,...) at ttysleep+33

ttread(3378062336,3957267644,0) at ttread+1167

ttyread(3377925120,3957267644,0,3957267476,3226488553,...) at ttyread+47

scread(3377925120,3957267644,0,3228776192,0,...) at scread+34

giant_read(3377925120,3957267644,0) at giant_read+45

devfs_read_f(3382342928,3957267644,3376727936,0,3385377536) at devfs_read_f+98

dofileread(3385377536,0,3382342928,3957267644,4294967295,...) at dofileread+133

kern_readv(3385377536,0,3957267644,3217026591,1,...) at kern_readv+54

read(3385377536,3957267716) at read+69

syscall(59,59,59,0,134538624,...) at syscall+559

Xint0x80_syscall() at Xint0x80_syscall+31

--- syscall (3, FreeBSD ELF32, read), eip = 672428879, esp = 3217026556, ebp = -

Tracing command login pid 858 tid 100059 td 0xc9991a80

sched_switch(3382254208,0,1) at sched_switch+331

mi_switch(1,0,0,3956730892,3226748917,...) at mi_switch+426

sleepq_switch(3382249032) at sleepq_switch+113

sleepq_wait_sig(3382249032) at sleepq_wait_sig+29

msleep(3382249032,3382249136,348,3228297560,0) at msleep+510

kern_wait(3382254208,878,3956731012,0,0) at kern_wait+1699

wait4(3382254208,3956731140) at wait4+42

syscall(59,59,59,582,1,...) at syscall+559

Xint0x80_syscall() at Xint0x80_syscall+31

--- syscall (7, FreeBSD ELF32, wait4), eip = 672093903, esp = 3217026396, ebp =-

Tracing command cron pid 815 tid 100080 td 0xc9992c00

sched_switch(3382258688,0,1) at sched_switch+331

mi_switch(1,0,0,3956866108,3226749082,...) at mi_switch+426

sleepq_switch(3228779692) at sleepq_switch+113

sleepq_timedwait_sig(3228779692) at sleepq_timedwait_sig+30

msleep(3228779692,0,348,3228286296,60001,...) at msleep+468

kern_nanosleep(3382258688,3956866252,3956866244) at kern_nanosleep+171

nanosleep(3382258688,3956866308) at nanosleep+79

syscall(59,59,59,134545548,60,...) at syscall+559

Xint0x80_syscall() at Xint0x80_syscall+31

--- syscall (240, FreeBSD ELF32, nanosleep), eip = 672337699, esp = 3217026268,-

Tracing command sendmail pid 808 tid 100065 td 0xc96ea900

sched_switch(3379472640,0,1) at sched_switch+331

mi_switch(1,0,0,3920903260,3226748917,...) at mi_switch+426

sleepq_switch(3382772836) at sleepq_switch+113

sleepq_wait_sig(3382772836) at sleepq_wait_sig+29

msleep(3382772836,3382772888,360,3228284168,0) at msleep+510

kern_sigsuspend(3379472640,0,0,0,0,...) at kern_sigsuspend+163

sigsuspend(3379472640,3920903428) at sigsuspend+51

syscall(59,59,59,0,3217018880,...) at syscall+559

Xint0x80_syscall() at Xint0x80_syscall+31

--- syscall (341, FreeBSD ELF32, sigsuspend), eip = 673683383, esp = 3217018860-

Tracing command sendmail pid 804 tid 100041 td 0xc9566600

sched_switch(3377882624,0,1) at sched_switch+331

mi_switch(1,0,0,3920562976,3226749082,...) at mi_switch+426

sleepq_switch(3229095940) at sleepq_switch+113

sleepq_timedwait_sig(3229095940,3229095940,5001,3920563008,582,...) at sleepq_t0

cv_timedwait_sig(3229095940,3229095904,5001) at cv_timedwait_sig+290

kern_select(3377882624,4,3217016064,0,0,...) at kern_select+1169

select(3377882624,3920563460) at select+68

syscall(59,59,59,1,1,...) at syscall+559

Xint0x80_syscall() at Xint0x80_syscall+31

--- syscall (93, FreeBSD ELF32, select), eip = 674198223, esp = 3217015820, ebp-

Tracing command sshd pid 798 tid 100082 td 0xc9992900

sched_switch(3382257920,0,1) at sched_switch+331

mi_switch(1,0,0,3956841260,3226748917,...) at mi_switch+426

sleepq_switch(3229095940) at sleepq_switch+113

sleepq_wait_sig(3229095940,582,3228776128,3228296630,771,...) at sleepq_wait_si9

cv_wait_sig(3229095940,3229095904) at cv_wait_sig+281

kern_select(3382257920,5,134723872,0,0,...) at kern_select+1191

select(3382257920,3956841732) at select+68

syscall(59,59,59,1,10,...) at syscall+559

Xint0x80_syscall() at Xint0x80_syscall+31

--- syscall (93, FreeBSD ELF32, select), eip = 674546383, esp = 3217023340, ebp-

Tracing command usbd pid 761 tid 100050 td 0xc96eb600

sched_switch(3379475968,0,1) at sched_switch+331

mi_switch(1,0,0,3921001248,3226749082,...) at mi_switch+426

sleepq_switch(3229095940) at sleepq_switch+113

sleepq_timedwait_sig(3229095940,3229095940,30001,3921001280,582,...) at sleepq_0

cv_timedwait_sig(3229095940,3229095904,30001) at cv_timedwait_sig+290

kern_select(3379475968,6,3217026464,3217026336,0,...) at kern_select+1169

select(3379475968,3921001732) at select+68

syscall(59,59,59,3217026336,30,...) at syscall+559

Xint0x80_syscall() at Xint0x80_syscall+31

--- syscall (93, FreeBSD ELF32, select), eip = 672379599, esp = 3217026252, ebp-

Tracing command smbd pid 735 tid 100078 td 0xc9a12000

sched_switch(3382779904,0,1) at sched_switch+331

mi_switch(1,0,0,3956943964,3226748917,...) at mi_switch+426

sleepq_switch(3385356876) at sleepq_switch+113

sleepq_wait_sig(3385356876) at sleepq_wait_sig+29

msleep(3385356876,3385356928,360,3228284168,0) at msleep+510

kern_sigsuspend(3382779904,1073746048,0,0,0,...) at kern_sigsuspend+163

sigsuspend(3382779904,3956944132) at sigsuspend+51

syscall(59,59,59,137408512,137408512,...) at syscall+559

Xint0x80_syscall() at Xint0x80_syscall+31

--- syscall (341, FreeBSD ELF32, sigsuspend), eip = 676951991, esp = 3217021452-

Tracing command smbd pid 714 tid 100086 td 0xc9992300

sched_switch(3382256384,0,1) at sched_switch+331

mi_switch(1,0,0,3956792108,3226748917,...) at mi_switch+426

sleepq_switch(3229095940) at sleepq_switch+113

sleepq_wait_sig(3229095940,582,3228776128,3228296630,771,...) at sleepq_wait_si9

cv_wait_sig(3229095940,3229095904) at cv_wait_sig+281

kern_select(3382256384,22,3217021808,0,0,...) at kern_select+1191

select(3382256384,3956792580) at select+68

syscall(59,59,59,137408512,0,...) at syscall+559

Xint0x80_syscall() at Xint0x80_syscall+31

--- syscall (93, FreeBSD ELF32, select), eip = 677466831, esp = 3217021372, ebp-

Tracing command nmbd pid 710 tid 100089 td 0xc9c8c180

sched_switch(3385377152,0,1) at sched_switch+331

mi_switch(1,0,0,3957254944,3226749082,...) at mi_switch+426

sleepq_switch(3229095940) at sleepq_switch+113

sleepq_timedwait_sig(3229095940,3229095940,10001,3957254976,582,...) at sleepq_0

cv_timedwait_sig(3229095940,3229095904,10001) at cv_timedwait_sig+290

kern_select(3385377152,13,3217025088,0,0,...) at kern_select+1169

select(3385377152,3957255428) at select+68

syscall(59,59,59,13,3217025088,...) at syscall+559

Xint0x80_syscall() at Xint0x80_syscall+31

--- syscall (93, FreeBSD ELF32, select), eip = 674271951, esp = 3217024796, ebp-

Tracing command syslogd pid 631 tid 100044 td 0xc9566180

sched_switch(3377881472,0,1) at sched_switch+331

mi_switch(1,0,0,3920526124,3226748917,...) at mi_switch+426

sleepq_switch(3229095940) at sleepq_switch+113

sleepq_wait_sig(3229095940,582,3228776128,3228296630,771,...) at sleepq_wait_si9

cv_wait_sig(3229095940,3229095904) at cv_wait_sig+281

kern_select(3377881472,9,134570080,0,0,...) at kern_select+1191

select(3377881472,3920526596) at select+68

syscall(59,59,59,134570084,134570080,...) at syscall+559

Xint0x80_syscall() at Xint0x80_syscall+31

--- syscall (93, FreeBSD ELF32, select), eip = 672445135, esp = 3217021484, ebp-

Tracing command devd pid 565 tid 100060 td 0xc9991900

sched_switch(3382253824,0,1) at sched_switch+331

mi_switch(1,0,0,3956718380,3226748917,...) at mi_switch+426

sleepq_switch(3229095940) at sleepq_switch+113

sleepq_wait_sig(3229095940,582,3228776128,3228296630,771,...) at sleepq_wait_si9

cv_wait_sig(3229095940,3229095904) at cv_wait_sig+281

kern_select(3382253824,5,3217025488,0,0,...) at kern_select+1191

select(3382253824,3956718852) at select+68

syscall(59,59,59,0,8,...) at syscall+559

Xint0x80_syscall() at Xint0x80_syscall+31

--- syscall (93, FreeBSD ELF32, select), eip = 134698999, esp = 3217025388, ebp-

Tracing command adjkerntz pid 143 tid 100064 td 0xc96eaa80

sched_switch(3379473024,0,1) at sched_switch+331

mi_switch(1,0,0,3920915548,3226748917,...) at mi_switch+426

sleepq_switch(3382772300) at sleepq_switch+113

sleepq_wait_sig(3382772300) at sleepq_wait_sig+29

msleep(3382772300,3382772352,360,3228284168,0) at msleep+510

kern_sigsuspend(3379473024,0,0,0,0,...) at kern_sigsuspend+163

sigsuspend(3379473024,3920915716) at sigsuspend+51

syscall(59,59,59,0,25200,...) at syscall+559

Xint0x80_syscall() at Xint0x80_syscall+31

--- syscall (341, FreeBSD ELF32, sigsuspend), eip = 671856567, esp = 3217026460-

Tracing command schedcpu pid 42 tid 100046 td 0xc951bd80

sched_switch(3377577344,0,1) at sched_switch+331

mi_switch(1,0,0,3919867052,3226749014,...) at mi_switch+426

sleepq_switch(3919867124,3228776128,0,3228292512,566,...) at sleepq_switch+113

sleepq_timedwait(3919867124) at sleepq_timedwait+34

msleep(3919867124,0,0,3228262727,1000) at msleep+489

schedcpu_thread(0,3919867192,3228776128,0,3228273492,...) at schedcpu_thread+38

fork_exit(3226695036,0,3919867192) at fork_exit+97

fork_trampoline() at fork_trampoline+8

--- trap 1, eip = 0, esp = 3919867244, ebp = 0 ---

Tracing command softdepflush pid 41 tid 100047 td 0xc951bc00

sched_switch(3377576960,0,1) at sched_switch+331

mi_switch(1,0,0,3919854756,3226749014,...) at mi_switch+426

sleepq_switch(3229151220,3228776128,0,3228292512,566,...) at sleepq_switch+113

sleepq_timedwait(3229151220) at sleepq_timedwait+34

msleep(3229151220,3229151168,68,3228349621,1000) at msleep+489

softdep_flush(0,3919854904,3228776128,0,3228273492,...) at softdep_flush+499

fork_exit(3227615092,0,3919854904) at fork_exit+97

fork_trampoline() at fork_trampoline+8

--- trap 1, eip = 0, esp = 3919854956, ebp = 0 ---

Tracing command vnlru pid 40 tid 100048 td 0xc951ba80

sched_switch(3377576576,0,1) at sched_switch+331

mi_switch(1,0,1,3919842460,3226749014,...) at mi_switch+426

sleepq_switch(3379465360,3228776128,0,3228292512,566,...) at sleepq_switch+113

sleepq_timedwait(3379465360) at sleepq_timedwait+34

msleep(3379465360,3229098336,592,3228312087,1000,3229098464) at msleep+489

vnlru_proc(0,3919842616,3228776128,0,3228273492,...) at vnlru_proc+223

fork_exit(3226951860,0,3919842616) at fork_exit+97

fork_trampoline() at fork_trampoline+8

--- trap 1, eip = 0, esp = 3919842668, ebp = 0 ---

Tracing command syncer pid 39 tid 100049 td 0xc951b900

sched_switch(3377576192,0,1) at sched_switch+331

mi_switch(1,0,1,3919830160,3226748861,...) at mi_switch+426

sleepq_switch(3228779036,3228776128,0,3228292512,529,...) at sleepq_switch+113

sleepq_wait(3228779036,0,3226599612,3229098400,8,...) at sleepq_wait+33

msleep(3228779036,0,104,3228312226,0) at msleep+525

sched_sync(0,3919830328,3228776128,0,3228273492,...) at sched_sync+777

fork_exit(3226957540,0,3919830328) at fork_exit+97

fork_trampoline() at fork_trampoline+8

--- trap 1, eip = 0, esp = 3919830380, ebp = 0 ---

Tracing command bufdaemon pid 38 tid 100027 td 0xc951b180

sched_switch(3377574272,0,1) at sched_switch+331

mi_switch(1,0,1,3919768756,3226749014,...) at mi_switch+426

sleepq_switch(3229097324,3228776128,0,3228292512,566,...) at sleepq_switch+113

sleepq_timedwait(3229097324) at sleepq_timedwait+34

msleep(3229097324,3229097344,68,3228307860,1000) at msleep+489

buf_daemon(0,3919768888,3228776128,0,3228273492,...) at buf_daemon+264

fork_exit(3226903868,0,3919768888) at fork_exit+97

fork_trampoline() at fork_trampoline+8

--- trap 1, eip = 0, esp = 3919768940, ebp = 0 ---

Tracing command pagezero pid 37 tid 100028 td 0xc951b000

sched_switch(3377573888,0,1) at sched_switch+331

mi_switch(1,0,0,3919756452,3226749014,...) at mi_switch+426

sleepq_switch(3229155172,3228776128,0,3228292512,566,...) at sleepq_switch+113

sleepq_timedwait(3229155172) at sleepq_timedwait+34

msleep(3229155172,3229153856,512,3228365548,300000,3229153856,0,3228365522,157)9

vm_pagezero(0,3919756600,3228776128,0,3228273492,...) at vm_pagezero+108

fork_exit(3227806548,0,3919756600) at fork_exit+97

fork_trampoline() at fork_trampoline+8

--- trap 1, eip = 0, esp = 3919756652, ebp = 0 ---

Tracing command idlepoll pid 36 tid 100029 td 0xc9454d80

sched_switch(3376762240,0,1) at sched_switch+331

mi_switch(1,0,0,3892346024,3226749014,...) at mi_switch+426

sleepq_switch(3228776544,3228776128,0,3228292512,566,...) at sleepq_switch+113

sleepq_timedwait(3228776544) at sleepq_timedwait+34

msleep(3228776544,0,0,3228281840,3000) at msleep+489

poll_idle(0,3892346168,3228776128,0,3228273492,...) at poll_idle+226

fork_exit(3226607484,0,3892346168) at fork_exit+97

fork_trampoline() at fork_trampoline+8

--- trap 1, eip = 0, esp = 3892346220, ebp = 0 ---

Tracing command vmdaemon pid 35 tid 100030 td 0xc9454c00

sched_switch(3376761856,0,1) at sched_switch+331

mi_switch(1,0,1,3892333716,3226748861,...) at mi_switch+426

sleepq_switch(3229153972,3228776128,0,3228292512,529,...) at sleepq_switch+113

sleepq_wait(3229153972,0,3226599551,3228776192,3228365165,...) at sleepq_wait+33

msleep(3229153972,0,104,3228307860,0) at msleep+525

vm_daemon(0,3892333880,3228776128,0,3228273492,...) at vm_daemon+54

fork_exit(3227802892,0,3892333880) at fork_exit+97

fork_trampoline() at fork_trampoline+8

--- trap 1, eip = 0, esp = 3892333932, ebp = 0 ---

Tracing command pagedaemon pid 34 tid 100031 td 0xc9454a80

sched_switch(3376761472,0,1) at sched_switch+331

mi_switch(1,0,0,3892321456,3226749014,...) at mi_switch+426

sleepq_switch(3229153904,3228776128,0,3228292512,566,...) at sleepq_switch+113

sleepq_timedwait(3229153904) at sleepq_timedwait+34

msleep(3229153904,3229153856,68,3228307860,5000) at msleep+489

vm_pageout(0,3892321592,3228776128,0,3228273492,...) at vm_pageout+608

fork_exit(3227802060,0,3892321592) at fork_exit+97

fork_trampoline() at fork_trampoline+8

--- trap 1, eip = 0, esp = 3892321644, ebp = 0 ---

Tracing command irq7: ppc0 pid 33 tid 100032 td 0xc9454900

fork_trampoline() at fork_trampoline

Tracing command swi0: sio pid 32 tid 100033 td 0xc9454780

sched_switch(3376760704,0,1) at sched_switch+331

mi_switch(1,0) at mi_switch+426

ithread_loop(3377944496,3892297016,3228776128,0,3228273492,...) at ithread_loop2

fork_exit(3226555672,3377944496,3892297016) at fork_exit+97

fork_trampoline() at fork_trampoline+8

--- trap 1, eip = 0, esp = 3892297068, ebp = 0 ---

Tracing command fdc0 pid 31 tid 100034 td 0xc9454600

sched_switch(3376760320,0,1) at sched_switch+331

mi_switch(1,0,0,3892284500,3226749014,...) at mi_switch+426

sleepq_switch(3377805884,3228776128,0,3228292512,566,...) at sleepq_switch+113

sleepq_timedwait(3377805884) at sleepq_timedwait+34

msleep(3377805884,3377806064,76,3228262727,1000) at msleep+489

fdc_worker(3377805824,3377806064,0,3228371674,1166) at fdc_worker+458

fdc_thread(3377805824,3892284728,3228776128,0,3228273492,...) at fdc_thread+84

fork_exit(3227861752,3377805824,3892284728) at fork_exit+97

fork_trampoline() at fork_trampoline+8

--- trap 1, eip = 0, esp = 3892284780, ebp = 0 ---

Tracing command irq1: atkbd0 pid 30 tid 100035 td 0xc9454480

sched_switch(3376759936,0,1) at sched_switch+331

mi_switch(1,0) at mi_switch+426

ithread_loop(3377944864,3892272440,3228776128,0,3228273492,...) at ithread_loop2

fork_exit(3226555672,3377944864,3892272440) at fork_exit+97

fork_trampoline() at fork_trampoline+8

--- trap 1, eip = 0, esp = 3892272492, ebp = 0 ---

Tracing command irq24: bge1 pid 29 tid 100036 td 0xc9566d80

sched_switch(3377884544,0,1) at sched_switch+331

mi_switch(1,0) at mi_switch+426

ithread_loop(3377837568,3920624952,3228776128,0,3228273492,...) at ithread_loop2

fork_exit(3226555672,3377837568,3920624952) at fork_exit+97

fork_trampoline() at fork_trampoline+8

--- trap 1, eip = 0, esp = 3920625004, ebp = 0 ---

Tracing command irq27: bge0 pid 28 tid 100037 td 0xc9566c00

sched_switch(3377884160,0,1) at sched_switch+331

mi_switch(1,0) at mi_switch+426

ithread_loop(3377838352,3920612664,3228776128,0,3228273492,...) at ithread_loop2

fork_exit(3226555672,3377838352,3920612664) at fork_exit+97

fork_trampoline() at fork_trampoline+8

--- trap 1, eip = 0, esp = 3920612716, ebp = 0 ---

Tracing command irq15: ata1 pid 27 tid 100017 td 0xc9450900

fork_trampoline() at fork_trampoline

Tracing command irq14: ata0 pid 26 tid 100018 td 0xc9450780

fork_trampoline() at fork_trampoline

Tracing command irq16: atapci0 pid 25 tid 100019 td 0xc9450600

sched_switch(3376743936,0,1) at sched_switch+331

mi_switch(1,0) at mi_switch+426

ithread_loop(3377734416,3892161848,3228776128,0,3228273492,...) at ithread_loop2

fork_exit(3226555672,3377734416,3892161848) at fork_exit+97

fork_trampoline() at fork_trampoline+8

--- trap 1, eip = 0, esp = 3892161900, ebp = 0 ---

Tracing command usb1 pid 24 tid 100020 td 0xc9450480

sched_switch(3376743552,0,1) at sched_switch+331

mi_switch(1,0,1,3892149420,3226749014,...) at mi_switch+426

sleepq_switch(3377725968,3228776128,0,3228292512,566,...) at sleepq_switch+113

sleepq_timedwait(3377725968) at sleepq_timedwait+34

msleep(3377725968,0,92,3228259087,60000,3377742272) at msleep+489

usb_event_thread(3377742272,3892149560,3228776128,0,3228273492,...) at usb_even2

fork_exit(3226327968,3377742272,3892149560) at fork_exit+97

fork_trampoline() at fork_trampoline+8

--- trap 1, eip = 0, esp = 3892149612, ebp = 0 ---

Tracing command usbtask pid 23 tid 100021 td 0xc9450300

sched_switch(3376743168,0,1) at sched_switch+331

mi_switch(1,0,1,3892137140,3226748861,...) at mi_switch+426

sleepq_switch(3228766148,3228776128,0,3228292512,529,...) at sleepq_switch+113

sleepq_wait(3228766148,0,3226599551,3228776192,3228259073,...) at sleepq_wait+33

msleep(3228766148,0,92,3228259094,0,...) at msleep+525

usb_task_thread(0,3892137272,3228776128,0,3228273492,...) at usb_task_thread+71

fork_exit(3226328112,0,3892137272) at fork_exit+97

fork_trampoline() at fork_trampoline+8

--- trap 1, eip = 0, esp = 3892137324, ebp = 0 ---

Tracing command usb0 pid 22 tid 100022 td 0xc9450180

sched_switch(3376742784,0,1) at sched_switch+331

mi_switch(1,0,1,3892124844,3226749014,...) at mi_switch+426

sleepq_switch(3377709584,3228776128,0,3228292512,566,...) at sleepq_switch+113

sleepq_timedwait(3377709584) at sleepq_timedwait+34

msleep(3377709584,0,92,3228259087,60000,3377645696) at msleep+489

usb_event_thread(3377645696,3892124984,3228776128,0,3228273492,...) at usb_even2

fork_exit(3226327968,3377645696,3892124984) at fork_exit+97

fork_trampoline() at fork_trampoline+8

--- trap 1, eip = 0, esp = 3892125036, ebp = 0 ---

Tracing command irq19: ohci0 ohci1 pid 21 tid 100023 td 0xc951b780

fork_trampoline() at fork_trampoline

Tracing command irq9: acpi0 pid 20 tid 100024 td 0xc951b600

fork_trampoline() at fork_trampoline

Tracing command swi6: task queue pid 19 tid 100025 td 0xc951b480

sched_switch(3377575040,0,1) at sched_switch+331

mi_switch(1,0) at mi_switch+426

ithread_loop(3376511392,3919793464,3228776128,0,3228273492,...) at ithread_loop2

fork_exit(3226555672,3376511392,3919793464) at fork_exit+97

fork_trampoline() at fork_trampoline+8

--- trap 1, eip = 0, esp = 3919793516, ebp = 0 ---

Tracing command kqueue taskq pid 9 tid 100026 td 0xc951b300

sched_switch(3377574656,0,1) at sched_switch+331

mi_switch(1,0,0,3919781032,3226748861,...) at mi_switch+426

sleepq_switch(3377065088,3228776128,0,3228292512,529,...) at sleepq_switch+113

sleepq_wait(3377065088,0,3228293171,73,8,...) at sleepq_wait+33

msleep(3377065088,3377065116,0,3228262727,0) at msleep+525

taskqueue_thread_loop(3228769632,3919781176,3228776128,0,3228273492,...) at tas0

fork_exit(3226753588,3228769632,3919781176) at fork_exit+97

fork_trampoline() at fork_trampoline+8

--- trap 1, eip = 0, esp = 3919781228, ebp = 0 ---

Tracing command acpi_task_2 pid 8 tid 100008 td 0xc944f300

sched_switch(3376739072,0,1) at sched_switch+331

mi_switch(1,0,0,3892014248,3226748861,...) at mi_switch+426

sleepq_switch(3377065344,3228776128,0,3228292512,529,...) at sleepq_switch+113

sleepq_wait(3377065344,0,3228293171,73,8,...) at sleepq_wait+33

msleep(3377065344,3377065372,0,3228262727,0) at msleep+525

taskqueue_thread_loop(3230234752,3892014392,3228776128,0,3228273492,...) at tas0

fork_exit(3226753588,3230234752,3892014392) at fork_exit+97

fork_trampoline() at fork_trampoline+8

--- trap 1, eip = 0, esp = 3892014444, ebp = 0 ---

Tracing command acpi_task_1 pid 7 tid 100009 td 0xc944f180

sched_switch(3376738688,0,1) at sched_switch+331

mi_switch(1,0,0,3892001960,3226748861,...) at mi_switch+426

sleepq_switch(3377065344,3228776128,0,3228292512,529,...) at sleepq_switch+113

sleepq_wait(3377065344,0,3228293171,73,8,...) at sleepq_wait+33

msleep(3377065344,3377065372,0,3228262727,0) at msleep+525

taskqueue_thread_loop(3230234752,3892002104,3228776128,0,3228273492,...) at tas0

fork_exit(3226753588,3230234752,3892002104) at fork_exit+97

fork_trampoline() at fork_trampoline+8

--- trap 1, eip = 0, esp = 3892002156, ebp = 0 ---

Tracing command acpi_task_0 pid 6 tid 100010 td 0xc944f000

sched_switch(3376738304,0,1) at sched_switch+331

mi_switch(1,0,0,3891989672,3226748861,...) at mi_switch+426

sleepq_switch(3377065344,3228776128,0,3228292512,529,...) at sleepq_switch+113

sleepq_wait(3377065344,0,3228293171,73,3377600608,...) at sleepq_wait+33

msleep(3377065344,3377065372,0,3228262727,0) at msleep+525

taskqueue_thread_loop(3230234752,3891989816,3228776128,0,3228273492,...) at tas0

fork_exit(3226753588,3230234752,3891989816) at fork_exit+97

fork_trampoline() at fork_trampoline+8

--- trap 1, eip = 0, esp = 3891989868, ebp = 0 ---

Tracing command swi2: cambio pid 18 tid 100011 td 0xc9454300

sched_switch(3376759552,0,1) at sched_switch+331

mi_switch(1,0) at mi_switch+426

ithread_loop(3376511456,3892260152,3228776128,0,3228273492,...) at ithread_loop2

fork_exit(3226555672,3376511456,3892260152) at fork_exit+97

fork_trampoline() at fork_trampoline+8

--- trap 1, eip = 0, esp = 3892260204, ebp = 0 ---

Tracing command swi5: + pid 17 tid 100012 td 0xc9454180

fork_trampoline() at fork_trampoline

Tracing command thread taskq pid 5 tid 100013 td 0xc9454000

sched_switch(3376758784,0,1) at sched_switch+331

mi_switch(1,0,0,3892235432,3226748861,...) at mi_switch+426

sleepq_switch(3377103104,3228776128,0,3228292512,529,...) at sleepq_switch+113

sleepq_wait(3377103104,0,3228293171,73,8,...) at sleepq_wait+33

msleep(3377103104,3377103132,0,3228262727,0) at msleep+525

taskqueue_thread_loop(3228801864,3892235576,3228776128,0,3228273492,...) at tas0

fork_exit(3226753588,3228801864,3892235576) at fork_exit+97

fork_trampoline() at fork_trampoline+8

--- trap 1, eip = 0, esp = 3892235628, ebp = 0 ---

Tracing command swi6: Giant taskq pid 16 tid 100014 td 0xc9450d80

kdb_enter(3228375620) at kdb_enter+43

scgetc(3229266944,2) at scgetc+1348

sckbdevent(3377072384,0,3229266944) at sckbdevent+456

kbdmux_intr(3377072384,0,3377105560,3892223160,3226752756,...) at kbdmux_intr+28

kbdmux_kbd_intr(3377072384,1) at kbdmux_kbd_intr+26

taskqueue_run(3377103360,3892223196,3226555506,0,3376511600,...) at taskqueue_r0

taskqueue_swi_giant_run(0) at taskqueue_swi_giant_run+14

ithread_execute_handlers(3377065984,3377103232) at ithread_execute_handlers+206

ithread_loop(3376511600,3892223288,3228776128,0,3228273492,...) at ithread_loop9

fork_exit(3226555672,3376511600,3892223288) at fork_exit+97

fork_trampoline() at fork_trampoline+8

--- trap 1, eip = 0, esp = 3892223340, ebp = 0 ---

Tracing command yarrow pid 15 tid 100015 td 0xc9450c00

sched_switch(3376745472,0,1) at sched_switch+331

mi_switch(1,0,0,3892210848,3226749014,...) at mi_switch+426

sleepq_switch(3228757952,3228776128,0,3228292512,566,...) at sleepq_switch+113

sleepq_timedwait(3228757952) at sleepq_timedwait+34

msleep(3228757952,0,0,3228262727,100) at msleep+489

random_kthread(0,3892211000,3228776128,0,3228273492,...) at random_kthread+369

fork_exit(3226233400,0,3892211000) at fork_exit+97

fork_trampoline() at fork_trampoline+8

--- trap 1, eip = 0, esp = 3892211052, ebp = 0 ---

Tracing command g_down pid 4 tid 100016 td 0xc9450a80

sched_switch(3376745088,0,1) at sched_switch+331

mi_switch(1,0,0,3892198556,3226749014,...) at mi_switch+426

sleepq_switch(3228767432,3228776128,0,3228292512,566,...) at sleepq_switch+113

sleepq_timedwait(3228767432) at sleepq_timedwait+34

msleep(3228767432,3228767144,588,3228262727,100) at msleep+489

g_io_schedule_down(3376745088) at g_io_schedule_down+86

g_down_procbody(0,3892198712,3228776128,0,3228273492,...) at g_down_procbody+70

fork_exit(3226426420,0,3892198712) at fork_exit+97

fork_trampoline() at fork_trampoline+8

--- trap 1, eip = 0, esp = 3892198764, ebp = 0 ---

Tracing command g_up pid 3 tid 100000 td 0xc9450000

sched_switch(3376742400,0,1) at sched_switch+331

mi_switch(1,0,0,3892112544,3226749014,...) at mi_switch+426

sleepq_switch(3228767428,3228776128,0,3228292512,566,...) at sleepq_switch+113

sleepq_timedwait(3228767428) at sleepq_timedwait+34

msleep(3228767428,3228767208,588,3228262727,100) at msleep+489

g_io_schedule_up(3376742400) at g_io_schedule_up+204

g_up_procbody(0,3892112696,3228776128,0,3228273492,...) at g_up_procbody+70

fork_exit(3226426344,0,3892112696) at fork_exit+97

fork_trampoline() at fork_trampoline+8

--- trap 1, eip = 0, esp = 3892112748, ebp = 0 ---

Tracing command g_event pid 2 tid 100001 td 0xc944fd80

sched_switch(3376741760,0,1) at sched_switch+331

mi_switch(1,0,0,3892100260,3226749014,...) at mi_switch+426

sleepq_switch(3228767420,3228776128,0,3228292512,566,...) at sleepq_switch+113

sleepq_timedwait(3228767420) at sleepq_timedwait+34

msleep(3228767420,0,76,3228262727,100) at msleep+489

g_event_procbody(0,3892100408,3228776128,0,3228273492,...) at g_event_procbody+0

fork_exit(3226426496,0,3892100408) at fork_exit+97

fork_trampoline() at fork_trampoline+8

--- trap 1, eip = 0, esp = 3892100460, ebp = 0 ---

Tracing command swi3: vm pid 14 tid 100002 td 0xc944fc00

fork_trampoline() at fork_trampoline

Tracing command swi4: clock sio pid 13 tid 100003 td 0xc944fa80

sched_switch(3376740992,0,1) at sched_switch+331

mi_switch(1,0) at mi_switch+426

ithread_loop(3376511136,3892075832,3228776128,0,3228273492,...) at ithread_loop2

fork_exit(3226555672,3376511136,3892075832) at fork_exit+97

fork_trampoline() at fork_trampoline+8

--- trap 1, eip = 0, esp = 3892075884, ebp = 0 ---

Tracing command swi1: net pid 12 tid 100004 td 0xc944f900

sched_switch(3376740608,0,1) at sched_switch+331

mi_switch(1,0) at mi_switch+426

ithread_loop(3376511152,3892063544,3228776128,0,3228273492,...) at ithread_loop2

fork_exit(3226555672,3376511152,3892063544) at fork_exit+97

fork_trampoline() at fork_trampoline+8

--- trap 1, eip = 0, esp = 3892063596, ebp = 0 ---

Tracing command idle: cpu0 pid 11 tid 100005 td 0xc944f780

sched_switch(3376740224,3376743936,6) at sched_switch+331

mi_switch(6,3376743936,3376744280,3376743936,3892050952,...) at mi_switch+426

maybe_preempt(3376743936) at maybe_preempt+196

sched_add(3376743936,4,3376743936,3377734416,3377063168,...) at sched_add+240

setrunqueue(3376743936,4) at setrunqueue+99

intr_event_schedule_thread(3377063168) at intr_event_schedule_thread+163

intr_execute_handlers(3376699000,3892051072,16,3892051140,3227972099,...) at in0

lapic_handle_intr(50) at lapic_handle_intr+46

Xapic_isr1() at Xapic_isr1+51

--- interrupt, eip = 3230175589, esp = 3892051140, ebp = 3892051140 ---

acpi_cpu_c1(3228776128,0,3892051172,1,4294967294,...) at acpi_cpu_c1+5

acpi_cpu_idle(3892051196,3226552629,0,3376736888,3892051236,...) at acpi_cpu_id8

cpu_idle(0,3376736888,3892051236,3226552149,0,...) at cpu_idle+40

idle_proc(0,3892051256,3228776128,0,3228273492,...) at idle_proc+77

fork_exit(3226552552,0,3892051256) at fork_exit+97

fork_trampoline() at fork_trampoline+8

--- trap 1, eip = 0, esp = 3892051308, ebp = 0 ---

Tracing command idle: cpu1 pid 10 tid 100006 td 0xc944f600

sched_switch(3376739840,0,1) at sched_switch+331

mi_switch(1,0,3228776128,0,3228273740,...) at mi_switch+426

idle_proc(0,3892038968,3228776128,0,3228273492,...) at idle_proc+123

fork_exit(3226552552,0,3892038968) at fork_exit+97

fork_trampoline() at fork_trampoline+8

--- trap 1, eip = 0, esp = 3892039020, ebp = 0 ---

Tracing command init pid 1 tid 100007 td 0xc944f480

sched_switch(3376739456,0,1) at sched_switch+331

mi_switch(1,0,0,3892026380,3226748917,...) at mi_switch+426

sleepq_switch(3376754688) at sleepq_switch+113

sleepq_wait_sig(3376754688) at sleepq_wait_sig+29

msleep(3376754688,3376754792,348,3228297560,0) at msleep+510

kern_wait(3376739456,4294967295,3892026500,0,0) at kern_wait+1699

wait4(3376739456,3892026628) at wait4+42

syscall(59,59,59,0,0,...) at syscall+559

Xint0x80_syscall() at Xint0x80_syscall+31

--- syscall (7, FreeBSD ELF32, wait4), eip = 134557511, esp = 3217026428, ebp =-

Tracing command swapper pid 0 tid 0 td 0xc07317e0

sched_switch(3228768224,0,1) at sched_switch+331

mi_switch(1,0) at mi_switch+426

scheduler(0,12708864,12705792,0,3225640485,...) at scheduler+385

mi_startup() at mi_startup+150

begin() at begin+44

db>