new twist on IPI deadlock

[John Baldwin]

On Saturday 21 August 2004 04:12 pm, Doug White wrote:
> Got this on my xeon today, with hyperthreading disabled, and dropping to
> ddb with NMI after a hang.  Looks like smp_rendezvous_action() colliding
> with smp_tlb_shootdown().
>
> smp_rendezvous and smp_tlb_shootdown use different IPI vectors and
> different locks, but I wonder if the operations aren't orthogonal, and
> doing multiple IPIs at once can cause unexpected behavior.

Cute!  This might actually explain the SMP deadlocks with KSE apps.  The fix 
is probably to make the TLB code use the same mutex as the SMP rendezvous 
code.

> kernel trap 19 with interrupts disabled
> NMI ... going to debugger
> [thread 100168]
> Stopped at      smp_rendezvous_action+0x30:     cmpl    mp_ncpus,%eax
> db>    tr
> smp_rendezvous_action(fd) at smp_rendezvous_action+0x30
> smp_rendezvous(0,c06a0724,0,c2a66420) at smp_rendezvous+0xd7
> i386_ldt_grow(c2a66420,12,8,dfe2a000,c2a63f60) at i386_ldt_grow+0x1b1
> i386_set_ldt(c2a66420,bfbfe968,c2a63de0,0,dfd61d40) at i386_set_ldt+0x2de
> sysarch(c2a66420,dfd61d14,2,0,206) at sysarch+0x67
> syscall(2f,2f,2f,2807f010,0) at syscall+0x287
> Xint0x80_syscall() at Xint0x80_syscall+0x1f
> --- syscall (165, FreeBSD ELF32, sysarch), eip = 0x2807196f, esp =
> 0xbfbfe954, ebp = 0x-
> [...]
> db> tr 28658
> sched_switch(f6,dc511000,dc512000) at sched_switch+0x9b
> smp_invlpg_range(dc511000,dc512000) at smp_invlpg_range+0x1c
> pmap_invalidate_range(c0775e20,dc511000,dc512000,c2294780,c2277170) at
> pmap_invalidate_5
> pmap_qenter(dc511000,c2294790,1) at pmap_qenter+0x50
> sf_buf_alloc(c1a329e0,0,0,0,0) at sf_buf_alloc+0x1a9
> uiomove_fromphys(c28a9600,3000,8d8,dfed5c88,0) at uiomove_fromphys+0x92
> pipe_read(c2a68bf4,dfed5c88,c2b7c400,0,c349a2c0) at pipe_read+0x238
> dofileread(c349a2c0,c2a68bf4,0,812a000,4000) at dofileread+0x95
> read(c349a2c0,dfed5d14,3,0,296) at read+0x3b
> syscall(2f,2f,2f,80da500,80f7034) at syscall+0x287
> Xint0x80_syscall() at Xint0x80_syscall+0x1f
> --- syscall (3, FreeBSD ELF32, read), eip = 0x80a5783, esp = 0xbfbfe69c,
> ebp = 0xbfbfe6-
> db> tr 28690
> smp_rendezvous_action(fd) at smp_rendezvous_action+0x30
> smp_rendezvous(0,c06a0724,0,c2a66420) at smp_rendezvous+0xd7
> i386_ldt_grow(c2a66420,12,8,dfe2a000,c2a63f60) at i386_ldt_grow+0x1b1
> i386_set_ldt(c2a66420,bfbfe968,c2a63de0,0,dfd61d40) at i386_set_ldt+0x2de
> sysarch(c2a66420,dfd61d14,2,0,206) at sysarch+0x67
> syscall(2f,2f,2f,2807f010,0) at syscall+0x287
> Xint0x80_syscall() at Xint0x80_syscall+0x1f
> --- syscall (165, FreeBSD ELF32, sysarch), eip = 0x2807196f, esp =
> 0xbfbfe954, ebp = 0x-

-- 
John Baldwin <jhb at FreeBSD.org>  <><  http://www.FreeBSD.org/~jhb/
"Power Users Use the Power to Serve"  =  http://www.FreeBSD.org