LOR (intr table and sio) and instability

[Rong-en Fan]

I'm running today's 6-stable on a amd64 SMP (Pentium-D) machine.
When turning on witness, I got a LOR on half way of booting:

ata0-slave: pio=PIO4 wdma=WDMA2 udma=UDMA100 cable=40 wire
ata2-master: pio=PIO4 wdma=WDMA2 udma=UDMA133 cable=40 wire
ad4: 157066MB <HDT722516DLA380 V43OA96A> at ata2-master SATA150
ad4: 321672960 sectors [319120C/16H/63S] 16 sectors/interrupt 1 depth queue
SMP: AP CPU #1 Launched!
cpu1 AP:
     ID: 0x01000000   VER: 0x00050014 LDR: 0x00000000 DFR: 0xffffffff
  lint0: 0x00010700 lint1: 0x00000400 TPR: 0x00000000 SVR: 0x000001ff
  timer: 0x000200ef therm: 0x00010000 err: 0x00010000 pcm: 0x00010000
lock order reversal:
 1st 0xffffffff805643c0 intr table (intr table) @
/home/admin/usr/src/sys/amd64/amd64/intr_machdep.c:417
 2nd 0xffffffff8056fb00 sio (sio) @ /home/admin/usr/src/sys/dev/sio/sio.c:2586
KDB: stack backtrace:
witness_checkorder() at witness_checkorder+0x4e1
_mtx_lock_spin_flags() at _mtx_lock_spin_flags+0x5d
siocnputc() at siocnputc+0xf5
cnputc() at cnputc+0x60
putchar() at putchar+0xcb
kvprintf() at kvprintf+0x9b
printf() at printf+0xe7
intr_assign_next_cpu() at intr_assign_next_cpu+0x7a
intr_shuffle_irqs() at intr_shuffle_irqs+0x6b
mi_startup() at mi_startup+0xc0
btext() at btext+0x2c
INTR: Assigning IRQ 1 to local APIC 0
ioapic0: Assigning ISA IRQ 1 to local APIC 0
INTR: Assigning IRQ 3 to local APIC 1
ioapic0: Assigning ISA IRQ 3 to local APIC 1
INTR: Assigning IRQ 4 to local APIC 0
ioapic0: Assigning ISA IRQ 4 to local APIC 0
INTR: Assigning IRQ 9 to local APIC 1
ioapic0: Assigning ISA IRQ 9 to local APIC 1
INTR: Assigning IRQ 12 to local APIC 0
ioapic0: Assigning ISA IRQ 12 to local APIC 0
INTR: Assigning IRQ 14 to local APIC 1
ioapic0: Assigning ISA IRQ 14 to local APIC 1
INTR: Assigning IRQ 15 to local APIC 0
ioapic0: Assigning ISA IRQ 15 to local APIC 0
INTR: Assigning IRQ 17 to local APIC 1
ioapic0: Assigning PCI IRQ 17 to local APIC 1
GEOM: new disk ad4
Trying to mount root from ufs:/dev/ad4s1a

I checked the LOR page, and there no such report. I'm wondering
if this is related to the console hang every few days on this machine.
When it hangs, keyboard does not work. The only way is to break
into ddb and reset it. I just hooked up the serial console few days ago.
No hang yet. But I got a panic this noon:

Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address  = 0x18c
fault code             = supervisor read, page not present
instruction pointer    = 0x8:0xffffffff801f107e
stack pointer          = 0x10:0xffffffffb1839b30
frame pointer          = 0x10:0xffffffffb1839b60
code segment           = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags       = resume, IOPL = 0
current process                = 9 (thread taskq)


[thread pid 9 tid 100011 ]
Stopped at      _mtx_lock_sleep+0x80:   movl    0x18824(%r12),%r8d
db>  bt
Tracing pid 9 tid 100011 td 0xffffff007b218980
_mtx_lock_sleep() at _mtx_lock_sleep+0x80
unp_gc() at unp_gc+0x4c4
taskqueue_run() at taskqueue_run+0xd5
taskqueue_thread_loop() at taskqueue_thread_loop+0x88
fork_exit() at fork_exit+0x8b
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffb1839d00, rbp = 0 ---

I searched the mail archive, but it seems there is no similar report.
I call 'doadump' in ddb, but after rebooting, savecore says there is
no dump there? (i have dump_dev="AUTO"

If you need more information, please let me know.

Thanks,
Rong-En Fan