5.2R SMP kernel crashes when paging on swap

[Dong Lin]

Does anyone have any suggestions on how to debug the following
vm_pager problem in a SMP kernel? Thanks in advance.

The kernel crashes when vm_pageout_scan kicks. I have a hard time
locating the offending line because there is no panic. The machine
just goes blank and reboots.

If I add DELAY() at the end of vm_pageout_scan (after Giant unlock),
it does not crash but causes the following warning on the console:

lock order reversal:
1st vm object(vm object) @ vm/swap_pager.c:1325
2nd swap_pager swhash (swap_pager swhash) @ vm/swap_pager.c:1840
3rd vm object (vm object) @ vm/uma_core.c:873
Stack backtrace:
backtrace()
witness_lock()
_mtx_lock_flags()
obj_alloc()
slab_zalloc()
uma_zone_slab()
uma_zalloc_internal()
uma_zalloc_arg()
swp_pager_meta_build()
swap_pager_putpages()
default_pager_putpages()
vm_pageout_flush()
vm_pageout_clean()
vm_pageout_scan()
vm_pageout()
fork_exit()
fork_trampoline()

This dual processor machine works fine with a non-SMP kernel. The SMP
kernel works on a uniprocessor machine as well.

my SMP kernel:
GENERIC with BOOTP and BOOTP_NFSROOT added to run diskless.

my machine configuration:
all fs are mounted via md or NFS. The only local storage is a swap
disk.

nfsserver:/disk4/diskless_root  69481814 61293081 2630188    96%    /
devfs                                  1        1       0   100%    /dev
/dev/md0                            7406     1720    5094    25%    /etc
procfs                                 4        4       0   100%    /proc
/dev/md1                           31470      242   28712     1%    /var
nfsserver:/usr                   2804906  1406602 1173912    55%    /usr

% swapinfo 
Device          1K-blocks     Used    Avail Capacity
/dev/ad0s3b        528144     4600   523544     1%

MPTable: <INTEL    Kitty Hawk  >
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel Pentium III (864.46-MHz 686-class CPU)
  Origin = "GenuineIntel"  Id = 0x686  Stepping = 6
  Features=0x383fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CM
OV,PAT,PSE36,MMX,FXSR,SSE>
real memory  = 134041600 (127 MB)
avail memory = 120590336 (115 MB)
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
 cpu0 (BSP): APIC ID:  1
 cpu1 (AP): APIC ID:  0
ioapic0: Assuming intbase of 0
ioapic0 <Version 2.0> irqs 0-23 on motherboard
Pentium Pro MTRR support enabled