lock order reversal and swap problems

[Artem Koutchine]

This evening, before going hove i decided to load-test the new server.
I remind the server is:
dual xeon 2.66, 2GB ECC ram, supermicro x5dpeg2 mb,
3ware 8506-4LP sata radi controller, 4 sata seagate disks,
eveyrthing is install in a RAID5 array.

I run:
3 bonnies in a loop,
4 byteneches in a loop,
6 my own perl scripts to test the memory. Each allocated about 256MB
ran, fill it with somethingm then free and do it over and over again.

After 3 minutes of running i saw this:

Apr  7 20:16:34 new kernel: lock order reversal
Apr  7 20:16:34 new kernel: 1st 0xc79446b4 vm object (vm object) @
/usr/src/sys/vm/swap_pager.c:1313
Apr  7 20:16:35 new kernel: 2nd 0xc08b9e00 swap_pager swhash (swap_pager
swhash) @ /usr/src/sys/vm/swap_pager.c:1803
Apr  7 20:16:35 new kernel: 3rd 0xc7898294 vm object (vm object) @
/usr/src/sys/vm/uma_core.c:886
Apr  7 20:16:35 new kernel: Stack backtrace:
Apr  7 20:16:35 new kernel: backtrace(0,1,c0888798,c08898f0,c08174bc) at
backtrace+0x12
Apr  7 20:16:35 new kernel: witness_checkorder(c7898294,9,c07d43dd,376) at
witness_checkorder+0x593
Apr  7 20:16:35 new kernel:
_mtx_lock_flags(c7898294,0,c07d43dd,376,c78888d4) at _mtx_lock_flags+0x67
Apr  7 20:16:35 new kernel: obj_alloc(c78888c0,1000,e32ffa2f,101,c78888d4)
at obj_alloc+0x31
Apr  7 20:16:35 new kernel: slab_zalloc(c78888c0,1,c78888c0,c78888d4,1) at
slab_zalloc+0x82
Apr  7 20:16:35 new kernel:
uma_zone_slab(c78888c0,1,c78888d4,0,c07d43dd,68f) at uma_zone_slab+0xb5
Apr  7 20:16:35 new kernel: uma_zalloc_internal(c78888c0,0,1,c78888d4,0) at
uma_zalloc_internal+0x28
Apr  7 20:16:35 new kernel: uma_zalloc_arg(c78888c0,0,1) at
uma_zalloc_arg+0x2f8
Apr  7 20:16:35 new kernel: swp_pager_meta_build(c79446b4,38,0,2,0) at
swp_pager_meta_build+0x108
Apr  7 20:16:36 new kernel:
swap_pager_putpages(c79446b4,e32ffc00,4,0,e32ffb70) at
swap_pager_putpages+0x2b4
Apr  7 20:16:36 new kernel:
default_pager_putpages(c79446b4,e32ffc00,4,0,e32ffb70) at
default_pager_putpages+0x18
Apr  7 20:16:36 new kernel: vm_pageout_flush(e32ffc00,4,0,0,e32ffbdc) at
vm_pageout_flush+0x11e
Apr  7 20:16:36 new kernel: vm_pageout_clean(c2c26308) at
vm_pageout_clean+0x2a5
Apr  7 20:16:36 new kernel: vm_pageout_scan(0) at vm_pageout_scan+0x54b
Apr  7 20:16:36 new kernel: vm_pageout(0,e32ffd48,0,c071cc3c,0) at
vm_pageout+0x2d2
Apr  7 20:16:36 new kernel: fork_exit(c071cc3c,0,e32ffd48) at fork_exit+0xa8
Apr  7 20:16:36 new kernel: fork_trampoline() at fork_trampoline+0x8
Apr  7 20:16:36 new kernel: --- trap 0x1, eip = 0, esp = 0xe32ffd7c, ebp =
0 ---

and then hundreds and thousands:

Apr  7 20:20:29 new kernel: swap_pager_getswapspace(16): failed
Apr  7 20:20:32 new last message repeated 44 times

while  actually there were 500MB of swap space left.

What is going on?

Regards,
Artem