6.1-stable hangs and LORs

[Brad Waite]

Hi guys,

I'm going to take another stab at getting some help.

For the last 6 months my FBSD gateway has been locking up every few 
days, usually about once a week.  No panic, no reboots, just a hard lock 
with no response on the console or over the net.

I've replaced literally every piece of hardware with the exception of 
the case and power supply.  No change.

I've upgraded from 5.3- to 6.0- to 6.1-STABLE.  No change.

I've researched as much as I know how and still come up with hardly 
anything.  I have turned on BREAK_TO_DEBUGGER, WITNESS and INVARIANTS 
and the only indication I've gotten is a lock order reversal that's 
*similar* to http://sources.zabbadoz.net/freebsd/lor/017.html.  The line 
numbers in pf.c don't match up with LOR 017, but that's about all I can 
tell.

I'm reasonably certain the issue is with pf, since I have 3 other 
non-gateway servers humming along with no problems.  The hardware is 
nearly identical - their RAID cards are different, but I've tried 
running my gateway on just a single SCSI drive and had the same lockup 
issue.  Of course, the issue could be somewhere else, but I'm at a loss 
as to how to find it.

I'm running my console over serial so I can log anything that's 
necessary.  I've been able to break to the debugger, but to be honest, I 
don't know what to look for.  I've seen several posts on the lists about 
  posting the output of debug commands, but I figured it to be in poor 
taste to just dump my output here before someone asked.

I'm getting a lot of heat from the boss since our VoIP phones don't work 
when the gateway locks up.

If someone can help identify and/or eliminate this issue, I'm more than 
happy to do everything I can to provide the necessary information.

Thanks.