6.1-stable hangs and LORs

[Roland Smith]

On Sun, Jun 11, 2006 at 12:09:05PM -0600, Brad Waite wrote:
> Hi guys,
> 
> I'm going to take another stab at getting some help.
> 
> For the last 6 months my FBSD gateway has been locking up every few 
> days, usually about once a week.  No panic, no reboots, just a hard lock 
> with no response on the console or over the net.
> 
> I've replaced literally every piece of hardware with the exception of 
> the case and power supply.  No change.

One of my machines had random lockup problems, an then the power supply
died. The problems were gone after it was replaced. You might want to
swap out the power supply. And test the RAM.

> I've upgraded from 5.3- to 6.0- to 6.1-STABLE.  No change.
> 
> I've researched as much as I know how and still come up with hardly 
> anything.  I have turned on BREAK_TO_DEBUGGER, WITNESS and INVARIANTS 
> and the only indication I've gotten is a lock order reversal that's 
> *similar* to http://sources.zabbadoz.net/freebsd/lor/017.html.  The line 
> numbers in pf.c don't match up with LOR 017, but that's about all I can 
> tell.
> 
> I'm reasonably certain the issue is with pf, since I have 3 other 
> non-gateway servers humming along with no problems.  The hardware is 
> nearly identical - their RAID cards are different, but I've tried 
> running my gateway on just a single SCSI drive and had the same lockup 
> issue.  Of course, the issue could be somewhere else, but I'm at a loss 
> as to how to find it.

Could you try swapping two machines? That would be the ultimate check if
it's hardware related.
 
> I'm running my console over serial so I can log anything that's 
> necessary.  I've been able to break to the debugger, but to be honest, I 
> don't know what to look for.  I've seen several posts on the lists about 
>  posting the output of debug commands, but I figured it to be in poor 
> taste to just dump my output here before someone asked.

Posting a stack backtrace (bt) might be a good start.

> I'm getting a lot of heat from the boss since our VoIP phones don't work 
> when the gateway locks up.

Sometimes POTS isn't so bad after all. ;-)

Roland
-- 
R.F.Smith                                   http://www.xs4all.nl/~rsmith/
[plain text _non-HTML_ PGP/GnuPG encrypted/signed email much appreciated]
pgp: 1A2B 477F 9970 BA3C 2914  B7CE 1277 EFB0 C321 A725 (KeyID: C321A725)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 187 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20060611/a07b61a5/attachment.pgp