Kernel panic on PowerEdge 1950 under certain stress load

[Benjie Chen]

Hi FreeBSD hackers and engineers,

I am experiencing a kernel panic that comes on when my new PowerEdge 1950
FreeBSD 6.2 setup is under a certain stress load. I've emailed a few people
on the list who have given me useful comments, some of which I am still
following up. But I wanted to send a general cry for help to see if there
are more knowledge out there about this problem.

FreeBSD 6.2 on PowerEdge 1950, RAID1 setup with mfi driver (PERC5i). 4GB
RAM. I am currently running i386, and not amd64, due to various reasons.

I've ran exhaustively memory tests, disk tests, and network tests and cannot
produce the kernel panic. I worked with Dell support to run memory test 1
DIMM at a time and cannot find any problem.  With 1 DIMM at a time, I could
still get the kernel panic under my work load.

My work load is heavily hitting a web site running on the machine and
requiring the web service to do MySQL requests. On the side, I am running a
bunch of scripts that mostly read from the MySQL database but also write to
it occasionally. Not memory intensive -- still have usually about 1GB free
memory, but fairly disk intensive. I don't get disk errors. Anywhere from
between 10 minutes to 4 or 5 hours into the test, I get the kernel panic.
Again, still no disk errors. I turned off soft-update, still happens.

Kernel panic is at 0xC066C731, which from nm shows it's in mtx_lock_spin
 c066c7b4 T _mtx_lock_spin
 c066c85c T _mtx_unlock_sleep

So this could mean that independent stress tests will not result in panic if
there aren't enough concurrency to cause the problem.

There are a few other complaints about kernel panics at the same IP on the
web (google 0xc066c731)... I was wondering if anyone had dealt with this
before and if there are any work arounds?

Thanks,
Benjie