Serious Dell Sadness - H200, H700, and H800

[Neil Schelly]

We have reached some conclusion on this issue, and a positive one at that.  Big Credit here goes to Scott Long, who was able to help us debug the issue with a patch to the driver that has completely resolved the issue for us.  He gave permission for me to post/distribute this patch, and sees no reason it couldn't be made a part of the MFI driver base.  I've pasted it at the bottom of this message.

His explanation centers around out-of-band interrupt synchronization on the PCI bus.  Interrupts associated with the completion of I/O operations from the card to the CPU are getting lost/ignored.  By issuing a dummy read operation (thus forcing a flush of data buffers), this issue is largely averted.  He strongly suspects that the controller firmware is de-asserting an interrupt prematurely, so that the OS never responds to the I/O operation and things just hang.  Once something like mfiutil is run, it reads from the device, unlocking the bus, and things continue as normal.  The patch adds extraneous read operations into the end of the interrupt handler, which keeps things flowing more normally, albeit with a slight performance hit by having the extra read operations.

I am unsure if this completely eliminates the race condition, but it will at least have to happen in a much smaller window of time with this patch.  We have been unable to reproduce the problem while running this version.  From the sound of his explanation, it's also possible this problem doesn't exist except when accessing the card via PCI semantics.  If the device were operating in MSI mode (PCI Express), where interrupt handling is significantly different, this may not come up at all.

Thanks again to Scott Long for the help.  Here's patch:

Index: mfi.c
===================================================================
RCS file: /usr/ncvs/src/sys/dev/mfi/mfi.c,v
retrieving revision 1.54
diff -u -r1.54 mfi.c
--- mfi.c 7 Dec 2009 21:24:07 -0000 1.54
+++ mfi.c 13 Mar 2011 04:12:35 -0000
@@ -928,6 +928,12 @@
if (sc->mfi_check_clear_intr(sc))
return;

+ /*
+ * Do a dummy read to flush the interrupt ACK that we just performed,
+ * ensuring that everything is really, truly consistent.
+ */
+ (void)sc->mfi_read_fw_status(sc);
+
pi = sc->mfi_comms->hw_pi;
ci = sc->mfi_comms->hw_ci;
mtx_lock(&sc->mfi_io_lock);

--
Neil Schelly
Director of Uptime
Dynamic Network Services, Inc.
W: 603-296-1581
M: 508-410-4776
http://www.dyndns.com