3Ware 9000 series hangs under load

[John Baldwin]

On Wednesday 29 October 2008 12:07:28 pm Oliver Lehmann wrote:
> Hi,
> 
> I've problems with my 3ware controller. Havingg heavy I/O load (e.g.
> running 40 port builds the day over with tinderbox which involves
> un-taring a whole FreeBSD tree 40 times), my system hangs with the well
> known
> 
> swap_pager: indefinite wait buffer: bufobj: 0, blkno: 2, size: 4096
> swap_pager: indefinite wait buffer: bufobj: 0, blkno: 2, size: 4096
> 
> error. I'v opened a ticket at 3ware and after half a month of
> dummy-testings (are your drives fine, can you run a stress test), it
> looks like i was redirected to someone from the 2nd lvl support and he
> told me:
> 
>   There are 2 things that you can try,
>   1, disable apic in your bootloader.conf file, or RMA the controller.
> 
>   The error that you have is generally caused by an interrupt problem,
>   defective backplane, bad drive or bad controller.
> 
> and after I told him that I intend to use the 2 CPUs I have and not
> falling back to one CPU for ever he responded:
> 
>   Yes I do understand about disabling APIC, but the feature is sometimes
>   not stable in all dual proc systems.  There are many variables, the
>   CPU's have to be matched down to the Lot #, the motherboard must have a
>   good design and the kernel supporting APIC must be stable. But, it is a
>   good test to see if it is software or hardware.
> 
> So what I did now, was compiling a kernel w/o apic/smp and I'm running
> this configuration now for 3 days stressing the system w/o running into
> the swap_pager problem. Can it be still a controller problem or is it
> more likley a problem of FreeBSDs smp/apic implementation or the board
> I'm using (Intel L440GX).
> 
> I'm asking because I'm not sure which problem it is now and before
> telling it 3ware and having them responding "ok it is a FreeBSD problem"
> or "ok it is a board problem" I'd like to know what can be the case here.
> 
> (please keep me CCed, I'm not subscribed to smp@)
> 
> Further information (and the history) on this topic can be found here
> (and following):
> 
> http://lists.freebsd.org/pipermail/freebsd-stable/2008-September/045500.html

FYI, you can disable APIC support w/o recompiling your kernel.  Just 
set 'hint.apic.0.disabled=1' in the loader.  If the problem is that the card 
stops triggering interrupts after being up for a while, then it is likely not 
a FreeBSD bug.  If FreeBSD doesn't get the interrupt routing and setup 
correct then the card will not work at all starting at boot.  You can also 
try just disabling SMP while leaving APIC enabled by 
setting 'kern.smp.disabled=1' from the loader.  If that fixes the issue, then 
it may be that the 3ware driver simply has a race condition that is more 
easily triggered on SMP boxes.

-- 
John Baldwin