syncer causing latency spikes

[Steven Hartland]

----- Original Message ----- 
From: "John Baldwin" <jhb at freebsd.org>

> On Wednesday, July 17, 2013 3:18:52 pm Konstantin Belousov wrote:
>> On Wed, Jul 17, 2013 at 02:07:55PM -0400, Mark Johnston wrote:
>> > During such an fsync, DTrace shows me that syncer sleeps of 50-200ms are
>> > happening up to 8 or 10 times a second. When this happens, a bunch of
>> > postgres threads become blocked in vn_write() waiting for the vnode lock
>> > to become free. It looks like the write-clustering code is limited to
>> > using (nswbuf / 2) pbufs, and FreeBSD prevents one from setting nswbuf
>> > to anything greater than 256.
>> Syncer is probably just a victim of profiling.  Would postgres called
>> fsync(2), you then blame the fsync code for the pauses.
>> 
>> Just add a tunable to allow the user to manually-tune the nswbuf,
>> regardless of the buffer cache sizing.  And yes, nswbuf default max
>> probably should be bumped to something like 1024, at least on 64bit
>> architectures which do not starve for kernel memory.
> 
> Also, if you are seeing I/O stalls with mfi(4), then you might need a
> firmware update for your mfi(4) controller.  cc'ing smh@ who knows more about 
> that particular issue (IIRC).

Indeed if your seeing any IO timeouts in /var/log/messages or the console
there is a know issue in older mfi(4) firmware which can cause extended
IO stalls.

We believe that fixed firmware packages include "APP" version > *.130.*
For Dell branded HW that corrisponds with FW package version 21.2.1-0000

    Regards
    Steve