ZFS "stalls" -- and maybe we should be talking about defaults?

[Karl Denninger]

On 3/4/2013 9:25 PM, Steven Hartland wrote:
> ----- Original Message ----- From: "Karl Denninger" <karl at denninger.net>
>
>> Stick this in /boot/loader.conf and see if your lockups goes away:
>>
>> vfs.zfs.write_limit_override=1024000000
> ...
>
>> If it turns out that the write_limit_override tunable is the one
>> responsible for stopping the hangs I can drop the ARC limit tunable
>> although I'm not sure I want to; I don't see much if any performance
>> penalty from leaving it where it is and if the larger cache isn't
>> helping anything then why use it?  I'm inclined to stick an SSD in the
>> cabinet as a cache drive instead of dedicating RAM to this -- even
>> though it's not AS fast as RAM it's still MASSIVELY quicker than getting
>> data off a rotating plate of rust.
>
> Now interesting you should say that I've seen a stall recently on ZFS
> only box running on 6 x SSD RAIDZ2.
>
> The stall was caused by fairly large mysql import, with nothing else
> running.
>
> Then it happened I thought the machine had wedged, but minutes (not
> seconds) later, everything sprung into action again.

That's exactly what I can reproduce here; the stalls are anywhere from a
few seconds to well north of a half-minute.  It looks like the machine
is hung -- but it is not.

The machine in question normally runs with zero swap allocated but it
always has 1.5Gb of shared memory allocated to Postgres ("shared_buffers
= 1500MB" in its config file)

I wonder if the ARC cache management code is misbehaving when shared
segments are in use?

-- 
-- Karl Denninger
/The Market Ticker ®/ <http://market-ticker.org>
Cuda Systems LLC