Varnish proxy goes catatonic under heavy load

[Konstantin Belousov]

On Thu, Nov 06, 2014 at 07:11:38AM +0000, Matthew Seaman wrote:
> On 05/11/2014 12:00, Steven Hartland wrote:
> > As a guess you exhausted all mbufs, 10 has much better defaults for
> > these so I'd recommend updating.
> > 
> > If you can get in via IPMI or something similar you should be able to
> > confirm.
> > 
> > A trick I've used in the past to recover from such a issue is to hard
> > bounce the nic ports on the switch which seemed to free enough to be
> > able to ssh in.
> 
> Yes, you nailed it.  We managed to recreate the effect in the lab, and
> 10.0 behaves much better under horrible overload.  While horribly slow,
> we can still get to a shell prompt via ssh, and when we turn off the
> load, the system recovers straight away.
> 
> We'll be upgrading to 10.x ASAP.
> 

I do not remember exact point in the stable/9 lifetime when the
debug.vn_io_fault_enable was merged.  If it is present in your system,
frob its value to 1 and see.  I highly suspect that if varnish is in
'mmap' mode (whatever it is called), and you use UFS, it may help.

I am suggesting this before upgrading to 10 only because I want to
know whether the vn_io_fault code helps in this situation.  There
are rumors that it does, but I never seen the confirmation.