Varnish proxy goes catatonic under heavy load
kostikbel at gmail.com
Thu Nov 6 08:31:59 UTC 2014
On Thu, Nov 06, 2014 at 07:11:38AM +0000, Matthew Seaman wrote:
> On 05/11/2014 12:00, Steven Hartland wrote:
> > As a guess you exhausted all mbufs, 10 has much better defaults for
> > these so I'd recommend updating.
> > If you can get in via IPMI or something similar you should be able to
> > confirm.
> > A trick I've used in the past to recover from such a issue is to hard
> > bounce the nic ports on the switch which seemed to free enough to be
> > able to ssh in.
> Yes, you nailed it. We managed to recreate the effect in the lab, and
> 10.0 behaves much better under horrible overload. While horribly slow,
> we can still get to a shell prompt via ssh, and when we turn off the
> load, the system recovers straight away.
> We'll be upgrading to 10.x ASAP.
I do not remember exact point in the stable/9 lifetime when the
debug.vn_io_fault_enable was merged. If it is present in your system,
frob its value to 1 and see. I highly suspect that if varnish is in
'mmap' mode (whatever it is called), and you use UFS, it may help.
I am suggesting this before upgrading to 10 only because I want to
know whether the vn_io_fault code helps in this situation. There
are rumors that it does, but I never seen the confirmation.
More information about the freebsd-stable