Varnish proxy goes catatonic under heavy load

[Matthew Seaman]

On 11/06/14 08:31, Konstantin Belousov wrote:
> I do not remember exact point in the stable/9 lifetime when the
> debug.vn_io_fault_enable was merged.  If it is present in your system,
> frob its value to 1 and see.  I highly suspect that if varnish is in
> 'mmap' mode (whatever it is called), and you use UFS, it may help.

Seems it is not present in 9.1-RELEASE, but it is in 9.2-RELEASE.  I've
toggled that sysctl on the one machine running 9.2-RELEASE, but I doubt
we're going to get any useful results from it before we start upgrading
-- the traffic flood that triggered this was exceptional, and an
isolated incident.

> I am suggesting this before upgrading to 10 only because I want to
> know whether the vn_io_fault code helps in this situation.  There
> are rumors that it does, but I never seen the confirmation.

Hmmm.... well, our theory about this is that we see the effect when the
total traffic is sufficiently high that we're hitting the network
capacity, and dropping some packets.  (The actual traffic load on an
individual server was big, but nothing like saturating the network. It's
the total that was maxing out our uplink to the Internet.)

We simulated the effect by sticking a test box on a 10Mb/s connection
and threw a lot of requests for a largeish (1MB) file at it.  The packet
loss seems to be important -- presumably it's clogging up the available
mbufs with old packets that haven't received an ACK yet, so have to be
held onto in case they need to be resent.

	Cheers,

	Matthew




-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 949 bytes
Desc: OpenPGP digital signature
URL: <http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20141106/e7330cca/attachment.sig>