11.0 stuck on high network load

[Slawa Olhovchenkov]

On Mon, Sep 05, 2016 at 11:50:28AM -0600, Warner Losh wrote:

> > How to interoperable with USB stack (I am beware USB keyboard may be locked)?
> 
> I've just done serial console, so I'm not sure. I think that it works...

IPMI or hardware?

> >> I've seen this at Netflix on one variant of our flash offload box with
> >> a Intel e5-2697v2 running with the Chelsio driver. We're working
> >> around it by having fewer receive threads than CPUs in the system. The
> >> only way the boxes would come back was with watchdog. The load was
> >> streaming video > ~36Gbps out 4 lagged 10G ports. Console is totally
> >> unresponsive as well. This is on our FreeBSD-10 stable based fork.
> >> >From my debugging, we go from totally fine as far as I can tell from
> >> ps, etc in the moments leading to the hang to being totally wedged. It
> >> seems a very sudden-onset condition. Sound at all familiar?
> >>
> >> Warner
> >
> > Not sure.
> > This is less power box and can be servered only 20Gbit, using Intel
> > card (lagg 2x10H). Day ago I am using on this box 10-STABLE w/o such
> > issuse. (Not cleancly remember, may be some month ago this box crashed
> > by this issuse -- at the that time I am don't have any ideas about crash)
> 
> OK.
> 
> > May be stuck caused by some poor (too big) memory request from nginx
> > (atempt parsing some malformed files). Or frequent nginx core dump
> > (from this malformed files).
> 
> OK. We're using nginx too, with our modified sendfile.

I am don't use sendfile and use ZFS.

> > 11.0 on two different more power box servered from 40 to 55Gbit w/o stuck.
> > But w/o malformed files (t.e. w/o bogus memory request and w/o nginx
> > crash). Not sure about correlation.
> 
> In our case it seems like a timing issue between too many threads. The
> same hardware can handle 1x40G no probem...

I am already reconfigure NIC to have total number of recive thread as
half of total CPU cores. I am don't see high number of AIO tasks
immediate before stuck (around 100).