11.0 stuck on high network load

[Slawa Olhovchenkov]

On Mon, Sep 19, 2016 at 10:32:13PM +0200, Julien Charbon wrote:

> 
> > @ CPU_CLK_UNHALTED_CORE [4653445 samples]
> > 
> > 51.86%  [2413083]  lock_delay @ /boot/kernel.VSTREAM/kernel
> >  100.0%  [2413083]   __rw_wlock_hard
> >   100.0%  [2413083]    tcp_tw_2msl_scan
> >    99.99%  [2412958]     pfslowtimo
> >     100.0%  [2412958]      softclock_call_cc
> >      100.0%  [2412958]       softclock
> >       100.0%  [2412958]        intr_event_execute_handlers
> >        100.0%  [2412958]         ithread_loop
> >         100.0%  [2412958]          fork_exit
> >    00.01%  [125]         tcp_twstart
> >     100.0%  [125]          tcp_do_segment
> >      100.0%  [125]           tcp_input
> >       100.0%  [125]            ip_input
> >        100.0%  [125]             swi_net
> >         100.0%  [125]              intr_event_execute_handlers
> >          100.0%  [125]               ithread_loop
> >           100.0%  [125]                fork_exit
> 
>  The only write lock tcp_tw_2msl_scan() tries to get is a
> INP_WLOCK(inp).  Thus here, tcp_tw_2msl_scan() seems to be stuck
> spinning on INP_WLOCK (or pfslowtimo() is going crazy and calls
> tcp_tw_2msl_scan() at high rate but this will be quite unexpected).
> 
>  Thus my hypothesis is that something is holding the INP_WLOCK and not
> releasing it, and tcp_tw_2msl_scan() is spinning on it.
> 
>  If you can, could you compile the kernel with below options:
> 
> options        DDB                     # Support DDB.
> options        DEADLKRES               # Enable the deadlock resolver
> options        INVARIANTS              # Enable calls of extra sanity
> checking
> options        INVARIANT_SUPPORT       # Extra sanity checks of internal
> structures, required by INVARIANTS
> options        WITNESS                 # Enable checks to detect
> deadlocks and cycles
> options        WITNESS_SKIPSPIN        # Don't run witness on spinlocks
> for speed

Currently this host run with 100% CPU load (on all cores), i.e.
enabling WITNESS will be significant drop performance.
Can I use only some subset of options?

Also, I can some troubles to DDB enter in this case.
May be kgdb will be success (not tryed yet)?

>  And once the issue is reproduce, run in ddb run the below commands:
> 
> show pcpu
> show allpcpu
> show locks
> show alllocks
> show lockchain
> show allchains
> show all trace
> 
>  This is to see if the contention is indeed on the tcp_tw_2msl_scan's
> INP_WLOCK.