11.0 stuck on high network load

[Slawa Olhovchenkov]

On Wed, Sep 21, 2016 at 09:11:24AM +0200, Julien Charbon wrote:

> 
>  You can also use Dtrace and lockstat (especially with the lockstat -s
> option):
> 
> https://wiki.freebsd.org/DTrace/One-Liners#Kernel_Locks
> https://www.freebsd.org/cgi/man.cgi?query=lockstat&manpath=FreeBSD+11.0-RELEASE
> 
>  But I am less familiar with Dtrace/lockstat tools.

I am still use old kernel and got lockdown again.
Try using lockstat (I am save more output), interesting may be next:

R/W writer spin on writer: 190019 events in 1.070 seconds (177571 events/sec)

-------------------------------------------------------------------------------
Count indv cuml rcnt     nsec Lock                   Caller                  
140839  74%  74% 0.00    24659 tcpinp                 tcp_tw_2msl_scan+0xc6   

      nsec ------ Time Distribution ------ count     Stack                   
      4096 |                               913       tcp_twstart+0xa3        
      8192 |@@@@@@@@@@@@                   58191     tcp_do_segment+0x201f   
     16384 |@@@@@@                         29594     tcp_input+0xe1c         
     32768 |@@@@                           23447     ip_input+0x15f          
     65536 |@@@                            16197     
    131072 |@                              8674      
    262144 |                               3358      
    524288 |                               456       
   1048576 |                               9         
-------------------------------------------------------------------------------
Count indv cuml rcnt     nsec Lock                   Caller                  
49180  26% 100% 0.00    15929 tcpinp                 tcp_tw_2msl_scan+0xc6   

      nsec ------ Time Distribution ------ count     Stack                   
      4096 |                               157       pfslowtimo+0x54         
      8192 |@@@@@@@@@@@@@@@                24796     softclock_call_cc+0x179 
     16384 |@@@@@@                         11223     softclock+0x44          
     32768 |@@@@                           7426      intr_event_execute_handlers+0x95
     65536 |@@                             3918      
    131072 |                               1363      
    262144 |                               278       
    524288 |                               19        
-------------------------------------------------------------------------------


> >>  #1. Try above kernel options at least once, and see what you can get.
> > 
> > OK, I am try this after some time.
> > 
> >>  #2. If #1 is a total failure try below patch:  It won't solve anything,
> >> it just makes tcp_tw_2msl_scan() less greedy when there is contention on
> >> the INP write lock.  If it makes the debugging more feasible, continue
> >> to #3.
> > 
> > OK, thanks.
> > What purpose to not skip locked tcptw in this loop?
> 
>  If I understand your question correctly:  According to your pmcstat
> result, tcp_tw_2msl_scan() currently struggles with a write lock
> (__rw_wlock_hard) and the only write lock used tcp_tw_2msl_scan() is
> INP_WLOCK.  No sign of contention on TW_RLOCK(V_tw_lock) currently.
> 
> 51.86%  [2413083]  lock_delay @ /boot/kernel.VSTREAM/kernel
>  100.0%  [2413083]   __rw_wlock_hard
>   100.0%  [2413083]    tcp_tw_2msl_scan
> 
> --
> Julien