network performance

[Stefan Lambrev]

Greetings,

Stefan Lambrev wrote:
> Greetings,
>
> In my desire to increase network throughput, and to be able to handle 
> more then ~250-270kpps
> I started experimenting with lagg and link aggregation control 
> protocol (lacp).
> To my surprise this doesn't increase the amount of packets my server 
> can handle
>
> Here is what netstat reports:
>
> netstat -w1 -I lagg0
>            input        (lagg0)           output
>   packets  errs      bytes    packets  errs      bytes colls
>    267180     0   16030806     254056     0   14735542     0
>    266875     0   16012506     253829     0   14722260     0
>
> netstat -w1 -I em0
>            input          (em0)           output
>   packets  errs      bytes    packets  errs      bytes colls
>    124789 72976    7487340     115329     0    6690468     0
>    126860 67350    7611600     114769     0    6658002     0
>
> netstat -w1 -I em2
>            input          (em2)           output
>   packets  errs      bytes    packets  errs      bytes colls
>    123695 65533    7421700     113575     0    6584856     0
>    130277 62646    7816626     113648     0    6592280     0
>    123545 64171    7412706     113714     0    6596174     0
>
> Using lagg doesn't improve situation at all, and also errors are not 
> reported.
> Also using lagg increased content switches:
>
> procs      memory      page                   disk   faults      cpu
> r b w     avm    fre   flt  re  pi  po    fr  sr ad4   in   sy   cs us 
> sy id
> 1 0 0   81048 1914640    52   0   0   0    50   0   0 3036 37902 
> 13512  1 20 79
> 0 0 0   81048 1914640    13   0   0   0     0   0   0 9582   83 22166  
> 0 56 44
> 0 0 0   81048 1914640    13   0   0   0     0   0   0 9594   80 22028  
> 0 55 45
> 0 0 0   81048 1914640    13   0   0   0     0   0   0 9593   82 22095  
> 0 56 44
>
> Top showed for CPU states +55%   system, which is quite high?
>
> I'll use hwpmc and lock_profiling to see where the kernel spends it's 
> time.
>
Greetings,

Here is what hwpmc shows (without using lagg):

  %   cumulative   self              self     total
 time   seconds   seconds    calls  ms/call  ms/call  name
 14.7  325801.00 325801.00        0  100.00%           MD5Transform [1]
  8.4  512008.00 186207.00        0  100.00%           _mtx_unlock_flags [2]
  6.1  646787.00 134779.00        0  100.00%           _mtx_lock_flags [3]
  5.6  769909.00 123122.00        0  100.00%           uma_zalloc_arg [4]
  5.0  879853.00 109944.00        0  100.00%           rn_match [5]
  3.5  957294.00 77441.00        0  100.00%           memcpy [6]
  3.1 1025989.00 68695.00        0  100.00%           bzero [7]
  2.8 1087273.00 61284.00        0  100.00%           em_encap [8]
  2.6 1145231.00 57958.00        0  100.00%           ip_output [9]
  2.5 1200105.00 54874.00        0  100.00%           
bus_dmamap_load_mbuf_sg [10]
  2.3 1251626.00 51521.00        0  100.00%           syncache_add [11]
  2.1 1297826.50 46200.50        0  100.00%           syncache_lookup [12]
  2.1 1343661.50 45835.00        0  100.00%           tcp_input [13]
  1.8 1383912.00 40250.50        0  100.00%           ip_input [14]
  1.5 1417997.00 34085.00        0  100.00%           syncache_respond [15]
  1.5 1451114.50 33117.50        0  100.00%           uma_zfree_internal 
[16]
  1.5 1484046.00 32931.50        0  100.00%           critical_exit [17]
  1.5 1516899.00 32853.00        0  100.00%           MD5Update [18]

em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=19b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4>
        ether 00:15:17:58:11:a5
        inet 10.3.3.1 netmask 0xffffff00 broadcast 10.3.3.255
        media: Ethernet autoselect (1000baseTX <full-duplex>)
        status: active

Is it normal so much time to be spent in MD5Transform with tx/rx enabled?

LOCK_PROFILING results here - http://89.186.204.158/lock_profiling2.txt

-- 

Best Wishes,
Stefan Lambrev
ICQ# 24134177