svn commit: r285051 - head/sys/netinet

Ermal Luçi eri at freebsd.org
Mon Aug 3 15:05:43 UTC 2015


Hello Olivier,

its strange seeing so much contention on the arp tables on your PMC stats.
Do you run ping(to prepopulate arp) or static arp to remove the noise from
that interaction?

Also do you run with flowtable active?

On Mon, Aug 3, 2015 at 3:06 PM, Olivier Cochard-Labbé <olivier at cochard.me>
wrote:

> On Tue, Jul 28, 2015 at 2:42 PM, Gleb Smirnoff <glebius at freebsd.org>
> wrote:
>
>>
>> Here is suggested patch. Ermal and Oliver, can you please test/benchmark
>> it?
>>
>
> ​Hi,
>
> this patch reduce performanece :-(
>
> Here are the results regarding forwarding:
>
> x r285046.pps.forwarding (IPSEC compiled but not used)
> + r285051.pps.forwarding (IPSEC compiled but not used)
> * r285051-glebius-patched.pps.forwarding (IPSEC compiled but not used)
>
> +--------------------------------------------------------------------------------+
> |xxx     xx                ****
> *                                           + +++|
> ||_M_A___|
> |_A_|                                             |AM||
>
> +--------------------------------------------------------------------------------+
>     N           Min           Max        Median           Avg        Stddev
> x   5        397733        406951        399300      401613.8     4324.9755
> +   5        478095        482079        480869      480543.6     1666.0282
> Difference at 95.0% confidence
>         78929.8 +/- 4779.72
>         19.6532% +/- 1.19013%
>         (Student's t, pooled s = 3277.27)
> *   5        424720        430745        427014      427378.4     2351.7439
> Difference at 95.0% confidence
>         25764.6 +/- 5076.98
>         6.41527% +/- 1.26415%
>         (Student's t, pooled s = 3481.1)
>>
>
> PMC stats during forwarding bench:
> [root at netgate]/data# pmcannotate pmc.forwarding.out
> /data/debug/boot/kernel/kernel.symbols
> CONVERSION STATISTICS:
>  #samples/total                           33880
> Profile trace for function: __rw_rlock() [6.29%]
> Profile trace for function: ip_forward() [4.68%]
> Profile trace for function: ip_output() [4.64%]
> Profile trace for function: binuptime() [4.05%]
> Profile trace for function: igb_mq_start_locked() [3.79%]
> Profile trace for function: igb_rxeof() [3.46%]
> Profile trace for function: tsc_get_timecount_low_lfence() [3.25%]
> Profile trace for function: ether_output() [3.03%]
> Profile trace for function: rtalloc1_fib() [2.77%]
> Profile trace for function: random_ivy_read() [2.64%]
> Profile trace for function: _rw_runlock_cookie() [2.64%]
> Profile trace for function: ether_nh_input() [2.63%]
> Profile trace for function: ip_input() [2.55%]
> Profile trace for function: key_allocsp_default() [2.39%]
> Profile trace for function: igb_mq_start() [2.39%]
> Profile trace for function: bzero() [2.08%]
> Profile trace for function: uma_zalloc_arg() [1.95%]
> Profile trace for function: memcpy() [1.84%]
> Profile trace for function: _mtx_lock_spin_cookie() [1.83%]
> Profile trace for function: bcopy() [1.76%]
> Profile trace for function: random_harvest_queue() [1.63%]
> Profile trace for function: __mtx_lock_sleep() [1.56%]
> Profile trace for function: uma_zfree_arg() [1.47%]
> Profile trace for function: arpresolve() [1.39%]
> Profile trace for function: in_cksumdata() [1.25%]
> Profile trace for function: bounce_bus_dmamap_load_buffer() [1.22%]
> Profile trace for function: bcmp() [1.13%]
> Profile trace for function: rtalloc_ign_fib() [1.11%]
> Profile trace for function: rn_match() [1.03%]
> Profile trace for function: netisr_dispatch_src() [1.03%]
> Profile trace for function: critical_exit() [1.02%]
> Profile trace for function: bus_dmamap_load_mbuf_sg() [0.87%]
> Profile trace for function: spinlock_exit() [0.79%]
> Profile trace for function: in_cksum_skip() [0.75%]
> Profile trace for function: ip_ipsec_output() [0.75%]
> Profile trace for function: acpi_cpu_c1() [0.74%]
> Profile trace for function: in_broadcast() [0.74%]
> Profile trace for function: spinlock_enter() [0.74%]
> Profile trace for function: igb_refresh_mbufs() [0.71%]
> Profile trace for function: in_lltable_lookup() [0.71%]
> Profile trace for function: ip_fastforward() [0.68%]
> Profile trace for function: m_adj() [0.65%]
> Profile trace for function: ether_demux() [0.65%]
> Profile trace for function: _key_freesp() [0.61%]
> Profile trace for function: lockstat_nsecs() [0.60%]
> Profile trace for function: m_freem() [0.58%]
> Profile trace for function: critical_enter() [0.56%]
> Profile trace for function: m_copydata() [0.55%]
> Profile trace for function: mb_free_ext() [0.54%]
> Profile trace for function: pmap_kextract() [0.50%]
>
>>
> ​And about fastforwarding:
>> x 285046.pps.fastforwarding (IPSEC compiled but not used)
> + 285051.pps.fastforwarding (IPSEC compiled but not used)
> * r285051-glebius-patched.pps.fastforwarding (IPSEC compiled but not used)
>
> +--------------------------------------------------------------------------------+
> |*
> +            |
> |*     * * *                                                        ++ +x
> x+ xx x|
> ||____AM__|
> |_MA__|__AM_| |
>
> +--------------------------------------------------------------------------------+
>     N           Min           Max        Median           Avg        Stddev
> x   5        742683        754709        750940      749015.6     4944.9059
> +   5        736459        747593        738197      740112.4     4511.1611
> Difference at 95.0% confidence
>         -8903.2 +/- 6902.82
>         -1.18865% +/- 0.921585%
>         (Student's t, pooled s = 4733)
> *   5        638356        652855        646668        645263     6599.8604
> Difference at 95.0% confidence
>         -103753 +/- 8504.75
>         -13.8519% +/- 1.13546%
>         (Student's t, pooled s = 5831.39)
>
> And pmcstat during fastforwarding:
>
> [root at netgate]/data# pmcannotate pmc.fastforwading.out
> /data/debug/boot/kernel/kernel.symbols
> CONVERSION STATISTICS:
>  #samples/total                           37656
> Profile trace for function: __rw_rlock() [9.12%]
> Profile trace for function: ip_fastforward() [5.80%]
> Profile trace for function: binuptime() [5.73%]
> Profile trace for function: rtalloc1_fib() [4.92%]
> Profile trace for function: tsc_get_timecount_low_lfence() [4.71%]
> Profile trace for function: igb_rxeof() [4.51%]
> Profile trace for function: _rw_runlock_cookie() [4.14%]
> Profile trace for function: igb_mq_start_locked() [4.12%]
> Profile trace for function: ether_output() [3.36%]
> Profile trace for function: ether_nh_input() [3.11%]
> Profile trace for function: igb_mq_start() [3.02%]
> Profile trace for function: random_ivy_read() [2.88%]
> Profile trace for function: bzero() [2.70%]
> Profile trace for function: memcpy() [2.54%]
> Profile trace for function: random_harvest_queue() [2.32%]
> Profile trace for function: __mtx_lock_sleep() [1.91%]
> Profile trace for function: bcmp() [1.78%]
> Profile trace for function: _mtx_lock_spin_cookie() [1.73%]
> Profile trace for function: igb_txeof() [1.70%]
> Profile trace for function: rn_match() [1.69%]
> Profile trace for function: bounce_bus_dmamap_load_buffer() [1.52%]
> Profile trace for function: rtalloc_ign_fib() [1.30%]
> Profile trace for function: mb_free_ext() [1.29%]
> Profile trace for function: arpresolve() [1.28%]
> Profile trace for function: bcopy() [1.22%]
> Profile trace for function: spinlock_exit() [1.06%]
> Profile trace for function: mb_ctor_pack() [1.04%]
> Profile trace for function: uma_zalloc_arg() [1.02%]
> Profile trace for function: spinlock_enter() [1.00%]
> Profile trace for function: bus_dmamap_load_mbuf_sg() [0.93%]
> Profile trace for function: critical_exit() [0.90%]
> Profile trace for function: igb_refresh_mbufs() [0.89%]
> Profile trace for function: acpi_cpu_c1() [0.89%]
> Profile trace for function: netisr_dispatch_src() [0.81%]
> Profile trace for function: uma_zfree_arg() [0.80%]
> Profile trace for function: lockstat_nsecs() [0.79%]
> Profile trace for function: ether_demux() [0.76%]
> Profile trace for function: in_lltable_lookup() [0.75%]
> Profile trace for function: ip_findroute() [0.66%]
> Profile trace for function: in_localip() [0.65%]
> Profile trace for function: _mtx_trylock_flags_() [0.57%]
> Profile trace for function: _bus_dmamap_load_buffer() [0.54%]
>
> ​Regards,
>
> Olivier​
>



-- 
Ermal


More information about the svn-src-all mailing list