svn commit: r266757 - in head/sys: conf dev/cxgbe dev/cxgbe/common modules/cxgbe/if_cxgbe

[Navdeep Parhar]

On 05/27/14 17:36, Luigi Rizzo wrote:
> 
> 
> 
> On Tue, May 27, 2014 at 8:18 PM, Navdeep Parhar <np at freebsd.org
> <mailto:np at freebsd.org>> wrote:
> 
>     Author: np
>     Date: Tue May 27 18:18:41 2014
>     New Revision: 266757
>     URL: http://svnweb.freebsd.org/changeset/base/266757
> 
>     Log:
>       cxgbe(4): netmap support for Terminator 5 (T5) based 10G/40G cards.
>       Netmap gets its own hardware-assisted virtual interface and won't take
>       over or disrupt the "normal" interface in any way.  You can use both
>       simultaneously.
> 
>       For kernels with DEV_NETMAP, cxgbe(4) carves out an ncxl<N> interface
>       (note the 'n' prefix) in the hardware to accompany each cxl<N>
>       interface.  These two ifnet's per port share the same wire but really
>       are separate interfaces in the hardware and software.  Each gets
>     its own
>       L2 MAC addresses (unicast and multicast), MTU, checksum caps, etc.
>      You
>       should run netmap on the 'n' interfaces only, that's what they are
>     for.
> 
>       With this, pkt-gen is able to transmit > 45Mpps out of a single
>     40G port
> 
> 
> ​and just for the records, this is with 1 core in userland.
> Very cool.
> 
> Do you have an easy way to check what kind of throughput
> you get when the netmap interface has one hw queue ?

Both tx and rx are at around ~33Mpps right now with one hardware queue.
 That's more than enough to saturate a 40G link with 128B sized frames.

Regards,
Navdeep

> 
> cheers
> luigi
> 
>       of a T580 card.  2 port tx is at ~56Mpps total (28M + 28M) as of now.
>       Single port receive is at 33Mpps but this is very much a work in
>       progress.  I expect it to be closer to 40Mpps once done.  In any case
>       the current effort can already saturate multiple 10G ports of a T5
>     card
>       at the smallest legal packet size.  T4 gear is totally untested.
> 
>       trantor:~# ./pkt-gen -i ncxl0 -f tx -D 00:07:43:ab:cd:ef
>       881.952141 main [1621] interface is ncxl0
>       881.952250 extract_ip_range [275] range is 10.0.0.1:0
>     <http://10.0.0.1:0> to 10.0.0.1:0 <http://10.0.0.1:0>
>       881.952253 extract_ip_range [275] range is 10.1.0.1:0
>     <http://10.1.0.1:0> to 10.1.0.1:0 <http://10.1.0.1:0>
>       881.962540 main [1804] mapped 334980KB at 0x801dff000
>       Sending on netmap:ncxl0: 4 queues, 1 threads and 1 cpus.
>       10.0.0.1 -> 10.1.0.1 (00:00:00:00:00:00 -> 00:07:43:ab:cd:ef)
>       881.962562 main [1882] Sending 512 packets every  0.000000000 s
>       881.962563 main [1884] Wait 2 secs for phy reset
>       884.088516 main [1886] Ready...
>       884.088535 nm_open [457] overriding ifname ncxl0 ringid 0x0 flags 0x1
>       884.088607 sender_body [996] start
>       884.093246 sender_body [1064] drop copy
>       885.090435 main_thread [1418] 45206353 pps (45289533 pkts in
>     1001840 usec)
>       886.091600 main_thread [1418] 45322792 pps (45375593 pkts in
>     1001165 usec)
>       887.092435 main_thread [1418] 45313992 pps (45351784 pkts in
>     1000834 usec)
>       888.094434 main_thread [1418] 45315765 pps (45406397 pkts in
>     1002000 usec)
>       889.095434 main_thread [1418] 45333218 pps (45378551 pkts in
>     1001000 usec)
>       890.097434 main_thread [1418] 45315247 pps (45405877 pkts in
>     1002000 usec)
>       891.099434 main_thread [1418] 45326515 pps (45417168 pkts in
>     1002000 usec)
>       892.101434 main_thread [1418] 45333039 pps (45423705 pkts in
>     1002000 usec)
>       893.103434 main_thread [1418] 45324105 pps (45414708 pkts in
>     1001999 usec)
>       894.105434 main_thread [1418] 45318042 pps (45408723 pkts in
>     1002001 usec)
>       895.106434 main_thread [1418] 45332430 pps (45377762 pkts in
>     1001000 usec)
>       896.107434 main_thread [1418] 45338072 pps (45383410 pkts in
>     1001000 usec)
>       ...
> 
>       Relnotes:     Yes
>       Sponsored by: Chelsio Communications.
> 
>     Added:
>       head/sys/dev/cxgbe/t4_netmap.c   (contents, props changed)
>     Modified:
>       head/sys/conf/files
>       head/sys/dev/cxgbe/adapter.h
>       head/sys/dev/cxgbe/common/common.h
>       head/sys/dev/cxgbe/common/t4_hw.c
>       head/sys/dev/cxgbe/t4_main.c
>       head/sys/dev/cxgbe/t4_sge.c
>       head/sys/modules/cxgbe/if_cxgbe/Makefile
>