Netmap ixgbe driver problem on 10Gbps X540-AT2 adapters (Linux only?)
Walter de Donato
walter.dedonato at unina.it
Fri Mar 22 10:18:07 UTC 2013
Dear Tahir,
I've already tried to disable Rx/Tx pause and autonegotiation but the
result in transmission is always the same.
I did the following:
$ sudo ifconfig eth2 up
$ sudo ethtool -A eth2 autoneg off
$ sudo ethtool -A eth2 rx off
$ sudo ethtool -A eth2 tx off
These are the resulting dmesg output lines:
[253131.445124] ixgbe 0000:0c:00.0: eth2: NIC Link is Up 10 Gbps, Flow
Control: RX/TX
[253147.935426] ixgbe 0000:0c:00.0: eth2: NIC Link is Up 10 Gbps, Flow
Control: TX
[253163.930776] ixgbe 0000:0c:00.0: eth2: NIC Link is Up 10 Gbps, Flow
Control: None
Here is what I obtain:
$ sudo ./pkt-gen -i eth2 -n 100000 -f tx
extract_ip_range [136] extract IP range from 10.0.0.1
extract_ip_range [171] range is 10.0.0.1 0 to 10.0.0.1 0
extract_ip_range [136] extract IP range from 10.1.0.1
extract_ip_range [171] range is 10.1.0.1 0 to 10.1.0.1 0
extract_mac_range [177] extract MAC range from a0:36:9f:0d:f0:98
extract_mac_range [192] a0:36:9f:0d:f0:98 starts at a0:36:9f:d:f0:98
extract_mac_range [177] extract MAC range from ff:ff:ff:ff:ff:ff
extract_mac_range [192] ff:ff:ff:ff:ff:ff starts at ff:ff:ff:ff:ff:ff
main [1401] map size is 334980 Kb
main [1423] mapping 334980 Kbytes
Sending on eth2: 4 queues, 1 threads and 1 cpus.
10.0.0.1 -> 10.1.0.1 (a0:36:9f:0d:f0:98 -> ff:ff:ff:ff:ff:ff)
main [1477] Wait 2 secs for phy reset
main [1479] Ready...
sender_body [683] start
main_thread [1078] 7155 pps (7162 pkts in 1001020 usec)
main_thread [1078] 3484 pps (3488 pkts in 1001030 usec)
main_thread [1078] 3487 pps (3491 pkts in 1001024 usec)
main_thread [1078] 3466 pps (3469 pkts in 1000847 usec)
main_thread [1078] 3485 pps (3490 pkts in 1001521 usec)
main_thread [1078] 3322 pps (3328 pkts in 1001765 usec)
main_thread [1078] 3293 pps (3297 pkts in 1001322 usec)
main_thread [1078] 3492 pps (3498 pkts in 1001763 usec)
main_thread [1078] 3485 pps (3489 pkts in 1001168 usec)
main_thread [1078] 3351 pps (3356 pkts in 1001402 usec)
main_thread [1078] 3297 pps (3301 pkts in 1001314 usec)
main_thread [1078] 3933 pps (3938 pkts in 1001390 usec)
main_thread [1078] 3481 pps (3486 pkts in 1001528 usec)
main_thread [1078] 3493 pps (3497 pkts in 1001177 usec)
main_thread [1078] 3310 pps (3314 pkts in 1001292 usec)
main_thread [1078] 3485 pps (3491 pkts in 1001812 usec)
main_thread [1078] 3326 pps (3331 pkts in 1001547 usec)
main_thread [1078] 3480 pps (3485 pkts in 1001456 usec)
main_thread [1078] 3489 pps (3493 pkts in 1001209 usec)
main_thread [1078] 3377 pps (3381 pkts in 1001040 usec)
main_thread [1078] 3450 pps (3454 pkts in 1001048 usec)
main_thread [1078] 3378 pps (3382 pkts in 1001050 usec)
main_thread [1078] 3470 pps (3474 pkts in 1001052 usec)
main_thread [1078] 3323 pps (3327 pkts in 1001056 usec)
main_thread [1078] 3482 pps (3486 pkts in 1001050 usec)
main_thread [1078] 3472 pps (3476 pkts in 1001066 usec)
main_thread [1078] 3497 pps (3501 pkts in 1001044 usec)
main_thread [1078] 3115 pps (3115 pkts in 1000035 usec)
main_thread [1078] 0 pps (0 pkts in 1001057 usec)
Sent 100000 packets, 60 bytes each, in 28.42 seconds.
Speed: 3.52 Kpps Bandwidth: 1.69 Mbps (raw 2.36 Mbps)
While executing pkt-gen I see the following dmesg output lines:
[253741.892031] ixgbe 0000:0c:00.0: eth2: Reset adapter
[253742.874686] ixgbe 0000:0c:00.0: eth2: Reset adapter
[253743.864635] ixgbe 0000:0c:00.0: eth2: Reset adapter
[253744.854513] ixgbe 0000:0c:00.0: eth2: Reset adapter
[253745.844527] ixgbe 0000:0c:00.0: eth2: Reset adapter
[253746.830323] ixgbe 0000:0c:00.0: eth2: Reset adapter
[253747.821223] ixgbe 0000:0c:00.0: eth2: Reset adapter
[253748.813229] ixgbe 0000:0c:00.0: eth2: Reset adapter
[253749.804317] ixgbe 0000:0c:00.0: eth2: Reset adapter
[253750.794286] ixgbe 0000:0c:00.0: eth2: Reset adapter
[253751.784302] ixgbe 0000:0c:00.0: eth2: Reset adapter
[253752.774324] ixgbe 0000:0c:00.0: eth2: Reset adapter
[253753.764258] ixgbe 0000:0c:00.0: eth2: Reset adapter
[253754.753735] ixgbe 0000:0c:00.0: eth2: Reset adapter
[253755.748011] ixgbe 0000:0c:00.0: eth2: Reset adapter
[253756.734927] ixgbe 0000:0c:00.0: eth2: Reset adapter
[253757.724876] ixgbe 0000:0c:00.0: eth2: Reset adapter
[253758.713267] ixgbe 0000:0c:00.0: eth2: Reset adapter
[253759.704793] ixgbe 0000:0c:00.0: eth2: Reset adapter
[253760.694759] ixgbe 0000:0c:00.0: eth2: Reset adapter
[253761.684743] ixgbe 0000:0c:00.0: eth2: Reset adapter
[253762.674699] ixgbe 0000:0c:00.0: eth2: Reset adapter
[253763.664642] ixgbe 0000:0c:00.0: eth2: Reset adapter
[253764.654488] ixgbe 0000:0c:00.0: eth2: Reset adapter
[253765.644462] ixgbe 0000:0c:00.0: eth2: Reset adapter
[253766.634425] ixgbe 0000:0c:00.0: eth2: Reset adapter
[253767.624392] ixgbe 0000:0c:00.0: eth2: Reset adapter
[253768.614364] ixgbe 0000:0c:00.0: eth2: Reset adapter
[253769.611107] ixgbe 0000:0c:00.0: eth2: Reset adapter
[253769.715298] 893.960705 netmap_ring_reinit [966] called for eth2
[253769.716646] 893.962054 netmap_ring_reinit [966] called for eth2
[253769.725844] 893.971252 netmap_ioctl [1236] deprecated, data is ded4df20
[253776.141936] ixgbe 0000:0c:00.0: eth2: NIC Link is Up 10 Gbps, Flow
Control: None
It seems that the adapter is reset every second, but no packet goes on the
wire.
I'm starting to look into the code to investigate this issue.
If you have any suggestion on where to look at it would save me a lot of
time.
I also rechecked the rx mode.
It is working at least up to 50Kpps (as I'm generation on the other side
with nping and the original driver).
Pkg-gen/netmap is configured to receive only the frames directed to the
physical MAC address of the interface.
I don't know if and how a promiscuous mode can be enabled with netmap.
Thanks,
-Walter
2013/3/22 Tahir Rauf <tahir.rauf1 at gmail.com>
> Dear Walter,
>
> Can you please disable Rx/Tx pause frames on your NIC. You can use
> "ethtool" to do so. Please run the example again after disabling Rx/Tx
> frames and let me know about the results.
>
> Regards
>
>
> On Thu, Mar 21, 2013 at 8:20 PM, Walter de Donato <
> walter.dedonato at gmail.com> wrote:
>
>> Not that lucky since my first goal is to transmit packets...
>> I also noticed that the same car d is configured with 4 rings on 32bit
>> kernels and 16 rings on 64bit kernels.
>> Do you have any idea to explain this difference?
>>
>> -Walter
>>
>> 2013/3/21 Tahir Rauf <tahir.rauf1 at gmail.com>
>>
>>> Dear Walter,
>>>
>>> You are lucky to have* X540-AT2 :). *Mine is different (82598EB) and
>>> can't receive packets properly.
>>> Yes, my tx is working and is able to send packets on line rate.
>>>
>>> Regards
>>>
>>>
>>> On Thu, Mar 21, 2013 at 8:04 PM, Walter de Donato <
>>> walter.dedonato at gmail.com> wrote:
>>>
>>>> Here is the lshw output:
>>>>
>>>> *-network:0
>>>> description: Ethernet interface
>>>> product: Ethernet Controller 10 Gigabit X540-AT2
>>>> vendor: Intel Corporation
>>>> physical id: 0
>>>> bus info: pci at 0000:0c:00.0
>>>> logical name: eth2
>>>> version: 01
>>>> serial: a0:36:9f:0d:f0:98
>>>> capacity: 1Gbit/s
>>>> width: 64 bits
>>>> clock: 33MHz
>>>> capabilities: pm msi msix pciexpress vpd bus_master cap_list rom
>>>> ethernet physical tp 100bt-fd 1000bt-fd autonegotiation
>>>> configuration: autonegotiation=on broadcast=yes driver=ixgbe
>>>> driverversion=3.6.7-k duplex=full firmware=0x800002ef latency=0 link=yes
>>>> multicast=yes port=twisted pair
>>>> resources: irq:17 memory:d8400000-d85fffff
>>>> memory:d83fc000-d83fffff memory:fc500000-fc57ffff memory:fc400000-fc4fffff
>>>> *-network:1
>>>> description: Ethernet interface
>>>> product: Ethernet Controller 10 Gigabit X540-AT2
>>>> vendor: Intel Corporation
>>>> physical id: 0.1
>>>> bus info: pci at 0000:0c:00.1
>>>> logical name: eth3
>>>> version: 01
>>>> serial: a0:36:9f:0d:f0:9a
>>>> capacity: 1Gbit/s
>>>> width: 64 bits
>>>> clock: 33MHz
>>>> capabilities: pm msi msix pciexpress vpd bus_master cap_list rom
>>>> ethernet physical tp 100bt-fd 1000bt-fd autonegotiation
>>>> configuration: autonegotiation=on broadcast=yes driver=ixgbe
>>>> driverversion=3.6.7-k duplex=full firmware=0x800002ef ip=192.168.78.2
>>>> latency=0 link=yes multicast=yes port=twisted pair
>>>> resources: irq:16 memory:d8000000-d81fffff
>>>> memory:d83f8000-d83fbfff memory:d8200000-d827ffff
>>>>
>>>> Did you check if in tx mode your pkt-gen effectively sends the packets
>>>> on the wire?
>>>> What rates are you observing with it?
>>>>
>>>> -Walter
>>>>
>>>> 2013/3/21 Tahir Rauf <tahir.rauf1 at gmail.com>
>>>>
>>>>> Dear walter,
>>>>>
>>>>> Can you please run following command to get your NIC information.
>>>>> sudo lshw -class network
>>>>>
>>>>> I have *82598EB *10GB AF dual port network connection NIC and my rx
>>>>> is not working.
>>>>>
>>>>> Regards
>>>>>
>>>>>
>>>>> On Thu, Mar 21, 2013 at 7:46 PM, Walter de Donato <
>>>>> walter.dedonato at gmail.com> wrote:
>>>>>
>>>>>> I tried the rx command and it seems to work:
>>>>>> $ sudo ./pkt-gen -i eth3 -f rx -n 100000 -l 60 -w 1
>>>>>> extract_ip_range [136] extract IP range from 10.0.0.1
>>>>>> extract_ip_range [171] range is 10.0.0.1 0 to 10.0.0.1 0
>>>>>> extract_ip_range [136] extract IP range from 10.1.0.1
>>>>>> extract_ip_range [171] range is 10.1.0.1 0 to 10.1.0.1 0
>>>>>> extract_mac_range [177] extract MAC range from a0:36:9f:0d:f0:9a
>>>>>> extract_mac_range [192] a0:36:9f:0d:f0:9a starts at a0:36:9f:d:f0:9a
>>>>>> extract_mac_range [177] extract MAC range from ff:ff:ff:ff:ff:ff
>>>>>> extract_mac_range [192] ff:ff:ff:ff:ff:ff starts at ff:ff:ff:ff:ff:ff
>>>>>> main [1401] map size is 334980 Kb
>>>>>> main [1423] mapping 334980 Kbytes
>>>>>> Receiving from eth3: 4 queues, 1 threads and 1 cpus.
>>>>>> main [1477] Wait 1 secs for phy reset
>>>>>> main [1479] Ready...
>>>>>> main_thread [1078] 0 pps (0 pkts in 1001020 usec)
>>>>>> receiver_body [833] waiting for initial packets, poll returns 0 0
>>>>>> main_thread [1078] 0 pps (0 pkts in 1001059 usec)
>>>>>> receiver_body [833] waiting for initial packets, poll returns 0 0
>>>>>> main_thread [1078] 0 pps (0 pkts in 1001047 usec)
>>>>>> receiver_body [833] waiting for initial packets, poll returns 0 0
>>>>>> main_thread [1078] 2 pps (2 pkts in 1001118 usec)
>>>>>> main_thread [1078] 1 pps (1 pkts in 1001040 usec)
>>>>>> main_thread [1078] 1 pps (1 pkts in 1001038 usec)
>>>>>> main_thread [1078] 1 pps (1 pkts in 1001030 usec)
>>>>>> main_thread [1078] 1 pps (1 pkts in 1001028 usec)
>>>>>> main_thread [1078] 1 pps (1 pkts in 1001051 usec)
>>>>>> main_thread [1078] 1 pps (1 pkts in 1001068 usec)
>>>>>> main_thread [1078] 2 pps (2 pkts in 1001040 usec)
>>>>>> main_thread [1078] 2 pps (2 pkts in 1001035 usec)
>>>>>> main_thread [1078] 2 pps (2 pkts in 1001042 usec)
>>>>>> main_thread [1078] 2 pps (2 pkts in 1001083 usec)
>>>>>> main_thread [1078] 2 pps (2 pkts in 1001050 usec)
>>>>>> main_thread [1078] 2 pps (2 pkts in 1001073 usec)
>>>>>> main_thread [1078] 2 pps (2 pkts in 1001052 usec)
>>>>>> main_thread [1078] 2 pps (2 pkts in 1001041 usec)
>>>>>> main_thread [1078] 2 pps (2 pkts in 1001051 usec)
>>>>>> main_thread [1078] 1 pps (1 pkts in 1001058 usec)
>>>>>> main_thread [1078] 1 pps (1 pkts in 1001020 usec)
>>>>>> main_thread [1078] 2 pps (2 pkts in 1001021 usec)
>>>>>> main_thread [1078] 2 pps (2 pkts in 1001022 usec)
>>>>>> main_thread [1078] 2 pps (2 pkts in 1001072 usec)
>>>>>> main_thread [1078] 2 pps (2 pkts in 1001020 usec)
>>>>>> main_thread [1078] 3 pps (3 pkts in 1001041 usec)
>>>>>> main_thread [1078] 3 pps (3 pkts in 1001023 usec)
>>>>>> main_thread [1078] 4 pps (4 pkts in 1001025 usec)
>>>>>> main_thread [1078] 3 pps (3 pkts in 1001032 usec)
>>>>>> main_thread [1078] 3 pps (3 pkts in 1001058 usec)
>>>>>> main_thread [1078] 2 pps (2 pkts in 1001042 usec)
>>>>>> main_thread [1078] 2 pps (2 pkts in 1001018 usec)
>>>>>> main_thread [1078] 2 pps (2 pkts in 1001024 usec)
>>>>>> main_thread [1078] 0 pps (0 pkts in 1001058 usec)
>>>>>> Received 58 packets, in 29.16 seconds.
>>>>>> Speed: 1.99 pps
>>>>>>
>>>>>>
>>>>>> The proble is that on the other side I had to use ping instead of
>>>>>> pkt-gen.
>>>>>>
>>>>>> If I try to use pkt-gen I don't receive any packet.
>>>>>> I just checked that pkt-gen is not effectively sending the packets on
>>>>>> the wire,
>>>>>> on the other side using tcpdump I don't receive any packet.
>>>>>>
>>>>>> So the problem on the tx side has not been completely solved in my
>>>>>> case.
>>>>>> Can you do the same test please?
>>>>>>
>>>>>> -Walter
>>>>>>
>>>>>> 2013/3/21 Tahir Rauf <tahir.rauf1 at gmail.com>
>>>>>>
>>>>>>> Dear Walter,
>>>>>>>
>>>>>>> What about the Rx, Can you please try and inform me.
>>>>>>> sudo ./pkt-gen -i eth2 -f rx -n 100000 -l 60 -w 1
>>>>>>>
>>>>>>> Regards
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Mar 21, 2013 at 7:06 PM, Walter de Donato <
>>>>>>> walter.dedonato at gmail.com> wrote:
>>>>>>>
>>>>>>>> Thanks Thair,
>>>>>>>>
>>>>>>>> with the fixed version the interface correctly attaches to the
>>>>>>>> netmap driver and it is able to transmit packets.
>>>>>>>>
>>>>>>>> Anyway, I'm observing really poor performance:
>>>>>>>> $ sudo ./pkt-gen -i eth2 -f tx -n 100000 -l 60 -w 1
>>>>>>>> extract_ip_range [136] extract IP range from 10.0.0.1
>>>>>>>> extract_ip_range [171] range is 10.0.0.1 0 to 10.0.0.1 0
>>>>>>>> extract_ip_range [136] extract IP range from 10.1.0.1
>>>>>>>> extract_ip_range [171] range is 10.1.0.1 0 to 10.1.0.1 0
>>>>>>>> extract_mac_range [177] extract MAC range from a0:36:9f:0d:f0:98
>>>>>>>> extract_mac_range [192] a0:36:9f:0d:f0:98 starts at a0:36:9f:d:f0:98
>>>>>>>> extract_mac_range [177] extract MAC range from ff:ff:ff:ff:ff:ff
>>>>>>>> extract_mac_range [192] ff:ff:ff:ff:ff:ff starts at
>>>>>>>> ff:ff:ff:ff:ff:ff
>>>>>>>> main [1401] map size is 334980 Kb
>>>>>>>> main [1423] mapping 334980 Kbytes
>>>>>>>> Sending on eth2: 4 queues, 1 threads and 1 cpus.
>>>>>>>> 10.0.0.1 -> 10.1.0.1 (a0:36:9f:0d:f0:98 -> ff:ff:ff:ff:ff:ff)
>>>>>>>> main [1477] Wait 1 secs for phy reset
>>>>>>>> main [1479] Ready...
>>>>>>>> sender_body [683] start
>>>>>>>> main_thread [1078] 7101 pps (7108 pkts in 1001019 usec)
>>>>>>>> main_thread [1078] 3482 pps (3486 pkts in 1001031 usec)
>>>>>>>> main_thread [1078] 3490 pps (3494 pkts in 1001026 usec)
>>>>>>>> main_thread [1078] 3483 pps (3487 pkts in 1001026 usec)
>>>>>>>> main_thread [1078] 3340 pps (3343 pkts in 1001019 usec)
>>>>>>>> main_thread [1078] 3455 pps (3459 pkts in 1001025 usec)
>>>>>>>> main_thread [1078] 3493 pps (3497 pkts in 1001024 usec)
>>>>>>>> main_thread [1078] 3494 pps (3498 pkts in 1001022 usec)
>>>>>>>> main_thread [1078] 3353 pps (3357 pkts in 1001056 usec)
>>>>>>>> main_thread [1078] 3488 pps (3492 pkts in 1001093 usec)
>>>>>>>> main_thread [1078] 3474 pps (3478 pkts in 1001064 usec)
>>>>>>>> main_thread [1078] 3485 pps (3489 pkts in 1001102 usec)
>>>>>>>> main_thread [1078] 3479 pps (3483 pkts in 1001064 usec)
>>>>>>>> main_thread [1078] 3335 pps (3339 pkts in 1001111 usec)
>>>>>>>> main_thread [1078] 3484 pps (3488 pkts in 1001074 usec)
>>>>>>>> main_thread [1078] 3484 pps (3488 pkts in 1001030 usec)
>>>>>>>> main_thread [1078] 3470 pps (3474 pkts in 1001046 usec)
>>>>>>>> main_thread [1078] 3486 pps (3490 pkts in 1001040 usec)
>>>>>>>> main_thread [1078] 3482 pps (3486 pkts in 1001035 usec)
>>>>>>>> main_thread [1078] 3278 pps (3281 pkts in 1001045 usec)
>>>>>>>> main_thread [1078] 3476 pps (3480 pkts in 1001021 usec)
>>>>>>>> main_thread [1078] 3490 pps (3494 pkts in 1001036 usec)
>>>>>>>> main_thread [1078] 3477 pps (3481 pkts in 1001083 usec)
>>>>>>>> main_thread [1078] 3493 pps (3497 pkts in 1001048 usec)
>>>>>>>> main_thread [1078] 3353 pps (3357 pkts in 1001082 usec)
>>>>>>>> main_thread [1078] 3490 pps (3494 pkts in 1001051 usec)
>>>>>>>> main_thread [1078] 3465 pps (3469 pkts in 1001060 usec)
>>>>>>>> main_thread [1078] 3008 pps (3011 pkts in 1001030 usec)
>>>>>>>> main_thread [1078] 0 pps (0 pkts in 1001039 usec)
>>>>>>>> Sent 100000 packets, 60 bytes each, in 28.58 seconds.
>>>>>>>> Speed: 3.50 Kpps Bandwidth: 1.68 Mbps (raw 2.35 Mbps)
>>>>>>>>
>>>>>>>> With the rtl8169 1G driver (which they say to have the worst
>>>>>>>> performance)
>>>>>>>> I easily obtained Speed: 359.24Kpps. Bandwidth: 172.44Mbps with 64
>>>>>>>> bytes packets.
>>>>>>>>
>>>>>>>> Also, if I increase the packet size this is what happens:
>>>>>>>> $ sudo ./pkt-gen -i eth2 -f tx -t 1000000 -l 508
>>>>>>>> main [1270] -t deprecated, please use -f tx -n 1000000
>>>>>>>> extract_ip_range [136] extract IP range from 10.0.0.1
>>>>>>>> extract_ip_range [171] range is 10.0.0.1 0 to 10.0.0.1 0
>>>>>>>> extract_ip_range [136] extract IP range from 10.1.0.1
>>>>>>>> extract_ip_range [171] range is 10.1.0.1 0 to 10.1.0.1 0
>>>>>>>> extract_mac_range [177] extract MAC range from a0:36:9f:0d:f0:98
>>>>>>>> extract_mac_range [192] a0:36:9f:0d:f0:98 starts at a0:36:9f:d:f0:98
>>>>>>>> extract_mac_range [177] extract MAC range from ff:ff:ff:ff:ff:ff
>>>>>>>> extract_mac_range [192] ff:ff:ff:ff:ff:ff starts at
>>>>>>>> ff:ff:ff:ff:ff:ff
>>>>>>>> main [1401] map size is 334980 Kb
>>>>>>>> main [1423] mapping 334980 Kbytes
>>>>>>>> Sending on eth2: 4 queues, 1 threads and 1 cpus.
>>>>>>>> 10.0.0.1 -> 10.1.0.1 (a0:36:9f:0d:f0:98 -> ff:ff:ff:ff:ff:ff)
>>>>>>>> main [1477] Wait 2 secs for phy reset
>>>>>>>> main [1479] Ready...
>>>>>>>> sender_body [683] start
>>>>>>>> main_thread [1078] 2042 pps (2044 pkts in 1001016 usec)
>>>>>>>> main_thread [1078] 0 pps (0 pkts in 1001049 usec)
>>>>>>>> sender_body [729] poll error/timeout on queue 0
>>>>>>>> main_thread [1078] 0 pps (0 pkts in 1001035 usec)
>>>>>>>> main_thread [1096] ouch, thread 0 exited with error
>>>>>>>> Sent 2044 packets, 508 bytes each, in -1363874673.38 seconds.
>>>>>>>> Speed: -0.00 pps Bandwidth: -0.01 bps (raw -0.01 bps)
>>>>>>>>
>>>>>>>> Don't you experience the same issues?
>>>>>>>>
>>>>>>>> -Walter
>>>>>>>>
>>>>>>>> 2013/3/21 Tahir Rauf <tahir.rauf1 at gmail.com>
>>>>>>>>
>>>>>>>>> Dear Walter,
>>>>>>>>>
>>>>>>>>> Please try following code.
>>>>>>>>> http://info.iet.unipi.it/~luigi/doc/20130217-netmap.tgz
>>>>>>>>> It will fix your problem with Ixgbe tx function.
>>>>>>>>>
>>>>>>>>> Please let me know, if you encounter any problem.
>>>>>>>>>
>>>>>>>>> Regards
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, Mar 21, 2013 at 6:28 PM, Walter de Donato <
>>>>>>>>> walter.dedonato at gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Dear Tahir,
>>>>>>>>>>
>>>>>>>>>> thank you for the prompt reply.
>>>>>>>>>> I downloaded the source code from the official website:
>>>>>>>>>> http://info.iet.unipi.it/~luigi/doc/20120813-netmap.tgz
>>>>>>>>>> Can you please share with me the fix about the tx function?
>>>>>>>>>> I was just starting to look at the patched code of the drivers.
>>>>>>>>>>
>>>>>>>>>> Thanks in advance,
>>>>>>>>>> -Walter
>>>>>>>>>>
>>>>>>>>>> 2013/3/21 Tahir Rauf <tahir.rauf1 at gmail.com>
>>>>>>>>>>
>>>>>>>>>>> Hi dedonato,
>>>>>>>>>>>
>>>>>>>>>>> I just observed that you are having trouble with tx function of
>>>>>>>>>>> pkt-gen. Whereas, we already have fixed that problem. We are facing issues
>>>>>>>>>>> with rx only now.
>>>>>>>>>>> Please tell me that from where you get the code of netmap?
>>>>>>>>>>>
>>>>>>>>>>> Regards
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Mar 21, 2013 at 5:49 PM, Tahir Rauf <
>>>>>>>>>>> tahir.rauf1 at gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Dear Walter,
>>>>>>>>>>>>
>>>>>>>>>>>> I tried to fix the problem but i don't have any success yet.
>>>>>>>>>>>> Also if you are trying to fix this problem, please share with
>>>>>>>>>>>> me that which path you are going to take for fixing. It might be possible
>>>>>>>>>>>> that I already have tried that path. In that case, I can share my results
>>>>>>>>>>>> with you, thus saving your time.
>>>>>>>>>>>>
>>>>>>>>>>>> Regards
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, Mar 21, 2013 at 5:08 PM, <walter.dedonato at gmail.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Dear Tahir and Luigi,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I have exactly the same problem with a similar hardware/os
>>>>>>>>>>>>> configuration reported in the following.
>>>>>>>>>>>>> Do you hav any uptade on this issue?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> -Walter
>>>>>>>>>>>>>
>>>>>>>>>>>>> Netmap version: 20120813
>>>>>>>>>>>>> OS: Ubuntu 12.04.2 LTS
>>>>>>>>>>>>> Kernel: 3.2.0-38-generic-pae #61-Ubuntu SMP Tue Feb 19
>>>>>>>>>>>>> 12:39:51 UTC 2013 i686 i686 i386 GNU/Linux
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Network interfaces:
>>>>>>>>>>>>> eth2
>>>>>>>>>>>>> 0c:00.0 Ethernet controller: Intel Corporation Ethernet
>>>>>>>>>>>>> Controller 10 Gigabit X540-AT2 (rev 01)
>>>>>>>>>>>>> Subsystem: Intel Corporation Ethernet 10G 2P X540-t
>>>>>>>>>>>>> Adapter
>>>>>>>>>>>>> Flags: bus master, fast devsel, latency 0, IRQ 17
>>>>>>>>>>>>> Memory at d8400000 (64-bit, prefetchable) [size=2M]
>>>>>>>>>>>>> Memory at d83fc000 (64-bit, prefetchable) [size=16K]
>>>>>>>>>>>>> Expansion ROM at fc500000 [disabled] [size=512K]
>>>>>>>>>>>>> Capabilities: [40] Power Management version 3
>>>>>>>>>>>>> Capabilities: [50] MSI: Enable- Count=1/1 Maskable+
>>>>>>>>>>>>> 64bit+
>>>>>>>>>>>>> Capabilities: [70] MSI-X: Enable+ Count=64 Masked-
>>>>>>>>>>>>> Capabilities: [a0] Express Endpoint, MSI 00
>>>>>>>>>>>>> Capabilities: [e0] Vital Product Data
>>>>>>>>>>>>> Capabilities: [100] Advanced Error Reporting
>>>>>>>>>>>>> Capabilities: [140] Device Serial Number
>>>>>>>>>>>>> a0-36-9f-ff-ff-0d-f0-98
>>>>>>>>>>>>> Capabilities: [150] Alternative Routing-ID
>>>>>>>>>>>>> Interpretation (ARI)
>>>>>>>>>>>>> Capabilities: [160] Single Root I/O Virtualization
>>>>>>>>>>>>> (SR-IOV)
>>>>>>>>>>>>> Capabilities: [1d0] Access Control Services
>>>>>>>>>>>>> Kernel driver in use: ixgbe
>>>>>>>>>>>>> Kernel modules: ixgbe
>>>>>>>>>>>>>
>>>>>>>>>>>>> eth3
>>>>>>>>>>>>> 0c:00.1 Ethernet controller: Intel Corporation Ethernet
>>>>>>>>>>>>> Controller 10 Gigabit X540-AT2 (rev 01)
>>>>>>>>>>>>> Subsystem: Intel Corporation Ethernet 10G 2P X540-t
>>>>>>>>>>>>> Adapter
>>>>>>>>>>>>> Flags: bus master, fast devsel, latency 0, IRQ 16
>>>>>>>>>>>>> Memory at d8000000 (64-bit, prefetchable) [size=2M]
>>>>>>>>>>>>> Memory at d83f8000 (64-bit, prefetchable) [size=16K]
>>>>>>>>>>>>> Expansion ROM at d8200000 [disabled] [size=512K]
>>>>>>>>>>>>> Capabilities: [40] Power Management version 3
>>>>>>>>>>>>> Capabilities: [50] MSI: Enable- Count=1/1 Maskable+
>>>>>>>>>>>>> 64bit+
>>>>>>>>>>>>> Capabilities: [70] MSI-X: Enable+ Count=64 Masked-
>>>>>>>>>>>>> Capabilities: [a0] Express Endpoint, MSI 00
>>>>>>>>>>>>> Capabilities: [e0] Vital Product Data
>>>>>>>>>>>>> Capabilities: [100] Advanced Error Reporting
>>>>>>>>>>>>> Capabilities: [140] Device Serial Number
>>>>>>>>>>>>> a0-36-9f-ff-ff-0d-f0-98
>>>>>>>>>>>>> Capabilities: [150] Alternative Routing-ID
>>>>>>>>>>>>> Interpretation (ARI)
>>>>>>>>>>>>> Capabilities: [160] Single Root I/O Virtualization
>>>>>>>>>>>>> (SR-IOV)
>>>>>>>>>>>>> Capabilities: [1d0] Access Control Services
>>>>>>>>>>>>> Kernel driver in use: ixgbe
>>>>>>>>>>>>> Kernel modules: ixgbe
>>>>>>>>>>>>>
>>>>>>>>>>>>> After loading the modules this is the (interesting) lsmod
>>>>>>>>>>>>> output:
>>>>>>>>>>>>> $ lsmod
>>>>>>>>>>>>> Module Size Used by
>>>>>>>>>>>>> ixgbe 160150 0
>>>>>>>>>>>>> netmap_lin 102855 1 ixgbe
>>>>>>>>>>>>> dca 14728 1 ixgbe
>>>>>>>>>>>>> mdio 13559 1 ixgbe
>>>>>>>>>>>>>
>>>>>>>>>>>>> And the following is the (interesting) dmesg output:
>>>>>>>>>>>>> [175397.625200] 521.870604 netmap_new_obj_allocator [426]
>>>>>>>>>>>>> objsize 1024 clustsize 4096 objects 4
>>>>>>>>>>>>> [175397.625501] 521.870912 netmap_new_obj_allocator [504]
>>>>>>>>>>>>> Pre-allocated 128 clusters (4/512KB) for 'netmap_if'
>>>>>>>>>>>>> [175397.625653] 521.871065 netmap_new_obj_allocator [426]
>>>>>>>>>>>>> objsize 36864 clustsize 36864 objects 1
>>>>>>>>>>>>> [175397.629906] 521.875312 netmap_new_obj_allocator [504]
>>>>>>>>>>>>> Pre-allocated 200 clusters (36/7200KB) for 'netmap_ring'
>>>>>>>>>>>>> [175397.630064] 521.875477 netmap_new_obj_allocator [426]
>>>>>>>>>>>>> objsize 2048 clustsize 4096 objects 2
>>>>>>>>>>>>> [175397.708572] 521.953976 netmap_new_obj_allocator [504]
>>>>>>>>>>>>> Pre-allocated 50000 clusters (4/200000KB) for 'netmap_buf'
>>>>>>>>>>>>> [175397.708737] 521.954147 netmap_memory_init [554] Have 512
>>>>>>>>>>>>> KB for interfaces, 7200 KB for rings and 195 MB for buffers
>>>>>>>>>>>>> [175397.708885] netmap: loaded module with 202 Mbytes
>>>>>>>>>>>>> [175418.013476] ixgbe: Intel(R) 10 Gigabit PCI Express Network
>>>>>>>>>>>>> Driver - version 3.6.7-k
>>>>>>>>>>>>> [175418.013479] ixgbe: Copyright (c) 1999-2011 Intel
>>>>>>>>>>>>> Corporation.
>>>>>>>>>>>>> [175418.013531] ixgbe 0000:0c:00.0: PCI INT B -> GSI 17
>>>>>>>>>>>>> (level, low) -> IRQ 17
>>>>>>>>>>>>> [175418.013543] ixgbe 0000:0c:00.0: setting latency timer to 64
>>>>>>>>>>>>> [175418.425782] ixgbe 0000:0c:00.0: irq 271 for MSI/MSI-X
>>>>>>>>>>>>> [175418.425791] ixgbe 0000:0c:00.0: irq 272 for MSI/MSI-X
>>>>>>>>>>>>> [175418.425798] ixgbe 0000:0c:00.0: irq 273 for MSI/MSI-X
>>>>>>>>>>>>> [175418.425806] ixgbe 0000:0c:00.0: irq 274 for MSI/MSI-X
>>>>>>>>>>>>> [175418.425813] ixgbe 0000:0c:00.0: irq 275 for MSI/MSI-X
>>>>>>>>>>>>> [175418.425829] ixgbe 0000:0c:00.0: Multiqueue Enabled: Rx
>>>>>>>>>>>>> Queue count = 4, Tx Queue count = 4
>>>>>>>>>>>>> [175418.485972] ixgbe 0000:0c:00.0: (PCI Express:2.5GT/s:Width
>>>>>>>>>>>>> x8) a0:36:9f:0d:f0:98
>>>>>>>>>>>>> [175418.646372] ixgbe 0000:0c:00.0: MAC: 3, PHY: 3, PBA No:
>>>>>>>>>>>>> G44743-006
>>>>>>>>>>>>> [175418.788262] ixgbe 0000:0c:00.0: Intel(R) 10 Gigabit
>>>>>>>>>>>>> Network Connection
>>>>>>>>>>>>> [175418.788285] ixgbe 0000:0c:00.1: PCI INT A -> GSI 16
>>>>>>>>>>>>> (level, low) -> IRQ 16
>>>>>>>>>>>>> [175418.788296] ixgbe 0000:0c:00.1: setting latency timer to 64
>>>>>>>>>>>>> [175419.213734] ixgbe 0000:0c:00.1: irq 276 for MSI/MSI-X
>>>>>>>>>>>>> [175419.213742] ixgbe 0000:0c:00.1: irq 277 for MSI/MSI-X
>>>>>>>>>>>>> [175419.213750] ixgbe 0000:0c:00.1: irq 278 for MSI/MSI-X
>>>>>>>>>>>>> [175419.213758] ixgbe 0000:0c:00.1: irq 279 for MSI/MSI-X
>>>>>>>>>>>>> [175419.213766] ixgbe 0000:0c:00.1: irq 280 for MSI/MSI-X
>>>>>>>>>>>>> [175419.213782] ixgbe 0000:0c:00.1: Multiqueue Enabled: Rx
>>>>>>>>>>>>> Queue count = 4, Tx Queue count = 4
>>>>>>>>>>>>> [175419.273937] ixgbe 0000:0c:00.1: (PCI Express:2.5GT/s:Width
>>>>>>>>>>>>> x8) a0:36:9f:0d:f0:9a
>>>>>>>>>>>>> [175419.434299] ixgbe 0000:0c:00.1: MAC: 3, PHY: 3, PBA No:
>>>>>>>>>>>>> G44743-006
>>>>>>>>>>>>> [175419.576197] ixgbe 0000:0c:00.1: Intel(R) 10 Gigabit
>>>>>>>>>>>>> Network Connection
>>>>>>>>>>>>>
>>>>>>>>>>>>> This is the pkt-gen output:
>>>>>>>>>>>>> $ sudo ./pkt-gen -i eth2 tx -n 10000
>>>>>>>>>>>>> extract_ip_range [119] extract IP range from 10.0.0.1
>>>>>>>>>>>>> extract_ip_range [129] 10.0.0.1 starts at 10.0.0.1
>>>>>>>>>>>>> extract_ip_range [119] extract IP range from 10.1.0.1
>>>>>>>>>>>>> extract_ip_range [129] 10.1.0.1 starts at 10.1.0.1
>>>>>>>>>>>>> extract_mac_range [135] extract MAC range from
>>>>>>>>>>>>> a0:36:9f:0d:f0:98
>>>>>>>>>>>>> extract_mac_range [150] a0:36:9f:0d:f0:98 starts at
>>>>>>>>>>>>> a0:36:9f:d:f0:98
>>>>>>>>>>>>> extract_mac_range [135] extract MAC range from
>>>>>>>>>>>>> ff:ff:ff:ff:ff:ff
>>>>>>>>>>>>> extract_mac_range [150] ff:ff:ff:ff:ff:ff starts at
>>>>>>>>>>>>> ff:ff:ff:ff:ff:ff
>>>>>>>>>>>>> main [1053] map size is 207712 Kb
>>>>>>>>>>>>> main [1059] Unable to get if info for eth2
>>>>>>>>>>>>> main [1066] bad nthreads 1, have 0 queues
>>>>>>>>>>>>> main [1075] mmapping 207712 Kbytes
>>>>>>>>>>>>> main [1094] Unable to register interface eth2
>>>>>>>>>>>>> Receiving from eth2: 0 queues, 1 threads and 1 cpus.
>>>>>>>>>>>>> main [1128] Wait 2 secs for phy reset
>>>>>>>>>>>>> main [1130] Ready...
>>>>>>>>>>>>> main [1181] Unable to register eth2
>>>>>>>>>>>>> main [1242] 0 pps
>>>>>>>>>>>>> Received 0 packets, in 0.00 seconds.
>>>>>>>>>>>>> Speed: -nanpps.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Dmesg after trying to use pkt-gen on eth2:
>>>>>>>>>>>>> [175537.838503] ADDRCONF(NETDEV_UP): eth2: link is not ready
>>>>>>>>>>>>> [175542.802288] ixgbe 0000:0c:00.0: eth2: NIC Link is Up 10
>>>>>>>>>>>>> Gbps, Flow Control: RX/TX
>>>>>>>>>>>>> [175542.802607] ADDRCONF(NETDEV_CHANGE): eth2: link becomes
>>>>>>>>>>>>> ready
>>>>>>>>>>>>> [175553.256029] eth2: no IPv6 routers present
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Tahir Rauf
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Tahir Rauf
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Tahir Rauf
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Tahir Rauf
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Tahir Rauf
>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>> Tahir Rauf
>>>
>>>
>>
>
>
> --
> Tahir Rauf
>
>
More information about the freebsd-net
mailing list