Netmap ixgbe driver problem on 10Gbps X540-AT2 adapters (Linux only?)
Michio Honda
micchie at sfc.wide.ad.jp
Fri Mar 22 10:41:54 UTC 2013
You need to specify "-w 4" on running pkt-gen to wait for that the link resets.
Cheers,
- Michio
On Mar 22, 2013, at 11:16 AM, Walter de Donato wrote:
> Dear Tahir,
>
> I've already tried to disable Rx/Tx pause and autonegotiation but the
> result in transmission is always the same.
>
> I did the following:
> $ sudo ifconfig eth2 up
> $ sudo ethtool -A eth2 autoneg off
> $ sudo ethtool -A eth2 rx off
> $ sudo ethtool -A eth2 tx off
>
> These are the resulting dmesg output lines:
> [253131.445124] ixgbe 0000:0c:00.0: eth2: NIC Link is Up 10 Gbps, Flow
> Control: RX/TX
> [253147.935426] ixgbe 0000:0c:00.0: eth2: NIC Link is Up 10 Gbps, Flow
> Control: TX
> [253163.930776] ixgbe 0000:0c:00.0: eth2: NIC Link is Up 10 Gbps, Flow
> Control: None
>
> Here is what I obtain:
> $ sudo ./pkt-gen -i eth2 -n 100000 -f tx
> extract_ip_range [136] extract IP range from 10.0.0.1
> extract_ip_range [171] range is 10.0.0.1 0 to 10.0.0.1 0
> extract_ip_range [136] extract IP range from 10.1.0.1
> extract_ip_range [171] range is 10.1.0.1 0 to 10.1.0.1 0
> extract_mac_range [177] extract MAC range from a0:36:9f:0d:f0:98
> extract_mac_range [192] a0:36:9f:0d:f0:98 starts at a0:36:9f:d:f0:98
> extract_mac_range [177] extract MAC range from ff:ff:ff:ff:ff:ff
> extract_mac_range [192] ff:ff:ff:ff:ff:ff starts at ff:ff:ff:ff:ff:ff
> main [1401] map size is 334980 Kb
> main [1423] mapping 334980 Kbytes
> Sending on eth2: 4 queues, 1 threads and 1 cpus.
> 10.0.0.1 -> 10.1.0.1 (a0:36:9f:0d:f0:98 -> ff:ff:ff:ff:ff:ff)
> main [1477] Wait 2 secs for phy reset
> main [1479] Ready...
> sender_body [683] start
> main_thread [1078] 7155 pps (7162 pkts in 1001020 usec)
> main_thread [1078] 3484 pps (3488 pkts in 1001030 usec)
> main_thread [1078] 3487 pps (3491 pkts in 1001024 usec)
> main_thread [1078] 3466 pps (3469 pkts in 1000847 usec)
> main_thread [1078] 3485 pps (3490 pkts in 1001521 usec)
> main_thread [1078] 3322 pps (3328 pkts in 1001765 usec)
> main_thread [1078] 3293 pps (3297 pkts in 1001322 usec)
> main_thread [1078] 3492 pps (3498 pkts in 1001763 usec)
> main_thread [1078] 3485 pps (3489 pkts in 1001168 usec)
> main_thread [1078] 3351 pps (3356 pkts in 1001402 usec)
> main_thread [1078] 3297 pps (3301 pkts in 1001314 usec)
> main_thread [1078] 3933 pps (3938 pkts in 1001390 usec)
> main_thread [1078] 3481 pps (3486 pkts in 1001528 usec)
> main_thread [1078] 3493 pps (3497 pkts in 1001177 usec)
> main_thread [1078] 3310 pps (3314 pkts in 1001292 usec)
> main_thread [1078] 3485 pps (3491 pkts in 1001812 usec)
> main_thread [1078] 3326 pps (3331 pkts in 1001547 usec)
> main_thread [1078] 3480 pps (3485 pkts in 1001456 usec)
> main_thread [1078] 3489 pps (3493 pkts in 1001209 usec)
> main_thread [1078] 3377 pps (3381 pkts in 1001040 usec)
> main_thread [1078] 3450 pps (3454 pkts in 1001048 usec)
> main_thread [1078] 3378 pps (3382 pkts in 1001050 usec)
> main_thread [1078] 3470 pps (3474 pkts in 1001052 usec)
> main_thread [1078] 3323 pps (3327 pkts in 1001056 usec)
> main_thread [1078] 3482 pps (3486 pkts in 1001050 usec)
> main_thread [1078] 3472 pps (3476 pkts in 1001066 usec)
> main_thread [1078] 3497 pps (3501 pkts in 1001044 usec)
> main_thread [1078] 3115 pps (3115 pkts in 1000035 usec)
> main_thread [1078] 0 pps (0 pkts in 1001057 usec)
> Sent 100000 packets, 60 bytes each, in 28.42 seconds.
> Speed: 3.52 Kpps Bandwidth: 1.69 Mbps (raw 2.36 Mbps)
>
> While executing pkt-gen I see the following dmesg output lines:
> [253741.892031] ixgbe 0000:0c:00.0: eth2: Reset adapter
> [253742.874686] ixgbe 0000:0c:00.0: eth2: Reset adapter
> [253743.864635] ixgbe 0000:0c:00.0: eth2: Reset adapter
> [253744.854513] ixgbe 0000:0c:00.0: eth2: Reset adapter
> [253745.844527] ixgbe 0000:0c:00.0: eth2: Reset adapter
> [253746.830323] ixgbe 0000:0c:00.0: eth2: Reset adapter
> [253747.821223] ixgbe 0000:0c:00.0: eth2: Reset adapter
> [253748.813229] ixgbe 0000:0c:00.0: eth2: Reset adapter
> [253749.804317] ixgbe 0000:0c:00.0: eth2: Reset adapter
> [253750.794286] ixgbe 0000:0c:00.0: eth2: Reset adapter
> [253751.784302] ixgbe 0000:0c:00.0: eth2: Reset adapter
> [253752.774324] ixgbe 0000:0c:00.0: eth2: Reset adapter
> [253753.764258] ixgbe 0000:0c:00.0: eth2: Reset adapter
> [253754.753735] ixgbe 0000:0c:00.0: eth2: Reset adapter
> [253755.748011] ixgbe 0000:0c:00.0: eth2: Reset adapter
> [253756.734927] ixgbe 0000:0c:00.0: eth2: Reset adapter
> [253757.724876] ixgbe 0000:0c:00.0: eth2: Reset adapter
> [253758.713267] ixgbe 0000:0c:00.0: eth2: Reset adapter
> [253759.704793] ixgbe 0000:0c:00.0: eth2: Reset adapter
> [253760.694759] ixgbe 0000:0c:00.0: eth2: Reset adapter
> [253761.684743] ixgbe 0000:0c:00.0: eth2: Reset adapter
> [253762.674699] ixgbe 0000:0c:00.0: eth2: Reset adapter
> [253763.664642] ixgbe 0000:0c:00.0: eth2: Reset adapter
> [253764.654488] ixgbe 0000:0c:00.0: eth2: Reset adapter
> [253765.644462] ixgbe 0000:0c:00.0: eth2: Reset adapter
> [253766.634425] ixgbe 0000:0c:00.0: eth2: Reset adapter
> [253767.624392] ixgbe 0000:0c:00.0: eth2: Reset adapter
> [253768.614364] ixgbe 0000:0c:00.0: eth2: Reset adapter
> [253769.611107] ixgbe 0000:0c:00.0: eth2: Reset adapter
> [253769.715298] 893.960705 netmap_ring_reinit [966] called for eth2
> [253769.716646] 893.962054 netmap_ring_reinit [966] called for eth2
> [253769.725844] 893.971252 netmap_ioctl [1236] deprecated, data is ded4df20
> [253776.141936] ixgbe 0000:0c:00.0: eth2: NIC Link is Up 10 Gbps, Flow
> Control: None
>
> It seems that the adapter is reset every second, but no packet goes on the
> wire.
>
> I'm starting to look into the code to investigate this issue.
> If you have any suggestion on where to look at it would save me a lot of
> time.
>
> I also rechecked the rx mode.
> It is working at least up to 50Kpps (as I'm generation on the other side
> with nping and the original driver).
> Pkg-gen/netmap is configured to receive only the frames directed to the
> physical MAC address of the interface.
> I don't know if and how a promiscuous mode can be enabled with netmap.
>
> Thanks,
> -Walter
> 2013/3/22 Tahir Rauf <tahir.rauf1 at gmail.com>
>
>> Dear Walter,
>>
>> Can you please disable Rx/Tx pause frames on your NIC. You can use
>> "ethtool" to do so. Please run the example again after disabling Rx/Tx
>> frames and let me know about the results.
>>
>> Regards
>>
>>
>> On Thu, Mar 21, 2013 at 8:20 PM, Walter de Donato <
>> walter.dedonato at gmail.com> wrote:
>>
>>> Not that lucky since my first goal is to transmit packets...
>>> I also noticed that the same car d is configured with 4 rings on 32bit
>>> kernels and 16 rings on 64bit kernels.
>>> Do you have any idea to explain this difference?
>>>
>>> -Walter
>>>
>>> 2013/3/21 Tahir Rauf <tahir.rauf1 at gmail.com>
>>>
>>>> Dear Walter,
>>>>
>>>> You are lucky to have* X540-AT2 :). *Mine is different (82598EB) and
>>>> can't receive packets properly.
>>>> Yes, my tx is working and is able to send packets on line rate.
>>>>
>>>> Regards
>>>>
>>>>
>>>> On Thu, Mar 21, 2013 at 8:04 PM, Walter de Donato <
>>>> walter.dedonato at gmail.com> wrote:
>>>>
>>>>> Here is the lshw output:
>>>>>
>>>>> *-network:0
>>>>> description: Ethernet interface
>>>>> product: Ethernet Controller 10 Gigabit X540-AT2
>>>>> vendor: Intel Corporation
>>>>> physical id: 0
>>>>> bus info: pci at 0000:0c:00.0
>>>>> logical name: eth2
>>>>> version: 01
>>>>> serial: a0:36:9f:0d:f0:98
>>>>> capacity: 1Gbit/s
>>>>> width: 64 bits
>>>>> clock: 33MHz
>>>>> capabilities: pm msi msix pciexpress vpd bus_master cap_list rom
>>>>> ethernet physical tp 100bt-fd 1000bt-fd autonegotiation
>>>>> configuration: autonegotiation=on broadcast=yes driver=ixgbe
>>>>> driverversion=3.6.7-k duplex=full firmware=0x800002ef latency=0 link=yes
>>>>> multicast=yes port=twisted pair
>>>>> resources: irq:17 memory:d8400000-d85fffff
>>>>> memory:d83fc000-d83fffff memory:fc500000-fc57ffff memory:fc400000-fc4fffff
>>>>> *-network:1
>>>>> description: Ethernet interface
>>>>> product: Ethernet Controller 10 Gigabit X540-AT2
>>>>> vendor: Intel Corporation
>>>>> physical id: 0.1
>>>>> bus info: pci at 0000:0c:00.1
>>>>> logical name: eth3
>>>>> version: 01
>>>>> serial: a0:36:9f:0d:f0:9a
>>>>> capacity: 1Gbit/s
>>>>> width: 64 bits
>>>>> clock: 33MHz
>>>>> capabilities: pm msi msix pciexpress vpd bus_master cap_list rom
>>>>> ethernet physical tp 100bt-fd 1000bt-fd autonegotiation
>>>>> configuration: autonegotiation=on broadcast=yes driver=ixgbe
>>>>> driverversion=3.6.7-k duplex=full firmware=0x800002ef ip=192.168.78.2
>>>>> latency=0 link=yes multicast=yes port=twisted pair
>>>>> resources: irq:16 memory:d8000000-d81fffff
>>>>> memory:d83f8000-d83fbfff memory:d8200000-d827ffff
>>>>>
>>>>> Did you check if in tx mode your pkt-gen effectively sends the packets
>>>>> on the wire?
>>>>> What rates are you observing with it?
>>>>>
>>>>> -Walter
>>>>>
>>>>> 2013/3/21 Tahir Rauf <tahir.rauf1 at gmail.com>
>>>>>
>>>>>> Dear walter,
>>>>>>
>>>>>> Can you please run following command to get your NIC information.
>>>>>> sudo lshw -class network
>>>>>>
>>>>>> I have *82598EB *10GB AF dual port network connection NIC and my rx
>>>>>> is not working.
>>>>>>
>>>>>> Regards
>>>>>>
>>>>>>
>>>>>> On Thu, Mar 21, 2013 at 7:46 PM, Walter de Donato <
>>>>>> walter.dedonato at gmail.com> wrote:
>>>>>>
>>>>>>> I tried the rx command and it seems to work:
>>>>>>> $ sudo ./pkt-gen -i eth3 -f rx -n 100000 -l 60 -w 1
>>>>>>> extract_ip_range [136] extract IP range from 10.0.0.1
>>>>>>> extract_ip_range [171] range is 10.0.0.1 0 to 10.0.0.1 0
>>>>>>> extract_ip_range [136] extract IP range from 10.1.0.1
>>>>>>> extract_ip_range [171] range is 10.1.0.1 0 to 10.1.0.1 0
>>>>>>> extract_mac_range [177] extract MAC range from a0:36:9f:0d:f0:9a
>>>>>>> extract_mac_range [192] a0:36:9f:0d:f0:9a starts at a0:36:9f:d:f0:9a
>>>>>>> extract_mac_range [177] extract MAC range from ff:ff:ff:ff:ff:ff
>>>>>>> extract_mac_range [192] ff:ff:ff:ff:ff:ff starts at ff:ff:ff:ff:ff:ff
>>>>>>> main [1401] map size is 334980 Kb
>>>>>>> main [1423] mapping 334980 Kbytes
>>>>>>> Receiving from eth3: 4 queues, 1 threads and 1 cpus.
>>>>>>> main [1477] Wait 1 secs for phy reset
>>>>>>> main [1479] Ready...
>>>>>>> main_thread [1078] 0 pps (0 pkts in 1001020 usec)
>>>>>>> receiver_body [833] waiting for initial packets, poll returns 0 0
>>>>>>> main_thread [1078] 0 pps (0 pkts in 1001059 usec)
>>>>>>> receiver_body [833] waiting for initial packets, poll returns 0 0
>>>>>>> main_thread [1078] 0 pps (0 pkts in 1001047 usec)
>>>>>>> receiver_body [833] waiting for initial packets, poll returns 0 0
>>>>>>> main_thread [1078] 2 pps (2 pkts in 1001118 usec)
>>>>>>> main_thread [1078] 1 pps (1 pkts in 1001040 usec)
>>>>>>> main_thread [1078] 1 pps (1 pkts in 1001038 usec)
>>>>>>> main_thread [1078] 1 pps (1 pkts in 1001030 usec)
>>>>>>> main_thread [1078] 1 pps (1 pkts in 1001028 usec)
>>>>>>> main_thread [1078] 1 pps (1 pkts in 1001051 usec)
>>>>>>> main_thread [1078] 1 pps (1 pkts in 1001068 usec)
>>>>>>> main_thread [1078] 2 pps (2 pkts in 1001040 usec)
>>>>>>> main_thread [1078] 2 pps (2 pkts in 1001035 usec)
>>>>>>> main_thread [1078] 2 pps (2 pkts in 1001042 usec)
>>>>>>> main_thread [1078] 2 pps (2 pkts in 1001083 usec)
>>>>>>> main_thread [1078] 2 pps (2 pkts in 1001050 usec)
>>>>>>> main_thread [1078] 2 pps (2 pkts in 1001073 usec)
>>>>>>> main_thread [1078] 2 pps (2 pkts in 1001052 usec)
>>>>>>> main_thread [1078] 2 pps (2 pkts in 1001041 usec)
>>>>>>> main_thread [1078] 2 pps (2 pkts in 1001051 usec)
>>>>>>> main_thread [1078] 1 pps (1 pkts in 1001058 usec)
>>>>>>> main_thread [1078] 1 pps (1 pkts in 1001020 usec)
>>>>>>> main_thread [1078] 2 pps (2 pkts in 1001021 usec)
>>>>>>> main_thread [1078] 2 pps (2 pkts in 1001022 usec)
>>>>>>> main_thread [1078] 2 pps (2 pkts in 1001072 usec)
>>>>>>> main_thread [1078] 2 pps (2 pkts in 1001020 usec)
>>>>>>> main_thread [1078] 3 pps (3 pkts in 1001041 usec)
>>>>>>> main_thread [1078] 3 pps (3 pkts in 1001023 usec)
>>>>>>> main_thread [1078] 4 pps (4 pkts in 1001025 usec)
>>>>>>> main_thread [1078] 3 pps (3 pkts in 1001032 usec)
>>>>>>> main_thread [1078] 3 pps (3 pkts in 1001058 usec)
>>>>>>> main_thread [1078] 2 pps (2 pkts in 1001042 usec)
>>>>>>> main_thread [1078] 2 pps (2 pkts in 1001018 usec)
>>>>>>> main_thread [1078] 2 pps (2 pkts in 1001024 usec)
>>>>>>> main_thread [1078] 0 pps (0 pkts in 1001058 usec)
>>>>>>> Received 58 packets, in 29.16 seconds.
>>>>>>> Speed: 1.99 pps
>>>>>>>
>>>>>>>
>>>>>>> The proble is that on the other side I had to use ping instead of
>>>>>>> pkt-gen.
>>>>>>>
>>>>>>> If I try to use pkt-gen I don't receive any packet.
>>>>>>> I just checked that pkt-gen is not effectively sending the packets on
>>>>>>> the wire,
>>>>>>> on the other side using tcpdump I don't receive any packet.
>>>>>>>
>>>>>>> So the problem on the tx side has not been completely solved in my
>>>>>>> case.
>>>>>>> Can you do the same test please?
>>>>>>>
>>>>>>> -Walter
>>>>>>>
>>>>>>> 2013/3/21 Tahir Rauf <tahir.rauf1 at gmail.com>
>>>>>>>
>>>>>>>> Dear Walter,
>>>>>>>>
>>>>>>>> What about the Rx, Can you please try and inform me.
>>>>>>>> sudo ./pkt-gen -i eth2 -f rx -n 100000 -l 60 -w 1
>>>>>>>>
>>>>>>>> Regards
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Mar 21, 2013 at 7:06 PM, Walter de Donato <
>>>>>>>> walter.dedonato at gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Thanks Thair,
>>>>>>>>>
>>>>>>>>> with the fixed version the interface correctly attaches to the
>>>>>>>>> netmap driver and it is able to transmit packets.
>>>>>>>>>
>>>>>>>>> Anyway, I'm observing really poor performance:
>>>>>>>>> $ sudo ./pkt-gen -i eth2 -f tx -n 100000 -l 60 -w 1
>>>>>>>>> extract_ip_range [136] extract IP range from 10.0.0.1
>>>>>>>>> extract_ip_range [171] range is 10.0.0.1 0 to 10.0.0.1 0
>>>>>>>>> extract_ip_range [136] extract IP range from 10.1.0.1
>>>>>>>>> extract_ip_range [171] range is 10.1.0.1 0 to 10.1.0.1 0
>>>>>>>>> extract_mac_range [177] extract MAC range from a0:36:9f:0d:f0:98
>>>>>>>>> extract_mac_range [192] a0:36:9f:0d:f0:98 starts at a0:36:9f:d:f0:98
>>>>>>>>> extract_mac_range [177] extract MAC range from ff:ff:ff:ff:ff:ff
>>>>>>>>> extract_mac_range [192] ff:ff:ff:ff:ff:ff starts at
>>>>>>>>> ff:ff:ff:ff:ff:ff
>>>>>>>>> main [1401] map size is 334980 Kb
>>>>>>>>> main [1423] mapping 334980 Kbytes
>>>>>>>>> Sending on eth2: 4 queues, 1 threads and 1 cpus.
>>>>>>>>> 10.0.0.1 -> 10.1.0.1 (a0:36:9f:0d:f0:98 -> ff:ff:ff:ff:ff:ff)
>>>>>>>>> main [1477] Wait 1 secs for phy reset
>>>>>>>>> main [1479] Ready...
>>>>>>>>> sender_body [683] start
>>>>>>>>> main_thread [1078] 7101 pps (7108 pkts in 1001019 usec)
>>>>>>>>> main_thread [1078] 3482 pps (3486 pkts in 1001031 usec)
>>>>>>>>> main_thread [1078] 3490 pps (3494 pkts in 1001026 usec)
>>>>>>>>> main_thread [1078] 3483 pps (3487 pkts in 1001026 usec)
>>>>>>>>> main_thread [1078] 3340 pps (3343 pkts in 1001019 usec)
>>>>>>>>> main_thread [1078] 3455 pps (3459 pkts in 1001025 usec)
>>>>>>>>> main_thread [1078] 3493 pps (3497 pkts in 1001024 usec)
>>>>>>>>> main_thread [1078] 3494 pps (3498 pkts in 1001022 usec)
>>>>>>>>> main_thread [1078] 3353 pps (3357 pkts in 1001056 usec)
>>>>>>>>> main_thread [1078] 3488 pps (3492 pkts in 1001093 usec)
>>>>>>>>> main_thread [1078] 3474 pps (3478 pkts in 1001064 usec)
>>>>>>>>> main_thread [1078] 3485 pps (3489 pkts in 1001102 usec)
>>>>>>>>> main_thread [1078] 3479 pps (3483 pkts in 1001064 usec)
>>>>>>>>> main_thread [1078] 3335 pps (3339 pkts in 1001111 usec)
>>>>>>>>> main_thread [1078] 3484 pps (3488 pkts in 1001074 usec)
>>>>>>>>> main_thread [1078] 3484 pps (3488 pkts in 1001030 usec)
>>>>>>>>> main_thread [1078] 3470 pps (3474 pkts in 1001046 usec)
>>>>>>>>> main_thread [1078] 3486 pps (3490 pkts in 1001040 usec)
>>>>>>>>> main_thread [1078] 3482 pps (3486 pkts in 1001035 usec)
>>>>>>>>> main_thread [1078] 3278 pps (3281 pkts in 1001045 usec)
>>>>>>>>> main_thread [1078] 3476 pps (3480 pkts in 1001021 usec)
>>>>>>>>> main_thread [1078] 3490 pps (3494 pkts in 1001036 usec)
>>>>>>>>> main_thread [1078] 3477 pps (3481 pkts in 1001083 usec)
>>>>>>>>> main_thread [1078] 3493 pps (3497 pkts in 1001048 usec)
>>>>>>>>> main_thread [1078] 3353 pps (3357 pkts in 1001082 usec)
>>>>>>>>> main_thread [1078] 3490 pps (3494 pkts in 1001051 usec)
>>>>>>>>> main_thread [1078] 3465 pps (3469 pkts in 1001060 usec)
>>>>>>>>> main_thread [1078] 3008 pps (3011 pkts in 1001030 usec)
>>>>>>>>> main_thread [1078] 0 pps (0 pkts in 1001039 usec)
>>>>>>>>> Sent 100000 packets, 60 bytes each, in 28.58 seconds.
>>>>>>>>> Speed: 3.50 Kpps Bandwidth: 1.68 Mbps (raw 2.35 Mbps)
>>>>>>>>>
>>>>>>>>> With the rtl8169 1G driver (which they say to have the worst
>>>>>>>>> performance)
>>>>>>>>> I easily obtained Speed: 359.24Kpps. Bandwidth: 172.44Mbps with 64
>>>>>>>>> bytes packets.
>>>>>>>>>
>>>>>>>>> Also, if I increase the packet size this is what happens:
>>>>>>>>> $ sudo ./pkt-gen -i eth2 -f tx -t 1000000 -l 508
>>>>>>>>> main [1270] -t deprecated, please use -f tx -n 1000000
>>>>>>>>> extract_ip_range [136] extract IP range from 10.0.0.1
>>>>>>>>> extract_ip_range [171] range is 10.0.0.1 0 to 10.0.0.1 0
>>>>>>>>> extract_ip_range [136] extract IP range from 10.1.0.1
>>>>>>>>> extract_ip_range [171] range is 10.1.0.1 0 to 10.1.0.1 0
>>>>>>>>> extract_mac_range [177] extract MAC range from a0:36:9f:0d:f0:98
>>>>>>>>> extract_mac_range [192] a0:36:9f:0d:f0:98 starts at a0:36:9f:d:f0:98
>>>>>>>>> extract_mac_range [177] extract MAC range from ff:ff:ff:ff:ff:ff
>>>>>>>>> extract_mac_range [192] ff:ff:ff:ff:ff:ff starts at
>>>>>>>>> ff:ff:ff:ff:ff:ff
>>>>>>>>> main [1401] map size is 334980 Kb
>>>>>>>>> main [1423] mapping 334980 Kbytes
>>>>>>>>> Sending on eth2: 4 queues, 1 threads and 1 cpus.
>>>>>>>>> 10.0.0.1 -> 10.1.0.1 (a0:36:9f:0d:f0:98 -> ff:ff:ff:ff:ff:ff)
>>>>>>>>> main [1477] Wait 2 secs for phy reset
>>>>>>>>> main [1479] Ready...
>>>>>>>>> sender_body [683] start
>>>>>>>>> main_thread [1078] 2042 pps (2044 pkts in 1001016 usec)
>>>>>>>>> main_thread [1078] 0 pps (0 pkts in 1001049 usec)
>>>>>>>>> sender_body [729] poll error/timeout on queue 0
>>>>>>>>> main_thread [1078] 0 pps (0 pkts in 1001035 usec)
>>>>>>>>> main_thread [1096] ouch, thread 0 exited with error
>>>>>>>>> Sent 2044 packets, 508 bytes each, in -1363874673.38 seconds.
>>>>>>>>> Speed: -0.00 pps Bandwidth: -0.01 bps (raw -0.01 bps)
>>>>>>>>>
>>>>>>>>> Don't you experience the same issues?
>>>>>>>>>
>>>>>>>>> -Walter
>>>>>>>>>
>>>>>>>>> 2013/3/21 Tahir Rauf <tahir.rauf1 at gmail.com>
>>>>>>>>>
>>>>>>>>>> Dear Walter,
>>>>>>>>>>
>>>>>>>>>> Please try following code.
>>>>>>>>>> http://info.iet.unipi.it/~luigi/doc/20130217-netmap.tgz
>>>>>>>>>> It will fix your problem with Ixgbe tx function.
>>>>>>>>>>
>>>>>>>>>> Please let me know, if you encounter any problem.
>>>>>>>>>>
>>>>>>>>>> Regards
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Thu, Mar 21, 2013 at 6:28 PM, Walter de Donato <
>>>>>>>>>> walter.dedonato at gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Dear Tahir,
>>>>>>>>>>>
>>>>>>>>>>> thank you for the prompt reply.
>>>>>>>>>>> I downloaded the source code from the official website:
>>>>>>>>>>> http://info.iet.unipi.it/~luigi/doc/20120813-netmap.tgz
>>>>>>>>>>> Can you please share with me the fix about the tx function?
>>>>>>>>>>> I was just starting to look at the patched code of the drivers.
>>>>>>>>>>>
>>>>>>>>>>> Thanks in advance,
>>>>>>>>>>> -Walter
>>>>>>>>>>>
>>>>>>>>>>> 2013/3/21 Tahir Rauf <tahir.rauf1 at gmail.com>
>>>>>>>>>>>
>>>>>>>>>>>> Hi dedonato,
>>>>>>>>>>>>
>>>>>>>>>>>> I just observed that you are having trouble with tx function of
>>>>>>>>>>>> pkt-gen. Whereas, we already have fixed that problem. We are facing issues
>>>>>>>>>>>> with rx only now.
>>>>>>>>>>>> Please tell me that from where you get the code of netmap?
>>>>>>>>>>>>
>>>>>>>>>>>> Regards
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, Mar 21, 2013 at 5:49 PM, Tahir Rauf <
>>>>>>>>>>>> tahir.rauf1 at gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Dear Walter,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I tried to fix the problem but i don't have any success yet.
>>>>>>>>>>>>> Also if you are trying to fix this problem, please share with
>>>>>>>>>>>>> me that which path you are going to take for fixing. It might be possible
>>>>>>>>>>>>> that I already have tried that path. In that case, I can share my results
>>>>>>>>>>>>> with you, thus saving your time.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Regards
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Thu, Mar 21, 2013 at 5:08 PM, <walter.dedonato at gmail.com>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Dear Tahir and Luigi,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I have exactly the same problem with a similar hardware/os
>>>>>>>>>>>>>> configuration reported in the following.
>>>>>>>>>>>>>> Do you hav any uptade on this issue?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>> -Walter
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Netmap version: 20120813
>>>>>>>>>>>>>> OS: Ubuntu 12.04.2 LTS
>>>>>>>>>>>>>> Kernel: 3.2.0-38-generic-pae #61-Ubuntu SMP Tue Feb 19
>>>>>>>>>>>>>> 12:39:51 UTC 2013 i686 i686 i386 GNU/Linux
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Network interfaces:
>>>>>>>>>>>>>> eth2
>>>>>>>>>>>>>> 0c:00.0 Ethernet controller: Intel Corporation Ethernet
>>>>>>>>>>>>>> Controller 10 Gigabit X540-AT2 (rev 01)
>>>>>>>>>>>>>> Subsystem: Intel Corporation Ethernet 10G 2P X540-t
>>>>>>>>>>>>>> Adapter
>>>>>>>>>>>>>> Flags: bus master, fast devsel, latency 0, IRQ 17
>>>>>>>>>>>>>> Memory at d8400000 (64-bit, prefetchable) [size=2M]
>>>>>>>>>>>>>> Memory at d83fc000 (64-bit, prefetchable) [size=16K]
>>>>>>>>>>>>>> Expansion ROM at fc500000 [disabled] [size=512K]
>>>>>>>>>>>>>> Capabilities: [40] Power Management version 3
>>>>>>>>>>>>>> Capabilities: [50] MSI: Enable- Count=1/1 Maskable+
>>>>>>>>>>>>>> 64bit+
>>>>>>>>>>>>>> Capabilities: [70] MSI-X: Enable+ Count=64 Masked-
>>>>>>>>>>>>>> Capabilities: [a0] Express Endpoint, MSI 00
>>>>>>>>>>>>>> Capabilities: [e0] Vital Product Data
>>>>>>>>>>>>>> Capabilities: [100] Advanced Error Reporting
>>>>>>>>>>>>>> Capabilities: [140] Device Serial Number
>>>>>>>>>>>>>> a0-36-9f-ff-ff-0d-f0-98
>>>>>>>>>>>>>> Capabilities: [150] Alternative Routing-ID
>>>>>>>>>>>>>> Interpretation (ARI)
>>>>>>>>>>>>>> Capabilities: [160] Single Root I/O Virtualization
>>>>>>>>>>>>>> (SR-IOV)
>>>>>>>>>>>>>> Capabilities: [1d0] Access Control Services
>>>>>>>>>>>>>> Kernel driver in use: ixgbe
>>>>>>>>>>>>>> Kernel modules: ixgbe
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> eth3
>>>>>>>>>>>>>> 0c:00.1 Ethernet controller: Intel Corporation Ethernet
>>>>>>>>>>>>>> Controller 10 Gigabit X540-AT2 (rev 01)
>>>>>>>>>>>>>> Subsystem: Intel Corporation Ethernet 10G 2P X540-t
>>>>>>>>>>>>>> Adapter
>>>>>>>>>>>>>> Flags: bus master, fast devsel, latency 0, IRQ 16
>>>>>>>>>>>>>> Memory at d8000000 (64-bit, prefetchable) [size=2M]
>>>>>>>>>>>>>> Memory at d83f8000 (64-bit, prefetchable) [size=16K]
>>>>>>>>>>>>>> Expansion ROM at d8200000 [disabled] [size=512K]
>>>>>>>>>>>>>> Capabilities: [40] Power Management version 3
>>>>>>>>>>>>>> Capabilities: [50] MSI: Enable- Count=1/1 Maskable+
>>>>>>>>>>>>>> 64bit+
>>>>>>>>>>>>>> Capabilities: [70] MSI-X: Enable+ Count=64 Masked-
>>>>>>>>>>>>>> Capabilities: [a0] Express Endpoint, MSI 00
>>>>>>>>>>>>>> Capabilities: [e0] Vital Product Data
>>>>>>>>>>>>>> Capabilities: [100] Advanced Error Reporting
>>>>>>>>>>>>>> Capabilities: [140] Device Serial Number
>>>>>>>>>>>>>> a0-36-9f-ff-ff-0d-f0-98
>>>>>>>>>>>>>> Capabilities: [150] Alternative Routing-ID
>>>>>>>>>>>>>> Interpretation (ARI)
>>>>>>>>>>>>>> Capabilities: [160] Single Root I/O Virtualization
>>>>>>>>>>>>>> (SR-IOV)
>>>>>>>>>>>>>> Capabilities: [1d0] Access Control Services
>>>>>>>>>>>>>> Kernel driver in use: ixgbe
>>>>>>>>>>>>>> Kernel modules: ixgbe
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> After loading the modules this is the (interesting) lsmod
>>>>>>>>>>>>>> output:
>>>>>>>>>>>>>> $ lsmod
>>>>>>>>>>>>>> Module Size Used by
>>>>>>>>>>>>>> ixgbe 160150 0
>>>>>>>>>>>>>> netmap_lin 102855 1 ixgbe
>>>>>>>>>>>>>> dca 14728 1 ixgbe
>>>>>>>>>>>>>> mdio 13559 1 ixgbe
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> And the following is the (interesting) dmesg output:
>>>>>>>>>>>>>> [175397.625200] 521.870604 netmap_new_obj_allocator [426]
>>>>>>>>>>>>>> objsize 1024 clustsize 4096 objects 4
>>>>>>>>>>>>>> [175397.625501] 521.870912 netmap_new_obj_allocator [504]
>>>>>>>>>>>>>> Pre-allocated 128 clusters (4/512KB) for 'netmap_if'
>>>>>>>>>>>>>> [175397.625653] 521.871065 netmap_new_obj_allocator [426]
>>>>>>>>>>>>>> objsize 36864 clustsize 36864 objects 1
>>>>>>>>>>>>>> [175397.629906] 521.875312 netmap_new_obj_allocator [504]
>>>>>>>>>>>>>> Pre-allocated 200 clusters (36/7200KB) for 'netmap_ring'
>>>>>>>>>>>>>> [175397.630064] 521.875477 netmap_new_obj_allocator [426]
>>>>>>>>>>>>>> objsize 2048 clustsize 4096 objects 2
>>>>>>>>>>>>>> [175397.708572] 521.953976 netmap_new_obj_allocator [504]
>>>>>>>>>>>>>> Pre-allocated 50000 clusters (4/200000KB) for 'netmap_buf'
>>>>>>>>>>>>>> [175397.708737] 521.954147 netmap_memory_init [554] Have 512
>>>>>>>>>>>>>> KB for interfaces, 7200 KB for rings and 195 MB for buffers
>>>>>>>>>>>>>> [175397.708885] netmap: loaded module with 202 Mbytes
>>>>>>>>>>>>>> [175418.013476] ixgbe: Intel(R) 10 Gigabit PCI Express Network
>>>>>>>>>>>>>> Driver - version 3.6.7-k
>>>>>>>>>>>>>> [175418.013479] ixgbe: Copyright (c) 1999-2011 Intel
>>>>>>>>>>>>>> Corporation.
>>>>>>>>>>>>>> [175418.013531] ixgbe 0000:0c:00.0: PCI INT B -> GSI 17
>>>>>>>>>>>>>> (level, low) -> IRQ 17
>>>>>>>>>>>>>> [175418.013543] ixgbe 0000:0c:00.0: setting latency timer to 64
>>>>>>>>>>>>>> [175418.425782] ixgbe 0000:0c:00.0: irq 271 for MSI/MSI-X
>>>>>>>>>>>>>> [175418.425791] ixgbe 0000:0c:00.0: irq 272 for MSI/MSI-X
>>>>>>>>>>>>>> [175418.425798] ixgbe 0000:0c:00.0: irq 273 for MSI/MSI-X
>>>>>>>>>>>>>> [175418.425806] ixgbe 0000:0c:00.0: irq 274 for MSI/MSI-X
>>>>>>>>>>>>>> [175418.425813] ixgbe 0000:0c:00.0: irq 275 for MSI/MSI-X
>>>>>>>>>>>>>> [175418.425829] ixgbe 0000:0c:00.0: Multiqueue Enabled: Rx
>>>>>>>>>>>>>> Queue count = 4, Tx Queue count = 4
>>>>>>>>>>>>>> [175418.485972] ixgbe 0000:0c:00.0: (PCI Express:2.5GT/s:Width
>>>>>>>>>>>>>> x8) a0:36:9f:0d:f0:98
>>>>>>>>>>>>>> [175418.646372] ixgbe 0000:0c:00.0: MAC: 3, PHY: 3, PBA No:
>>>>>>>>>>>>>> G44743-006
>>>>>>>>>>>>>> [175418.788262] ixgbe 0000:0c:00.0: Intel(R) 10 Gigabit
>>>>>>>>>>>>>> Network Connection
>>>>>>>>>>>>>> [175418.788285] ixgbe 0000:0c:00.1: PCI INT A -> GSI 16
>>>>>>>>>>>>>> (level, low) -> IRQ 16
>>>>>>>>>>>>>> [175418.788296] ixgbe 0000:0c:00.1: setting latency timer to 64
>>>>>>>>>>>>>> [175419.213734] ixgbe 0000:0c:00.1: irq 276 for MSI/MSI-X
>>>>>>>>>>>>>> [175419.213742] ixgbe 0000:0c:00.1: irq 277 for MSI/MSI-X
>>>>>>>>>>>>>> [175419.213750] ixgbe 0000:0c:00.1: irq 278 for MSI/MSI-X
>>>>>>>>>>>>>> [175419.213758] ixgbe 0000:0c:00.1: irq 279 for MSI/MSI-X
>>>>>>>>>>>>>> [175419.213766] ixgbe 0000:0c:00.1: irq 280 for MSI/MSI-X
>>>>>>>>>>>>>> [175419.213782] ixgbe 0000:0c:00.1: Multiqueue Enabled: Rx
>>>>>>>>>>>>>> Queue count = 4, Tx Queue count = 4
>>>>>>>>>>>>>> [175419.273937] ixgbe 0000:0c:00.1: (PCI Express:2.5GT/s:Width
>>>>>>>>>>>>>> x8) a0:36:9f:0d:f0:9a
>>>>>>>>>>>>>> [175419.434299] ixgbe 0000:0c:00.1: MAC: 3, PHY: 3, PBA No:
>>>>>>>>>>>>>> G44743-006
>>>>>>>>>>>>>> [175419.576197] ixgbe 0000:0c:00.1: Intel(R) 10 Gigabit
>>>>>>>>>>>>>> Network Connection
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> This is the pkt-gen output:
>>>>>>>>>>>>>> $ sudo ./pkt-gen -i eth2 tx -n 10000
>>>>>>>>>>>>>> extract_ip_range [119] extract IP range from 10.0.0.1
>>>>>>>>>>>>>> extract_ip_range [129] 10.0.0.1 starts at 10.0.0.1
>>>>>>>>>>>>>> extract_ip_range [119] extract IP range from 10.1.0.1
>>>>>>>>>>>>>> extract_ip_range [129] 10.1.0.1 starts at 10.1.0.1
>>>>>>>>>>>>>> extract_mac_range [135] extract MAC range from
>>>>>>>>>>>>>> a0:36:9f:0d:f0:98
>>>>>>>>>>>>>> extract_mac_range [150] a0:36:9f:0d:f0:98 starts at
>>>>>>>>>>>>>> a0:36:9f:d:f0:98
>>>>>>>>>>>>>> extract_mac_range [135] extract MAC range from
>>>>>>>>>>>>>> ff:ff:ff:ff:ff:ff
>>>>>>>>>>>>>> extract_mac_range [150] ff:ff:ff:ff:ff:ff starts at
>>>>>>>>>>>>>> ff:ff:ff:ff:ff:ff
>>>>>>>>>>>>>> main [1053] map size is 207712 Kb
>>>>>>>>>>>>>> main [1059] Unable to get if info for eth2
>>>>>>>>>>>>>> main [1066] bad nthreads 1, have 0 queues
>>>>>>>>>>>>>> main [1075] mmapping 207712 Kbytes
>>>>>>>>>>>>>> main [1094] Unable to register interface eth2
>>>>>>>>>>>>>> Receiving from eth2: 0 queues, 1 threads and 1 cpus.
>>>>>>>>>>>>>> main [1128] Wait 2 secs for phy reset
>>>>>>>>>>>>>> main [1130] Ready...
>>>>>>>>>>>>>> main [1181] Unable to register eth2
>>>>>>>>>>>>>> main [1242] 0 pps
>>>>>>>>>>>>>> Received 0 packets, in 0.00 seconds.
>>>>>>>>>>>>>> Speed: -nanpps.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Dmesg after trying to use pkt-gen on eth2:
>>>>>>>>>>>>>> [175537.838503] ADDRCONF(NETDEV_UP): eth2: link is not ready
>>>>>>>>>>>>>> [175542.802288] ixgbe 0000:0c:00.0: eth2: NIC Link is Up 10
>>>>>>>>>>>>>> Gbps, Flow Control: RX/TX
>>>>>>>>>>>>>> [175542.802607] ADDRCONF(NETDEV_CHANGE): eth2: link becomes
>>>>>>>>>>>>>> ready
>>>>>>>>>>>>>> [175553.256029] eth2: no IPv6 routers present
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Tahir Rauf
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Tahir Rauf
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Tahir Rauf
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Tahir Rauf
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Tahir Rauf
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Tahir Rauf
>>>>
>>>>
>>>
>>
>>
>> --
>> Tahir Rauf
>>
>>
> _______________________________________________
> freebsd-net at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org"
>
More information about the freebsd-net
mailing list