Possible transmit/stats problem in igb driver.

[Sreekanth Rupavatharam]

Inline

>Apart from stats, do you see anything else going wrong? i.e. do you
>actually see less packets (arp replies??) than expected?

[SR] The packets are not going out on the wire. The tool doesn’t receive the packets. That’s how I started noticing the issue. 

>Taking your example, tx_packets is something we count in the drivers and
>total_pkts_txd is calculated in the card and we just read it off of it
>to report (E1000_TPT).

[SR] Correct. My main question would be under what circumstance would the packet handed off to hardware will *not* be transmitted?. Especially considering there are no transmit errors or pause frames received. There are no dma tx failures either. That’s the baffling part. I tried another exercise where I used ping of various sizes going out, but that doesn’t seem to trigger the problem.


>To understand your setup better, ixia is the sender and your box with
>igb(4) is the receiver and your are sending arp requests to it.

Yes, correct. 

>Can you post following for working (size <= 64bytes) and non-working
>(size > 64bytes) cases for before/after?
>
>sysctl dev.igb | grep tx_packets
>sysctl dev.igb | grep total_pkts_txd
>sysctl dev.igb | grep rx_packets
>sysctl dev.igb | grep total_pkts_recvd


Before(not working): 
dev.igb.1.queue0.tx_packets: 24907933
dev.igb.1.queue0.rx_packets: 18086575
dev.igb.1.mac_stats.total_pkts_recvd: 25057359
dev.igb.1.mac_stats.total_pkts_txd: 16647169

After(not working):
dev.igb.1.queue0.tx_packets: 24913324
dev.igb.1.queue0.rx_packets: 18091832
dev.igb.1.mac_stats.total_pkts_recvd: 25062618
dev.igb.1.mac_stats.total_pkts_txd: 16647545
>netstat -sp arp

The difference is  5391 for queue0.tx_packets but for mac_stats.total_pkts_txd  is 376
Everything else is matching up. 

Before (working)
dev.igb.1.queue0.tx_packets: 25359165
dev.igb.1.queue0.rx_packets: 18526094
dev.igb.1.mac_stats.total_pkts_recvd: 25508763
dev.igb.1.mac_stats.total_pkts_txd: 16831587


After(working)
dev.igb.1.queue0.tx_packets: 25364597
dev.igb.1.queue0.rx_packets: 18531398
dev.igb.1.mac_stats.total_pkts_recvd: 25514009
dev.igb.1.mac_stats.total_pkts_txd: 16836833


Another interesting stat is 
before_notworking:dev.igb.1.interrupts.tx_queue_empty: 16646890
after_notworking:dev.igb.1.interrupts.tx_queue_empty: 16647266

The difference here is exactly 376 which is the number of packets that the device actually claims to have transmitted. It’s as though it didn’t see the other packets en-queued in the ring descriptor.


I can’t do netstat just for arp as these are coming in a tunnel(Packets don’t’ show up as arp on the interface). However, I did see the packet rate was about 500 packets/sec


>(You can netstat -z to clear the counters between runs.)
>
>Cheers,
>Hiren