Possible transmit/stats problem in igb driver.

[Sreekanth Rupavatharam]

Don’t see any errors as such. Here’s the complete stats.


Thanks,

-Sreekanth


From: Jack Vogel <jfvogel at gmail.com>
Date: Thursday, June 2, 2016 at 2:40 PM
To: Sreekanth Rupavatharam <rupavath at juniper.net>
Cc: hiren panchasara <hiren at strugglingcoder.info>, "freebsd-net at freebsd.org" <freebsd-net at freebsd.org>
Subject: Re: Possible transmit/stats problem in igb driver.

Hmmm, so I no longer have access to any of this, but I'm wondering if there is some
other stat that might report errors or some sort that would match the missing/bad packets?
Maybe they were considered runts or something and dropped.
Good luck :)
Jack

On Thu, Jun 2, 2016 at 2:11 PM, Sreekanth Rupavatharam <rupavath at juniper.net<mailto:rupavath at juniper.net>> wrote:
Inline

>Apart from stats, do you see anything else going wrong? i.e. do you
>actually see less packets (arp replies??) than expected?

[SR] The packets are not going out on the wire. The tool doesn’t receive the packets. That’s how I started noticing the issue.

>Taking your example, tx_packets is something we count in the drivers and
>total_pkts_txd is calculated in the card and we just read it off of it
>to report (E1000_TPT).

[SR] Correct. My main question would be under what circumstance would the packet handed off to hardware will *not* be transmitted?. Especially considering there are no transmit errors or pause frames received. There are no dma tx failures either. That’s the baffling part. I tried another exercise where I used ping of various sizes going out, but that doesn’t seem to trigger the problem.


>To understand your setup better, ixia is the sender and your box with
>igb(4) is the receiver and your are sending arp requests to it.

Yes, correct.

>Can you post following for working (size <= 64bytes) and non-working
>(size > 64bytes) cases for before/after?
>
>sysctl dev.igb | grep tx_packets
>sysctl dev.igb | grep total_pkts_txd
>sysctl dev.igb | grep rx_packets
>sysctl dev.igb | grep total_pkts_recvd


Before(not working):
dev.igb.1.queue0.tx_packets: 24907933
dev.igb.1.queue0.rx_packets: 18086575
dev.igb.1.mac_stats.total_pkts_recvd: 25057359
dev.igb.1.mac_stats.total_pkts_txd: 16647169

After(not working):
dev.igb.1.queue0.tx_packets: 24913324
dev.igb.1.queue0.rx_packets: 18091832
dev.igb.1.mac_stats.total_pkts_recvd: 25062618
dev.igb.1.mac_stats.total_pkts_txd: 16647545
>netstat -sp arp

The difference is  5391 for queue0.tx_packets but for mac_stats.total_pkts_txd  is 376
Everything else is matching up.

Before (working)
dev.igb.1.queue0.tx_packets: 25359165
dev.igb.1.queue0.rx_packets: 18526094
dev.igb.1.mac_stats.total_pkts_recvd: 25508763
dev.igb.1.mac_stats.total_pkts_txd: 16831587


After(working)
dev.igb.1.queue0.tx_packets: 25364597
dev.igb.1.queue0.rx_packets: 18531398
dev.igb.1.mac_stats.total_pkts_recvd: 25514009
dev.igb.1.mac_stats.total_pkts_txd: 16836833


Another interesting stat is
before_notworking:dev.igb.1.interrupts.tx_queue_empty: 16646890
after_notworking:dev.igb.1.interrupts.tx_queue_empty: 16647266

The difference here is exactly 376 which is the number of packets that the device actually claims to have transmitted. It’s as though it didn’t see the other packets en-queued in the ring descriptor.


I can’t do netstat just for arp as these are coming in a tunnel(Packets don’t’ show up as arp on the interface). However, I did see the packet rate was about 500 packets/sec


>(You can netstat -z to clear the counters between runs.)
>
>Cheers,
>Hiren

-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: after_notworking.txt
URL: <http://lists.freebsd.org/pipermail/freebsd-net/attachments/20160602/8aabc455/attachment-0002.txt>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: before_notworking.txt
URL: <http://lists.freebsd.org/pipermail/freebsd-net/attachments/20160602/8aabc455/attachment-0003.txt>