Possible transmit/stats problem in igb driver.

[hiren panchasara]

+ Sean, Eric

On 06/02/16 at 09:11P, Sreekanth Rupavatharam wrote:
> Inline
> 
> >Apart from stats, do you see anything else going wrong? i.e. do you
> >actually see less packets (arp replies??) than expected?
> 
> [SR] The packets are not going out on the wire. The tool doesn?t receive the packets. That?s how I started noticing the issue. 
> 
> >Taking your example, tx_packets is something we count in the drivers and
> >total_pkts_txd is calculated in the card and we just read it off of it
> >to report (E1000_TPT).
> 
> [SR] Correct. My main question would be under what circumstance would the packet handed off to hardware will *not* be transmitted?. Especially considering there are no transmit errors or pause frames received. There are no dma tx failures either. That?s the baffling part. I tried another exercise where I used ping of various sizes going out, but that doesn?t seem to trigger the problem.
> 
> 
> >To understand your setup better, ixia is the sender and your box with
> >igb(4) is the receiver and your are sending arp requests to it.
> 
> Yes, correct. 
> 
> >Can you post following for working (size <= 64bytes) and non-working
> >(size > 64bytes) cases for before/after?
> >
> >sysctl dev.igb | grep tx_packets
> >sysctl dev.igb | grep total_pkts_txd
> >sysctl dev.igb | grep rx_packets
> >sysctl dev.igb | grep total_pkts_recvd
> 
> 
> Before(not working): 
> dev.igb.1.queue0.tx_packets: 24907933
> dev.igb.1.queue0.rx_packets: 18086575
> dev.igb.1.mac_stats.total_pkts_recvd: 25057359
> dev.igb.1.mac_stats.total_pkts_txd: 16647169
> 
> After(not working):
> dev.igb.1.queue0.tx_packets: 24913324
> dev.igb.1.queue0.rx_packets: 18091832
> dev.igb.1.mac_stats.total_pkts_recvd: 25062618
> dev.igb.1.mac_stats.total_pkts_txd: 16647545
> >netstat -sp arp
> 
> The difference is  5391 for queue0.tx_packets but for mac_stats.total_pkts_txd  is 376
> Everything else is matching up. 
> 
> Before (working)
> dev.igb.1.queue0.tx_packets: 25359165
> dev.igb.1.queue0.rx_packets: 18526094
> dev.igb.1.mac_stats.total_pkts_recvd: 25508763
> dev.igb.1.mac_stats.total_pkts_txd: 16831587
> 
> 
> After(working)
> dev.igb.1.queue0.tx_packets: 25364597
> dev.igb.1.queue0.rx_packets: 18531398
> dev.igb.1.mac_stats.total_pkts_recvd: 25514009
> dev.igb.1.mac_stats.total_pkts_txd: 16836833
> 
> 
> Another interesting stat is 
> before_notworking:dev.igb.1.interrupts.tx_queue_empty: 16646890
> after_notworking:dev.igb.1.interrupts.tx_queue_empty: 16647266
> 
> The difference here is exactly 376 which is the number of packets that the device actually claims to have transmitted. It?s as though it didn?t see the other packets en-queued in the ring descriptor.
> 

Very interesting. Do you tune defaults at all? What does sysctl hw.igb
say? Not sure if bumping up txd would help.

Adding Sean and Eric to throw some light.

> 
> I can?t do netstat just for arp as these are coming in a tunnel(Packets don?t? show up as arp on the interface). However, I did see the packet rate was about 500 packets/sec
>

Cheers,
Hiren
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 603 bytes
Desc: not available
URL: <http://lists.freebsd.org/pipermail/freebsd-net/attachments/20160602/cddec0a6/attachment.sig>