em(4): difference between missed_packets and rx_overrun

[hiren panchasara]

On 03/27/15 at 10:33P, hiren panchasara wrote:
> + jfv, erj from Intel.
> 
> On 03/26/15 at 01:08P, hiren panchasara wrote:
> > This is what we are seeing on em(4) 82574L chipset running stable/10:
> > 
> > dev.em.0.mac_stats.missed_packets: 1441927
> > dev.em.0.interrupts.rx_overrun: 153
> > 
> > From the datasheet:
> > http://www.intel.com/content/www/us/en/ethernet-controllers/82574l-gbe-controller-datasheet.html
> > 
> > 10.2.7.4 Missed Packets Count - MPC (0x04010; R)
> > Counts the number of missed packets. Packets are missed when the receive
> > FIFO has insufficient space to store the incoming packet. This could be
> > caused because of too few buffers allocated, or because there is
> > insufficient bandwidth on the IO bus. Events setting this counter
> > cause RXO, the receiver overrun interrupt, to be set. This register
> > does not increment if receives are not enabled.
> > 
> > 10.2.4.1 Interrupt Cause Read Register - ICR (0x000C0; RC/WC)
> > RXO Receiver Overrun
> > Set on receive data FIFO overrun. Could be caused either because
> > there are no available buffers or because PCIe receive bandwidth is
> > inadequate.
> > 
> > So, first one is a count and another one is an interrupt. Are these 2
> > related? Both seem to be happen when on card FIFO gets full. We see no
> > evidence of RX queue on the host being full based on
> > dev.em.0.mac_stats.recv_no_buff.
> > 
> > Many a times we see missed_packets increasing without rx_overrun
> > changing.
> > 
> > The spec says there is a 40KB buffer on card which seems to be used by
> > both RX and TX? Is is split between them for 20KB each? OR is it
> > possible that when we are doing high rate TX, we use up that buffer and
> > RX suffers from that?

So, I found some seemingly useful things in the spec:
10.2.12 Diagnostic Register Descriptions

I tried to read on-chip FIFO head/tail to make a sense of what we  have
there when we see mpc (missed packet count) increase. In
em_update_stats_counters() whenever I see mpc increasing, I go print
following registers.

fhead = E1000_READ_REG(&adapter->hw, E1000_RDFH);
ftail = E1000_READ_REG(&adapter->hw, E1000_RDFT);
fheads = E1000_READ_REG(&adapter->hw, E1000_RDFHS);
ftails = E1000_READ_REG(&adapter->hw, E1000_RDFTS);
fcount = E1000_READ_REG(&adapter->hw, E1000_RDFPC);

But I do not see any pattern here. Some times I see a large differenec
between head and tail and still no mpc increase and sometimes mpc++ when
head/tail are following eachother closely.

Also, fcount E1000_RDFPC which is supposed to be "The number of received
packets currently in the RX FIFO." is always 0 OR a moderately low
number. That doesn't make sense to me. If FIFO is not full why do we
get so many missed packets?

cheers,
Hiren
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 618 bytes
Desc: not available
URL: <http://lists.freebsd.org/pipermail/freebsd-net/attachments/20150331/5ddb5a61/attachment.sig>