re(4) driver dropping packets when reading NFS files

[Pyun YongHyeon]

On Wed, Nov 03, 2010 at 07:27:20PM -0400, Rick Macklem wrote:
> > 
> > I'm more interested in number of dropped frames. See below how to
> > extract that information.
> > 
> 
> I've attached the stats. I'm guessing that the
> Rx missed frames : 14792
> is the culprit.
> 

Because that counter is 16bit it's also possible it wrapped
multiple times. Could you verify that?

> This was for a read of a fairly large file via NFS over TCP,
> getting a read rate of about 450Kbytes/sec. (No DEVICE_POLLING option.)
> (with your patch applied)
> 
> > 
> > > (It almost looks like it only handles the first received packet,
> > > although
> > >  it appears to be using a receive ring of 64 buffers.)
> > >
> > 
> > No, re(4) uses 256 TX/RX buffers for RTL810xE controllers.
> > 
> 
> Oops, my mistake. At a quick glance, I had thought rl_type was
> set to 8139, but I now see it's 8169.
> 
> Btw, I printed out the hwrev and its a RL_HWREV_8102EL_SPIN1,
> if that is of any use to you.
> 
> > 
> > Ok, here is patch.
> > http://people.freebsd.org/~yongari/re/re.intr.patch
> > 
> > The patch has the following changes.
> > o 64bit DMA support for PCIe controllers.
> > o Hardware MAC statistics counter support. You can extract these
> > counters with "sysctl dev.re.0.stats=1". You can check the
> > output on console or dmesg. It seems extracting these counters
> > take a lot of time so I didn't try to accumulate the counters.
> > You can see how many frames are dropped from the output. I saw a
> > lot FAE(frame alignment errors) under high RX load and I can't
> > explain how this can happen. This may indicate PHY hardware is
> > poor or it may need DSP fixups. Realtek seems to maintain large
> > set of DSP fixups for each PHY revisions and re(4) does not
> > have the magic code at this moment.
> > o Overhaul MSI interrupt handler such that make it give fairness
> > to TX as well as serving RX. Because re(4) controllers do not
> > have interrupt moderation mechanism, naive interrupt handler can
> > generate more than 125k intrs/sec under high load. Fortunately,
> > Bill implemented TX interrupt moderation with a timer register
> > and it seems to work well on TX path. One drawback of the
> > approach is it will require extra timer register accesses in
> > fast path. There is no second timer register to use in RX path
> > so no RX interrupt moderation is done in driver such that it can
> > generate about 25k intrs/sec under high RX load. However, I
> > think most systems can handle that interrupt load. Note, this
> > feature is activated only when MSI is in use and DEVICE_POLLING
> > is not defined.
> > 
> > >From my limited testing, it seems it works as expected. Would you
> > give it try and let me know how well it behaves with NFS?
> > 
> Without DEVICE_POLLING it behaves just like the unpatched one.
> 

Hmm, that's strange. Are you sure you rebuilt kernel without polling
option? Just disabling polling with ifconfig(8) has no effect to
make patch work.

> I'm going to look at the driver tomorrow and try some hacks on it, rick

> re0 statistics:
> Transmit good frames : 100966
> Receive good frames : 133470
> Tx errors : 0
> Rx errors : 0
> Rx missed frames : 14792

If the counter was not wrapped, it seem you lost more than 10% out of
total RX frames. This is a lot loss and there should be a way to
mitigate it.

> Rx frame alignment errs : 0
> Tx single collisions : 0
> Tx multiple collisions : 0
> Rx unicast frames : 133463
> Rx broadcast frames : 0
> Rx multicast frames : 7
> Tx aborts : 0
> Tx underruns : 0