BeagleBone slow inbound net I/O

[Bernd Walter]

On Sat, Mar 14, 2015 at 02:59:54PM +0100, Bernd Walter wrote:
> On Sat, Mar 14, 2015 at 07:23:20AM -0600, Ian Lepore wrote:
> > On Sat, 2015-03-14 at 03:15 -0400, Brett Wynkoop wrote:
> > > On Fri, 13 Mar 2015 23:02:25 -0700
> > > Tim Kientzle <tim at kientzle.com> wrote:
> > > 
> > > > 
> > > > > On Mar 11, 2015, at 1:51 PM, Brett Wynkoop <freebsd-arm at wynn.com>
> > > > > wrote:
> > > > > 
> > > > > Have I managed to find a network driver issue?  Any ideas how to
> > > > > gather more information to help get to the bottom of things?
> > > > > 
> > > > 
> > > > $ sysctl dev.cpsw
> > > > 
> > > > This will dump detailed statistics from the Ethernet hardware and
> > > > driver.
> > > > 
> > > > Tim
> > > > 
> > > 
> > > After a short time while doing nfs i/o
> > > 
> > > 
> > > [wynkoop at beaglebone ~]$ sysctl dev.cpsw | grep -i error
> > > dev.cpsw.0.stats.RxCrcErrors: 40
> > > dev.cpsw.0.stats.RxAlignErrors: 32
> > > dev.cpsw.0.stats.CarrierSenseErrors: 0
> > [...]
> > > [wynkoop at beaglebone ~]$ sysctl dev.cpsw | grep -i error
> > > dev.cpsw.0.stats.RxCrcErrors: 262
> > > dev.cpsw.0.stats.RxAlignErrors: 231
> > > dev.cpsw.0.stats.CarrierSenseErrors: 0
> > > [wynkoop at beaglebone ~]$ 
> > > 
> > > So we can see climbing errors.  I am not sure how this compares to the
> > > results of others. The above was during the first few minutes of a
> > > buildworld from an nfs share.
> > > 
> > > At the same time on the console:
> > > 
> > > Mar 14 03:07:47 beaglebone amd[1163]: mountd rpc failed: RPC: Can't
> > > decode result Mar 14 03:11:48 beaglebone amd[1399]: mountd rpc failed:
> > > RPC: Can't decode result
> > > 
> > > which makes sense with the above errors I think.
> 
> It doesn't make sense with the ethernet CRC alone, since ethernet CRC
> failures are basicly dropped packets.
> If the RPC answer can't be parsed then ethernet packet CRC was Ok, when
> verified by the MAC and corrupted later.
> 
> > On mine:
> > 
> > root at bb:/usr/ports/benchmarks/iperf # sysctl dev.cpsw | grep Err
> > dev.cpsw.0.stats.RxCrcErrors: 0
> > dev.cpsw.0.stats.RxAlignErrors: 0
> > dev.cpsw.0.stats.CarrierSenseErrors: 0
> > 
> > That's after 3 days of uptime including doing builds over nfs, and all
> > the iperf testing I was doing yesterday (no errors after megabytes of
> > transfers).
> > 
> > I wonder if your power supply is failing and injecting transient
> > glitches under heavy load or something?
> 
> Power seems likely.
> The RPC error and filesystem corruptions make sense with broken RAM too,
> but the ethernet CRC should be checked during transfer to RAM.
> The subsystems with failures are very different and mostly isolated from
> software and HW-logic.
> 
> About ZFS:
> It is designed to handle data corruption to some degree and has an extremly
> different workload, so possible that it works by luck.

Just noticed in another of your mails that you have uncorrected errors
with ZFS too.
It is just that ZFS notices the corruption with it's CRC, plus has copies
for metadata, so it doesn't need to panic as UFS.
But as a user I would panic with such a system ;-)

-- 
B.Walter <bernd at bwct.de> http://www.bwct.de
Modbus/TCP Ethernet I/O Baugruppen, ARM basierte FreeBSD Rechner uvm.