Re: 60+% ping packet loss on Pi3 under -current and stable-13

From: bob prohaska <fbsd_at_www.zefox.net>
Date: Sun, 01 May 2022 18:12:54 UTC
On Sat, Apr 30, 2022 at 06:39:57PM -0700, Bakul Shah wrote:
> On Apr 29, 2022, at 7:12 PM, bob prohaska <fbsd@www.zefox.net> wrote:
> > 
> > Since about December of 2021 I've been noticing problems with
> > wired network connectivity on a pair of raspberry pi 3 machines
> > using wired network connections. One runs stable-13.1, the other
> > runs -current, both are up to date as of a few days ago.
> > 
> > Essentially both machines fail to respond to inbound network
> > connections via ssh or ping after reboot. If I get on the 
> > serial console and start an outbound ping to anywhere, both
> > machines respond to incoming pings with about a 65% packet
> > loss. 

> Suggest running tcpdump on the rpi3 to see what is going on
> when connected to the public vs private net. 
>
 
Public net first, since that's where the machine is now. Gateway.zefox.net
is the name of my router's public interface, dcn.org belongs to my isp and
fusionbroadband is their service provider..
 
While on the -current Pi3 serial console (with no outbound ping running) 
and no inbound traffic from my hosts I see after a couple minutes:

root@www:/mnt # tcpdump
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ue0, link-type EN10MB (Ethernet), capture size 262144 bytes
10:39:40.887853 ARP, Request who-has www.zefox.org tell gateway.zefox.net, length 46
10:39:40.887929 ARP, Reply www.zefox.org is-at b8:27:eb:71:46:4e (oui Unknown), length 28
10:39:40.893220 ARP, Request who-has 50-1-20-1.dsl.static.fusionbroadband.com tell www.zefox.org, length 28
10:39:40.915469 ARP, Reply 50-1-20-1.dsl.static.fusionbroadband.com is-at 00:1b:90:d2:4a:c4 (oui Unknown), length 50
10:39:40.915529 IP www.zefox.org.50714 > spoke.dcn.davis.ca.us.domain: 51409+ PTR? 28.20.1.50.in-addr.arpa. (41)
10:39:40.943602 IP spoke.dcn.davis.ca.us.domain > www.zefox.org.50714: 51409 1/3/6 PTR www.zefox.org. (265)
10:39:40.945416 IP www.zefox.org.15986 > spoke.dcn.davis.ca.us.domain: 44966+ PTR? 31.20.1.50.in-addr.arpa. (41)
10:39:40.973487 IP spoke.dcn.davis.ca.us.domain > www.zefox.org.15986: 44966 1/3/6 PTR gateway.zefox.net. (266)
10:39:40.975037 IP www.zefox.org.57611 > spoke.dcn.davis.ca.us.domain: 31749+ PTR? 1.20.1.50.in-addr.arpa. (40)
10:39:46.288219 IP www.zefox.org.49710 > wheel.dcn.davis.ca.us.domain: 31749+ PTR? 1.20.1.50.in-addr.arpa. (40)
10:39:46.316239 IP wheel.dcn.davis.ca.us.domain > www.zefox.org.49710: 31749 1/3/6 PTR 50-1-20-1.dsl.static.fusionbroadband.com. (291)
10:39:46.318267 IP www.zefox.org.17061 > spoke.dcn.davis.ca.us.domain: 37579+ PTR? 2.253.150.168.in-addr.arpa. (44)
10:39:46.346851 IP spoke.dcn.davis.ca.us.domain > www.zefox.org.17061: 37579* 1/2/2 PTR spoke.dcn.davis.ca.us. (145)
10:39:46.348674 IP www.zefox.org.40440 > spoke.dcn.davis.ca.us.domain: 20572+ PTR? 1.253.150.168.in-addr.arpa. (44)
10:39:51.420705 IP www.zefox.org.64019 > wheel.dcn.davis.ca.us.domain: 20572+ PTR? 1.253.150.168.in-addr.arpa. (44)
10:39:51.448850 IP wheel.dcn.davis.ca.us.domain > www.zefox.org.64019: 20572* 1/2/2 PTR wheel.dcn.davis.ca.us. (145)
10:40:40.147603 ARP, Request who-has 50-1-20-1.dsl.static.fusionbroadband.com tell ns1.zefox.net, length 46
10:40:40.148844 IP www.zefox.org.46127 > spoke.dcn.davis.ca.us.domain: 12186+ PTR? 29.20.1.50.in-addr.arpa. (41)
10:40:40.176486 IP spoke.dcn.davis.ca.us.domain > www.zefox.org.46127: 12186 1/3/6 PTR ns1.zefox.net. (262)
10:40:57.688225 ARP, Request who-has www.zefox.org tell gateway.zefox.net, length 46
10:40:57.688305 ARP, Reply www.zefox.org is-at b8:27:eb:71:46:4e (oui Unknown), length 28
10:42:14.488727 ARP, Request who-has www.zefox.org tell gateway.zefox.net, length 46
10:42:14.488804 ARP, Reply www.zefox.org is-at b8:27:eb:71:46:4e (oui Unknown), length 28
10:42:43.761226 ARP, Request who-has 50-1-20-1.dsl.static.fusionbroadband.com tell www.zefox.com, length 46
10:42:43.762522 IP www.zefox.org.56181 > spoke.dcn.davis.ca.us.domain: 28779+ PTR? 26.20.1.50.in-addr.arpa. (41)
10:42:43.790361 IP spoke.dcn.davis.ca.us.domain > www.zefox.org.56181: 28779 1/3/6 PTR www.zefox.com. (265)
10:43:31.289103 ARP, Request who-has www.zefox.org tell gateway.zefox.net, length 46
10:43:31.289181 ARP, Reply www.zefox.org is-at b8:27:eb:71:46:4e (oui Unknown), length 28

If I now start an inbound ping from one of my hosts it gets no reply and 
tcpdump reports no additional traffic. With an outbound ping running there's
at least a sparse reply.

^C
28 packets captured
28 packets received by filter
0 packets dropped by kernel
root@www:/mnt # 

The "oui unknown" looks like some sort of failure.....
Can you ping www.zefox.org? I have no outside vantage point.
There is still no outbound ping running and I would expect
you'll get no or very sparse reply. 


Thus far only the two Pi3s suffer from connectivity problems; Pi2s and a Pi4 have
no difficulty on the same address block. Is there a switch for tcpdump  that will
limit records to relevant traffic? Otherwise it's a flood.

These results were obtained after standing idle overnight and
are rather different (in ways I don't understand) from behavior
immediately after reboot, I'll have to repeat as I learn more.

Thanks for reading, and any more suggestions!

bob prohaska