Re: 13.1R problems on Pi3

[This ends with a possible config.txt way to control the
default Ether MAC address used, independent of which OS
happens to be running. Possibly better than using the
only-FreeBSD technique.]

On 2022-Jul-4, at 11:47, Mark Millard <marklmi@yahoo.com> wrote:

> On 2022-Jul-4, at 11:25, bob prohaska <fbsd@www.zefox.net> wrote:
> 
>> On Mon, Jul 04, 2022 at 12:17:15PM -0400, Karl Denninger wrote:
>>> 
>>> On 7/4/2022 11:28, bob prohaska wrote:
>>>> On Sun, Jul 03, 2022 at 10:36:35PM -0400, Karl Denninger wrote:
>>>> 
>>>> Can any sense be made of the few ping responses obtained when ntp
>>>> is coming up? It's looks as if something happens after ntp runs
>>>> that blocks subsequent network traffic, but why starting an outbound
>>>> ping should partly unblock things is obscure to me.
>>> 
>>> Yes.?? The odds are reasonably high that there is confusion as to which MAC
>>> address maps to which device.?? This implies there's a loop between the two
>>> switches (e.g. there is more than one way for packets to get into and out of
>>> each said switch to the other) or the two devices are claiming the same MAC
>>> address and thus when each "speaks" and performs ARP it "grabs" the map
>>> which works until the next one pipes up and it grabs it.
>>> 
>> 
>> Looks like that's the problem. There's only one cable between switches, but
>> here's what I get from ifconfig on each host:
>> 
>> On the machine running 13.1-R attached to switch 2:
>> bob@www:~ % ifconfig
>> lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
>> 	options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
>> 	inet6 ::1 prefixlen 128
>> 	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
>> 	inet 127.0.0.1 netmask 0xff000000
>> 	groups: lo
>> 	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
>> ue0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
>> 	options=80009<RXCSUM,VLAN_MTU,LINKSTATE>
>>>>>>>>> 	ether b8:27:eb:71:46:4e
>> 	inet 50.1.20.28 netmask 0xffffff00 broadcast 50.1.20.255
>> 	media: Ethernet autoselect (100baseTX <full-duplex>)
>> 	status: active
>> 	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
>> bob@www:~ % hostname
>> www.zefox.org
>> bob@www:~ % 
>> bob@www:~ % uname -a
>> FreeBSD www.zefox.org 13.1-RELEASE FreeBSD 13.1-RELEASE releng/13.1-n250148-fc952ac2212 GENERIC arm64
>> bob@www:~ %
>> 
>> On the machine running an updated stable/13 system attached to switch 1: 
>> bob@pelorus:~ % ifconfig
>> lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
>> 	options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
>> 	inet6 ::1 prefixlen 128
>> 	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
>> 	inet 127.0.0.1 netmask 0xff000000
>> 	groups: lo
>> 	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
>> ue0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
>> 	options=80009<RXCSUM,VLAN_MTU,LINKSTATE>
>>>>>>>> 	ether b8:27:eb:71:46:4e
>> 	inet 50.1.20.24 netmask 0xffffff00 broadcast 50.1.20.255
>> 	media: Ethernet autoselect (100baseTX <full-duplex>)
>> 	status: active
>> 	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
>> bob@pelorus:~ % hostname
>> pelorus.zefox.org
>> bob@pelorus:~ % 
>> bob@pelorus:~ % uname -a
>> FreeBSD pelorus.zefox.org 13.1-STABLE FreeBSD 13.1-STABLE #6 stable/13-n251601-2353343b324: Sun Jul  3 21:43:04 PDT 2022     bob@pelorus.zefox.org:/usr/obj/usr/src/arm64.aarch64/sys/GENERIC arm64

Did you get the two RPi3*'s in the same purchase? Ending up with
2 that both happen to have b8:27:eb:71:46:4e seems odd unless
possibly the means of setting the values at the factory was
not varying the values like it should in some production lot (or
over some range of lots). Independent purchases at different
times would make it seem even odder.

>> Thinking it over, I added the extra switch some time ago and didn't 
>> immediately notice any problems. Both Pi3s started out on the first
>> switch (NetGear), with no obvdious problems. Later I probably moved 
>> one Pi3 to the second switch (D-Link) and started to notice troubles. 
>> Does this story make sense? 
>> 
>>> Each interface device from the factory is supposed to have a unique MAC
>>> address.?? This can, for most interfaces, be overridden (modern Android
>>> phones "randomize" it if told to as a "security" measure) but for obvious
>>> reasons doing that can lead to problems. Collisions where multiple devices
>>> are using the same MAC will lead to exactly the sort of thing you're seeing
>>> because the switch is sending the packets to the wrong place.
>>> 
>>> I've got a decent number of Pis of everything back to the "2" here and most
>>> of the time several of them are on my network at once.?? I've not seen this
>>> problem but I wouldn't exclude that both are claiming the same MAC and, if
>>> so, that's what's causing the problem.
>>> 
>> [example ifconfig output snipped]
>>> 
>>> That MUST be unique on your LAN; the prefix (first three octets) is a vendor
>>> code /*and the last three should never be duplicated by a vendor. */If you
>>> are not setting it in /etc/rc.conf or elsewhere and there /are /duplicates
>>> then a very bad thing happened when those units were manufactured -- set one
>>> of them to something else.
>>> 
>> 
>> Any pointers to MAC-setting methods appreciated.....
> 
> My example is not the best fit because it is for DHCP
> but in /etc/rc.conf I use (but showing "??"s):
> 
> ifconfig_dwc0="ether ??:??:??:??:??:?? DHCP"
> 
> to avoid its random assignment at power up.

"its": a Rock64, not a RPi* .

> So for you I would guess:
> 
> ifconfig_ue0="ether ??:??:??:??:??:?? inet 50.1.20.28 netmask 255.255.255.0"
> 

Of course, I listed a specific inet of yours.


It appears that for before the RPi4B* based RPi*'s, the serial number
and the mac address were related. From the network booting material
at:

https://www.raspberrypi.com/documentation/computers/remote-access.html#ethernet-mac-address

there is:

QUOTE
Ethernet MAC address

Before configuring network boot, make a note of the serial number and mac address so that the board can be identified by the TFTP/DHCP server.

On Raspberry Pi 4 the MAC address is programmed at manufacture and there is no link between the MAC address and serial number. Both the MAC address and serial numbers are displayed on the bootloader HDMI diagnostics screen.

To find the Ethernet MAC address:

ethtool -P eth0
To find the serial number:

grep Serial /proc/cpuinfo | cut -d ' ' -f 2 | cut -c 8-16
END QUOTE

You might want to use some variant of RaspiOS to look up the
serial numbers and mac addresses on both RPi3*'s: Did you also
end up with duplicated serial numbers?

Another way to look is to use (in a RaspiOS variant):

# vcgencmd otp_dump

Row "28:" has the serial number and the last 6 digits
should match the last 6 from the MAC address for a
RPi3* . Rows "64:" and "65:" together hold the MAC
address.

Some folks have end up with rows "65:" and "66:" matching,
which should not happen but has been associate with odd
MAC address results: the MAC address ends up not matching
the 6 digits from the serial number, for example.


I've found references to an undocumented control in config.txt :

force_mac_address=??:??:??:??:??:??

See, for example, https://forums.raspberrypi.com/viewtopic.php?t=327562

Apparently, force_mac_address controls what value shows up in
the device tree for what the ethernet0 alias points to in the
device tree.

Note that having a odd .dtb file could lead to force_mac_address
not working. (The RPi* firmware reads the file and then makes
a device tree with some modifications applied.)

===
Mark Millard
marklmi at yahoo.com