FYI: RPi4 (8 GiByte) USB3 vs. head -r363123: still a no-go for booting a USB3 / in my experiments

Sun Jul 19 22:08:24 UTC 2020

> Certainly makes it interesting to figure out what
> combination of things should be used overall.

Sorry, yes, the dtb I mostly use is ahead of FreeBSD's pkg content. The 8 GB rpi4 model I obtained refuses to boot with the start4.elf and other boot firwmare from November, so I just obtained the newest version for the whole boot partition. i.e., the newest versions of those files from the Raspberry Pi Foundation. That broke ethernet (by moving to rgmii-rxid) which I then fixed in D25251. I appreciate this complicates the situation.

> Until such a time, is there a good way to limit a RPi4 to
> the lower <= 3 GiBytes overall for the u-boot case?

You could set hw.physmem="3G" in /boot/loader.conf. I don't know if that guarantees FreeBSD will specifically see the lowest 3 GB. The bus tag should not be too hard to fix though.

    -- RHC.

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Sunday, 19 July 2020 22:51, Mark Millard <marklmi at yahoo.com> wrote:

>
>
> On 2020-Jul-19, at 04:30, Robert Crowston <crowston atprotonmail.com> wrote:
>
> > The ethernet problem could be caused by the recent change in upstream dtb, to change the ethernet driver to operate in rgmii-rxid mode instead of rgmii mode. See
> > https://reviews.freebsd.org/D25251. I have not noticed this performance problem you report myself, when using upstream dtbs from June 2020.
>
> https://reviews.freebsd.org/D25251 was added to head as
> -r362353 and my testing was based on head -r3630123 .
> So my report is based on D25251.
>
> I do see that sys/gnu/dts/arm64/broadcom/bcm2711-rpi-4-b.dts
> is base on Linux 5.5 (checked-in to FreeBSD 4 months ago).
>
> But the port Makefile sysutils/rpi-firmware/Makefile indicates:
> FW_TAG= 2042453 for PORTVERSION= 1.20190925.g20200109 .
> Looking on github 2042453 is the 2019-Nov-22 commit for
> "kernel: Bump to 4.19.85". It seems to be from between official
> Tags 1.0290925 and 1.20200114 .
>
> The most recent official Tag is 1.20200601+arm64 and has
> materials in part based on "kernel: Bump to 5.4.42". (That
> is a raspberrypi/linux number.)
>
> The most recent "kernel: Bump to" for master is for 5.4.51
> (committed on 2020-Jul-13). One of the items in that is:
>
> kernel: ARM: dts: Make bcm2711 dts more like 5.7
>
> (I assume 5.7 is an upstream linux version indication.)
> So newer than where
> sys/gnu/dts/arm64/broadcom/bcm2711-rpi-4-b.dts is based.
>
> Certainly makes it interesting to figure out what
> combination of things should be used overall. Your wording
> suggests to me 1.20200601+arm64 (i.e., f382cc1), if your
> wording is for an officially tagged version.
>
> > You are right that we are not handling the 3 GB DMA limit in the pcie driver. Unfortunately, it did not seem easy to thread the appropriate bus tag through without rewriting half the inherited driver stack, and in my testing the USB driver always allocated its DMA buffers in the lower 3 GB without being told. But obviously it is the wrong to rely on luck, so I’ll have a think about it.
>
> Until such a time, is there a good way to limit a RPi4 to
> the lower <= 3 GiBytes overall for the u-boot case? (There
> is an explicit selection for the uefi case.) Having a mode
> of operation that avoids the unreliable status could prove
> important for the duration.
>
> >     — RHC.
> >
> >
> > On Sun, Jul 19, 2020 at 03:18, Mark Millard marklmi at yahoo.com wrote:
> >
> > > > On 2020-Jul-18, at 14:37, Robert Crowston <crowston at protonmail.com> wrote:
> > > > I believe this differential (https://reviews.freebsd.org/D25261) would resolve it, but I haven't got around to addressing the comments there yet.
> > > > -- RHC.
> > >
> > > Yep, a -r363123 kernel with that also it booted / from
> > > the USB3 SSD. (The kernel came from the mmcsd0.)
> > > Notes for this u-boot-rpi4 based context:
> > > Unlike for uefi v1.17, genet0 and mmcsd0 show up.
> > > There is a big asymmetry for sending vs. receiving
> > > over genet0 (192.168.1.126 here):
> > >
> > > iperf3 -c 192.168.1.120 --get-server-output -B 192.168.1.126
> > >
> > > =============================================================
> > >
> > > Connecting to host 192.168.1.120, port 5201
> > > [ 5] local 192.168.1.126 port 56649 connected to 192.168.1.120 port 5201
> > > [ ID] Interval Transfer Bitrate Retr Cwnd
> > > [ 5] 0.00-1.00 sec 22.7 MBytes 191 Mbits/sec 85 38.4 KBytes
> > > [ 5] 1.00-2.00 sec 23.4 MBytes 196 Mbits/sec 146 25.7 KBytes
> > > [ 5] 2.00-3.00 sec 23.2 MBytes 195 Mbits/sec 226 27.1 KBytes
> > > [ 5] 3.00-4.00 sec 23.3 MBytes 195 Mbits/sec 229 95.0 KBytes
> > > [ 5] 4.00-5.00 sec 23.6 MBytes 198 Mbits/sec 228 83.7 KBytes
> > > [ 5] 5.00-6.00 sec 23.7 MBytes 199 Mbits/sec 209 7.13 KBytes
> > > [ 5] 6.00-7.00 sec 23.7 MBytes 199 Mbits/sec 207 49.8 KBytes
> > > [ 5] 7.00-8.00 sec 23.6 MBytes 198 Mbits/sec 217 1.43 KBytes
> > > [ 5] 8.00-9.00 sec 23.6 MBytes 198 Mbits/sec 224 71.3 KBytes
> > > [ 5] 9.00-10.00 sec 23.5 MBytes 197 Mbits/sec 242 8.55 KBytes
> > >
> > > [ ID] Interval Transfer Bitrate Retr
> > > [ 5] 0.00-10.00 sec 234 MBytes 197 Mbits/sec 2013 sender
> > > [ 5] 0.00-10.27 sec 234 MBytes 191 Mbits/sec receiver
> > >
> > > Server output:
> > >
> > > ---------------
> > >
> > > Server listening on 5201
> > >
> > > -------------------------
> > >
> > > Accepted connection from 192.168.1.126, port 57882
> > > [ 5] local 192.168.1.120 port 5201 connected to 192.168.1.126 port 56649
> > > [ ID] Interval Transfer Bitrate
> > > [ 5] 0.00-1.00 sec 16.4 MBytes 138 Mbits/sec
> > > [ 5] 1.00-2.00 sec 23.4 MBytes 196 Mbits/sec
> > > [ 5] 2.00-3.00 sec 23.2 MBytes 195 Mbits/sec
> > > [ 5] 3.00-4.00 sec 23.1 MBytes 194 Mbits/sec
> > > [ 5] 4.00-5.00 sec 23.6 MBytes 198 Mbits/sec
> > > [ 5] 5.00-6.00 sec 23.7 MBytes 199 Mbits/sec
> > > [ 5] 6.00-7.00 sec 23.7 MBytes 199 Mbits/sec
> > > [ 5] 7.00-8.00 sec 23.6 MBytes 198 Mbits/sec
> > > [ 5] 8.00-9.00 sec 23.6 MBytes 198 Mbits/sec
> > > [ 5] 9.00-10.00 sec 23.5 MBytes 197 Mbits/sec
> > > [ 5] 10.00-10.27 sec 6.30 MBytes 197 Mbits/sec
> > >
> > > [ ID] Interval Transfer Bitrate
> > > [ 5] 0.00-10.27 sec 234 MBytes 191 Mbits/sec receiver
> > > iperf Done.
> > > Rock64orRPi4# iperf3 -R -c 192.168.1.120 --get-server-output -B 192.168.1.126
> > > Connecting to host 192.168.1.120, port 5201
> > > Reverse mode, remote host 192.168.1.120 is sending
> > > [ 5] local 192.168.1.126 port 25404 connected to 192.168.1.120 port 5201
> > > [ ID] Interval Transfer Bitrate
> > > [ 5] 0.00-1.00 sec 111 MBytes 933 Mbits/sec
> > > [ 5] 1.00-2.00 sec 111 MBytes 933 Mbits/sec
> > > [ 5] 2.00-3.00 sec 111 MBytes 933 Mbits/sec
> > > [ 5] 3.00-4.00 sec 111 MBytes 933 Mbits/sec
> > > [ 5] 4.00-5.00 sec 111 MBytes 933 Mbits/sec
> > > [ 5] 5.00-6.00 sec 111 MBytes 933 Mbits/sec
> > > [ 5] 6.00-7.00 sec 111 MBytes 933 Mbits/sec
> > > [ 5] 7.00-8.00 sec 111 MBytes 933 Mbits/sec
> > > [ 5] 8.00-9.00 sec 111 MBytes 933 Mbits/sec
> > > [ 5] 9.00-10.00 sec 111 MBytes 933 Mbits/sec
> > >
> > > [ ID] Interval Transfer Bitrate Retr
> > > [ 5] 0.00-10.26 sec 1.09 GBytes 910 Mbits/sec 923 sender
> > > [ 5] 0.00-10.00 sec 1.09 GBytes 933 Mbits/sec receiver
> > >
> > > Server output:
> > >
> > > ---------------
> > >
> > > Server listening on 5201
> > >
> > > -------------------------
> > >
> > > Accepted connection from 192.168.1.126, port 41483
> > > [ 5] local 192.168.1.120 port 5201 connected to 192.168.1.126 port 25404
> > > [ ID] Interval Transfer Bitrate Retr Cwnd
> > > [ 5] 0.00-1.00 sec 83.7 MBytes 702 Mbits/sec 62 1.61 MBytes
> > > [ 5] 1.00-2.00 sec 111 MBytes 932 Mbits/sec 92 1.61 MBytes
> > > [ 5] 2.00-3.00 sec 111 MBytes 934 Mbits/sec 93 175 KBytes
> > > [ 5] 3.00-4.00 sec 111 MBytes 933 Mbits/sec 95 88.4 KBytes
> > > [ 5] 4.00-5.00 sec 111 MBytes 933 Mbits/sec 94 268 KBytes
> > > [ 5] 5.00-6.00 sec 111 MBytes 932 Mbits/sec 89 291 KBytes
> > > [ 5] 6.00-7.00 sec 111 MBytes 933 Mbits/sec 92 4.28 KBytes
> > > [ 5] 7.00-8.00 sec 111 MBytes 933 Mbits/sec 92 1.27 MBytes
> > > [ 5] 8.00-9.00 sec 111 MBytes 933 Mbits/sec 94 1.61 MBytes
> > > [ 5] 9.00-10.00 sec 111 MBytes 933 Mbits/sec 96 1.42 MBytes
> > > [ 5] 10.00-10.26 sec 29.3 MBytes 931 Mbits/sec 24 1.61 MBytes
> > >
> > > [ ID] Interval Transfer Bitrate Retr
> > > [ 5] 0.00-10.26 sec 1.09 GBytes 910 Mbits/sec 923 sender
> > > iperf Done.
> > > My attempt to duplicate an about 11 GiByte .tar file failed:
> > >
> > > cp -aRx /usr/obj/clang-armv7-on-aarch64.tar /usr/obj/mmjnk.tar
> > >
> > > ===============================================================
> > >
> > > g_vfs_done():gpt/Rock64root[READ(offset=274911461376, length=32768)]error = 5
> > > c/usr/obj/clang-armv7-on-UFS /dev/gpt/Rock64root (/) cylinder checksum failed: cg 270, cgp: 0x0 != bp: 0xa023cccf
> > > aarch64.tar: Input/output error
> > > UFS /dev/gpt/Rock64root (/) cylinder checksum failed: cg 270, cgp: 0x0 != bp: 0xa023cccf
> > > pid 1123 (ntpd), jid 0, uid 0: exited on signal 6 (core dumped)
> > > pid 1240 (login), jid 0, uid 0: exited on signal 11 (core dumped)
> > > Same media as used for Rock64 experiments. I've had no
> > > evidence of problems there.
> > > However uefi based booting the RPi4 also has problems
> > > unless uefi is set to force 3 GiBytes of RAM (at most).
> > > The problem has usually been visible as the diff
> > > after the copy showing 4 KiByte or slightly less
> > > having differences according to diff or cmp or the
> > > like. There is evidence of the content and where
> > > changing without updates to the files, suggesting
> > > garbage read data.
> > > Again, same media for 3GiByte RAM or on the Rock64:
> > > no evidence of problems for such operations.
> > > My initial guess is that the handling of the RPi4
> > > DMA limitations is not yet correct overall,
> > > apparently for both uefi and u-boot style booting.
> > >
> > > > ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> > > > On Wednesday, 15 July 2020 10:09, Mark Millard via freebsd-arm freebsd-arm at freebsd.org wrote:
> > > > . . .
>
> ==
>
> Mark Millard
> marklmi at yahoo.com
> ( dsl-only.net went
> away in early 2018-Mar)