Any hints about how to get (a RPi4B) PCIe/XHCI combination to deal with a "PCI addr: 0x400000000, CPU addr: 0x0" from dma-ranges?

From: Mark Millard <marklmi_at_yahoo.com>
Date: Fri, 20 Jan 2023 07:23:48 UTC
For my own edification, I've been looking into having RPi4B's use the
dma-ranges device tree information to set up the inbound configuration
for its PCIe/XHCI context. This would allow the modern "C0T" parts
that have more than 3 GiBytes of RAM to avoid bounce buffering
associated with the only-lower-3-GiBytes works PCIe wrapper-logic
problem that exists for "B0T" parts.

For a "C0T" 8 GiByte RPI4B the live Device Tree sections involved
look like:

. . .
        #address-cells = <0x00000002>;
        #size-cells = <0x00000001>;
. . .
        scb {
                compatible = "simple-bus";
                #address-cells = <0x00000002>;
                #size-cells = <0x00000002>;
                . . .
                pcie@7d500000 {
                        compatible = "brcm,bcm2711-pcie", "brcm,bcm7445-pcie";
                        reg = <0x00000000 0x7d500000 0x00000000 0x00009310>;
                        device_type = "pci";
                        #address-cells = <0x00000003>;
                        #interrupt-cells = <0x00000001>;
                        #size-cells = <0x00000002>;
                        . . .
                        ranges = <0x02000000 0x00000000 0xc0000000 0x00000006 0x00000000 0x00000000 0x40000000>;
                        dma-ranges = <0x02000000 0x00000004 0x00000000 0x00000000 0x00000000 0x00000002 0x00000000>;
                        . . .
                        pci@0,0 {
                                device_type = "pci";
                                #address-cells = <0x00000003>;
                                #size-cells = <0x00000002>;
                                ranges;
                                reg = <0x00000000 0x00000000 0x00000000 0x00000000 0x00000000>;
                                usb@0,0 {
                                        reg = <0x00000000 0x00000000 0x00000000 0x00000000 0x00000000>;
                                        resets = <0x0000002e 0x00000000>;
                                };
                        };
                };
. . .
                xhci@7e9c0000 {
                        compatible = "generic-xhci";
                        status = "disabled";
                        reg = <0x00000000 0x7e9c0000 0x00000000 0x00100000>;
                        interrupts = <0x00000000 0x000000b0 0x00000004>;
                        power-domains = <0x00000013 0x00000006>;
                        phandle = <0x000000e2>;
                };
. . .


Showing in boot -v output like form, the pcib0 would look like
(showing both Inbound and Outbound):

pcib0: Inbound pcie dma-range: PCI addr: 0x400000000, CPU addr: 0x0, Size: 0x200000000
pcib0: parsing FDT for ECAM0:
pcib0: 	PCI addr: 0xc0000000, CPU addr: 0x600000000, Size: 0x40000000

Note the:

PCI addr: 0x400000000, . . ., Size: 0x200000000
vs.
PCI addr:  0xc0000000, . . ., Size:  0x40000000

which nicely avoid PCI addr overlaps and leave the Inbound context
with a size of 8 GiBytes, matching the RAM size.

Unfortunately, for as little as I've figured out, I have to override
(just) the 0x400000000 and use 0x0 (to numerically match the CPU
addr values):

pcib0: Inbound pcie dma-range: PCI addr: 0x0, CPU addr: 0x0, Size: 0x200000000
pcib0: parsing FDT for ECAM0:
pcib0: 	PCI addr: 0xc0000000, CPU addr: 0x600000000, Size: 0x40000000

Note the:

PCI addr: 0x00000000, . . ., Size: 0x200000000
vs.
PCI addr: 0xc0000000, . . ., Size:  0x40000000

With this they do overlap. While it boots this way, it is only
because it has not yet run into the overlap. (I can do USB
tests that lead to a crash.)

It appears that I need a hint about how to get the XHCI to deal
with the "PCI addr: 0x400000000, CPU addr: 0x0" combination,
translating addresses between spaces appropriately. I am already
setting the PCIe RC Inbound PCIe related registers. But I expect
that there is more to it in order for the XHCI activity to use
that mapping. The hint might just be pointing me at something to
read that I've not managed to find.


For reference, the 0x400000000 context starts its boot failure
with a "Run timeout":

. . .
bcm_xhci0: (New XHCI DeviceId=0x34831106)
bcm_xhci0: Run timeout.
bcm_xhci0: XHCI halt/start/probe failed err=18
device_attach: bcm_xhci0 attach returned 6
. . .



===
Mark Millard
marklmi at yahoo.com