From nobody Wed Feb 14 16:08:37 2024 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TZjmT2M8wz5BFhr; Wed, 14 Feb 2024 16:08:41 +0000 (UTC) (envelope-from jhb@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TZjmT1r5hz45dx; Wed, 14 Feb 2024 16:08:41 +0000 (UTC) (envelope-from jhb@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1707926921; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=h/9057JLSucUzDn0y2NbBakKBsDjDVq0e9ZdHAvInfg=; b=eb1klHDE9V+Mrn52KhDuXMf24igekCq5PL3xerBqtNKHKwuLwoSNHopkeN71y5NR05mO9R e2p3zhk0eNQKWXmjFdc9BIWabbfoC+JhzAL3A/cRH8Kp36SPsPehNWF0ChpRMdj78NijSi zb4CQvf2nk5HSdCqI9z8l4lNHGGIHHYb3mk/JY4qBRWPkjchoandiVxbrzTR3hd26gKtB7 BH83Pm0x6sCNfG471g23qFdMS9chDRdyPZa5IXMUs2/WybF3YkjsjYQRi/LqTDkRRcCYtL +vy4O9kVr+j57WTZhwWc91L0n5BBJuhf08LlArJja4NSsaRw4euWciwfxAjyYQ== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1707926921; a=rsa-sha256; cv=none; b=m5QuIKaGUme/KVnjhrF9h/5uQTk6X1nXKYiJUpYR9Ogfj7x0D7icsnCKGIYBRnJd9ACdy/ zZJ10gBWXLrlyNTwnit6+ujCsqqp0VbwOLcUbCS53MKq1DS4XelDdYbeAWrjNHYktxJI+t +GwDFvTVf1l1UaPpEpNX7IpbjOGkbS7zODtqq+OXtFGNNYlWwjsA4AvWywyZ3f7EqR3G9T 8pAasI/FxgOD+Qz6kjAN9dv7bb7Ei3YunKmEudrURZpZ9eQCmsBngtutpBdbSlA5DaoEne qfOYadsmsduN+pnDppanyNjvR+V12YiYz59fhDFzl0oZ/JOYMu3U3DPzRN0+Bw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1707926921; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=h/9057JLSucUzDn0y2NbBakKBsDjDVq0e9ZdHAvInfg=; b=uAFe+XVcMYwBqBiqFB7UQ+MXuGXXOeIFC5OwODxmSeMOoiK1QIBE5GPGNJh6/le+UIQFZc hHHHc2LXaBSmYobF9IRnclZOGUGPUvWU1JxBgDns9k1SNRiZh700lWlHfpZXBPj8KrRXXX vzENceLr4xHEEYCtPJdqPHYB42ciHbMFvnK+h0sTl1DFscieciyesjDXBJAZbS2FPAQW2y XYHD7dZc9qRnuJZ4Kh+NEDPy2HST8OUl06LYchEmAUQW3LmxPcCBZYide5x39aZEs4/qz/ Em5GJd1DtD6jcwOe74gRW5QVKy8e4N2nM1huWjy0C+SGnsXia2sGaJX0fNhMgg== Received: from [IPV6:2601:644:937c:5920:4c63:23c7:5c22:d7ba] (unknown [IPv6:2601:644:937c:5920:4c63:23c7:5c22:d7ba]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: jhb) by smtp.freebsd.org (Postfix) with ESMTPSA id 4TZjmS3sx7zNsB; Wed, 14 Feb 2024 16:08:40 +0000 (UTC) (envelope-from jhb@FreeBSD.org) Message-ID: Date: Wed, 14 Feb 2024 08:08:37 -0800 List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: Recent commits reject RPi4B booting: pcib0 vs. pcib1 "rman_manage_region: request" leads to panic Content-Language: en-US To: Mark Millard Cc: FreeBSD ARM List , Current FreeBSD , Warner Losh References: <76AB969F-5BC5-4116-8AF4-3ED2CABEBBA5.ref@yahoo.com> <76AB969F-5BC5-4116-8AF4-3ED2CABEBBA5@yahoo.com> <1F704317-FDB8-4BDA-8A67-61CF48794DFE@yahoo.com> <9AFDF067-96E4-4E67-90D2-F40DAF3F138F@yahoo.com> <4C279710-5F88-4295-B1A4-7C395F3587E5@yahoo.com> <3A145420-399D-4EBD-9FF4-18924908AB1D@yahoo.com> <1298DF9C-0F82-4567-8E81-7332A608C7FC@yahoo.com> From: John Baldwin In-Reply-To: <1298DF9C-0F82-4567-8E81-7332A608C7FC@yahoo.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit On 2/12/24 5:57 PM, Mark Millard wrote: > On Feb 12, 2024, at 16:36, Mark Millard wrote: > >> On Feb 12, 2024, at 16:10, Mark Millard wrote: >> >>> On Feb 12, 2024, at 12:00, Mark Millard wrote: >>> >>>> [Gack: I was looking at the wrong vintage of source code, predating >>>> your changes: wrong system used.] >>>> >>>> >>>> On Feb 12, 2024, at 10:41, Mark Millard wrote: >>>> >>>>> On Feb 12, 2024, at 09:32, John Baldwin wrote: >>>>> >>>>>> On 2/9/24 8:13 PM, Mark Millard wrote: >>>>>>> Summary: >>>>>>> pcib0: mem 0x7d500000-0x7d50930f irq 80,81 on simplebus2 >>>>>>> pcib0: parsing FDT for ECAM0: >>>>>>> pcib0: PCI addr: 0xc0000000, CPU addr: 0x600000000, Size: 0x40000000 >>>>>>> . . . >>>>>>> rman_manage_region: request: start 0x600000000, end 0x6000fffff >>>>>>> panic: Failed to add resource to rman >>>>>> >>>>>> Hmmm, I suspect this is due to the way that bus_translate_resource works which is >>>>>> fundamentally broken. It rewrites the start address of a resource in-situ instead >>>>>> of keeping downstream resources separate from the upstream resources. For example, >>>>>> I don't see how you could ever release a resource in this design without completely >>>>>> screwing up your rman. That is, I expect trying to detach a PCI device behind a >>>>>> translating bridge that uses the current approach should corrupt the allocated >>>>>> resource ranges in an rman long before my changes. >>>>>> >>>>>> That said, that doesn't really explain the panic. Hmm, the panic might be because >>>>>> for PCI bridge windows the driver now passes RF_ACTIVE and the bus_translate_resource >>>>>> hack only kicks in the activate_resource method of pci_host_generic.c. >>>>>> >>>>>>> Detail: >>>>>>> . . . >>>>>>> pcib0: mem 0x7d500000-0x7d50930f irq 80,81 on simplebus2 >>>>>>> pcib0: parsing FDT for ECAM0: >>>>>>> pcib0: PCI addr: 0xc0000000, CPU addr: 0x600000000, Size: 0x40000000 >>>>>> >>>>>> This indicates this is a translating bus. >>>>>> >>>>>>> pcib1: irq 91 at device 0.0 on pci0 >>>>>>> rman_manage_region: request: start 0x1, end 0x1 >>>>>>> pcib0: rman_reserve_resource: start=0xc0000000, end=0xc00fffff, count=0x100000 >>>>>>> rman_reserve_resource_bound: request: [0xc0000000, 0xc00fffff], length 0x100000, flags 102, device pcib1 >>>>>>> rman_reserve_resource_bound: trying 0xffffffff <0xc0000000,0xfffff> >>>>>>> considering [0xc0000000, 0xffffffff] >>>>>>> truncated region: [0xc0000000, 0xc00fffff]; size 0x100000 (requested 0x100000) >>>>>>> candidate region: [0xc0000000, 0xc00fffff], size 0x100000 >>>>>>> allocating from the beginning >>>>>>> rman_manage_region: request: start 0x600000000, end 0x6000fffff >>> >>> What you later typed does not match: >>> >>> 0x600000000 >>> 0x6000fffff >>> >>> You later typed: >>> >>> 0x60000000 >>> 0x600fffffff >>> >>> This seems to have lead to some confusion from using the >>> wrong figure(s). >>> >>>>>> The fact that we are trying to reserve the CPU addresses in the rman is because >>>>>> bus_translate_resource rewrote the start address in the resource after it was allocated. >>>>>> >>>>>> That said, I can't see why rman_manage_region would actually fail. At this point the >>>>>> rman is empty (this is the first call to rman_manage_region for "pcib1 memory window"), >>>>>> so only the check that should be failing are the checks against rm_start and >>>>>> rm_end. For the memory window, rm_start is always 0, and rm_end is always >>>>>> 0xffffffff, so both the old (0xc00000000 - 0xc00fffff) and new (0x60000000 - 0x600fffffff) >>>>>> ranges are within those bounds. >>> >>> No: >>> >>> 0xffffffff >>> >>> .vs (actual): >>> >>> 0x600000000 >>> 0x6000fffff Ok, then this explains the failure if the "raw" addresses are above 4G. I have access to an emag I'm currently using to test fixes to pci_host_generic.c to avoid corrupting struct resource objects. I'll post the diff once I've got something verified to work. > It looks to me like in sys/dev/pci/pci_pci.c the: > > static void > pcib_probe_windows(struct pcib_softc *sc) > { > . . . > pcib_alloc_window(sc, &sc->mem, SYS_RES_MEMORY, 0, 0xffffffff); > . . . > > is just inappropriately restrictive about where in the system > address space a PCIe can validly be mapped to on the high end. > That, in turn, leads to the rejection on the RPi4B now that > the range use is checked. No, the physical register in PCI-PCI bridges is only 32-bits. Only the prefetchable BAR supports 64-bit addresses. This is why the host bridge is doing a translation from the CPU side (0x600000000) to the PCI BAR addresses (0xc0000000) so that the BAR addresses are down in the 32-bit address range. It's also true that many PCI devices only support 32-bit addresses in memory BARs. 64-bit BARs are an optional extension not universally supported. The translation here is somewhat akin to a type of MMU where the CPU addresses are mapped to PCI addresses. The problem here is that the PCI BAR resources need to "stay" as PCI addresses since we depend on being able to use rman_get_start/end to get the PCI addresses of allocated resources, but pci_host_generic.c currently rewrites the addresses. Probably I should remove rman_set_start/end entirely (Warner added them back in 2004) as the methods don't do anything to deal with the fallout that the rman.rm_list linked-list is no longer sorted by address once some addresses get rewritten, etc. -- John Baldwin