From nobody Wed Feb 14 18:17:43 2024 X-Original-To: freebsd-arm@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TZmdQ53SMz51hfL; Wed, 14 Feb 2024 18:17:46 +0000 (UTC) (envelope-from jhb@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TZmdQ48Jyz4SPB; Wed, 14 Feb 2024 18:17:46 +0000 (UTC) (envelope-from jhb@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1707934666; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=1DODsuObWTMBP56lJdKpk5bTPZocrl/yKB0jED9XwG8=; b=j7KF7iCdBM3q+xjFzu/qjalhYwG85ocoYiuk+CueGTuaRUUSMP0hJTSvhue8D7hd2OA+41 LqrAHqxdwS8j5f42u4+Qa6CDoS6Ad8HIcy0vPpcE2YVtoW5ghU0B24xq6H9PWiWt6wsmh7 l4nVl9PvtJNZ/kl3aWfhbtO1PEfEZc/V3UdR/W66uJnS+ndm4vdPvfInuZoV9O09OCUaqe DOD58cei3s6fr/aC+SWsSQw6z4eLEy6kzGLQH50brcNKMZ882TNEdw0GBnp5zminMbq5l9 1+U5C1vxFn752BoeccHEbHqRQ01UEadjDv89yV9vSp/xXQuUxx1zBpxLBku87w== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1707934666; a=rsa-sha256; cv=none; b=bIPdZdiYNu6LKdPUyt1+pFK1AJcftwCxx165gomfdHvVd9TNdM3BghKkeLPdBvnwDOcWYP 7mUGXgJxEU7xoKpNC4RFgmu2GJgxCCiJsn1H3Rm0noMJvTH4lH/rbwOAgxmGtfhv+WeE63 te20zMVlgGx83eIfms957h3OmQOA/8SL/+JynX5Swnym9mck72W5jpviFCGmnD6vKfZL5Z 3Sya6yTWyNKs3mc8Md9/RP3GTZm6JoYhlssjCUT3W0thc0JlaqXwPpQncotYG1pkk+W5lD grmk9LGou+e7KnVZ3cvAkTcrSGZ1h75NSkEg6oMZ3gy7EIUl7Fj3LECJqv+Qtg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1707934666; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=1DODsuObWTMBP56lJdKpk5bTPZocrl/yKB0jED9XwG8=; b=B4N5VvsKwOz1iO4el6vzX58bgtLdrCLmaZ6ZsBi0f60jduASnYBVL/UiNV0Hj98dBdbr1A POi91s2jxuYOgpDWDRiyLRDIgqlDpAQnsoQT8QXJQcNaeQqqhi+A6hEUGNKYVEWPAhsDOE 31M18h2g7OvwRDzDhVbVjHpxJLcpKGETfEQf4qdjZG46rkfFu6X8+cAWX6O6nn40svsGvL vecfYpYy0tjqkZDfe/1R+ph8AwIufdyZQkZtyYtCzf2WguUUDoIwdi+M7gAfNEXzUIWKkq wlBmhaqsv21KFOlarDHOk9dHsL9LnC0UYl56NoPqetFls48SO1BnEBoWadHRhg== Received: from [IPV6:2601:644:937c:5920:5974:74f9:c690:271e] (unknown [IPv6:2601:644:937c:5920:5974:74f9:c690:271e]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: jhb) by smtp.freebsd.org (Postfix) with ESMTPSA id 4TZmdP74cJzRgh; Wed, 14 Feb 2024 18:17:45 +0000 (UTC) (envelope-from jhb@FreeBSD.org) Message-ID: Date: Wed, 14 Feb 2024 10:17:43 -0800 List-Id: Porting FreeBSD to ARM processors List-Archive: https://lists.freebsd.org/archives/freebsd-arm List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-arm@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: Recent commits reject RPi4B booting: pcib0 vs. pcib1 "rman_manage_region: request" leads to panic Content-Language: en-US To: Mark Millard Cc: FreeBSD ARM List , Current FreeBSD , Warner Losh References: <76AB969F-5BC5-4116-8AF4-3ED2CABEBBA5.ref@yahoo.com> <76AB969F-5BC5-4116-8AF4-3ED2CABEBBA5@yahoo.com> <1F704317-FDB8-4BDA-8A67-61CF48794DFE@yahoo.com> <9AFDF067-96E4-4E67-90D2-F40DAF3F138F@yahoo.com> <4C279710-5F88-4295-B1A4-7C395F3587E5@yahoo.com> <3A145420-399D-4EBD-9FF4-18924908AB1D@yahoo.com> <1298DF9C-0F82-4567-8E81-7332A608C7FC@yahoo.com> <36ECB040-7E09-47A9-AF71-DE546A78E9CA@yahoo.com> From: John Baldwin In-Reply-To: <36ECB040-7E09-47A9-AF71-DE546A78E9CA@yahoo.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit On 2/14/24 9:57 AM, Mark Millard wrote: > On Feb 14, 2024, at 08:08, John Baldwin wrote: > >> On 2/12/24 5:57 PM, Mark Millard wrote: >>> On Feb 12, 2024, at 16:36, Mark Millard wrote: >>>> On Feb 12, 2024, at 16:10, Mark Millard wrote: >>>> >>>>> On Feb 12, 2024, at 12:00, Mark Millard wrote: >>>>> >>>>>> [Gack: I was looking at the wrong vintage of source code, predating >>>>>> your changes: wrong system used.] >>>>>> >>>>>> >>>>>> On Feb 12, 2024, at 10:41, Mark Millard wrote: >>>>>> >>>>>>> On Feb 12, 2024, at 09:32, John Baldwin wrote: >>>>>>> >>>>>>>> On 2/9/24 8:13 PM, Mark Millard wrote: >>>>>>>>> Summary: >>>>>>>>> pcib0: mem 0x7d500000-0x7d50930f irq 80,81 on simplebus2 >>>>>>>>> pcib0: parsing FDT for ECAM0: >>>>>>>>> pcib0: PCI addr: 0xc0000000, CPU addr: 0x600000000, Size: 0x40000000 >>>>>>>>> . . . >>>>>>>>> rman_manage_region: request: start 0x600000000, end 0x6000fffff >>>>>>>>> panic: Failed to add resource to rman >>>>>>>> >>>>>>>> Hmmm, I suspect this is due to the way that bus_translate_resource works which is >>>>>>>> fundamentally broken. It rewrites the start address of a resource in-situ instead >>>>>>>> of keeping downstream resources separate from the upstream resources. For example, >>>>>>>> I don't see how you could ever release a resource in this design without completely >>>>>>>> screwing up your rman. That is, I expect trying to detach a PCI device behind a >>>>>>>> translating bridge that uses the current approach should corrupt the allocated >>>>>>>> resource ranges in an rman long before my changes. >>>>>>>> >>>>>>>> That said, that doesn't really explain the panic. Hmm, the panic might be because >>>>>>>> for PCI bridge windows the driver now passes RF_ACTIVE and the bus_translate_resource >>>>>>>> hack only kicks in the activate_resource method of pci_host_generic.c. >>>>>>>> >>>>>>>>> Detail: >>>>>>>>> . . . >>>>>>>>> pcib0: mem 0x7d500000-0x7d50930f irq 80,81 on simplebus2 >>>>>>>>> pcib0: parsing FDT for ECAM0: >>>>>>>>> pcib0: PCI addr: 0xc0000000, CPU addr: 0x600000000, Size: 0x40000000 >>>>>>>> >>>>>>>> This indicates this is a translating bus. >>>>>>>> >>>>>>>>> pcib1: irq 91 at device 0.0 on pci0 >>>>>>>>> rman_manage_region: request: start 0x1, end 0x1 >>>>>>>>> pcib0: rman_reserve_resource: start=0xc0000000, end=0xc00fffff, count=0x100000 >>>>>>>>> rman_reserve_resource_bound: request: [0xc0000000, 0xc00fffff], length 0x100000, flags 102, device pcib1 >>>>>>>>> rman_reserve_resource_bound: trying 0xffffffff <0xc0000000,0xfffff> >>>>>>>>> considering [0xc0000000, 0xffffffff] >>>>>>>>> truncated region: [0xc0000000, 0xc00fffff]; size 0x100000 (requested 0x100000) >>>>>>>>> candidate region: [0xc0000000, 0xc00fffff], size 0x100000 >>>>>>>>> allocating from the beginning >>>>>>>>> rman_manage_region: request: start 0x600000000, end 0x6000fffff >>>>> >>>>> What you later typed does not match: >>>>> >>>>> 0x600000000 >>>>> 0x6000fffff >>>>> >>>>> You later typed: >>>>> >>>>> 0x60000000 >>>>> 0x600fffffff >>>>> >>>>> This seems to have lead to some confusion from using the >>>>> wrong figure(s). >>>>> >>>>>>>> The fact that we are trying to reserve the CPU addresses in the rman is because >>>>>>>> bus_translate_resource rewrote the start address in the resource after it was allocated. >>>>>>>> >>>>>>>> That said, I can't see why rman_manage_region would actually fail. At this point the >>>>>>>> rman is empty (this is the first call to rman_manage_region for "pcib1 memory window"), >>>>>>>> so only the check that should be failing are the checks against rm_start and >>>>>>>> rm_end. For the memory window, rm_start is always 0, and rm_end is always >>>>>>>> 0xffffffff, so both the old (0xc00000000 - 0xc00fffff) and new (0x60000000 - 0x600fffffff) >>>>>>>> ranges are within those bounds. >>>>> >>>>> No: >>>>> >>>>> 0xffffffff >>>>> >>>>> .vs (actual): >>>>> >>>>> 0x600000000 >>>>> 0x6000fffff >> >> Ok, then this explains the failure if the "raw" addresses are above 4G. I have >> access to an emag I'm currently using to test fixes to pci_host_generic.c to >> avoid corrupting struct resource objects. I'll post the diff once I've got >> something verified to work. >> >>> It looks to me like in sys/dev/pci/pci_pci.c the: >>> static void >>> pcib_probe_windows(struct pcib_softc *sc) >>> { >>> . . . >>> pcib_alloc_window(sc, &sc->mem, SYS_RES_MEMORY, 0, 0xffffffff); >>> . . . >>> is just inappropriately restrictive about where in the system >>> address space a PCIe can validly be mapped to on the high end. >>> That, in turn, leads to the rejection on the RPi4B now that >>> the range use is checked. >> >> No, the physical register in PCI-PCI bridges is only 32-bits. Only the >> prefetchable BAR supports 64-bit addresses. > > Just for my edification . . . > > As I understand, SYS_RES_MEMORY for the BCM2711 > means the 35 bit addressing space in the BCM2711, > not a PCIe device internal address range that > corresponds. Am I wrong about that? > > If I'm wrong, what does identify the 35 bit > addressing space in the BCM2711? > > If I'm correct, then the 0..0xffffffff > seems to be from the wrong address space up > front. Or, may be, the SYS_RES_MEMORY and the > 0xffffffff argments are not related as I > expected and the 0xffffffff is not a > SYS_RES_MEMORY value? We use SYS_RES_MEMORY for both address spaces. SYS_RES_MEMORY is more of an address space "type" and doesn't necessarily name a single, unique address space. The way to think about these address spaces is instances of 'struct rman'. There's a global 'struct rman' in the arm64 nexus driver that represents the CPU physical memory address space. The pci_host_generic driver contains its own 'struct rman' instances that represent the SYS_RES_MEMORY (for memory PCI BARs) and SYS_RES_IOPORT (for I/O port PCI BARs) address spaces. Put another way, SYS_RES_MEMORY names an I/O memory address space relative to a device's given position in the tree. For a given device node in the tree, SYS_RES_MEMORY is unique, but what it maps onto is defined by a parent bus device. -- John Baldwin