RE: [EXTERNAL] pcib msix allocation in arm64

> -----Original Message-----
> From: Andrew Turner <andrew@fubar.geek.nz>
> Sent: Thursday, November 3, 2022 6:21 PM
> To: Souradeep Chakrabarti <schakrabarti@microsoft.com>
> Cc: Warner Losh <imp@bsdimp.com>; Wei Hu <weh@microsoft.com>; freebsd-
> hackers@FreeBSD.org
> Subject: Re: [EXTERNAL] pcib msix allocation in arm64
> 
> Hi Souradeep,
> 
> For the vmbus_pcib driver you’ll need to implement the pcib_alloc_msi,
> pcib_release_msi, and the msix versions, and pcib_map_msi.
> 
> For alloc and release you can just call into the same intr_* function as
> pci_host_generic_acpi.c. You’ll need to pass in the xref for the interrupt controller.
> It needs to be the xref the MSI controller registered. For the GICv2m it’s
> ACPI_MSI_XREF.
[Souradeep] 
I am currently specifying gic_mbi_start and gic_mbi_end with SPI start and end number, in gic_v3_acpi_attach() and 
I can see my intr_alloc_msi() and intr_map_msi() are working. But then mlx driver is
getting stuck at CREATE_EQ . 

mlx5_core0: WARN: wait_func:967:(pid 0): CREATE_EQ(0x301) timeout. Will cause a leak of a command resource
mlx5_core0: WARN: mlx5_start_eqs:588:(pid 0): failed to create async EQ -60
mlx5_core0: ERR: mlx5_load_one:1157:(pid 0): Failed to start pages and async EQs
mlx5_core0: WARN: wait_func:967:(pid 0): MANAGE_PAGES(0x108) timeout. Will cause a leak of a command resource
mlx5_core0: ERR: reclaim_pages:444:(pid 0): failed reclaiming pages
mlx5_core0: WARN: mlx5_reclaim_startup_pages:574:(pid 0): failed reclaiming pages (-60)
mlx5_core0: ERR: init_one:1644:(pid 0): mlx5_load_one failed -60


gic_v3_acpi_attach( )
--
--
323     sc->gic_mbi_start = 64;
324     sc->gic_mbi_end = 920;
325
326     err = gic_v3_attach(dev);
327     if (err != 0)
328         goto error;
329
330     sc->gic_pic = intr_pic_register(dev, ACPI_INTR_XREF);
331     if (sc->gic_pic == NULL) {
332         device_printf(dev, "could not register PIC\n");
333         err = ENXIO;
334         goto error;
335     }
336     err = intr_msi_register(dev, ACPI_MSI_XREF);
337     if (err) {
338         device_printf(dev, "could not register MSI\n");
339     }
340     if (intr_pic_claim_root(dev, ACPI_INTR_XREF, arm_gic_v3_intr, sc,
341         GIC_LAST_SGI - GIC_FIRST_SGI + 1) != 0) {
342         err = ENXIO;
343         goto error;
344     }
We don't have SPI range mentioned in ACPI DSDT, so I need to specify it manually in the code.
It looks like MSI interrupt is not coming to the mlx. 
Is there any interrupt remapping blocked by FreeBSD kernel here?

> 
> For pcib_map_msi I am unsure if the current code is valid on arm64. If not you’ll
> need to call intr_map_msi.
> 
> Andrew
> 
> > On 2 Nov 2022, at 22:27, Souradeep Chakrabarti <schakrabarti@microsoft.com>
> wrote:
> >
> > Hi Andrew,
> > Thanks for the reply. Regarding generic_pcie_acpi_alloc_msix( ), it
> > can be called if the PCI device is child of the generic pcib (
> DRIVER_MODULE(pcib, acpi, generic_pcie_acpi_driver, 0, 0) .
> > But if the PCI device is communicating with a different pcib driver
> > (like vmbus_pcib), in that case do we need to implement all these functions of
> pci_host_generic_acpi.c ?
> >
> > Or there are some ways to reuse them?
> >
> >> -----Original Message-----
> >> From: Andrew Turner <andrew@fubar.geek.nz>
> >> Sent: Wednesday, November 2, 2022 6:54 PM
> >> To: Souradeep Chakrabarti <schakrabarti@microsoft.com>
> >> Cc: Warner Losh <imp@bsdimp.com>; Wei Hu <weh@microsoft.com>;
> >> freebsd- hackers@FreeBSD.org
> >> Subject: [EXTERNAL] Re: pcib msix allocation in arm64
> >>
> >>
> >>> On 2 Nov 2022, at 12:56, Souradeep Chakrabarti
> >>> <schakrabarti@microsoft.com>
> >> wrote:
> >>>
> >>> Hi,
> >>> I can see in x86 nexus.c has implemented pcib_alloc_msix using
> >> nexus_alloc_msix().
> >>> Which calls msix_alloc() for msix allocation.
> >>>
> >>> But in case of arm64 I don't find similar pcib_alloc_msix
> >>> implementation in
> >> nexus.c .
> >>> So, on arm64 what is correct way to get allocate msix ?
> >>
> >> For an arm64 system with ACPI it is most likely handled in
> >> generic_pcie_acpi_release_msix. For FDT it can depend on which PCI
> >> driver is used.
> >>
> >> In either case it will call into intr_release_msix that then calls
> >> into the MSI controller to allocate the vectors. For a GICv3 driver
> >> it will either be gicv3_its_alloc_msix if you have an ITS device, or
> >> gic_v3_alloc_msix if using MBI ranges.
> >>
> >> On ACPI we don’t currently support MBI ranges, although it looks like
> >> this could be handled by the existing gicv2m driver. This driver
> >> should already work as a child of the GICv3, however it appears to be
> >> FDT only, so will need some work to add ACPI support.
> >>
> >> Andrew
> >