From nobody Fri Nov 11 20:10:41 2022 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4N88wF75mqz4ftHG for ; Fri, 11 Nov 2022 20:10:53 +0000 (UTC) (envelope-from bT.43g9herc30=mb0c0bam51bh=hchst2t53z@em790814.fubar.geek.nz) Received: from e2i342.smtp2go.com (e2i342.smtp2go.com [103.2.141.86]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4N88wF4DZKz4LcF for ; Fri, 11 Nov 2022 20:10:52 +0000 (UTC) (envelope-from bT.43g9herc30=mb0c0bam51bh=hchst2t53z@em790814.fubar.geek.nz) Authentication-Results: mx1.freebsd.org; none DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=smtpservice.net; s=mgy720.a1-4.dyn; x=1668198352; h=Feedback-ID: X-Smtpcorp-Track:To:Date:Subject:Message-Id:From:Reply-To:Sender: List-Unsubscribe; bh=A6b4bXG0CJ7gIdEnMheEsN7oA2MfddOJhE/l5fGDG58=; b=xtCxtopm ruMWpvCfGPvRrlATWouBFnfjBybbKaiHARTWuS5Ppsy80N+LpPY4HX6cTKN1nCUu3Rag7laEolndo aS4EGC22QzCOUejEYkMR4L7nxHso2Ra1uVBn+PvQ0NTLTFTU/bb+GlQg1n+H1OpIp0MUwpIsX/FYR AC2aH7xnbNJ4trgwwEzoOIF9uUVG/jsW/IWZKnT6Pc7HEz9+24qefyABxeZQuJIQc24/t+us5OzB1 xq/vigzLUX5QMbQjiwMr9hugwHV1+vN9YHRm5dHb3gJkK0CBjrxb/dc2vQfdcZ3LztKBcY2hxYd56 Gnd5C6f+muO91xDy+yoiXwI1/g==; DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fubar.geek.nz; i=@fubar.geek.nz; q=dns/txt; s=s790814; t=1668197452; h=from : subject : to : message-id : date; bh=A6b4bXG0CJ7gIdEnMheEsN7oA2MfddOJhE/l5fGDG58=; b=JWy6po59OGk39KpljL6y09LL65OoQyTuy+RyvCrtw7QdV27oHNxNPKA+qr+1IUoBibVg5 68wS53/xUUbTqC7mTgFAHBq56eZDy0E5ioDMGwJLXHDsmtvpSwRWpAYit0kGJAkoXq6Nx/z E9ITthKAjr1TZsfpKgYikFx4eyyb9EA5TDfWbGlu601R/UPUvPEUyKx2qZ2Ylb+ZyLsaON4 V+erFm1mnNqomqN+//ImKbJEBFp5ShpgZITioEaSay5vwQC/9fvxF5E9sSEoQ9Gsmh0bt2E Hy6BV3pHjToJYLnKKyGTfn/S/TgCEoPv/8qi3YATpbIKOm9E9Ey9jrI1kMRA== Received: from [10.139.162.187] (helo=SmtpCorp) by smtpcorp.com with esmtpsa (TLS1.3:ECDHE_SECP256R1__RSA_PSS_RSAE_SHA256__AES_256_GCM:256) (Exim 4.94.2-S2G) (envelope-from ) id 1otaMR-TRjzqm-FB; Fri, 11 Nov 2022 20:10:47 +0000 Received: from [10.162.55.164] (helo=morbo.fubar.geek.nz) by smtpcorp.com with esmtpsa (TLS1.3:ECDHE_X25519__RSA_PSS_RSAE_SHA256__AES_256_GCM:256) (Exim 4.96-S2G) (envelope-from ) id 1otaMR-4XobWr-0s; Fri, 11 Nov 2022 20:10:47 +0000 Received: from smtpclient.apple (cpc91214-cmbg18-2-0-cust234.5-4.cable.virginm.net [81.102.75.235]) by morbo.fubar.geek.nz (Postfix) with ESMTPSA id 884D69AE8; Fri, 11 Nov 2022 20:10:42 +0000 (UTC) From: Andrew Turner Message-Id: Content-Type: multipart/alternative; boundary="Apple-Mail=_C72BF360-FA1E-44BC-ABBC-1D9F0B37BFF7" List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3696.120.41.1.1\)) Subject: Re: [EXTERNAL] pcib msix allocation in arm64 Date: Fri, 11 Nov 2022 20:10:41 +0000 In-Reply-To: Cc: Warner Losh , Wei Hu , "freebsd-hackers@FreeBSD.org" To: Souradeep Chakrabarti References: <146F5D18-6366-4953-A8D9-61FE7EC67F71@fubar.geek.nz> <947DBD67-6443-480F-82DA-2BDDF44C3D03@fubar.geek.nz> X-Mailer: Apple Mail (2.3696.120.41.1.1) X-Smtpcorp-Track: 1otauR4boPWr0s.PnxFYr1ZTENGC Feedback-ID: 790814m:790814amQcrys:790814sNwJM4lss3 X-Report-Abuse: Please forward a copy of this message, including all headers, to X-Rspamd-Queue-Id: 4N88wF4DZKz4LcF X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:23352, ipnet:103.2.140.0/22, country:US] X-ThisMailContainsUnwantedMimeParts: N --Apple-Mail=_C72BF360-FA1E-44BC-ABBC-1D9F0B37BFF7 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 > On 11 Nov 2022, at 19:07, Souradeep Chakrabarti = wrote: >=20 >=20 >=20 >=20 >> -----Original Message----- >> From: Souradeep Chakrabarti >> Sent: Thursday, November 10, 2022 3:20 PM >> To: Andrew Turner > >> Cc: Warner Losh >; Wei Hu = >; freebsd- >> hackers@FreeBSD.org >> Subject: RE: [EXTERNAL] pcib msix allocation in arm64 >>=20 >>=20 >>=20 >>=20 >>> -----Original Message----- >>> From: Andrew Turner >>> Sent: Thursday, November 3, 2022 6:21 PM >>> To: Souradeep Chakrabarti >>> Cc: Warner Losh ; Wei Hu ; >> freebsd- >>> hackers@FreeBSD.org >>> Subject: Re: [EXTERNAL] pcib msix allocation in arm64 >>>=20 >>> Hi Souradeep, >>>=20 >>> For the vmbus_pcib driver you=E2=80=99ll need to implement the = pcib_alloc_msi, >>> pcib_release_msi, and the msix versions, and pcib_map_msi. >>>=20 >>> For alloc and release you can just call into the same intr_* = function >>> as pci_host_generic_acpi.c. You=E2=80=99ll need to pass in the xref = for the >>> interrupt controller. It needs to be the xref the MSI controller >>> registered. For the GICv2m it=E2=80=99s ACPI_MSI_XREF. >>>=20 >> [Souradeep] >> I have used following in vmbus_pcib.c : >> ret =3D intr_alloc_msix(pcib, dev, ACPI_MSI_XREF, irq); But it is = failing in >> pic_lookup() with ESRCH by not finding pic with ACPI_MSI_XREF. >> Do I need to do anything for pic_lookup() to be successful? >>=20 > [Souradeep]=20 > I found out the reason here pic_lookup() for ACPI_MSI_XREF is failing, = as > we are not calling intr_msi_register() but intr_pic_register() is = getting called for the PCIB. > This is because we don't have ITS implemented in Hyper-V. Instead, = Hyper-V is presenting the > MSI/MSI-X as SPI because of it's own limitation. > Linux has solved it using custom irqchip driver to handle it.=20 > But I am not sure how to address it in FreeBSD.=20 > Can you please point me out something here? We currently only support SPIs in the GICv3 driver under FDT. To teach = the ACPI attachment to support them you=E2=80=99ll need to set = sc->gic_mbi_start and sc->gic_mbi_end in sys/arm64/arm64/gic_v3_acpi.c = to the correct value from the ACPI tables before calling gic_v3_attach. = After this returns successfully you can then call intr_msi_register. See gic_v3_fdt.c for how we do it under FDT. Andrew= --Apple-Mail=_C72BF360-FA1E-44BC-ABBC-1D9F0B37BFF7 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8

On 11 Nov 2022, at 19:07, Souradeep Chakrabarti <schakrabarti@microsoft.com> wrote:




-----Original Message-----
From: Souradeep Chakrabarti
Sent: Thursday, = November 10, 2022 3:20 PM
To: Andrew Turner <andrew@fubar.geek.nz>
Cc: Warner Losh = <imp@bsdimp.com>; = Wei Hu <weh@microsoft.com>; freebsd-
hackers@FreeBSD.org
Subject: RE: [EXTERNAL] pcib msix allocation in arm64




-----Original = Message-----
From: Andrew Turner <andrew@fubar.geek.nz>
Sent: Thursday, = November 3, 2022 6:21 PM
To: Souradeep Chakrabarti <schakrabarti@microsoft.com>
Cc: Warner = Losh <imp@bsdimp.com>; Wei Hu <weh@microsoft.com>;
freebsd-
hackers@FreeBSD.org
Subject: Re: [EXTERNAL] = pcib msix allocation in arm64

Hi = Souradeep,

For the vmbus_pcib driver = you=E2=80=99ll need to implement the pcib_alloc_msi,
pcib_release_msi, and the msix versions, and pcib_map_msi.

For alloc and release you can just call into = the same intr_* function
as pci_host_generic_acpi.c. = You=E2=80=99ll need to pass in the xref for the
interrupt = controller. It needs to be the xref the MSI controller
registered. For the GICv2m it=E2=80=99s ACPI_MSI_XREF.

[Souradeep]
I have = used following in vmbus_pcib.c :
ret =3D = intr_alloc_msix(pcib, dev, ACPI_MSI_XREF, irq); But it is failing in
pic_lookup() with ESRCH by not finding pic with = ACPI_MSI_XREF.
Do I need to do anything for pic_lookup() = to be successful?

[Souradeep] 
I found out the reason here = pic_lookup() for ACPI_MSI_XREF is failing, as
we are not calling = intr_msi_register() but intr_pic_register() is getting called for the = PCIB.
This is = because we don't have ITS implemented in Hyper-V. Instead, Hyper-V is = presenting the
MSI/MSI-X as SPI because of it's own limitation.
Linux has solved it using custom = irqchip driver to handle it. 
But I am not sure how to address = it in FreeBSD. 
Can you please point me out = something here?

We currently only support SPIs in the GICv3 driver = under FDT. To teach the ACPI attachment to support them you=E2=80=99ll = need to set sc->gic_mbi_start and sc->gic_mbi_end in = sys/arm64/arm64/gic_v3_acpi.c to the correct value from the ACPI tables = before calling gic_v3_attach. After this returns successfully you = can then call intr_msi_register.

See gic_v3_fdt.c for how we do it under = FDT.

Andrew
= --Apple-Mail=_C72BF360-FA1E-44BC-ABBC-1D9F0B37BFF7--