From nobody Mon Dec 05 15:13:56 2022 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4NQnBn1zSnz4jx0l for ; Mon, 5 Dec 2022 15:14:09 +0000 (UTC) (envelope-from bT.g127yoxc30=wsbnoru7raxx=48uxcsirsf@em790814.fubar.geek.nz) Received: from e2i580.smtp2go.com (e2i580.smtp2go.com [103.2.142.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4NQnBl0Kdbz3lJQ for ; Mon, 5 Dec 2022 15:14:06 +0000 (UTC) (envelope-from bT.g127yoxc30=wsbnoru7raxx=48uxcsirsf@em790814.fubar.geek.nz) Authentication-Results: mx1.freebsd.org; dkim=none ("invalid DKIM record") header.d=smtpservice.net header.s=mgy720.a1-4.dyn header.b=Hr3jXmt+; dkim=pass header.d=fubar.geek.nz header.s=s790814 header.b=fmOGsBwW; spf=pass (mx1.freebsd.org: domain of "bT.g127yoxc30=wsbnoru7raxx=48uxcsirsf@em790814.fubar.geek.nz" designates 103.2.142.68 as permitted sender) smtp.mailfrom="bT.g127yoxc30=wsbnoru7raxx=48uxcsirsf@em790814.fubar.geek.nz"; dmarc=pass (policy=none) header.from=fubar.geek.nz DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=smtpservice.net; s=mgy720.a1-4.dyn; x=1670254146; h=Feedback-ID: X-Smtpcorp-Track:To:Message-Id:Date:From:Subject:Reply-To:Sender: List-Unsubscribe; bh=iMn7UlAV99bCnI7RLFVmzrnZWK/9p2Mm5FfD3mUPq+U=; b=Hr3jXmt+ 76qcz2Hl5xqPpau38nlgpUbfWIyxFPmJOQ6dCGE82SiHMjmP55rPwjuXDNzXQDMrXZkEA3/xxutwU 7PYhyfJmu4OYU6bb7MQkyhc6F0Vtuq9YJLJrzqygPIh8/0hIqRn9J+uOlPnEkAohaHcJqQ904XOp0 ZPJ2Y2Awnyshn6JBAvA0A6CoWeMCtENzBXqu5e92srAftI6E8sOZXyjECvaRZv4C+dPB0rIQ1oZFF TXX+cKLDxYVtcxS0Hx7GATvwG7L6gRSoxIE+vaFYhlvQiCvzpjz7tH+FEacafKythUaCv+sTRuiMJ m87N+8rKmjY0O3A+5T5rk7jTBw==; DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fubar.geek.nz; i=@fubar.geek.nz; q=dns/txt; s=s790814; t=1670253246; h=from : subject : to : message-id : date; bh=iMn7UlAV99bCnI7RLFVmzrnZWK/9p2Mm5FfD3mUPq+U=; b=fmOGsBwW3fSQWlEisTyE0ELTmh5xumDT1F3+3u66CtpoqU3QWUGXWg6WIzfi1r7zzvpms FvvfjTAB3sHxWXkUDfeCV9SDhxv/oQJ939AHFNi41TfkURjQ/fbQ2VK1tgDTF5I2jJSQExz hm3HgczIjCP4YGT49M6stPev+XsRJItAA+PHinw6Y+li+HXvlhQLv7SGLIh9I3fQ9l1TR9e hH4GmgbblCQBixdiucnrbTL3L0mjPTAW11d37dew5kglh835uNxdpbQosfvWcZfN3+E+loe XL3L5tO2+O6ymdXqx/jckWHZwq6smnOez4ITp1gOknQaqFFuy1Bt1ICQOL9A== Received: from [10.176.58.103] (helo=SmtpCorp) by smtpcorp.com with esmtpsa (TLS1.3:ECDHE_SECP256R1__RSA_PSS_RSAE_SHA256__AES_256_GCM:256) (Exim 4.94.2-S2G) (envelope-from ) id 1p2DAP-qt4FoL-Lc; Mon, 05 Dec 2022 15:14:01 +0000 Received: from [10.162.55.164] (helo=morbo.fubar.geek.nz) by smtpcorp.com with esmtpsa (TLS1.3:ECDHE_X25519__RSA_PSS_RSAE_SHA256__AES_256_GCM:256) (Exim 4.96-S2G) (envelope-from ) id 1p2DAP-9ESiyE-1I; Mon, 05 Dec 2022 15:14:01 +0000 Received: from smtpclient.apple (cpc91214-cmbg18-2-0-cust234.5-4.cable.virginm.net [81.102.75.235]) by morbo.fubar.geek.nz (Postfix) with ESMTPSA id 3049C1C57F; Mon, 5 Dec 2022 15:13:57 +0000 (UTC) Content-Type: text/plain; charset=utf-8 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3696.120.41.1.1\)) Subject: Re: [EXTERNAL] pcib msix allocation in arm64 From: Andrew Turner In-Reply-To: Date: Mon, 5 Dec 2022 15:13:56 +0000 Cc: Warner Losh , Wei Hu , "freebsd-hackers@FreeBSD.org" Content-Transfer-Encoding: quoted-printable Message-Id: <877E8C1E-5CEA-44B6-ABC0-0C54D5B69A58@fubar.geek.nz> References: <146F5D18-6366-4953-A8D9-61FE7EC67F71@fubar.geek.nz> <947DBD67-6443-480F-82DA-2BDDF44C3D03@fubar.geek.nz> To: Souradeep Chakrabarti X-Mailer: Apple Mail (2.3696.120.41.1.1) X-Smtpcorp-Track: 1p2Dje9ESiyE1m.Mtq7ghInHyq7a Feedback-ID: 790814m:790814amQcrys:790814sNwJM4lss3 X-Report-Abuse: Please forward a copy of this message, including all headers, to X-Spamd-Result: default: False [-3.76 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.86)[-0.858]; MV_CASE(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[fubar.geek.nz,none]; RCVD_DKIM_ARC_DNSWL_MED(-0.50)[]; FORGED_SENDER(0.30)[andrew@fubar.geek.nz,bT.g127yoxc30=wsbnoru7raxx=48uxcsirsf@em790814.fubar.geek.nz]; RCVD_IN_DNSWL_MED(-0.20)[103.2.142.68:from]; R_SPF_ALLOW(-0.20)[+ip4:103.2.140.0/22]; R_DKIM_ALLOW(-0.20)[fubar.geek.nz:s=s790814]; MIME_GOOD(-0.10)[text/plain]; RCPT_COUNT_THREE(0.00)[4]; FROM_HAS_DN(0.00)[]; TO_DN_EQ_ADDR_SOME(0.00)[]; R_DKIM_PERMFAIL(0.00)[smtpservice.net:s=mgy720.a1-4.dyn]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; ARC_NA(0.00)[]; ASN(0.00)[asn:23352, ipnet:103.2.140.0/22, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DKIM_MIXED(0.00)[]; TO_DN_SOME(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_NEQ_ENVFROM(0.00)[andrew@fubar.geek.nz,bT.g127yoxc30=wsbnoru7raxx=48uxcsirsf@em790814.fubar.geek.nz]; RCVD_COUNT_THREE(0.00)[4]; MIME_TRACE(0.00)[0:+]; DKIM_TRACE(0.00)[smtpservice.net:~,fubar.geek.nz:+]; RWL_MAILSPIKE_POSSIBLE(0.00)[103.2.142.68:from]; RCVD_TLS_ALL(0.00)[] X-Rspamd-Queue-Id: 4NQnBl0Kdbz3lJQ X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N > On 1 Dec 2022, at 11:54, Souradeep Chakrabarti = wrote: >=20 >=20 >=20 >=20 >> -----Original Message----- >> From: Andrew Turner >> Sent: Thursday, November 3, 2022 6:21 PM >> To: Souradeep Chakrabarti >> Cc: Warner Losh ; Wei Hu ; = freebsd- >> hackers@FreeBSD.org >> Subject: Re: [EXTERNAL] pcib msix allocation in arm64 >>=20 >> Hi Souradeep, >>=20 >> For the vmbus_pcib driver you=E2=80=99ll need to implement the = pcib_alloc_msi, >> pcib_release_msi, and the msix versions, and pcib_map_msi. >>=20 >> For alloc and release you can just call into the same intr_* function = as >> pci_host_generic_acpi.c. You=E2=80=99ll need to pass in the xref for = the interrupt controller. >> It needs to be the xref the MSI controller registered. For the GICv2m = it=E2=80=99s >> ACPI_MSI_XREF. > [Souradeep]=20 > I am currently specifying gic_mbi_start and gic_mbi_end with SPI start = and end number, in gic_v3_acpi_attach() and=20 > I can see my intr_alloc_msi() and intr_map_msi() are working. But then = mlx driver is > getting stuck at CREATE_EQ .=20 >=20 > mlx5_core0: WARN: wait_func:967:(pid 0): CREATE_EQ(0x301) timeout. = Will cause a leak of a command resource > mlx5_core0: WARN: mlx5_start_eqs:588:(pid 0): failed to create async = EQ -60 > mlx5_core0: ERR: mlx5_load_one:1157:(pid 0): Failed to start pages and = async EQs > mlx5_core0: WARN: wait_func:967:(pid 0): MANAGE_PAGES(0x108) timeout. = Will cause a leak of a command resource > mlx5_core0: ERR: reclaim_pages:444:(pid 0): failed reclaiming pages > mlx5_core0: WARN: mlx5_reclaim_startup_pages:574:(pid 0): failed = reclaiming pages (-60) > mlx5_core0: ERR: init_one:1644:(pid 0): mlx5_load_one failed -60 >=20 >=20 > gic_v3_acpi_attach( ) > -- > -- > 323 sc->gic_mbi_start =3D 64; > 324 sc->gic_mbi_end =3D 920; > 325 > 326 err =3D gic_v3_attach(dev); > 327 if (err !=3D 0) > 328 goto error; > 329 > 330 sc->gic_pic =3D intr_pic_register(dev, ACPI_INTR_XREF); > 331 if (sc->gic_pic =3D=3D NULL) { > 332 device_printf(dev, "could not register PIC\n"); > 333 err =3D ENXIO; > 334 goto error; > 335 } > 336 err =3D intr_msi_register(dev, ACPI_MSI_XREF); > 337 if (err) { > 338 device_printf(dev, "could not register MSI\n"); > 339 } > 340 if (intr_pic_claim_root(dev, ACPI_INTR_XREF, arm_gic_v3_intr, = sc, > 341 GIC_LAST_SGI - GIC_FIRST_SGI + 1) !=3D 0) { > 342 err =3D ENXIO; > 343 goto error; > 344 } > We don't have SPI range mentioned in ACPI DSDT, so I need to specify = it manually in the code. > It looks like MSI interrupt is not coming to the mlx.=20 > Is there any interrupt remapping blocked by FreeBSD kernel here? Do you see the expected interrupt in arm_gic_v3_intr? If so it would be = useful to trace it though intr_isrc_dispatch. If no it may be because = the interrupt is not being triggered, or is masked. Andrew=