From nobody Wed Sep 27 14:11:44 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RwdpR27srz4vWWr for ; Wed, 27 Sep 2023 14:11:59 +0000 (UTC) (envelope-from john@sanren.ac.za) Received: from mail-pj1-x1030.google.com (mail-pj1-x1030.google.com [IPv6:2607:f8b0:4864:20::1030]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RwdpQ3f6Tz4fD2 for ; Wed, 27 Sep 2023 14:11:58 +0000 (UTC) (envelope-from john@sanren.ac.za) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=sanren-ac-za.20230601.gappssmtp.com header.s=20230601 header.b=MEU+PdZO; spf=pass (mx1.freebsd.org: domain of john@sanren.ac.za designates 2607:f8b0:4864:20::1030 as permitted sender) smtp.mailfrom=john@sanren.ac.za; dmarc=none Received: by mail-pj1-x1030.google.com with SMTP id 98e67ed59e1d1-2773af0c5dbso5454617a91.1 for ; Wed, 27 Sep 2023 07:11:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sanren-ac-za.20230601.gappssmtp.com; s=20230601; t=1695823915; x=1696428715; darn=freebsd.org; h=to:subject:message-id:date:from:mime-version:from:to:cc:subject :date:message-id:reply-to; bh=IgMmIyfOOD4ZY2NY+oVtmiR6zy4ZBIOjE9GLrrn1F5w=; b=MEU+PdZOG9SC6HuJN/XX3Kv410yEfmlkXW8/dAvzgudH78AYDseBosEPIGFSdoxdnG LNRiIdIbP/VhWaeggJxUSSGPxteASaTP07b+IGxPt8TTqustU8MaI/lwj/iah3LmLV4X BDRSIN1NCYZSocpUj/OO6w4vFOGMOMd2m1hRjjxPx4tjTjh64ZFxtAKMA/IPbMoxjIJM mkNDZyAnhWR056HHjRNPBgmXBE9QqXBkb/vFldM9cfKIRAmC1g1dCp6pUuf0ivUf9W/6 JZBkiDgdBYYHE0RO7advMSCpNtYSwnFS0pqYUcQJK+yeJTniW7YwgZJ+Rg0yAoYCXzcG 5HXw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1695823915; x=1696428715; h=to:subject:message-id:date:from:mime-version:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=IgMmIyfOOD4ZY2NY+oVtmiR6zy4ZBIOjE9GLrrn1F5w=; b=CDhTPe7QqHxR/5jWuz4h0RyHuhjnDbsJkjsa2IDBCw8plcVoQUVzzxQlxRFM24gCGf +42wWYjheW0eKGjuNmVesIixVmfJ/SqKOT095PAoxptFUXeQiynADlrbO+z54sehujAm ZCWYE+rlSV05rymuxw7tZDwoY9lETKzKTSARhriUftW3T+3j9VaOfVI1zcb7lkCOFQms cL2ViJ1O0WwPyqnOoHhqH92DkIlpGz/X77JHhWMRvnm8SH3RepgWdvUErG71hdNZdVNj LGo753h5VFsVLezxSvPysx4bbM9Xheqw4LdhauJfyTajXKqhdI0KWWUsxLsRm+GbrKeT 3UMw== X-Gm-Message-State: AOJu0YysDxAx//j7d4d9EQGUDRjoePxo8N6wFvS91YHvHi88d04InSCc 7DY39bk9ZJi+LcZa6RNrVGilb5qBSlTfqsTimIi51ZAGmi2m3fCBLYk= X-Google-Smtp-Source: AGHT+IGHjF2rMco0sHhS2vfQooa/BB914tCg01TcxqOhWAUKQCbfIbko+NJxM77lRRYgsRLVKAlpgCk574r+V3PIDoc= X-Received: by 2002:a17:90a:4f48:b0:268:93d:b936 with SMTP id w8-20020a17090a4f4800b00268093db936mr1605868pjl.18.1695823915308; Wed, 27 Sep 2023 07:11:55 -0700 (PDT) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 From: John Hay Date: Wed, 27 Sep 2023 16:11:44 +0200 Message-ID: Subject: make msi interrupts available to child devices in a pci card driver To: freebsd-hackers@freebsd.org Content-Type: multipart/alternative; boundary="00000000000012f29f060657c634" X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.48 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-0.98)[-0.976]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; R_DKIM_ALLOW(-0.20)[sanren-ac-za.20230601.gappssmtp.com:s=20230601]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::1030:from]; FROM_EQ_ENVFROM(0.00)[]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; RCVD_COUNT_ONE(0.00)[1]; MIME_TRACE(0.00)[0:+,1:+,2:~]; DKIM_TRACE(0.00)[sanren-ac-za.20230601.gappssmtp.com:+]; ARC_NA(0.00)[]; RCVD_TLS_LAST(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; DMARC_NA(0.00)[sanren.ac.za]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; TO_DN_NONE(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Queue-Id: 4RwdpQ3f6Tz4fD2 --00000000000012f29f060657c634 Content-Type: text/plain; charset="UTF-8" Hi, I would like to know how to make msi interrupts available to child devices in a pci card driver? I'm writing a driver for the OCP TimeCard. Apart from the timers, it also has 16650 UARTs and IIC devices. All of them are memory mapped in a single BAR. It also has 32 MSI interrupts, with each "function/device" connected to a specific MSI interrupt. I would like to add a child device for each of these "functions" and let each of them have their own interrupt, but I'm struggling to figure out the correct way to make the interrupts available to the child device. The memory mapping, using rman and creating subregions for the different devices is working and I can get a UART device connected in polled mode. The basic process in the card's attach() is much like what the puc(4) driver does (leaving error checking etc. out): bar->b_rid = PCIR_BAR(0); bar->b_type = SYS_RES_MEMORY; bar->b_res = bus_alloc_resource_any(sc->sc_dev, bar->b_type, &bar->b_rid, RF_ACTIVE); sc->sc_iomem.rm_type = RMAN_ARRAY; error = rman_init(&sc->sc_iomem); error = rman_manage_region(&sc->sc_iomem, start, end); for (idx = 0; idx < nports; idx++) { port->p_rres = rman_reserve_resource(&sc->sc_iomem, start, end, size, 0, NULL); bsh = rman_get_bushandle(bar->b_res); bst = rman_get_bustag(bar->b_res); port->p_rres = bus_space_subregion(bst, bsh, ofs, size, &bsh); rman_set_bushandle(port->p_rres, bsh); rman_set_bustag(port->p_rres, bst); port->p_dev = device_add_child(dev, NULL, -1); error = device_probe_and_attach(port->p_dev); I can also allocate MSI interrupts for the driver itself as a test, with no problem: sc->sc_msi =32; pci_alloc_msi(dev, &sc->sc_msi); printf("msi irqs %d\n", sc->sc_msi); for (idx = 0; idx < sc->sc_msi; idx++) { iport->pp_irid = idx + 1; iport->pp_ires = bus_alloc_resource_any(dev, SYS_RES_IRQ, &iport->pp_irid, RF_ACTIVE); bus_setup_intr(dev, iport->pp_ires, INTR_TYPE_TTY | INTR_MPSAFE, timecard_intr, NULL, iport, &iport->p_ihandle); error = pci_enable_busmaster(dev); But I have not been able to find a way, similar to the memory to make it available to a child device, that works. It might be because the child is not a direct child of the pci driver. One hackish way that I could get to work was if I used my dev instead of the child's inside the timecard_bus_alloc_resource() and timecard_bus_setup_intr() methods, something like this: static struct resource * timecard_bus_release_resource(device_t dev, device_t child, int type, int rid, struct resource *res) { if (type == SYS_RES_IRQ) return BUS_RELEASE_RESOURCE(device_get_parent(dev), dev, type, rid, res); } static int timecard_bus_setup_intr(device_t dev, device_t child, struct resource *res, int flags, driver_filter_t *filt, void (*ihand)(void *), void *arg, void **cookiep) { return bus_setup_intr(dev, res, flags, filt, ihand, arg, cookiep); } That works, but does not feel quite right and then the interrupt is not "owned" by the child, but by the timecard driver: # vmstat -ia | grep timecard irq41: timecard0 10408538 175 Regards John --00000000000012f29f060657c634 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi,

I would like to know how= to make msi interrupts available to child devices in a pci card driver?

I'm writing a driver for the OCP TimeCard. Apart= from the timers, it also has 16650 UARTs and IIC devices. All of them are = memory mapped in a single BAR. It also has 32 MSI interrupts, with each &qu= ot;function/device" connected to a specific MSI interrupt.
<= br>
I would like to add a child device for each of these "fu= nctions" and let each of them have their own interrupt, but I'm st= ruggling to figure out the correct way to make the interrupts available to = the child device.

The memory mapping, using r= man and creating subregions for the different devices is working and I can = get a UART device connected in polled mode. The basic process in the card&#= 39;s attach() is much like what the puc(4) driver does (leaving error check= ing etc. out):

<snip>
=C2=A0 = =C2=A0 =C2=A0 =C2=A0 bar->b_rid =3D PCIR_BAR(0);
=C2=A0 =C2=A0 =C2=A0= =C2=A0 bar->b_type =3D SYS_RES_MEMORY;
=C2=A0 =C2=A0 =C2=A0 =C2=A0 b= ar->b_res =3D bus_alloc_resource_any(sc->sc_dev, bar->b_type, &= ;bar->b_rid, RF_ACTIVE);
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 sc->sc_iomem.rm_type =3D RMAN_ARRAY;
=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 error =3D rman_init(&sc->sc_iomem);
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 error =3D rman_manage_region(= &sc->sc_iomem, start, end);
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 for (idx =3D 0; idx < nports; idx++) {
=C2=A0 =C2= =A0 =C2=A0 =C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 port->p_rre= s =3D rman_reserve_resource(&sc->sc_iomem, start, end, size, 0, NULL= );
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 bsh =3D rman_get_bushandle(bar->b_res);
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 bst =3D rman_get_bustag(bar->b_res);
= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 port->p_rres =3D bus_space_subregion(bst, bsh, ofs, size= , &bsh);
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 rman_set_bushandle(port->p_rres,= bsh);
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 rman_set_bustag(port->p_rres, bst);
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 port->p_dev =3D device_add_child(dev, NULL, -1)= ;
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 error =3D device_probe_and_attach(p= ort->p_dev);
</snip>

I can also= allocate MSI interrupts for the driver itself as a test, with no problem:<= /div>

<snip>
=C2=A0 =C2=A0 =C2=A0 =C2=A0= sc->sc_msi =3D32;
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = pci_alloc_msi(dev, &sc->sc_msi);
=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0 printf("msi irqs %d\n", sc->sc_msi);
<= div>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 for (idx =3D 0; idx < sc-= >sc_msi; idx++) {
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 iport->pp_irid =3D idx + 1;
=C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 iport->pp_ires =3D bus_alloc_resource_any(d= ev, SYS_RES_IRQ, &iport->pp_irid, RF_ACTIVE);
=C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 bus_setup_intr(dev, iport->= ;pp_ires, INTR_TYPE_TTY | INTR_MPSAFE, timecard_intr, NULL, iport, &ipo= rt->p_ihandle);
=C2=A0 =C2=A0 =C2=A0 =C2=A0 error =3D pci_enab= le_busmaster(dev);
</snip>

But I h= ave not been able to find a way, similar to the memory to make it available= to a child device, that works. It might be because the child is not a dire= ct child of the pci driver.

One hackish way that I= could get to work was if I used my dev instead of the child's inside t= he timecard_bus_alloc_resource() and timecard_bus_setup_intr() methods, som= ething like this:

<snip>
static st= ruct resource *
timecard_bus_release_resource(device_t dev, devic= e_t child, int type, int rid, struct resource *res)
{
i= f (type =3D=3D SYS_RES_IRQ)
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0 return BUS_RELEASE_RESOURCE(device_get_parent(dev), dev, type, ri= d, res);
}
static int
timecard_bus_setup_intr= (device_t dev, device_t child, struct resource *res,
=C2=A0 =C2= =A0 int flags, driver_filter_t *filt, void (*ihand)(void *), void *arg, voi= d **cookiep)
{
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 return bus_setup= _intr(dev, res, flags, filt, ihand, arg, cookiep);
}
&l= t;/snip>

That works, but does not feel quite ri= ght and then the interrupt is not "owned" by the child, but by th= e timecard driver:
<snip>
# vmstat -ia | grep= timecard
irq41: timecard0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A010408538 =C2=A0 =C2=A0 =C2=A0 =C2=A0175
</snip>

Regards

John
--00000000000012f29f060657c634--