From nobody Mon Feb 28 06:55:51 2022 X-Original-To: dev-commits-src-branches@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id ADE1719D3217; Mon, 28 Feb 2022 06:55:53 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4K6WP40lpMz4Rlc; Mon, 28 Feb 2022 06:55:51 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1646031353; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=3Vpva+gD/D2A0j62B7nResU/pfAkJMdte5svCdfTXT0=; b=Jl/iACbyamRfFzkLE/i0ZGAkf/n1xhyNKAECSKN3Ra8IJp2ZLRINbo0f9+VDSEx76IfEhH wXMynlJdeHLwV3ne/U5B0LKv0klVV2Vody+Rz99zaSNwHDGov84wiKgmqMdGEBQwQyY5Ai pQJKDRu9fgun7zh0O0BjyTqBwDW78K/Qu+deY63NEAwLkbCzn2ldrSFetxfJ9nnoLcTIrl jYyIlw9mHBP9GNkKRCz+1dZMSp81y9PzdKWd8y4cxheClzCLNmZkEvtkX/NVnf9lE3CNp8 iLVtVVWgU0OwzH4PceIL4Nh6/rAv14KsPH/9A9Fbg+g9fSTbRw0YuSpDwc5hwA== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 8B5BF16EDC; Mon, 28 Feb 2022 06:55:51 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.16.1/8.16.1) with ESMTP id 21S6tp46077965; Mon, 28 Feb 2022 06:55:51 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.16.1/8.16.1/Submit) id 21S6tpEl077964; Mon, 28 Feb 2022 06:55:51 GMT (envelope-from git) Date: Mon, 28 Feb 2022 06:55:51 GMT Message-Id: <202202280655.21S6tpEl077964@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-branches@FreeBSD.org From: Navdeep Parhar Subject: git: e005d417a282 - stable/13 - cxgbe(4): Fix bad races between sysctl and driver detach. List-Id: Commits to the stable branches of the FreeBSD src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-branches List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-branches@freebsd.org X-BeenThere: dev-commits-src-branches@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: np X-Git-Repository: src X-Git-Refname: refs/heads/stable/13 X-Git-Reftype: branch X-Git-Commit: e005d417a2822d4829cb133a8760a586025c4473 Auto-Submitted: auto-generated ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1646031353; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=3Vpva+gD/D2A0j62B7nResU/pfAkJMdte5svCdfTXT0=; b=aKG3jWKKks8RBOnOAFXg3797bH/zUC3FCbfOmBYUydOGWJvLAJHipzB1tIF97QrkF2dhj+ A0HxtSIqDHhmE/pRa9lGME4fscMsuQ9QaZwb3SO04xKKx3wXlKYyze44P7PAGWPBEX7HJd b5RldjA+OCk69ENH2oB3ESy+YaklRYk4gkT710e5eG5IqTnzj60KnfYESLLew4BhuHTWKs ZS/x3vdn8snRqY5iW2yQQTN45JK7+eUHFkLBTej82gN13WqZbZ8t1cgXgcxh8TkxkV3/0J PbLWRD0opSL8oVrMXLN/MhYR+TRcZ5ZsaFrwx6grVaR8wg0L0O+vvcIQNoqAww== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1646031353; a=rsa-sha256; cv=none; b=gJr+9Z89aIi4SdVj5nK/rGv0wtUMQopM/qbj2gBEvu7ojDi18oOQR3oI1y8okSg19xrAv5 CiO/FxwSmOvXxbkJ77p0T+/pCQXFLpQFp3b5CWowXyxKW388CActA8VqJ3Iqy0u5VqX+lp 5XG/EI5EzEcWW7MWq1qkjc4hZFFndHOd0b8MBRmZRLZQclIMkZ1Ukya9SdJhrZciHDV9ue S3ITiKIIOD2AdR9J0vH4AW5AxqOXmAl6RvWJ2wmShrG1diZL0JrxB5sG7PkSVa5Lu61vnM nOVLHhsdRXQCz3K8AlRMkJQi00IKBeGMsBeyiZYc1P91lW8S97R3T8nCUMcTBA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N The branch stable/13 has been updated by np: URL: https://cgit.FreeBSD.org/src/commit/?id=e005d417a2822d4829cb133a8760a586025c4473 commit e005d417a2822d4829cb133a8760a586025c4473 Author: Navdeep Parhar AuthorDate: 2022-01-13 22:21:49 +0000 Commit: Navdeep Parhar CommitDate: 2022-02-28 06:51:04 +0000 cxgbe(4): Fix bad races between sysctl and driver detach. The default sysctl context setup by newbus for a device is eventually freed by device_sysctl_fini, which runs after the device driver's detach routine. sysctl nodes associated with this context must not use any resources (like driver locks, hardware access, counters, etc.) that are released by driver detach. There are a lot of sysctl nodes like this in cxgbe(4) and the fix is to hang them off a context that is explicitly freed by the driver before it releases any resource that might be used by a sysctl. This fixes panics when running "sysctl dev.t6nex dev.cc" in a tight loop and loading/unloading the driver in parallel. Reported by: Suhas Lokesha Sponsored by: Chelsio Communications (cherry picked from commit a727d9531afbd58e304acc27e3717031e78bff90) --- sys/dev/cxgbe/adapter.h | 6 +++-- sys/dev/cxgbe/crypto/t4_crypto.c | 8 ++++--- sys/dev/cxgbe/t4_main.c | 48 +++++++++++----------------------------- sys/dev/cxgbe/t4_vf.c | 1 + 4 files changed, 23 insertions(+), 40 deletions(-) diff --git a/sys/dev/cxgbe/adapter.h b/sys/dev/cxgbe/adapter.h index 2626150858e0..1c2e52802060 100644 --- a/sys/dev/cxgbe/adapter.h +++ b/sys/dev/cxgbe/adapter.h @@ -159,7 +159,7 @@ enum { FW_OK = (1 << 1), CHK_MBOX_ACCESS = (1 << 2), MASTER_PF = (1 << 3), - ADAP_SYSCTL_CTX = (1 << 4), + /* 1 << 4 is unused, was ADAP_SYSCTL_CTX */ ADAP_ERR = (1 << 5), BUF_PACKING_OK = (1 << 6), IS_VF = (1 << 7), @@ -174,7 +174,7 @@ enum { /* VI flags */ DOOMED = (1 << 0), VI_INIT_DONE = (1 << 1), - VI_SYSCTL_CTX = (1 << 2), + /* 1 << 2 is unused, was VI_SYSCTL_CTX */ TX_USES_VM_WR = (1 << 3), VI_SKIP_STATS = (1 << 4), @@ -332,6 +332,8 @@ struct port_info { u_int tx_parse_error; int fcs_reg; uint64_t fcs_base; + + struct sysctl_ctx_list ctx; }; #define IS_MAIN_VI(vi) ((vi) == &((vi)->pi->vi[0])) diff --git a/sys/dev/cxgbe/crypto/t4_crypto.c b/sys/dev/cxgbe/crypto/t4_crypto.c index 885271d82de0..7c233897dff6 100644 --- a/sys/dev/cxgbe/crypto/t4_crypto.c +++ b/sys/dev/cxgbe/crypto/t4_crypto.c @@ -242,6 +242,8 @@ struct ccr_softc { counter_u64_t stats_sglist_error; counter_u64_t stats_process_error; counter_u64_t stats_sw_fallback; + + struct sysctl_ctx_list ctx; }; /* @@ -2118,14 +2120,12 @@ ccr_probe(device_t dev) static void ccr_sysctls(struct ccr_softc *sc) { - struct sysctl_ctx_list *ctx; + struct sysctl_ctx_list *ctx = &sc->ctx; struct sysctl_oid *oid, *port_oid; struct sysctl_oid_list *children; char buf[16]; int i; - ctx = device_get_sysctl_ctx(sc->dev); - /* * dev.ccr.X. */ @@ -2251,6 +2251,7 @@ ccr_attach(device_t dev) sc = device_get_softc(dev); sc->dev = dev; + sysctl_ctx_init(&sc->ctx); sc->adapter = device_get_softc(device_get_parent(dev)); for_each_port(sc->adapter, i) { ccr_init_port(sc, i); @@ -2317,6 +2318,7 @@ ccr_detach(device_t dev) crypto_unregister_all(sc->cid); + sysctl_ctx_free(&sc->ctx); mtx_destroy(&sc->lock); counter_u64_free(sc->stats_blkcipher_encrypt); counter_u64_free(sc->stats_blkcipher_decrypt); diff --git a/sys/dev/cxgbe/t4_main.c b/sys/dev/cxgbe/t4_main.c index 078515833045..391bc9de8bfc 100644 --- a/sys/dev/cxgbe/t4_main.c +++ b/sys/dev/cxgbe/t4_main.c @@ -1127,6 +1127,7 @@ t4_attach(device_t dev) sc = device_get_softc(dev); sc->dev = dev; + sysctl_ctx_init(&sc->ctx); TUNABLE_INT_FETCH("hw.cxgbe.dflags", &sc->debug_flags); if ((pci_get_device(dev) & 0xff00) == 0x5400) @@ -1180,10 +1181,10 @@ t4_attach(device_t dev) TASK_INIT(&sc->reset_task, 0, reset_adapter, sc); - sc->ctrlq_oid = SYSCTL_ADD_NODE(device_get_sysctl_ctx(sc->dev), + sc->ctrlq_oid = SYSCTL_ADD_NODE(&sc->ctx, SYSCTL_CHILDREN(device_get_sysctl_tree(sc->dev)), OID_AUTO, "ctrlq", CTLFLAG_RD | CTLFLAG_MPSAFE, NULL, "control queues"); - sc->fwq_oid = SYSCTL_ADD_NODE(device_get_sysctl_ctx(sc->dev), + sc->fwq_oid = SYSCTL_ADD_NODE(&sc->ctx, SYSCTL_CHILDREN(device_get_sysctl_tree(sc->dev)), OID_AUTO, "fwq", CTLFLAG_RD | CTLFLAG_MPSAFE, NULL, "firmware event queue"); @@ -1741,6 +1742,7 @@ t4_detach_common(device_t dev) } device_delete_children(dev); + sysctl_ctx_free(&sc->ctx); adapter_full_uninit(sc); if ((sc->flags & (IS_VF | FW_OK)) == FW_OK) @@ -2422,12 +2424,12 @@ cxgbe_vi_attach(device_t dev, struct vi_info *vi) { struct ifnet *ifp; struct sbuf *sb; - struct sysctl_ctx_list *ctx; + struct sysctl_ctx_list *ctx = &vi->ctx; struct sysctl_oid_list *children; struct pfil_head_args pa; struct adapter *sc = vi->adapter; - ctx = device_get_sysctl_ctx(vi->dev); + sysctl_ctx_init(ctx); children = SYSCTL_CHILDREN(device_get_sysctl_tree(vi->dev)); vi->rxq_oid = SYSCTL_ADD_NODE(ctx, children, OID_AUTO, "rxq", CTLFLAG_RD | CTLFLAG_MPSAFE, NULL, "NIC rx queues"); @@ -2577,6 +2579,8 @@ cxgbe_attach(device_t dev) struct vi_info *vi; int i, rc; + sysctl_ctx_init(&pi->ctx); + rc = cxgbe_vi_attach(dev, &pi->vi[0]); if (rc) return (rc); @@ -2618,6 +2622,7 @@ cxgbe_vi_detach(struct vi_info *vi) #endif cxgbe_uninit_synchronized(vi); callout_drain(&vi->tick); + sysctl_ctx_free(&vi->ctx); vi_full_uninit(vi); if_free(vi->ifp); @@ -2637,6 +2642,7 @@ cxgbe_detach(device_t dev) return (rc); device_delete_children(dev); + sysctl_ctx_free(&pi->ctx); doom_vi(sc, &pi->vi[0]); if (pi->flags & HAS_TRACEQ) { @@ -6531,11 +6537,6 @@ adapter_full_init(struct adapter *sc) ASSERT_SYNCHRONIZED_OP(sc); - if (!(sc->flags & ADAP_SYSCTL_CTX)) { - sysctl_ctx_init(&sc->ctx); - sc->flags |= ADAP_SYSCTL_CTX; - } - /* * queues that belong to the adapter (not any particular port). */ @@ -6590,12 +6591,6 @@ adapter_full_uninit(struct adapter *sc) { int i; - /* Do this before freeing the adapter queues. */ - if (sc->flags & ADAP_SYSCTL_CTX) { - sysctl_ctx_free(&sc->ctx); - sc->flags &= ~ADAP_SYSCTL_CTX; - } - t4_teardown_adapter_queues(sc); for (i = 0; i < nitems(sc->tq) && sc->tq[i]; i++) { @@ -6688,11 +6683,6 @@ vi_full_init(struct vi_info *vi) ASSERT_SYNCHRONIZED_OP(sc); - if (!(vi->flags & VI_SYSCTL_CTX)) { - sysctl_ctx_init(&vi->ctx); - vi->flags |= VI_SYSCTL_CTX; - } - /* * Allocate tx/rx/fl queues for this VI. */ @@ -6826,12 +6816,6 @@ vi_full_uninit(struct vi_info *vi) free(vi->nm_rss, M_CXGBE); } - /* Do this before freeing the VI queues. */ - if (vi->flags & VI_SYSCTL_CTX) { - sysctl_ctx_free(&vi->ctx); - vi->flags &= ~VI_SYSCTL_CTX; - } - t4_teardown_vi_queues(vi); vi->flags &= ~VI_INIT_DONE; } @@ -7194,13 +7178,11 @@ static char *caps_decoder[] = { void t4_sysctls(struct adapter *sc) { - struct sysctl_ctx_list *ctx; + struct sysctl_ctx_list *ctx = &sc->ctx; struct sysctl_oid *oid; struct sysctl_oid_list *children, *c0; static char *doorbells = {"\20\1UDB\2WCWR\3UDBWC\4KDB"}; - ctx = device_get_sysctl_ctx(sc->dev); - /* * dev.t4nex.X. */ @@ -7705,12 +7687,10 @@ t4_sysctls(struct adapter *sc) void vi_sysctls(struct vi_info *vi) { - struct sysctl_ctx_list *ctx; + struct sysctl_ctx_list *ctx = &vi->ctx; struct sysctl_oid *oid; struct sysctl_oid_list *children; - ctx = device_get_sysctl_ctx(vi->dev); - /* * dev.v?(cxgbe|cxl).X. */ @@ -7810,7 +7790,7 @@ vi_sysctls(struct vi_info *vi) static void cxgbe_sysctls(struct port_info *pi) { - struct sysctl_ctx_list *ctx; + struct sysctl_ctx_list *ctx = &pi->ctx; struct sysctl_oid *oid; struct sysctl_oid_list *children, *children2; struct adapter *sc = pi->adapter; @@ -7818,8 +7798,6 @@ cxgbe_sysctls(struct port_info *pi) char name[16]; static char *tc_flags = {"\20\1USER"}; - ctx = device_get_sysctl_ctx(pi->dev); - /* * dev.cxgbe.X. */ diff --git a/sys/dev/cxgbe/t4_vf.c b/sys/dev/cxgbe/t4_vf.c index 4ad5e9d7839d..95b984f3e3be 100644 --- a/sys/dev/cxgbe/t4_vf.c +++ b/sys/dev/cxgbe/t4_vf.c @@ -489,6 +489,7 @@ t4vf_attach(device_t dev) sc = device_get_softc(dev); sc->dev = dev; + sysctl_ctx_init(&sc->ctx); pci_enable_busmaster(dev); pci_set_max_read_req(dev, 4096); sc->params.pci.mps = pci_get_max_payload(dev);