From nobody Thu Oct 29 06:03:01 2026 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4jFYYd5zn5z6xrbt for ; Thu, 29 Oct 2026 06:03:01 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 4jFYYd3x09z4sFY for ; Thu, 29 Oct 2026 06:03:01 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1793253781; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=Mp4YyhZuDTpP5/ft+G6sVv7UUQqH4TPWGsSFstBkDNc=; b=XgzqmrmKL4ojZD0WuIHfA9TjJ10rWOTJ+2B/kMOfDIE2OWVHunM53bevcDG/UpSiQniVYc rJIKP0pKkoC1+ZmzldFl9tyRUfXlPVbK2ReemBbws9Johw6/5BKIz6jVUrDsLaTGeOlV0b U18EhQiCqhnOchiTDJgN33xZyNKdCbQH0QRV2C80rnzJTXi6AX7AmAaDXBMjRD5+XicMrQ d1PzRw8WKA+sTYeTKmzwkqOKlzdcta2KOVU3LAAPx2CoQiZ9khPWG40NkrSDRRDGCP1jrb rinypmyrY3XYYLsClxTs5xal/YAysZng7/T7Ka2Tggw8lZLjHbZir+ZZRytvTg== ARC-Seal: i=1; a=rsa-sha256; d=freebsd.org; s=dkim; cv=none; t=1793253781; b=BMrl3WuiDwgwy4eN5pi5tfMFLLJu1omqhYSCRpEmePKgzMW9u+Qx8Ir4+0xHSpXYKWdftx TvU5ocpYYNz0lKROims1FW3MAo8TQPj8foxMMLOcgu0HrOfNa7GqGLDynPUN4mxx409wbS kXdv7ZyQmf5pT2IsujOTRNlIaotYAQmLKm0CLmYSBXmihS7XaN9MV4GxWe5CIFUdw8Er6s zmh2JKHGkKuovPehxVNHXTmQ3G2qZ/u2A+WPt5TWrIYcogOFmwv1KRBLaczSNnr2+iWDw4 aWQG3L+HsKF0L777eiVAy84hgsOUIJDWWkGxsSprrTUFeQkbfWNOg9zx9+iimQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1793253781; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=Mp4YyhZuDTpP5/ft+G6sVv7UUQqH4TPWGsSFstBkDNc=; b=T0nKrQ3ixkvnYD9kivXLrf0/ffd6E8qyKFaH0A60tCx/AttNVoZC7E66IVR+HUyWVXoHHX +mxX+k0fCDdy+pvIyrh5jkWu2HeCGmp6VWI+xu9KrPVB+6mieuxy6GWtCX8+gIqBLcpcLC 9T/if4SQ+GeHKNi4Q8n6wh0lAtUlvX29EsPTVWDXiUHK9yJ/qMEQzrrapdABFR5zzv9/rm a15ZxaOarrUrFWi0pJ7rTYGdgUetIORUMUygP8iYoYOkxYEfZhipl6tFmRAt6NqEp2Bkxp sIv/YLdYxSiCBmutJQaOJidrDiRSPuW3mdZkxhkgIYy/rRfaj3hBP4epUVqlTA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) by mxrelay.nyi.freebsd.org (Postfix) with ESMTP id 4jFYYd3bJdzR9R for ; Thu, 29 Oct 2026 06:03:01 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from git (uid 1279) (envelope-from git@FreeBSD.org) id 3acb9 by gitrepo.freebsd.org (DragonFly Mail Agent v0.13+ on gitrepo.freebsd.org); Thu, 29 Oct 2026 06:03:01 +0000 To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-branches@FreeBSD.org Cc: Andrew Gallatin From: Kevin Bowling Subject: git: fff029b5f129 - stable/15 - iflib: Drop tx lock when freeing mbufs using simple_transmit List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all X-BeenThere: dev-commits-src-all@freebsd.org Sender: owner-dev-commits-src-all@FreeBSD.org List-Id: Commit messages for all branches of the src repository List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: kbowling X-Git-Repository: src X-Git-Refname: refs/heads/stable/15 X-Git-Reftype: branch X-Git-Commit: fff029b5f12937fc9ec4f57c605d37caf7e58ad2 Auto-Submitted: auto-generated Date: Thu, 29 Oct 2026 06:03:01 +0000 Message-Id: <6ae2e195.3acb9.35f1a59e@gitrepo.freebsd.org> The branch stable/15 has been updated by kbowling: URL: https://cgit.FreeBSD.org/src/commit/?id=fff029b5f12937fc9ec4f57c605d37caf7e58ad2 commit fff029b5f12937fc9ec4f57c605d37caf7e58ad2 Author: Andrew Gallatin AuthorDate: 2026-01-07 19:29:53 +0000 Commit: Kevin Bowling CommitDate: 2026-10-29 05:28:16 +0000 iflib: Drop tx lock when freeing mbufs using simple_transmit Freeing completed transmit mbufs can be time consuming (due to them being cold in cache, and due to ext free routines taking locks), especially when we batch tx completions. If we do this when holding the tx ring mutex, this can cause lock contention on the tx ring mutex when using iflib_simple_transmit. To resolve this, this patch opportunistically copies completed mbuf pointers into a new array (ifsd_m_defer) so they can be freed after dropping the transmit mutex. The ifsd_m_defer array is opportunistically used, and may be NULL. If its NULL, then we free mbufs in the old way. The ifsd_m_defer array is atomically nulled when a thread is using it, and atomically restored when the freeing thread is done with it. The use of atomics here avoids acquire/release of the tx lock to restore the array after freeing mbufs. Since we're no longer always freeing mbufs inline, peeking into them to see if a transmit used TSO or not will cause a useless cache miss, as nothing else in the mbuf is likely to be accessed soon. To avoid that cache miss, we encode a TSO or not TSO flag in the lower bits of the mbuf pointer stored in the ifsd_m array. Note that the IFLIB_NO_TSO flag exists primarily for sanity/debugging. iflib_completed_tx_reclaim() was refactored to break out iflib_txq_can_reclaim() and _iflib_completed_tx_reclaim() so the that the tx routine can call iflib_tx_credits_update() just once, rather than twice. Note that deferred mbuf freeing is not enabled by default, and can be enabled using the dev.$DEV.$UNIT.iflib.tx_defer_mfree sysctl. Differential Revision: https://reviews.freebsd.org/D54356 Sponsored by: Netflix Reviewed by: markj, kbowling, ziaee (cherry picked from commit 14d93f612f26f4e8454e393b75b0e4be0fc9d890) --- share/man/man4/iflib.4 | 8 ++- sys/net/iflib.c | 182 +++++++++++++++++++++++++++++++++++++++---------- 2 files changed, 154 insertions(+), 36 deletions(-) diff --git a/share/man/man4/iflib.4 b/share/man/man4/iflib.4 index 721eaab3ee24..e34235aa9cfe 100644 --- a/share/man/man4/iflib.4 +++ b/share/man/man4/iflib.4 @@ -1,4 +1,4 @@ -.Dd August 20, 2025 +.Dd January 7, 2026 .Dt IFLIB 4 .Os .Sh NAME @@ -94,6 +94,12 @@ If set to a non-zero value, task returns immediately and the transmit ring is serviced by a different task. This returns control to the caller faster and under high receive load, may result in fewer dropped RX frames. +.It Va tx_defer_mfree +Controls the threshold in packets before iflib will free the memory +(mbufs) for the packets that it has transmitted. +When this is nonzero, mbufs will be freed outside the transmit lock. +Setting this can reduce lock contention and CPU use when using simple_tx. +Note that this applies only when simple_tx is enabled. .It Va tx_reclaim_thresh Controls the threshold in packets before iflib will ask the driver how many transmitted packets can be reclaimed. diff --git a/sys/net/iflib.c b/sys/net/iflib.c index 8f0b0a13bc6e..6be0c21edb7e 100644 --- a/sys/net/iflib.c +++ b/sys/net/iflib.c @@ -242,6 +242,7 @@ struct iflib_ctx { bool ifc_cpus_are_physical_cores; bool ifc_core_offset_ref; bool ifc_sysctl_simple_tx; + bool ifc_sysctl_tx_defer_mfree; uint16_t ifc_sysctl_tx_reclaim_thresh; uint16_t ifc_sysctl_tx_reclaim_ticks; @@ -338,6 +339,8 @@ typedef struct iflib_sw_tx_desc_array { bus_dmamap_t *ifsd_map; /* bus_dma maps for packet */ bus_dmamap_t *ifsd_tso_map; /* bus_dma maps for TSO packet */ struct mbuf **ifsd_m; /* pkthdr mbufs */ + struct mbuf **ifsd_m_defer; /* deferred mbuf ptr */ + struct mbuf **ifsd_m_deferb;/* deferred mbuf backing ptr */ } if_txsd_vec_t; /* magic number that should be high enough for any hardware */ @@ -358,6 +361,20 @@ typedef struct iflib_sw_tx_desc_array { #define IFLIB_RESTART_BUDGET 8 + +/* + * Encode TSO or !TSO in the low bits of the tx ifsd_m pointer so as + * to avoid defref'ing the mbuf to determine the correct busdma resources + * to release + */ +#define IFLIB_TSO (1ULL << 0) +#define IFLIB_NO_TSO (2ULL << 0) +#define IFLIB_FLAGS_MASK (0x3ULL) +#define IFLIB_SAVE_MBUF(mbuf, flags) ((void *)(((uintptr_t)mbuf) | flags)) +#define IFLIB_GET_FLAGS(a) ((uintptr_t)a & IFLIB_FLAGS_MASK) +#define IFLIB_GET_MBUF(a) ((struct mbuf *)((uintptr_t)a & ~IFLIB_FLAGS_MASK)) + + #define IFC_LEGACY 0x001 #define IFC_QFLUSH 0x002 #define IFC_MULTISEG 0x004 @@ -383,7 +400,9 @@ struct iflib_txq { qidx_t ift_cidx_processed; qidx_t ift_pidx; uint8_t ift_gen; - uint8_t ift_br_offset; + uint8_t ift_br_offset:1, + ift_defer_mfree:1, + ift_spare_bits0:6; uint16_t ift_npending; uint16_t ift_db_pending; uint16_t ift_rs_pending; @@ -748,7 +767,8 @@ static void iflib_free_intr_mem(if_ctx_t ctx); #ifndef __NO_STRICT_ALIGNMENT static struct mbuf *iflib_fixup_rx(struct mbuf *m); #endif -static __inline int iflib_completed_tx_reclaim(iflib_txq_t txq); +static __inline int iflib_completed_tx_reclaim(iflib_txq_t txq, + struct mbuf **m_defer); static SLIST_HEAD(cpu_offset_list, cpu_offset) cpu_offsets = SLIST_HEAD_INITIALIZER(cpu_offsets); @@ -1799,7 +1819,16 @@ iflib_txsd_alloc(iflib_txq_t txq) err = ENOMEM; goto fail; } - + if (ctx->ifc_sysctl_simple_tx) { + if (!(txq->ift_sds.ifsd_m_defer = + (struct mbuf **) malloc(sizeof(struct mbuf *) * + scctx->isc_ntxd[txq->ift_br_offset], M_IFLIB, M_NOWAIT | M_ZERO))) { + device_printf(dev, "Unable to allocate TX mbuf map memory\n"); + err = ENOMEM; + goto fail; + } + } + txq->ift_sds.ifsd_m_deferb = txq->ift_sds.ifsd_m_defer; /* * Create the DMA maps for TX buffers. */ @@ -1892,6 +1921,10 @@ iflib_txq_destroy(iflib_txq_t txq) free(txq->ift_sds.ifsd_m, M_IFLIB); txq->ift_sds.ifsd_m = NULL; } + if (txq->ift_sds.ifsd_m_defer != NULL) { + free(txq->ift_sds.ifsd_m_defer, M_IFLIB); + txq->ift_sds.ifsd_m_defer = NULL; + } if (txq->ift_buf_tag != NULL) { bus_dma_tag_destroy(txq->ift_buf_tag); txq->ift_buf_tag = NULL; @@ -1908,10 +1941,10 @@ iflib_txq_destroy(iflib_txq_t txq) static void iflib_txsd_free(if_ctx_t ctx, iflib_txq_t txq, int i) { - struct mbuf **mp; + struct mbuf *m; - mp = &txq->ift_sds.ifsd_m[i]; - if (*mp == NULL) + m = IFLIB_GET_MBUF(txq->ift_sds.ifsd_m[i]); + if (m == NULL) return; if (txq->ift_sds.ifsd_map != NULL) { @@ -1925,9 +1958,8 @@ iflib_txsd_free(if_ctx_t ctx, iflib_txq_t txq, int i) bus_dmamap_unload(txq->ift_tso_buf_tag, txq->ift_sds.ifsd_tso_map[i]); } - m_freem(*mp); + m_freem(m); DBG_COUNTER_INC(tx_frees); - *mp = NULL; } static int @@ -3527,7 +3559,7 @@ iflib_remove_mbuf(iflib_txq_t txq) ntxd = txq->ift_size; pidx = txq->ift_pidx & (ntxd - 1); ifsd_m = txq->ift_sds.ifsd_m; - m = ifsd_m[pidx]; + m = IFLIB_GET_MBUF(ifsd_m[pidx]); ifsd_m[pidx] = NULL; bus_dmamap_unload(txq->ift_buf_tag, txq->ift_sds.ifsd_map[pidx]); if (txq->ift_sds.ifsd_tso_map != NULL) @@ -3594,6 +3626,7 @@ iflib_encap(iflib_txq_t txq, struct mbuf **m_headp) struct mbuf *m_head, **ifsd_m; bus_dmamap_t map; struct if_pkt_info pi; + uintptr_t flags; int remap = 0; int err, nsegs, ndesc, max_segs, pidx; @@ -3617,10 +3650,12 @@ iflib_encap(iflib_txq_t txq, struct mbuf **m_headp) map = txq->ift_sds.ifsd_tso_map[pidx]; MPASS(buf_tag != NULL); MPASS(max_segs > 0); + flags = IFLIB_TSO; } else { buf_tag = txq->ift_buf_tag; max_segs = scctx->isc_tx_nsegments; map = txq->ift_sds.ifsd_map[pidx]; + flags = IFLIB_NO_TSO; } if ((sctx->isc_flags & IFLIB_NEED_ETHER_PAD) && __predict_false(m_head->m_pkthdr.len < scctx->isc_min_frame_size)) { @@ -3693,7 +3728,7 @@ defrag: DBG_COUNTER_INC(encap_txd_encap_fail); return (err); } - ifsd_m[pidx] = m_head; + ifsd_m[pidx] = IFLIB_SAVE_MBUF(m_head, flags); if (m_head->m_pkthdr.csum_flags & CSUM_SND_TAG) pi.ipi_mbuf = m_head; else @@ -3704,7 +3739,7 @@ defrag: * cxgb */ if (__predict_false(nsegs > TXQ_AVAIL(txq))) { - (void)iflib_completed_tx_reclaim(txq); + (void)iflib_completed_tx_reclaim(txq, NULL); if (__predict_false(nsegs > TXQ_AVAIL(txq))) { txq->ift_no_desc_avail++; bus_dmamap_unload(buf_tag, map); @@ -3794,19 +3829,22 @@ defrag_failed: } static void -iflib_tx_desc_free(iflib_txq_t txq, int n) +iflib_tx_desc_free(iflib_txq_t txq, int n, struct mbuf **m_defer) { uint32_t qsize, cidx, gen; struct mbuf *m, **ifsd_m; + uintptr_t flags; cidx = txq->ift_cidx; gen = txq->ift_gen; qsize = txq->ift_size; - ifsd_m = txq->ift_sds.ifsd_m; + ifsd_m =txq->ift_sds.ifsd_m; while (n-- > 0) { - if ((m = ifsd_m[cidx]) != NULL) { - if (m->m_pkthdr.csum_flags & CSUM_TSO) { + if ((m = IFLIB_GET_MBUF(ifsd_m[cidx])) != NULL) { + flags = IFLIB_GET_FLAGS(ifsd_m[cidx]); + MPASS(flags != 0); + if (flags & IFLIB_TSO) { bus_dmamap_sync(txq->ift_tso_buf_tag, txq->ift_sds.ifsd_tso_map[cidx], BUS_DMASYNC_POSTWRITE); @@ -3821,7 +3859,12 @@ iflib_tx_desc_free(iflib_txq_t txq, int n) } /* XXX we don't support any drivers that batch packets yet */ MPASS(m->m_nextpkt == NULL); - m_freem(m); + if (m_defer == NULL) { + m_freem(m); + } else if (m != NULL) { + *m_defer = m; + m_defer++; + } ifsd_m[cidx] = NULL; #if MEMORY_LOGGING txq->ift_dequeued++; @@ -3838,28 +3881,20 @@ iflib_tx_desc_free(iflib_txq_t txq, int n) } static __inline int -iflib_completed_tx_reclaim(iflib_txq_t txq) +iflib_txq_can_reclaim(iflib_txq_t txq) { int reclaim, thresh; - uint32_t now; - if_ctx_t ctx = txq->ift_ctx; thresh = txq->ift_reclaim_thresh; KASSERT(thresh >= 0, ("invalid threshold to reclaim")); MPASS(thresh /*+ MAX_TX_DESC(txq->ift_ctx) */ < txq->ift_size); - now = ticks; - if (now <= (txq->ift_last_reclaim + txq->ift_reclaim_ticks) && + if (ticks <= (txq->ift_last_reclaim + txq->ift_reclaim_ticks) && txq->ift_in_use < thresh) - return (0); - txq->ift_last_reclaim = now; - /* - * Need a rate-limiting check so that this isn't called every time - */ - iflib_tx_credits_update(ctx, txq); + return (false); + iflib_tx_credits_update(txq->ift_ctx, txq); reclaim = DESC_RECLAIMABLE(txq); - - if (reclaim <= thresh /* + MAX_TX_DESC(txq->ift_ctx) */) { + if (reclaim <= thresh) { #ifdef INVARIANTS if (iflib_verbose_debug) { printf("%s processed=%ju cleaned=%ju tx_nsegments=%d reclaim=%d thresh=%d\n", __func__, @@ -3869,10 +3904,27 @@ iflib_completed_tx_reclaim(iflib_txq_t txq) #endif return (0); } - iflib_tx_desc_free(txq, reclaim); + return (reclaim); +} + +static __inline void +_iflib_completed_tx_reclaim(iflib_txq_t txq, struct mbuf **m_defer, int reclaim) +{ + txq->ift_last_reclaim = ticks; + iflib_tx_desc_free(txq, reclaim, m_defer); txq->ift_cleaned += reclaim; txq->ift_in_use -= reclaim; +} +static __inline int +iflib_completed_tx_reclaim(iflib_txq_t txq, struct mbuf **m_defer) +{ + int reclaim; + + reclaim = iflib_txq_can_reclaim(txq); + if (reclaim == 0) + return (0); + _iflib_completed_tx_reclaim(txq, m_defer, reclaim); return (reclaim); } @@ -3933,7 +3985,7 @@ iflib_txq_drain(struct ifmp_ring *r, uint32_t cidx, uint32_t pidx) DBG_COUNTER_INC(txq_drain_notready); return (0); } - reclaimed = iflib_completed_tx_reclaim(txq); + reclaimed = iflib_completed_tx_reclaim(txq, NULL); rang = iflib_txd_db_check(txq, reclaimed && txq->ift_db_pending); avail = IDXDIFF(pidx, cidx, r->size); @@ -4092,7 +4144,7 @@ _task_fn_tx(void *context) #endif if (ctx->ifc_sysctl_simple_tx) { mtx_lock(&txq->ift_mtx); - (void)iflib_completed_tx_reclaim(txq); + (void)iflib_completed_tx_reclaim(txq, NULL); mtx_unlock(&txq->ift_mtx); goto skip_ifmp; } @@ -7174,6 +7226,34 @@ iflib_handle_tx_reclaim_ticks(SYSCTL_HANDLER_ARGS) return (err); } +static int +iflib_handle_tx_defer_mfree(SYSCTL_HANDLER_ARGS) +{ + if_ctx_t ctx = (void *)arg1; + iflib_txq_t txq; + int i, err; + int defer; + + defer = ctx->ifc_sysctl_tx_defer_mfree; + err = sysctl_handle_int(oidp, &defer, arg2, req); + if (err != 0) { + return err; + } + + if (defer == ctx->ifc_sysctl_tx_defer_mfree) + return 0; + + ctx->ifc_sysctl_tx_defer_mfree = defer; + if (ctx->ifc_txqs == NULL) + return (err); + + txq = &ctx->ifc_txqs[0]; + for (i = 0; i < NTXQSETS(ctx); i++, txq++) { + txq->ift_defer_mfree = defer; + } + return (err); +} + #define NAME_BUFLEN 32 static void iflib_add_device_sysctl_pre(if_ctx_t ctx) @@ -7269,6 +7349,11 @@ iflib_add_device_sysctl_post(if_ctx_t ctx) 0, iflib_handle_tx_reclaim_ticks, "I", "Number of ticks before a TX reclaim is forced"); + SYSCTL_ADD_PROC(ctx_list, child, OID_AUTO, "tx_defer_mfree", + CTLTYPE_INT | CTLFLAG_RWTUN, ctx, + 0, iflib_handle_tx_defer_mfree, "I", + "Free completed transmits outside of TX ring lock"); + if (scctx->isc_ntxqsets > 100) qfmt = "txq%03d"; else if (scctx->isc_ntxqsets > 10) @@ -7538,7 +7623,7 @@ iflib_debugnet_poll(if_t ifp, int count) return (EBUSY); txq = &ctx->ifc_txqs[0]; - (void)iflib_completed_tx_reclaim(txq); + (void)iflib_completed_tx_reclaim(txq, NULL); NET_EPOCH_ENTER(et); for (i = 0; i < scctx->isc_nrxqsets; i++) @@ -7566,7 +7651,8 @@ iflib_simple_transmit(if_t ifp, struct mbuf *m) { if_ctx_t ctx; iflib_txq_t txq; - int error; + struct mbuf **m_defer; + int error, i, reclaimable; int bytes_sent = 0, pkt_sent = 0, mcast_sent = 0; @@ -7592,8 +7678,34 @@ iflib_simple_transmit(if_t ifp, struct mbuf *m) else if_inc_counter(ifp, IFCOUNTER_OERRORS, 1); } - (void)iflib_completed_tx_reclaim(txq); + m_defer = NULL; + reclaimable = iflib_txq_can_reclaim(txq); + if (reclaimable != 0) { + /* + * Try to set m_defer to the deferred mbuf reclaim array. If + * we can, the frees will happen outside the tx lock. If we + * can't, it means another thread is still proccessing frees. + */ + if (txq->ift_defer_mfree && + atomic_cmpset_acq_ptr((uintptr_t *)&txq->ift_sds.ifsd_m_defer, + (uintptr_t )txq->ift_sds.ifsd_m_deferb, 0)) { + m_defer = txq->ift_sds.ifsd_m_deferb; + } + _iflib_completed_tx_reclaim(txq, m_defer, reclaimable); + } mtx_unlock(&txq->ift_mtx); + + /* + * Process mbuf frees outside the tx lock + */ + if (m_defer != NULL) { + for (i = 0; m_defer[i] != NULL; i++) { + m_freem(m_defer[i]); + m_defer[i] = NULL; + } + atomic_store_rel_ptr((uintptr_t *)&txq->ift_sds.ifsd_m_defer, + (uintptr_t)m_defer); + } if_inc_counter(ifp, IFCOUNTER_OBYTES, bytes_sent); if_inc_counter(ifp, IFCOUNTER_OPACKETS, pkt_sent); if (mcast_sent)