[Bug 297307] if_pfsync: pfsync_sendout() silently and permanently fails to transmit when pfsync MTU exceeds MJUMPAGESIZE

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=297307

            Bug ID: 297307
           Summary: if_pfsync: pfsync_sendout() silently and permanently
                    fails to transmit when pfsync MTU exceeds MJUMPAGESIZE
           Product: Base System
           Version: CURRENT
          Hardware: Any
                OS: Any
            Status: New
          Severity: Affects Some People
          Priority: ---
         Component: kern
          Assignee: bugs@FreeBSD.org
          Reporter: mbishop@evequefou.be

# if_pfsync: pfsync_sendout() silently and permanently fails to transmit when
pfsync MTU exceeds MJUMPAGESIZE

## Synopsis

If a `pfsync` interface's MTU is set above `MJUMPAGESIZE` (4096 bytes on
amd64), `pfsync_sendout()` fails every packet allocation, never transmits
again, and cannot recover without destroying and recreating the interface
(i.e., a reboot, since `pfsync0` cannot be destroyed at runtime).


## Environment

- FreeBSD 14.3-RELEASE-p16 (`stable/26.1-n272152-9b6eef552f24`, via OPNsense
26.1.11_10, amd64)
- Believed applicable to any FreeBSD version with the current
`pfsync_sendout()`/`m_get2()` implementation
- Two-node CARP/pfsync pair, `pfsync0` MTU set to 8930 to match a jumbo-framed
sync interface

## Description

`pfsync_sendout()` (`sys/netpfil/pf/if_pfsync.c`) allocates its outgoing packet
as:

```c
m = m_get2(max_linkhdr + b->b_len, M_NOWAIT, MT_DATA, M_PKTHDR);
if (m == NULL) {
        if_inc_counter(sc->sc_ifp, IFCOUNTER_OERRORS, 1);
        V_pfsyncstats.pfsyncs_onomem++;
        return;
}
```

`m_get2()` (`sys/kern/kern_mbuf.c`) returns NULL for any request above
`MJUMPAGESIZE` (`PAGE_SIZE`, 4096 on amd64):

```c
if (size <= MCLBYTES)
        return (uma_zalloc_arg(zone_pack, &args, how));
if (size > MJUMPAGESIZE)
        return (NULL);
```

`pfsync_q_ins()` accumulates `b->b_len` up to the pfsync interface's configured
MTU before calling `pfsync_sendout()`. If that MTU is above ~4080 bytes
(`MJUMPAGESIZE - max_linkhdr`), every flush requests an allocation `m_get2()`
will always refuse. **No pfsync packet is ever constructed**, silently — the
failure is counted (`pfsyncs_onomem`) but nothing is logged, and `bpf_mtap()`
(which would make the failure visible to `tcpdump -i pfsync0`) is never
reached, since it only runs after a successful allocation.

**The failure is permanent, not transient.** The `m_get2() == NULL` branch
returns without resetting `b->b_len`. The only reset (`b->b_len =
PFSYNC_MINPKT`) happens after a successful send. The only other path that
clears it is `pfsync_drop()`, reachable only from `pfsync_clone_destroy()`.
Once `b_len` exceeds the threshold, every subsequent insert pushes it further
past it, and every subsequent flush fails identically — indefinitely, surviving
config reloads and even, in our case, a full VM restart with the interface
still misconfigured (recreating the interface with a *corrected* MTU does
resolve it, since that resets `b_len`; changing the MTU on the live,
already-latched instance does not).

**Symptom chain observed:** with pfsync effectively unable to transmit
anything, the pfsync bulk-update mechanism (used e.g. at boot, or when CARP
failover occurs) also fails every attempt (times out after ~65–90s), and
because `pfsync bulk start` self-demotes the requesting node's CARP priority
(`carp_demote_adj_p`) until a successful `pfsync bulk done` clears it, a bulk
transfer that can never succeed causes a full, repeating CARP mastership
handoff to the peer every time it's attempted.


## Steps to reproduce

1. Configure `pfsync0` (via `ifconfig pfsync0 mtu <N>` or by setting it to
match a syncdev interface with `N > ~4080`) with an MTU above `MJUMPAGESIZE -
max_linkhdr`.
2. Generate enough pf state churn that a pfsync send bucket's `b_len` exceeds
~4080 bytes before a flush (in practice: any nontrivial amount of real
traffic).
3. Observe `netstat -s -p pfsync`: `failures due to mbuf memory error` climbs
continuously; `state inserts sent` and related send counters stay frozen.
4. Observe `pfsync bulk start` always followed by `pfsync bulk fail`, never
`pfsync bulk done`, if CARP is in use.
5. Confirmed via DTrace (`fbt::pfsync_sendout:entry`, `fbt::pfsync_tx:entry`,
`fbt::m_get2:entry`): `pfsync_sendout` executes continuously (tens of thousands
of calls per few seconds under load), `pfsync_tx` is never reached, and
`m_get2` is repeatedly called with a size in the 8192–16384 range, all
returning NULL.


## Expected result

In descending order of desirability:

1. **Best:** a pfsync MTU above `MJUMPAGESIZE` actually works — the allocator
path supports it, and jumbo pfsync frames are transmitted as configured.
2. **Acceptable:** the interface self-clamps to the largest MTU the allocator
can actually satisfy, so an operator's oversized request degrades to a working
configuration rather than a broken one.
3. **Minimum acceptable:** an MTU the allocator cannot satisfy is rejected at
set time (`SIOCSIFMTU` / clone-create) with a visible error, rather than
accepted and left to fail silently at every subsequent send.
Independent of which of the above is implemented, `pfsync_sendout()` should not
permanently latch on an allocation failure — the queue-reset/drop behavior
described below is defense-in-depth that's worth having regardless of whether
(1), (2), or (3) is also fixed, since it protects against any other path that
could produce an oversized `b_len`.

## Actual result

Silent, total, permanent loss of pfsync transmission with no log output and no
visible indication beyond the `pfsyncs_onomem` counter, until the interface is
destroyed and recreated (reboot, in practice).


## Suggested fixes

In order matching the cascade above:

1. **Support the configured MTU.** Use an allocation path capable of satisfying
requests above `MJUMPAGESIZE` (e.g. `m_getjcl`/`m_getm2`) so a large,
deliberately-configured pfsync MTU actually works rather than being silently
rejected. This is the best outcome and removes the bug's precondition entirely.
2. **Self-clamp if (1) isn't feasible.** If pfsync's send path is intentionally
bounded by `MJUMPAGESIZE`, have the MTU-setting path clamp any requested value
down to `MJUMPAGESIZE - max_linkhdr` rather than accepting a larger value it
cannot serve.
3. **Reject loudly, at minimum.** If clamping is undesirable (e.g. because a
silently-lowered MTU could itself be confusing), reject an MTU above the
allocator's ceiling at `SIOCSIFMTU`/clone-create time with a clear error, so
the misconfiguration fails at the moment it's introduced instead of the first
time traffic volume triggers a flush.
4. **Defense-in-depth, regardless of (1)–(3):** reset or drop the accumulated
queue state on the `m_get2() == NULL` path in `pfsync_sendout()`, so an
allocation failure — from this cause or any other — degrades gracefully (drops
this batch, recovers next cycle) instead of permanently wedging the send path
until the interface is recreated.


## Additional notes

- This was reached downstream of an OPNsense-specific issue where `pfsync0`'s
MTU is unconditionally copied from the sync interface with no upper bound
(`interfaces_pfsync_configure()`, OPNsense `src/etc/inc/interfaces.inc`,
referencing an earlier fix at commit `15acbad935` for the reverse problem).
That OPNsense-side issue is what makes this reachable via normal configuration
(any jumbo-framed interface assigned to pfsync duty) rather than requiring a
deliberately unusual manual MTU. It's reported separately to OPNsense; this
report is about the underlying FreeBSD kernel behavior, which affects any
FreeBSD/pfsync consumer that can end up with an oversized pfsync MTU by any
means.
- `netstat -m` will show no mbuf/cluster exhaustion during this failure —
`m_get2()`'s size check rejects the request before it reaches UMA, so the usual
mbuf-pressure diagnostics look completely clean while the failure is ongoing.
This is worth calling out since it's an easy place for a diagnosis to go astray
(as happened in our own investigation before the `pfsyncs_onomem` counter and
the source were correlated).


## How this was found (AI transparency)

This root cause was identified through a collaborative debugging session
between the reporter and two Claude models (Anthropic): Claude Sonnet for
interactive troubleshooting, and Claude Opus for the source-level analysis once
the fault was narrowed to a single function. All commands, DTrace invocations,
and configuration changes were run by the reporter on live hardware; the AI
proposed hypotheses and interpreted output but never had direct system access.

The cause was located by: (1) using DTrace (`fbt` provider) to trace
`pfsync_sendout`, `pfsync_tx`, and related functions by call count, showing
`pfsync_sendout` executing continuously while `pfsync_tx` was never reached
despite a nonempty, actively-filling send queue; (2) fetching and reading the
actual `if_pfsync.c`/`kern_mbuf.c` source for the exact FreeBSD version in use,
which located the `m_get2()` call and the `MJUMPAGESIZE` ceiling; and (3)
confirming `pfsyncs_onomem` incremented at a rate matching the DTrace call
volume. The fix was then confirmed empirically: lowering the sync interface's
MTU and rebooting produced the three outcomes predicted from the source — send
counters resuming, the failure counter reaching zero, and bulk transfers
completing — observed together on the reporter's production hardware.

The source excerpts, `MJUMPAGESIZE` value, DTrace results, and before/after
counters in this report reflect real system output and real source code, not
paraphrase or invention.

-- 
You are receiving this mail because:
You are the assignee for the bug.