rack vs dummynet & pf - BAD stat: TCP out wire: ... State failure on: 1, dir=out

hi,

I have been running rack for a while now. Recently I added dummynet support
hoping to smooth out usage through the very crappy VDSL2 link. This resulted
in being unable to push podman images from the datacentre CI server to the
container registry on my PC. This reproducibly doesn't complete.

I fixed this with this pf rule, but perhaps this is not the right thing to do,
and maybe dummynet is the culprit.

```
pass out quick on $extl_if inet6 proto tcp to $my_pc port https keep state (sloppy)
```

## Issue

A podman push from a FreeBSD 16.0-CURRENT host (RACK stack, minimal pf with
default state tracking) to a registry behind a FreeBSD 15.1 firewall (pf plus
dummynet fq_codel shaping on a PPPoE uplink) reproducibly fails with 
`write: permission denied` as reported by podman.

Running `pfctl -x misc` on the sender (a05) we see:

```
pf: BAD state: TCP out wire: 2a02:ab8:201:14a0::1[443] 2a02:1b8:10:205::5[50043] stack: -
    [lo=3755612172 high=3755612556 win=8193 modulator=0 wscale=9]
    [lo=3659669939 high=3663864755 win=14566 modulator=0 wscale=8] 4:4 @39 A
    seq=3755612172 (3755612172) ack=3659669939 len=1408 ackskew=0
    pkts=30975:19650 dir=out,fwd
pf: State failure on: 1```

grep says that comes from https://cgit.freebsd.org/src/tree/sys/netpfil/pf/pf.c#n7618

```
printf("pf: State failure on: %c %c %c %c | %c %c\n",
    SEQ_GEQ(src->seqhi, data_end) ? ' ' : '1',
```

## network diagram

```
            podman push - upload 3-12 MB/s
  ───────────────────────────────────────────────────────────>
  <───────────────────────────────────────────────────────────
            ACKs, many are reordered by dummynet

 ┌───────────────┐  ┌───────────┐  ┌────┐  ┌───────────────┐  ┌────────────┐
 │ a05  arm64    │  │ r02 arm64 │  │ISP │  │ continuity    │  │ zot reg    │
 │ 16.0-CURRENT  │  │ 15.1-REL  │  │ ~4 │  │ 15.1-REL      │  │ amd64 RACK │
 │ arm64         │  │ RACK & pf │  │hops│  │ arm64, RACK   │  │ igc0       │
 │ RACK & pf     │  │ 2 x BGP   │  │ i  │  │ dpni0 -> ng0  │  │ wintermute │
 │ pf, strict    │  │           │  │ n  │  │ (PPPoE)       │  │            |
 │ lagg0         │  │           │  │ t  │  │ pf + dummynet │  │            |
 │ 2a02...:5     │  │           │  │ e  │  │ fq_codel      │  │            │
 │               │  │           │  │ r  │  │ 145M / 19M    │  │            │
 │ podman        │  │           │  │ n  │  │ haproxy TLS   │  │            │
 │               │  │           │  │ e  │  │ 2a02:...::1   │  │            │
 │               │  │           │  │ t  │  │               │  │            │
 └──────┬────────┘  └──┬─────┬──┘  └────┘  └─┬────┬─────┬──┘  └───┬────────┘
        │              │     │               │    │     │         │
     a05.pcap      r02-in  r02-out       dpni0  ng0  dpni1   zot.pcap
                                         (phys) (front)(back)
                                         before  after   to
                                         pf/dn   pf/dn   zot

 sender failure                          receiver cause
   RACK sends past snd_una+snd_wnd       ACKs leave ng0 out of order
   pf state-mismatch, dir=out
   tcp_output -> EACCES
   write(2): permission denied
```

There are a lot of ACKs leaving ng0 out of order! 

TCP timestamps show they are generated in order and reordered afterwards,
presumably by dummynet. Here's a snippet:

```
$ tshark -r rack/continuity-front.pcap \
  -Y 'tcp.srcport==443 && tcp.dstport==57450 && tcp.len==0' \
  -T fields -e frame.number -e frame.time_relative -e tcp.ack_raw \
  -e tcp.window_size_value -e tcp.options.timestamp.tsval \
  -e tcp.options.sack_le -e tcp.options.sack_re
...

# 106742 acks, 26711 out of order (25.0%); tsval older 22851, equal 3860, newer 0
#396278   55.304848 ack= 41074329 win= 109 ( 27904 B) edge= 41102233 ts=3166863965
#396285   55.305003 ack= 41067289 win= 136 ( 34816 B) edge= 41102233 ts=3166863964 sack 41070106-41071514  OOO behind=7040 (5.0 segs) tsdelta=1
#396286   55.305011 ack= 41068697 win= 131 ( 33536 B) edge= 41102233 ts=3166863964 sack 41070106-41071514  OOO behind=5632 (4.0 segs) tsdelta=1
#396287   55.305019 ack= 41071513 win= 120 ( 30720 B) edge= 41102233 ts=3166863964  OOO behind=2816 (2.0 segs) tsdelta=1
#396319   55.306813 ack= 41079961 win=  87 ( 22272 B) edge= 41102233 ts=3166863967 sack 41084186-41085594
#396322   55.306851 ack= 41079961 win=  87 ( 22272 B) edge= 41102233 ts=3166863967 sack 41081370,41084186-41082778,41085594
#396324   55.306890 ack= 41082777 win=  76 ( 19456 B) edge= 41102233 ts=3166863967 sack 41084186-41085594
#396328   55.307008 ack= 41085593 win=  65 ( 16640 B) edge= 41102233 ts=3166863967
#396340   55.307111 ack= 41077145 win=  98 ( 25088 B) edge= 41102233 ts=3166863966  OOO behind=8448 (6.0 segs) tsdelta=1
#396341   55.307119 ack= 41079961 win=  87 ( 22272 B) edge= 41102233 ts=3166863966  OOO behind=5632 (4.0 segs) tsdelta=1
#396363   55.308892 ack= 41088409 win=  54 ( 13824 B) edge= 41102233 ts=3166863968
#396364   55.308904 ack= 41091225 win=  43 ( 11008 B) edge= 41102233 ts=3166863968
```

## hypothesis

Is it possible that pf on a05 sees these reordered ACKs, and then drops the
out-of-window segments? tcp_output() would then return EACCES, and when that
lands inside the application's own write(2) the push dies.

I will try to get some black box logs over the weekend, on a05 after bumping
CURRENT again.

There are plenty of pcaps available :-)

A+
Dave
———
O for a muse of fire, that would ascend the brightest heaven of invention!