RELENG_6 and blocked packes with state-mismatch

[Jeremy Chadwick]

I'm having some problems with my pf rulesets on RELENG_6, where I see
some occasional blocked packets which also increment state-mismatch.
"Occasional" means maybe 3 or 4 packets every few minutes.  The machine
with the pf rules is 72.20.106.5 (also 72.20.106.8, which is an IP
alias).

Our ruleset is incredibly simple, so I'm a bit baffled as to how there
could be a TCP state mismatch.  I've used pfctl -xm to increase logging,
and here are some example packets which are getting blocked.

Jan 22 23:40:38 eos kernel: pf: BAD state: TCP 72.20.106.5:80 72.20.106.5:80 75.136.198.15:50665 [lo=606400253 high=606466492 win=501 modulator=1150870355 wscale=7] [lo=713095970 high=713158303 win=33120 modulator=41761135 wscale=1] 7:4 R seq=606400253 ack=713095970 len=0 ackskew=0 pkts=43:59 dir=in,fwd
Jan 22 23:40:38 eos kernel: pf: State failure on:         |
Jan 22 23:40:38 eos kernel: pf: BAD state: TCP 72.20.106.5:80 72.20.106.5:80 75.136.198.15:50665 [lo=606400253 high=606466492 win=501 modulator=1150870355 wscale=7] [lo=713095970 high=713158303 win=33120 modulator=41761135 wscale=1] 7:4 R seq=606400253 ack=713095970 len=0 ackskew=0 pkts=43:59 dir=in,fwd
Jan 22 23:40:38 eos kernel: pf: State failure on:         |
Jan 22 23:40:38 eos kernel: pf: BAD state: TCP 72.20.106.5:80 72.20.106.5:80 75.136.198.15:50666 [lo=1699814809 high=1699881048 win=501 modulator=4273956536 wscale=7] [lo=2035384330 high=2035447967 win=33120 modulator=4191871234 wscale=1] 7:4 R seq=1699814809 ack=2035384330 len=0 ackskew=0 pkts=37:41 dir=in,fwd
Jan 22 23:40:38 eos kernel: pf: State failure on:         |
Jan 22 23:40:38 eos kernel: pf: BAD state: TCP 72.20.106.5:80 72.20.106.5:80 75.136.198.15:50667 [lo=3735841199 high=3735906736 win=46 modulator=90037527 wscale=7] [lo=683911965 high=683917853 win=32768 modulator=3541623580 wscale=1] 4:2 R seq=3735841199 ack=683911965 len=0 ackskew=0 pkts=1:1 dir=in,fwd
Jan 22 23:40:38 eos kernel: pf: State failure on:         |
Jan 22 23:40:38 eos kernel: pf: BAD state: TCP 72.20.106.5:80 72.20.106.5:80 75.136.198.15:50668 [lo=3734587261 high=3734652798 win=46 modulator=3834798678 wscale=7] [lo=2009230346 high=2009236234 win=32768 modulator=3583619697 wscale=1] 4:2 R seq=3734587261 ack=2009230346 len=0 ackskew=0 pkts=1:1 dir=in,fwd
Jan 22 23:40:38 eos kernel: pf: State failure on:         |

Jan 22 23:40:59 eos kernel: pf: BAD state: TCP 72.20.106.5:80 72.20.106.5:80 75.136.198.15:50667 [lo=3735841199 high=3735906736 win=46 modulator=90037527 wscale=7] [lo=683911965 high=683917853 win=65535 modulator=3541623580 wscale=1] 4:2 R seq=3735841199 ack=683911965 len=0 ackskew=0 pkts=1:4 dir=in,fwd
Jan 22 23:40:59 eos kernel: pf: State failure on:         |
Jan 22 23:40:59 eos kernel: pf: BAD state: TCP 72.20.106.5:80 72.20.106.5:80 75.136.198.15:50668 [lo=3734587261 high=3734652798 win=46 modulator=3834798678 wscale=7] [lo=2009230346 high=2009236234 win=65535 modulator=3583619697 wscale=1] 4:2 R seq=3734587261 ack=2009230346 len=0 ackskew=0 pkts=1:4 dir=in,fwd
Jan 22 23:40:59 eos kernel: pf: State failure on:         |

Jan 22 23:45:56 eos kernel: pf: BAD state: TCP 72.20.106.5:80 72.20.106.5:80 71.62.42.150:54696 [lo=517642228 high=517707765 win=16425 modulator=4291220578 wscale=2] [lo=2300896510 high=2300962210 win=32768 modulator=18820549 wscale=1] 4:4 RA seq=517642228 ack=2300896510 len=0 ackskew=0 pkts=2:1 dir=in,fwd
Jan 22 23:45:56 eos kernel: pf: State failure on:         |
Jan 22 23:45:56 eos kernel: pf: BAD state: TCP 72.20.106.5:80 72.20.106.5:80 71.62.42.150:54699 [lo=755329106 high=755394643 win=16425 modulator=46409624 wscale=2] [lo=3951467432 high=3951533132 win=32768 modulator=4200940856 wscale=1] 4:4 RA seq=755329106 ack=3951467432 len=0 ackskew=0 pkts=2:1 dir=in,fwd
Jan 22 23:45:56 eos kernel: pf: State failure on:         |
Jan 22 23:45:56 eos kernel: pf: BAD state: TCP 72.20.106.5:80 72.20.106.5:80 71.62.42.150:54697 [lo=2644295101 high=2644360638 win=16425 modulator=3415384929 wscale=2] [lo=2718937398 high=2719003098 win=32768 modulator=345620445 wscale=1] 4:4 RA seq=2644295101 ack=2718937398 len=0 ackskew=0 pkts=2:1 dir=in,fwd
Jan 22 23:45:56 eos kernel: pf: State failure on:         |
Jan 22 23:45:56 eos kernel: pf: BAD state: TCP 72.20.106.5:80 72.20.106.5:80 71.62.42.150:54698 [lo=4259750290 high=4259815827 win=16425 modulator=323853463 wscale=2] [lo=3391337059 high=3391402759 win=32768 modulator=3588322356 wscale=1] 4:4 RA seq=4259750290 ack=3391337059 len=0 ackskew=0 pkts=2:1 dir=in,fwd
Jan 22 23:45:56 eos kernel: pf: State failure on:         |

Can someone help shed some light on what could be causing this, and/or
is it anything I need to worry about?  I'm concerned since
72.20.105.5:80 happens to be our production webserver, and I just
recently applied pf rules there (particularly the "block in log all"
clause).

If tcpdump is needed against one of the src IPs, let me know and I can
sniff a session to see what might be going on before the state mismatch
occurs.

-- 
| Jeremy Chadwick                                    jdc at parodius.com |
| Parodius Networking                           http://www.parodius.com/ |
| UNIX Systems Administrator                      Mountain View, CA, USA |
| Making life hard for others since 1977.                  PGP: 4BD6C0CB |


#	$FreeBSD: src/etc/pf.conf,v 1.1.2.1 2004/09/17 18:27:14 mlaier Exp $
#	$OpenBSD: pf.conf,v 1.21 2003/09/02 20:38:44 david Exp $
#
# See pf.conf(5) and /usr/share/examples/pf for syntax and examples.
# Required order: options, normalization, queueing, translation, filtering.
# Macros and tables may be defined and used anywhere.
# Note that translation rules are first match while filter rules are last match.

ext_if="bge0"
int_if="bge1"

# IANA-reserved netblocks.
# SSH brute-force attacks
table <iana-reserved> persist file "/conf/ME/pf.conf.iana-reserved"
table <ssh-deny> persist file "/conf/ME/pf.conf.ssh-deny"


# Options -- Internal options to pf itself.
set optimization normal
set loginterface $ext_if
set skip on lo0
set skip on $int_if

# This helps decrease state-mismatch entries caused by port number re-use;
# the pf state table keeps the state around for 100s (90s+10s internal)
# by default; drop this down to 25s (15s+10s internal).
set timeout { tcp.closed 15 }


# Normalization -- reassemble fragments and resolve/reduce traffic ambiguities.
#
scrub in  on $ext_if all fragment reassemble
scrub out on $ext_if random-id


# Filtering
#   - Block all inbound packets (on public interface only; see "set skip")
#   - Allow all outbound packets (on public interface only; see "set skip")
#
block in log all
pass out quick all modulate state

# Block traffic from IANA-reserved netblocks
block in log quick on $ext_if inet from { <iana-reserved> } to any

# Block traffic from SSH brute-force attackers
block in log quick on $ext_if inet proto tcp from { <ssh-deny> } to any port ssh flags S/SA

# Now we punch holes for services which we want to answer for on the
# public interface.  Look in /etc/services for service names.  The
# "sockstat -l" command might also come in handy.
#
pass in quick on $ext_if inet proto tcp from any to any port ssh modulate state flags S/SA
pass in quick on $ext_if inet proto tcp from any to any port domain modulate state flags S/SA
pass in quick on $ext_if inet proto udp from any to any port domain keep state
pass in quick on $ext_if inet proto tcp from any to any port { http, https } modulate state flags S/SA
pass in quick on $ext_if inet proto tcp from any to any port { smtp, smtps, submission } modulate state flags S/SA
pass in quick on $ext_if inet proto tcp from any to any port auth modulate state flags S/SA
pass in quick on $ext_if inet proto tcp from any to any port { imaps, pop3s } modulate state flags S/SA

# Punch holes for FTP.  The rule looks complex, so here it is explained:
# - Make sure pass rule only applies to 72.20.106.8 (ftp.sc1.parodius.com)
# - Permit incoming connections to port 21 (main FTP service)
# - Permit incoming connections to ports 49152-65535 (FTP passive mode)
# - TCP port 20 is actually for **outbound** connections in FTP active mode,
#   and since we allow all outbound traffic, we don't need a rule for it.
# - TCP ports 49152-65535 come from ftpd(8) and ip(4) manpages; there are
#   sysctl(8) knobs for theses, but we shouldn't mess with those.
#
pass in quick on $ext_if inet proto tcp from any to 72.20.106.8 port { ftp, 49152:65535 } modulate state flags S/SA

# We also want to respond to incoming ICMP packets.  This is necessary
# for a lot of reasons; not just for ping/traceroute, but additionally
# for things like path MTU discovery, network unreachable, source
# quench, and other control messages that TCP and UDP rely on.
#
pass in quick on $ext_if inet proto icmp from any to any keep state