[HEADS UP] merging projects/pf into head

[Ian FREISLICH]

=?ISO-8859-1?Q?Ermal_Lu=E7i?= wrote:
> > - the "pf: state key linking mismatch" which affects pf as far back
> > as we've been prepared to test (FreeBSD-8.0).  Although it only
> > became visible in the logs in -CURRENT before 9-RELEASE with the
> > pf import then.  It manifests as connections stalling randomly.
> >
> This has been an issue since new pf(4) import.

My contention is that this issue is also present in earlier pf.
It's just not logged verbosely:

[firewall1.jnb1] ~ # uname -a
FreeBSD firewall1.jnb1.gp-online.net 8.1-RELEASE FreeBSD 8.1-RELEASE #23: Tue Aug  7 20:21:54 SAST 2012     ianf at firewall1.jnb1.gp-online.net:/usr/obj/usr/src/sys/FIREWALL  amd64
[firewall1.jnb1] ~ # pfctl -s inf
Status: Enabled for 30 days 16:27:26          Debug: Urgent

State Table                          Total             Rate
  current entries                   377102               
  searches                    126189706387        47596.4/s
  inserts                       6358571792         2398.3/s
  removals                      6358194690         2398.2/s
Counters
  match                        23798723897         8976.4/s
  bad-offset                             0            0.0/s
  fragment                           29807            0.0/s
  short                              76362            0.0/s
  normalize                            234            0.0/s
  memory                                 0            0.0/s
  bad-timestamp                          0            0.0/s
  congestion                             0            0.0/s
  ip-option                          78290            0.0/s
  proto-cksum                     11023818            4.2/s
  state-mismatch                   4799367            1.8/s
  state-insert                       75295            0.0/s
  state-limit                           22            0.0/s
  src-limit                              0            0.0/s
  synproxy                               0            0.0/s

Every time the state-mismatch counter increments, the connection
stalls.  This manifests as as web pages needing to be reloaded
sometimes in order to complete downloading, or ssh connections being
reset.  While 4799367 is a small fraction of the total searches,
the chance of your flow being bitten is multiplied by each hop
through a FreeBSD router running pf.  While composing this email,
the state-mismatch counter increased by 11589.

We don't see this issue at all with Gleb's patches applied and
forwarding performance is greatly improved.

Whatever happens I'd like a way forward to be found because pf
deployed at the scale we're using it is unuseable post 2011-06-28
(and not ideal before).

> > There's not been a fix since it was first reported.  We're seeing
> > 0.08% of our connections dropped on the floor or about 4 per second.
> > As a result, we've been seriously considering replacing our FreeBSD
> > routers.
> 
> I have missed the report of this, can you point to details?

http://www.freebsd.org/cgi/query-pr.cgi?pr=163208

Comes to mind.  I'm sure there were some earlier reports, but I
can't find them in a hurry.  I'm also pretty sure there have been
reports on current at .

I posted to current@
http://www.freebsd.org/cgi/getmsg.cgi?fetch=164206+169604+/usr/local/www/db/text/2012/freebsd-current/20120812.freebsd-current

Which is how I came to this list on mail from Gleb.

I can tell you that this is not peculiar to 9 and later.  pf pre-9
was just silent about dropping the flows although the problem occurs
less frequently.

Ian

-- 
Ian Freislich