[HEADS UP] merging projects/pf into head

[Ermal Luçi]

On Fri, Sep 7, 2012 at 2:05 PM, Ian FREISLICH <ianf at clue.co.za> wrote:
> =?ISO-8859-1?Q?Ermal_Lu=E7i?= wrote:
>> > - the "pf: state key linking mismatch" which affects pf as far back
>> > as we've been prepared to test (FreeBSD-8.0).  Although it only
>> > became visible in the logs in -CURRENT before 9-RELEASE with the
>> > pf import then.  It manifests as connections stalling randomly.
>> >
>> This has been an issue since new pf(4) import.
>
> My contention is that this issue is also present in earlier pf.
> It's just not logged verbosely:
>
> [firewall1.jnb1] ~ # uname -a
> FreeBSD firewall1.jnb1.gp-online.net 8.1-RELEASE FreeBSD 8.1-RELEASE #23: Tue Aug  7 20:21:54 SAST 2012     ianf at firewall1.jnb1.gp-online.net:/usr/obj/usr/src/sys/FIREWALL  amd64
> [firewall1.jnb1] ~ # pfctl -s inf
> Status: Enabled for 30 days 16:27:26          Debug: Urgent
>
> State Table                          Total             Rate
>   current entries                   377102
>   searches                    126189706387        47596.4/s
>   inserts                       6358571792         2398.3/s
>   removals                      6358194690         2398.2/s
> Counters
>   match                        23798723897         8976.4/s
>   bad-offset                             0            0.0/s
>   fragment                           29807            0.0/s
>   short                              76362            0.0/s
>   normalize                            234            0.0/s
>   memory                                 0            0.0/s
>   bad-timestamp                          0            0.0/s
>   congestion                             0            0.0/s
>   ip-option                          78290            0.0/s
>   proto-cksum                     11023818            4.2/s
>   state-mismatch                   4799367            1.8/s
>   state-insert                       75295            0.0/s
>   state-limit                           22            0.0/s
>   src-limit                              0            0.0/s
>   synproxy                               0            0.0/s
>
> Every time the state-mismatch counter increments, the connection
> stalls.  This manifests as as web pages needing to be reloaded
> sometimes in order to complete downloading, or ssh connections being
> reset.  While 4799367 is a small fraction of the total searches,
> the chance of your flow being bitten is multiplied by each hop
> through a FreeBSD router running pf.  While composing this email,
> the state-mismatch counter increased by 11589.
>

This is not enough information to debug anything.
- Please post your ruleset
- A dump of your state table at the time
- Describe your environment to allow understanding
- Any kind of routing related
- Tcpdump would be helpful as well

Normally this issue, should exist in Gleb repo even though you are not
facing it loudly.
Nothing has changed in Gleb's repo related to this behaviour apart not
having the linked state functionality(right?),
which as you say does not seem the source of this since happens even
before 9.0 anyway.

I have not seen this reported in pfSense side of things either.
If you can try a quick test with pfSense, either just copying the
kernel and pfctl binary, and see if you have same behavior would be
helpful.

> We don't see this issue at all with Gleb's patches applied and
> forwarding performance is greatly improved.
>
That's a good thing in general and is good to have improvements just i
am a bit sceptic
about its changes in some areas.

> Whatever happens I'd like a way forward to be found because pf
> deployed at the scale we're using it is unuseable post 2011-06-28
> (and not ideal before).
>
>> > There's not been a fix since it was first reported.  We're seeing
>> > 0.08% of our connections dropped on the floor or about 4 per second.
>> > As a result, we've been seriously considering replacing our FreeBSD
>> > routers.
>>
>> I have missed the report of this, can you point to details?
>
> http://www.freebsd.org/cgi/query-pr.cgi?pr=163208
>
> Comes to mind.  I'm sure there were some earlier reports, but I
> can't find them in a hurry.  I'm also pretty sure there have been
> reports on current at .
>
> I posted to current@
> http://www.freebsd.org/cgi/getmsg.cgi?fetch=164206+169604+/usr/local/www/db/text/2012/freebsd-current/20120812.freebsd-current
>
> Which is how I came to this list on mail from Gleb.
>
> I can tell you that this is not peculiar to 9 and later.  pf pre-9
> was just silent about dropping the flows although the problem occurs
> less frequently.
>
> Ian
>
> --
> Ian Freislich



-- 
Ermal