pf reply-to tcp connections stall

[Cristian Bradiceanu]

On Wed, May 21, 2008 at 1:27 AM, Vlad GALU <dudu at dudu.ro> wrote:
> On 5/20/08, Cristian Bradiceanu <cbredi at bofhserver.net> wrote:
>> On Tue, May 20, 2008 at 7:20 PM, Jeremy Chadwick <koitsu at freebsd.org> wrote:
>>  > On Tue, May 20, 2008 at 06:30:58PM +0300, Cristian Bradiceanu wrote:
>>  >> I am trying to set up split routing on two Internet links, each with
>>  >> one IP address:
>>  >>
>>  >> em0 = wan1, $em0_gw gateway
>>  >> em1 = lan, NATed on em0 and em2
>>  >> em2 = wan2, default gateway
>>  >>
>>  >> pass in on em0 reply-to (em0 $em0_gw) inet proto tcp from any to em0 flags S/SA keep state
>>  >> pass in on em0 reply-to (em0 $em0_gw) inet proto udp from any to em0 keep state
>>  >> pass in on em0 reply-to (em0 $em0_gw) inet proto icmp from any to em0 keep state
>>  >>
>>  >> wan2 connections are working correct, no pf rules for policy routing
>>  >>
>>  >> wan1 tcp connections to IP of em0 (e.g. ssh) stall when a large amount
>>  >> of data is sent (e.g. running dmesg or cat file). States are created
>>  >> correctly. When ssh stalls there are some icmp packets out on lo0 with
>>  >> source and destination ip address of em0, which I believe is not
>>  >> correct (set skip on lo0 does not help). Also tried with tcp ...
>>  >> modulate state but same result.
>>  >
>>  > modulate state is known to be broken:
>>  >
>>  > http://wiki.freebsd.org/JeremyChadwick/Commonly_reported_issues
>>  >
>>  > Regarding the "when large amounts of data is sent, the connection
>>  > breaks" issue:
>>  >
>>  > I've reproduced this a few times on our systems (using the exact same
>>  > method you do: dmesg, cat'ing large files, or scp'ing -- anything using
>>  > large TCP packets), and it's always been caused by improper pf(4) rules
>>  > where state was broken.  In every case, the "state mismatch" counter
>>  > shown in pfctl -s info would increase.
>>
>>
>> state-mismatch counter does not increase, all "Counters" are 0 except
>>  match (pfctl -si).  When large amounts of data is sent the connection
>>  stalls and continues from time to time very slow; when it continues
>>  there are logged icmp packets out on lo0 from (em0) to (em0) which
>>  looks pretty weird to me.
>>
>>
>>  Cristian
>
>    This may be a PMTUD issue. Make sure your ICMP packets can travel
> back and forth unhindered and that there are no scrub rules that may
> clear out the DF flag on them.

There's no no-df scrub flag, also no icmp filters.

Cristian