pf reply-to tcp connections stall

[Cristian Bradiceanu]

On Tue, May 20, 2008 at 7:20 PM, Jeremy Chadwick <koitsu at freebsd.org> wrote:
> On Tue, May 20, 2008 at 06:30:58PM +0300, Cristian Bradiceanu wrote:
>> I am trying to set up split routing on two Internet links, each with
>> one IP address:
>>
>> em0 = wan1, $em0_gw gateway
>> em1 = lan, NATed on em0 and em2
>> em2 = wan2, default gateway
>>
>> pass in on em0 reply-to (em0 $em0_gw) inet proto tcp from any to em0 flags S/SA keep state
>> pass in on em0 reply-to (em0 $em0_gw) inet proto udp from any to em0 keep state
>> pass in on em0 reply-to (em0 $em0_gw) inet proto icmp from any to em0 keep state
>>
>> wan2 connections are working correct, no pf rules for policy routing
>>
>> wan1 tcp connections to IP of em0 (e.g. ssh) stall when a large amount
>> of data is sent (e.g. running dmesg or cat file). States are created
>> correctly. When ssh stalls there are some icmp packets out on lo0 with
>> source and destination ip address of em0, which I believe is not
>> correct (set skip on lo0 does not help). Also tried with tcp ...
>> modulate state but same result.
>
> modulate state is known to be broken:
>
> http://wiki.freebsd.org/JeremyChadwick/Commonly_reported_issues
>
> Regarding the "when large amounts of data is sent, the connection
> breaks" issue:
>
> I've reproduced this a few times on our systems (using the exact same
> method you do: dmesg, cat'ing large files, or scp'ing -- anything using
> large TCP packets), and it's always been caused by improper pf(4) rules
> where state was broken.  In every case, the "state mismatch" counter
> shown in pfctl -s info would increase.

state-mismatch counter does not increase, all "Counters" are 0 except
match (pfctl -si).  When large amounts of data is sent the connection
stalls and continues from time to time very slow; when it continues
there are logged icmp packets out on lo0 from (em0) to (em0) which
looks pretty weird to me.

Cristian