reply-to speed issue

[Alexey Lanetskiy]

Hello!

I have a freebsd box (7-release) acting as gateway.
The topology is very simple. There are 2 ifaces: em0 and em1, pointing to
gateway 1 (gw1) and gw2 correspondingly. Here is the "picture":

                    ,------------.
   (internal LAN)---* FreeBSD/pf *---(WAN / gw1), $ext_if1, $ext_ip1
                    |            *---(WAN / gw2), $ext_if2, $ext_ip2
                    `------------'

There are some servers inside internal LAN, so I have to respond the
request from WAN to the same iface. Well, I need following lines inside my
pf.conf:

nat on $ext_if1 from !(self) to any -> ($ext_if1:0)
nat on $ext_if2 from !(self) to any -> ($ext_if2:0)

# example of some internal service, hosted inside the LAN
rdr on $ext_if1 proto tcp to port $someport tag IF_1 \
                -> $ip_internal port $someport
rdr on $ext_if2 proto tcp to port $someport tag IF_2 \
                -> $ip_internal port $someport

block in all
block out all

# example of common services, hosted on freebsd box
pass in on $ext_if1 reply-to ($ext_if1 $ext_gw1) \
                proto tcp from <ext_white_ftp> \
                to $ext_ip1 port { ftp, ftp-data, 45000:50000 } \
                flags S/SA keep state
pass in on $ext_if2 reply-to ($ext_if2 $ext_gw2) \
                proto tcp from <ext_white_ftp> \
                to $ext_ip2 port { ftp, ftp-data, 45000:50000 } \
                flags S/SA keep state

pass in quick reply-to ($ext_if1 $ext_gw1) proto { udp, icmp } \
                tagged IF_1 keep state
pass in quick reply-to ($ext_if1 $ext_gw1) proto tcp \
                tagged IF_1 flags S/SA keep state
pass in quick reply-to ($ext_if2 $ext_gw2) proto { udp, icmp } \
                tagged IF_2 keep state
pass in quick reply-to ($ext_if2 $ext_gw2) proto tcp \
                tagged IF_2 flags S/SA keep state

Now it works. Connections from outside to both hosted @box & hosted @LAN
are estabilishing, data flows, but... strange speed issue detected.
Let's shut down pf (pfctl -d) and ftp to any of external ifaces: full
speed of iface in both directions.
Let's enable pf again, but use pf.conf without any "reply-to"
("route-to"s are still at their places): oops, something wrong with
outgoing stream. Look at this numbers: approx. 60kBytes/sec w/o "reply-to"
and only 3kBytes/sec with it. Not very nice, isn't it...

Let me say some words about the box itself.
box: SMP system on single core2duo CPU, 2 em & 1 rl nics.
freebsd: default sysctl setup, custom kernel built using GENERIC with
following difference:
options SCHED_ULE
device pf
options ALTQ
options ALTQ_CBQ
options ALTQ_RED
options ALTQ_RIO
options ALTQ_HFSC
options ALTQ_CDNR
options ALTQ_PRIQ
options ALTQ_NOPCC
pf: No queues running, very (less than 10 items) small tables, near 120
rules in pf.conf.

Here the question begins: what is the source of such a problem with
"reply-to". What should I test, may be on another box or in lab? What
manuals should I learn before configure pf any more if there are config
mistakes?

-- 
wbr, Alexey.