bridge + ip_alias --> SLOW!!!

[Andrea Venturoli]

Hello,
I've got a problem I cannot understand and hope someone can help me.

I've got a machine which must firewall a whole class C subnet.
The upstream router (100Mb/s fiber connection) is configured as xxx.xxx.xxx.254, so I've chosen xxx.xxx.xxx.1 for my
box and bridge with the other xxx.xxx.xxx.* IPs (10Mb/s copper).

(In all my tests I've setup the external NIC to 10Mb/s; I wouldn't do more anyway).

ifconfig gives

>fxp0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
>        ether 00:02:b3:5e:5c:ca
>        media: Ethernet 10baseT/UTP
>        status: active
>vr0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
>        inet xxx.xxx.xxx.1 netmask 0xffffff00 broadcast xxx.xxx.xxx.255
>        inet xxx.xxx.xxx.12 netmask 0xffffffff broadcast xxx.xxx.xxx.12
>        ether 00:40:f4:77:5f:c8
>        media: Ethernet 10baseT/UTP
>        status: active
>fxp1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
>        inet 192.168.106.1 netmask 0xffffff00 broadcast 192.168.106.255
>        ether 00:02:b3:5e:61:d0
>        media: Ethernet 100baseTX
>        status: active
>vr1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> mtu 1500
>        ether 00:40:f4:77:61:c5
>        media: Ethernet autoselect (none)
>        status: no carrier
>lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384
>        inet 127.0.0.1 netmask 0xff000000

vr1 is currently not used, fxp1 serves a private network, fxp0 and vr0 are bridged with the following:

cat /etc/sysctl.conf

>net.link.ether.bridge=1
>net.link.ether.bridge_cfg=vr0,fxp0
>net.link.ether.bridge_ipfw=1
>net.link.ether.ipfw=1

Notice I gave no IP to fxp0, since, from what I could understand, it is not needed.


uname -a gives:

>FreeBSD zzzzzz 4.10-RELEASE-p2 FreeBSD 4.10-RELEASE-p2 #7: Tue Aug 24 16:45:56 C
>EST 2004     root at zzzzzz:/usr/obj/usr/src/sys/ZZZZZZ  i386

and we are using ipfw2:

tail /usr/src/sys/i386/conf/ZZZZZZ

>options IPFIREWALL
>options IPFIREWALL_VERBOSE
>options TCP_DROP_SYNFIN
>options RANDOM_IP_ID
>options IPDIVERT
>options IPFW2
>options BRIDGE
>options DUMMYNET



As you can see vr0 also have an alias address (for reasons which are out of scope here) and with that the problem
begin.

I can achieve good speeds on the external side both ways (originating connections and working as a server) if I use
xxx.xxx.xxx.1, but xxx.xxx.xxx.12 is MUCH MUCH slower! No difference can be noted on the internal net or the private net
on fxp1.

Just to give an idea, I tested with iperf and this are the results:

internal net -> xxx.xxx.xxx.1     6.93 Mb/s
internal net -> xxx.xxx.xxx.12    6.94 Mb/s
internet     -> xxx.xxx.xxx.1     237 Kb/s
internet     -> xxx.xxx.xxx.12    60.3 Kb/s

So using the alias IP seems four times slower, but this is probably due to the bandwidth limit on the other side (I
could only test from an ADSL): if I surf the web, choosing one of the two IPs as source, I get a much bigger gap.

I tried with an "allow all" rule as the first in the ipfw chain, and got no improvement, so the firewall should (IMHO)
not be the problem.

I'm really lost, I cannot see any reason for this difference.
Any hint?

 bye & Thanks
        av.