IPFW Dummynet Bridge Limiting

Wed Jul 12 18:13:44 UTC 2006

Hey all,

            I have searched and searched and searched and can't seem to
come up with the answer to this little mystery I have going on here.
Maybe I could get some help from this large group of people who are much
smarter than I am. I have a FreeBSD machine running 6.1-RC that has
three NICs, two of which are acting as a bridge. It's a pretty standard
setup. What I am attempting to accomplish is bandwidth limiting using
dummynet over this bridge. Here's the network layout:

INTERNET ---- Core Router ---- Bridge (limiter) ---- Border Router ----
Customer Base

            The reason for the bridge between two routers is because we
also have our server farm between those routers. The customer base
consists of multiple routed networks and they all get public IPs. The
problem I'm having is that the bridge is not limiting any of the
customer IPs. I see packets flowing through the IPFW rules but they're
not being passed to the pipes. I will show the configuration
momentarily. The weird thing is, I am able to unplug the Border Router
from this whole setup and plug a laptop in to the bridge and set it up
so the laptop IP is limited. This setup works fine and I can limit the
laptop the way I expect the rest of the network to be. Here's my
configuration with the Border Router plugged in and the 216.19.50.37 IP
being used in the "Customer Base":

---Kernel Config---

options         SMP                     # Symmetric MultiProcessor
Kernel

options         IPFIREWALL              # Firewall support

options         IPFIREWALL_DEFAULT_TO_ACCEPT

options         IPDIVERT

options         DUMMYNET                # Traffic limiting

options         BRIDGE

options         HZ=1000                 # strongly recommended by
dummynet(4)

device          apic                    # I/O APIC

---Sysctl---

net.inet.ip.fw.enable=1

net.inet.ip.fw.one_pass=1

net.link.ether.bridge_cfg=em0,em1

net.link.ether.bridge.enable=1

net.link.ether.bridge_ipfw=1

net.inet.ip.fw.dyn_buckets=256

net.inet.ip.fw.curr_dyn_buckets=256

---rc.conf---

defaultrouter="[mydefaultrouter]"

hostname="[myhostname]"

ifconfig_bge0="[mymanagementinterface]"

cloned_interfaces="bridge0"

ifconfig_bridge0="addm em0 addm em1 up"

ifconfig_em0="up"

ifconfig_em1="up"

sshd_enable="YES"

firewall_enable="YES"

firewall_script="/etc/rc.firewall.bwmg"    # this just runs ipfw with
the rules supplied in custom_firewall below

firewall_quiet="NO"

firewall_logging="YES"

firewall_flags=""

---ifconfig----

-snip-

em0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500

        options=8<VLAN_MTU>

        ether 00:04:23:cb:60:aa

        media: Ethernet autoselect (100baseTX <full-duplex>)

        status: active

em1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500

        options=8<VLAN_MTU>

        ether 00:04:23:cb:60:ab

        media: Ethernet autoselect (100baseTX <full-duplex>)

        status: active

lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384

        inet 127.0.0.1 netmask 0xff000000 

bridge0: flags=8043<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500

        ether ac:de:48:ce:fe:5c

        priority 32768 hellotime 2 fwddelay 15 maxage 20

        member: em1 flags=3<LEARNING,DISCOVER>

        member: em0 flags=3<LEARNING,DISCOVER>

---custom_firewall---

-q flush

-q queue flush

-q pipe flush

add 1 allow all from any to any via lo0

add 2 deny all from any to 127.0.0.0/8

add 3 deny all from 127.0.0.0/8 to any

add 4 skipto 65534 all from any to any via bge0

add 65534 allow all from any to any

add 100 pipe 100 config bw 100Kbit/s

add 10 pipe 100 all from any to 216.19.50.37 recv em0

# ipfw show 10

00010      11430        925353 pipe 100 all from any to 216.19.50.37
recv em0

# ipfw pipe show 100

00100: 100.000 Kbit/s    0 ms   50 sl. 1 queues (1 buckets) droptail

    mask: 0x00 0x00000000/0x0000 -> 0x00000000/0x0000

BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes
Pkt/Byte Drp

  0 icmp 216.109.112.135/0        216.19.50.37/0     11434   925679  0
0   0

            I have tried many different configurations including
changing net.inet.ip.fw.one_pass to 0, changing the ipfw rule to recv
and xmit on BOTH devices of the bridge, changing the ipfw rule from all
to tcp and ip, and changing the rule from "any to 216.19.50.37" to
"216.19.50.37 to any" (recv and xmit on both interfaces). I've also
tried the kernel without IPDIVERT and with if_bridge. As I stated
before, the odd thing is that when I plug directly into it with an IP of
216.19.0.225 (can't use the other one here) and modify the rules to
reflect the new IP, the limiting works just fine. I have a feeling this
is where the problem is, but I can't quite think of any reason why this
wouldn't work. Previously, I had a Linux machine running TC installed in
place of this machine but I personally prefer FreeBSD and feel ipfw is
easier to configure than tc. The Linux machine worked just fine.

            Could anyone possibly help with this little problem? I'm
stuck. Also, if I forgot to include any information, I apologize. I'm a
bit spacey when I write emails. Just let me know what I missed and I can
explain further. Thanks.

Adam