Unknown Behavior of PF+ALTQ on a Bridge

[Rafael Henrique Faria]

Hi.

I'm working on a Brige between a router Cisco 7200, and a 3Com 7900 switch.
I have several subnetworks, and I need to balance the bandwidth between then.

The Brigde is running: "FreeBSD dell05 8.1-PRERELEASE FreeBSD
8.1-PRERELEASE #0: Tue Jun 22 13:59:17 BRT 2010
rafaelhfaria at dell05:/usr/obj/usr/src/sys/BRIDGE  amd64"

I have the following lines in /boot/loader.conf:
---
net.graph.maxalloc=512
net.graph.maxdgram=45000
net.graph.recvspace=45000
bridgestp_load="YES"
if_vlan_load="YES"
---

And my kernel is compiled with:
device          if_bridge
device          pf
device          pflog
options         ALTQ
options         ALTQ_CBQ
options         ALTQ_RED
options         ALTQ_RIO
options         ALTQ_HFSC
options         ALTQ_PRIQ
options         ALTQ_NOPCC
options         DEVICE_POLLING
options         HZ=1000
options         SHMSEG=16
options         SHMMNI=32
options         SHMMAX=2097152
options         SHMALL=4096
options         MAXFILES=8192

And the bridge configuration:
cloned_interfaces="bridge0 vlan1"
ifconfig_bridge0="addm bce0 stp bce0 addm bce1 stp bce1 up"
ifconfig_bce0="polling up"
ifconfig_bce1="polling up"
ifconfig_vlan1="inet 200.x.x.x netmask 0xFFFFFF00 broadcast
200.x.x.255 vlan 1 vlandev bce1"

bce0 is connected to the Cisco 7200 ($wan_if in pf)
bce1 is conencted to the 3Com 7900 ($lan_if in pf)

And my sysctl for bridge:
dell05# sysctl net.link.bridge
net.link.bridge.ipfw: 0
net.link.bridge.inherit_mac: 0
net.link.bridge.log_stp: 0
net.link.bridge.pfil_local_phys: 1
net.link.bridge.pfil_member: 1
net.link.bridge.pfil_bridge: 0
net.link.bridge.ipfw_arp: 0
net.link.bridge.pfil_onlyip: 0
dell05#

Ok...

Now, the problem.

With the following queue:
altq on $lan_if bandwidth 33Mb hfsc queue { down_sub1, down_sub2,
down_sub3, down_sub4, down_def }
   queue down_sub1   bandwidth 8Mb priority 1 qlimit 300 hfsc (
realtime 3.20Mb upperlimit 22.40Mb )
   queue down_sub2   bandwidth 8Mb priority 1 qlimit 300 hfsc (
realtime 3.20Mb upperlimit 22.40Mb )
   queue down_sub3  bandwidth 8Mb priority 1 qlimit 300 hfsc (
realtime 3.20Mb upperlimit 22.40Mb )
   queue down_sub4  bandwidth 8Mb priority 1 qlimit 300 hfsc (
realtime 3.20Mb upperlimit 22.40Mb )
   queue down_def     bandwidth 128Kb hfsc ( default )

And with the following rules:
pass in  log quick on $lan_if from <sub1> to any keep state queue ( down_sub1 )
pass out log quick on $wan_if from <sub1> to any keep state queue ( up_sub1 )
pass in  log quick on $wan_if from any to <sub1> keep state queue ( up_sub1 )
pass out log quick on $lan_if from any to <sub1> keep state queue ( down_sub1 )

(..) for each <sub1-4> I have the pass rules like those.


With the full use of the link, only a small part of the traffic gets
into the correct queue.

queue root_bce1 on bce1 bandwidth 33Mb priority 0 {down_sub1,
down_sub2, down_sub3, down_sub4, down_def}
  [ pkts:          0  bytes:          0  dropped pkts:      0 bytes:      0 ]
  [ qlength:   0/ 50 ]
  [ measured:     0.0 packets/s, 0 b/s ]
queue  down_sub1 on bce1 bandwidth 8Mb qlimit 300 hfsc( realtime
3.20Mb upperlimit 22.40Mb )
  [ pkts:      53177  bytes:   50082785  dropped pkts:      0 bytes:      0 ]
  [ qlength:   0/300 ]
  [ measured:   364.5 packets/s, 2.81Mb/s ]
queue  down_sub2 on bce1 bandwidth 8Mb qlimit 300 hfsc( realtime
3.20Mb upperlimit 22.40Mb )
  [ pkts:      90724  bytes:   79670459  dropped pkts:      0 bytes:      0 ]
  [ qlength:   0/300 ]
  [ measured:   744.6 packets/s, 5.20Mb/s ]
queue  down_sub3 on bce1 bandwidth 8Mb qlimit 300 hfsc( realtime
3.20Mb upperlimit 22.40Mb )
  [ pkts:      38333  bytes:   37384626  dropped pkts:      0 bytes:      0 ]
  [ qlength:   0/300 ]
  [ measured:   285.2 packets/s, 2.35Mb/s ]
queue  down_sub4 on bce1 bandwidth 8Mb qlimit 300 hfsc( realtime
3.20Mb upperlimit 22.40Mb )
  [ pkts:      80385  bytes:   69021129  dropped pkts:      0 bytes:      0 ]
  [ qlength:   0/300 ]
  [ measured:   585.1 packets/s, 3.92Mb/s ]
queue  down_def on bce1 bandwidth 128Kb hfsc( default )
  [ pkts:     268756  bytes:  336423531  dropped pkts:    121 bytes:  81921 ]
  [ qlength:   0/ 50 ]
  [ measured:  1615.4 packets/s, 16.49Mb/s ]

watching the pflog interface, I can see that the pass rules are
working, no traffic is getting out of one of the rules (I have put an
"pass log all" to check this).

All the rules are working... but they aren't sending the traffic to
the specified queue.

If someone have a glue for this...
Any suggestion are welcome.

Thank's in advance.

-- 
Rafael Henrique da Silva Faria
Grupo de Sistemas e Redes

Serviço Técnico de Informática
Faculdade de Ciências e Letras do Campus de Araraquara - UNESP