Unknown Behavior of PF+ALTQ on a Bridge

Thu Jun 24 17:18:31 UTC 2010

On Thu, Jun 24, 2010 at 14:04, Ermal Luçi <eri at freebsd.org> wrote:
> On Thu, Jun 24, 2010 at 3:12 PM, Rafael Henrique Faria
> <rafaelhfaria at cenadigital.com.br> wrote:
>> Hi.
>>
>> I'm working on a Brige between a router Cisco 7200, and a 3Com 7900 switch.
>> I have several subnetworks, and I need to balance the bandwidth between then.
>>
>> The Brigde is running: "FreeBSD dell05 8.1-PRERELEASE FreeBSD
>> 8.1-PRERELEASE #0: Tue Jun 22 13:59:17 BRT 2010
>> rafaelhfaria at dell05:/usr/obj/usr/src/sys/BRIDGE  amd64"
>>
>> I have the following lines in /boot/loader.conf:
>> ---
>> net.graph.maxalloc=512
>> net.graph.maxdgram=45000
>> net.graph.recvspace=45000
>> bridgestp_load="YES"
>> if_vlan_load="YES"
>> ---
>>
>> And my kernel is compiled with:
>> device          if_bridge
>> device          pf
>> device          pflog
>> options         ALTQ
>> options         ALTQ_CBQ
>> options         ALTQ_RED
>> options         ALTQ_RIO
>> options         ALTQ_HFSC
>> options         ALTQ_PRIQ
>> options         ALTQ_NOPCC
>> options         DEVICE_POLLING
>> options         HZ=1000
>> options         SHMSEG=16
>> options         SHMMNI=32
>> options         SHMMAX=2097152
>> options         SHMALL=4096
>> options         MAXFILES=8192
>>
>> And the bridge configuration:
>> cloned_interfaces="bridge0 vlan1"
>> ifconfig_bridge0="addm bce0 stp bce0 addm bce1 stp bce1 up"
>> ifconfig_bce0="polling up"
>> ifconfig_bce1="polling up"
>> ifconfig_vlan1="inet 200.x.x.x netmask 0xFFFFFF00 broadcast
>> 200.x.x.255 vlan 1 vlandev bce1"
>>
>> bce0 is connected to the Cisco 7200 ($wan_if in pf)
>> bce1 is conencted to the 3Com 7900 ($lan_if in pf)
>>
>> And my sysctl for bridge:
>> dell05# sysctl net.link.bridge
>> net.link.bridge.ipfw: 0
>> net.link.bridge.inherit_mac: 0
>> net.link.bridge.log_stp: 0
>> net.link.bridge.pfil_local_phys: 1
>> net.link.bridge.pfil_member: 1
>> net.link.bridge.pfil_bridge: 0
>> net.link.bridge.ipfw_arp: 0
>> net.link.bridge.pfil_onlyip: 0
>> dell05#
>>
>> Ok...
>>
>> Now, the problem.
>>
>> With the following queue:
>> altq on $lan_if bandwidth 33Mb hfsc queue { down_sub1, down_sub2,
>> down_sub3, down_sub4, down_def }
>>   queue down_sub1   bandwidth 8Mb priority 1 qlimit 300 hfsc (
>> realtime 3.20Mb upperlimit 22.40Mb )
>>   queue down_sub2   bandwidth 8Mb priority 1 qlimit 300 hfsc (
>> realtime 3.20Mb upperlimit 22.40Mb )
>>   queue down_sub3  bandwidth 8Mb priority 1 qlimit 300 hfsc (
>> realtime 3.20Mb upperlimit 22.40Mb )
>>   queue down_sub4  bandwidth 8Mb priority 1 qlimit 300 hfsc (
>> realtime 3.20Mb upperlimit 22.40Mb )
>>   queue down_def     bandwidth 128Kb hfsc ( default )
>>
>> And with the following rules:
>> pass in  log quick on $lan_if from <sub1> to any keep state queue ( down_sub1 )
>> pass out log quick on $wan_if from <sub1> to any keep state queue ( up_sub1 )
>> pass in  log quick on $wan_if from any to <sub1> keep state queue ( up_sub1 )
>> pass out log quick on $lan_if from any to <sub1> keep state queue ( down_sub1 )
>>
>> (..) for each <sub1-4> I have the pass rules like those.
>>
>>
>> With the full use of the link, only a small part of the traffic gets
>> into the correct queue.
>>
>> queue root_bce1 on bce1 bandwidth 33Mb priority 0 {down_sub1,
>> down_sub2, down_sub3, down_sub4, down_def}
>>  [ pkts:          0  bytes:          0  dropped pkts:      0 bytes:      0 ]
>>  [ qlength:   0/ 50 ]
>>  [ measured:     0.0 packets/s, 0 b/s ]
>> queue  down_sub1 on bce1 bandwidth 8Mb qlimit 300 hfsc( realtime
>> 3.20Mb upperlimit 22.40Mb )
>>  [ pkts:      53177  bytes:   50082785  dropped pkts:      0 bytes:      0 ]
>>  [ qlength:   0/300 ]
>>  [ measured:   364.5 packets/s, 2.81Mb/s ]
>> queue  down_sub2 on bce1 bandwidth 8Mb qlimit 300 hfsc( realtime
>> 3.20Mb upperlimit 22.40Mb )
>>  [ pkts:      90724  bytes:   79670459  dropped pkts:      0 bytes:      0 ]
>>  [ qlength:   0/300 ]
>>  [ measured:   744.6 packets/s, 5.20Mb/s ]
>> queue  down_sub3 on bce1 bandwidth 8Mb qlimit 300 hfsc( realtime
>> 3.20Mb upperlimit 22.40Mb )
>>  [ pkts:      38333  bytes:   37384626  dropped pkts:      0 bytes:      0 ]
>>  [ qlength:   0/300 ]
>>  [ measured:   285.2 packets/s, 2.35Mb/s ]
>> queue  down_sub4 on bce1 bandwidth 8Mb qlimit 300 hfsc( realtime
>> 3.20Mb upperlimit 22.40Mb )
>>  [ pkts:      80385  bytes:   69021129  dropped pkts:      0 bytes:      0 ]
>>  [ qlength:   0/300 ]
>>  [ measured:   585.1 packets/s, 3.92Mb/s ]
>> queue  down_def on bce1 bandwidth 128Kb hfsc( default )
>>  [ pkts:     268756  bytes:  336423531  dropped pkts:    121 bytes:  81921 ]
>>  [ qlength:   0/ 50 ]
>>  [ measured:  1615.4 packets/s, 16.49Mb/s ]
>>
>> watching the pflog interface, I can see that the pass rules are
>> working, no traffic is getting out of one of the rules (I have put an
>> "pass log all" to check this).
>>
>> All the rules are working... but they aren't sending the traffic to
>> the specified queue.
>>
>> If someone have a glue for this...
>> Any suggestion are welcome.
>>
>> Thank's in advance.
>
> Sorry but i do not see any evidence that what you claim is true!
>
> --
> Ermal
>

My subnets are all /24, so

table <sub1> const { 200.x.1.0/24 }
table <sub2> const { 200.x.2.0/24 }
table <sub3> const { 200.x.3.0/24 }
table <sub4> const { 200.x.4.0/24 }

In my network, I only have thoses subnets.

With:

pass all from <sub1> to any queue sub1
pass all from any to <sub1> queue sub1
pass all from <sub2> to any queue sub2
pass all from any to <sub2> queue sub2
pass all from <sub3> to any queue sub3
pass all from any to <sub3> queue sub3
pass all from <sub4> to any queue sub4
pass all from any to <sub4> queue sub4

pass all (sent to default queue)

The queues have to get all the traffic from my network. But it don't.

If I put an log option to the last pass all rule, and do a tcpdump to
pflog0, no packet is showed. So, the rules are working OK.

But with "pfctl -vvs queue", it shows:

sub1: 2.81Mb/s
sub2: 5.20Mb/s
sub3: 2.35Mb/s
sub4: 3.92Mb/s
default: 16.49Mb/s

As I can understand, with the pass rules, all the traffic from that
subnets, need to get into that queue.

So... with the pass rule of the <sub1>, all the traffic data from that
subnet, need to get into the queue sub1, the same with sub2, sub3, and
sub4.

But, Why, I have a high traffic in the default queue?

There is no packet at the last pass all rule. So, no packet is missing
the other rules.

What I want, it to get all the traffic from 200.x.1.0/24, into the
sub1 queue, and get limited by this queue, not the default queue. And
again, the same with sub2-4.

I'm using HFSC, but I'll try with CBQ.

-- 
Rafael Henrique da Silva Faria
Grupo de Sistemas e Redes

Serviço Técnico de Informática
Faculdade de Ciências e Letras do Campus de Araraquara - UNESP