Strange behavior of packet scheduling in ipfw3

[Николай Дмуха]

Hello!

The system is:
FreeBSD mysystem 8.0-STABLE-201005 FreeBSD 8.0-STABLE-201005 #0: Wed Jul 28
12:04:29 MSD 2010     root at mysystem:/usr/src/sys/amd64/compile/MYKERNEL
amd64

There is firewall "ipfw3" from Luigi Rizzo with packet scheduling.
There is part of firewall config (tariff with 1Mbit/s speed, for example),
below (the rules for another speeds are the same):
$IPFW pipe 11 config bw 1040Kbit/s mask dst-ip 0xffffffff
$IPFW pipe 12 config bw 1040Kbit/s mask src-ip 0xffffffff
########pipe 11
$IPFW sched 11 config type QFQ mask dst-ip 0xffffff00
$IPFW queue 111 config sched 11 weight 10
$IPFW queue 112 config sched 11 weight 8
$IPFW queue 113 config sched 11 weight 4
$IPFW queue 114 config sched 11 weight 1
$IPFW add queue 111 ip from any to table\(10\) via igb0 out proto udp
src-port 5060
$IPFW add queue 112 ip from any to table\(10\) via igb0 out proto tcp
src-port 80,443,8080
$IPFW add queue 113 ip from any to table\(10\) via igb0 out proto tcp
src-port 5223, 2009, 2106, 3724, 6112, 6881-6999, 7777, 27000-27050, 42292
$IPFW add queue 113 ip from any to table\(10\) via igb0 out proto icmp
$IPFW add queue 114 ip from any to table\(10\) via igb0 out
$IPFW add queue 111 ip from any to table\(10\) via igb2 out proto udp
src-port 5060
$IPFW add queue 112 ip from any to table\(10\) via igb2 out proto tcp
src-port 80,443,8080
$IPFW add queue 113 ip from any to table\(10\) via igb2 out proto tcp
src-port 5223, 2009, 2106, 3724, 6112, 6881-6999, 7777, 27000-27050, 42292
$IPFW add queue 113 ip from any to table\(10\) via igb2 out proto icmp
$IPFW add queue 114 ip from any to table\(10\) via igb2 out
########pipe 12
$IPFW sched 12 config type QFQ mask src-ip 0xffffff00
$IPFW queue 121 config sched 12 weight 10
$IPFW queue 122 config sched 12 weight 8
$IPFW queue 123 config sched 12 weight 4
$IPFW queue 124 config sched 12 weight 1
$IPFW add queue 1210 ip from table\(11\) to any via igb1 out proto udp
dst-port 5060
$IPFW add queue 122 ip from table\(11\) to any via igb1 out proto tcp
dst-port 80,443,8080
$IPFW add queue 123 ip from table\(11\) to any via igb1 out proto tcp
dst-port 5223, 2009, 2106, 3724, 6112, 6881-6999, 7777, 27000-27050, 42292
$IPFW add queue 123 ip from table\(11\) to any via igb1 out proto icmp
$IPFW add queue 124 ip from table\(11\) to any via igb1 out
$IPFW add queue 121 ip from table\(11\) to any via igb3 out proto udp
dst-port 5060
$IPFW add queue 122 ip from table\(11\) to any via igb3 out proto tcp
dst-port 80,443,8080
$IPFW add queue 123 ip from table\(11\) to any via igb3 out proto tcp
dst-port 5223, 2009, 2106, 3724, 6112, 6881-6999, 7777, 27000-27050, 42292
$IPFW add queue 123 ip from table\(11\) to any via igb3 out proto icmp
$IPFW add queue 124 ip from table\(11\) to any via igb3 out

Firstly, we have been tested firewall by ourself. And we had no any bad
results or any problems or maybe we have not seen them in our synthetic
tests. After that we have started this firewall in production. A few months
later we received a message from our subscriber with speed 1Mbit/s. He had a
problems with online game (big answer delay from the server). We spent a lot
of time to solve this problem. Finaly we solved it. The reason was in packet
scheduling:
1. we`ve tried to give to subscriber another channel (4Mbit/s) with packet
scheduling - there are no such problems;
2. we`ve tried to "turn off" the packet scheduling on 1Mbit channel - there
are no such problems.

The utilization of subscibers channel was always 0.4Mbit/s. But the traffic
from this subscriber was go on under the packet scheduling rules. That`s
very strange because of:
1. net.inet.ip.dummynet.io_fast=1;
2. subscribers channel utilization 0.4Mbit/s.

As I know with this option, with this firewall config and with this channel
utilization (0.4Mbit/s) traffic should bypass the pipe without packet
scheduling.

Why subscribers traffic with all these conditions doesn`t bypass through
pipe without any delays? Why his traffic was on packet scheduling rules?
Thanks.