cbq and borrowing not working

[Wes Santee]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Greetings.

I asked this same question on the pf at benzedrine.cx list and so far the
only answer I've received is "yeah, it happens here also, anyone know
what's going on?"  Perhaps someone here can shed some light.  My
apologies if you're on that list also and this question is a repeat.

I'm on 6-STABLE if that makes any difference.

Short Version
_____________

Assume altq is using only the cbq scheduler.  When a parent queue "A" is
not labeled for borrowing from its root queue, the child queues of "A"
will not fully borrow from their parent, even if there is plenty of
bandwidth available to the parent queue.

Does anyone know what is happening here, or how to solve it?

Long Version
____________

The setup from pf.conf is this:

altq on $ext_if bandwidth 54Mb cbq queue { internal, external }
queue internal bandwidth 53104Kb priority 7 cbq(borrow)
queue external bandwidth 896Kb priority 4 cbq { highq, defaultq, lowq }
   queue highq    bandwidth 40% priority 3 cbq(borrow)
   queue defaultq bandwidth 40% priority 2 cbq(default borrow)
   queue lowq     bandwidth 20% priority 1 qlimit 300 cbq(borrow)

The internal queue is for wireless traffic coming from and going to the
internal network.  The external queue (and its children) are for traffic
going upstream to my ISP.  The "external" queue is set to my max
upstream bandwidth to my ISP, so it can't borrow from the root queue.
All the child queue's under "external" can (and should) borrow against
the available upstream bandwidth amount if available.

The problem is that packets in lowq are filling up the queue, even
though there is plenty of bandwidth available to borrow from the parent
queue.

Here's a snapshot from pfctl -vvsqueue:

queue  internal bandwidth 53.10Mb priority 7 cbq( borrow )
  [ pkts:        401  bytes:     216532  dropped pkts:    0 bytes:   0 ]
  [ qlength:   0/ 50  borrows:      0  suspends:      0 ]
  [ measured:     0.0 packets/s, 0 b/s ]
queue  external bandwidth 896Kb priority 4 {highq, defaultq, lowq}
  [ pkts:          0  bytes:          0  dropped pkts:    0 bytes:   0 ]
  [ qlength:   0/ 50  borrows:      0  suspends:      0 ]
  [ measured:     0.0 packets/s, 0 b/s ]
queue   highq bandwidth 358.40Kb priority 3 cbq( borrow )
  [ pkts:          0  bytes:          0  dropped pkts:    0 bytes:   0 ]
  [ qlength:   0/ 50  borrows:      0  suspends:      0 ]
  [ measured:     0.0 packets/s, 0 b/s ]
queue   defaultq bandwidth 358.40Kb priority 2 cbq( borrow default )
  [ pkts:       7397  bytes:    3230740  dropped pkts:    0 bytes:   0 ]
  [ qlength:   0/ 50  borrows:     10  suspends:      9 ]
  [ measured:     1.0 packets/s, 4.16Kb/s ]
queue   lowq bandwidth 179.20Kb qlimit 300 cbq( borrow )
  [ pkts:      41643  bytes:   48966175  dropped pkts:    0 bytes:   0 ]
  [ qlength: 117/300  borrows:  31513  suspends:   6282 ]
  [ measured:    21.1 packets/s, 193.23Kb/s ]


Notice that queue "external", the parent of lowq, has plenty of
available bandwidth (if you add up the usage for its child queues).
However, lowq has still got a backlog of 117 packets, and refuses to go
over 193Kbit/sec.  It is borrowing, but not enough to keep the queue in
check.  If I don't set a qlimit of 300 on lowq, the queue will fill up
in about 5 seconds after starting pf.

Okay, now what happens if I change queue "external" to allow borrowing
from the root, ala:

altq on $ext_if bandwidth 54Mb cbq queue { internal, external }
queue internal bandwidth 53104Kb priority 7 cbq(borrow)
queue external bandwidth 896Kb priority 4 cbq(borrow) { highq, defaultq,
lowq }
   queue highq    bandwidth 40% priority 3 cbq(borrow)
   queue defaultq bandwidth 40% priority 2 cbq(default borrow)
   queue lowq     bandwidth 20% priority 1 cbq(borrow)

"pfctl -vvsqueue" paints a different picture:

queue  internal bandwidth 53.10Mb priority 7 qlimit 200 cbq( borrow )
  [ pkts:          0  bytes:          0  dropped pkts:    0 bytes:   0 ]
  [ qlength:   0/200  borrows:      0  suspends:      0 ]
  [ measured:     0.0 packets/s, 0 b/s ]
queue  external bandwidth 896Kb priority 4 cbq( borrow ) {highq,
defaultq, lowq}
  [ pkts:          0  bytes:          0  dropped pkts:    0 bytes:   0 ]
  [ qlength:   0/ 50  borrows:   1285  suspends:      0 ]
  [ measured:     0.0 packets/s, 0 b/s ]
queue   highq bandwidth 358.40Kb priority 3 cbq( borrow )
  [ pkts:          0  bytes:          0  dropped pkts:    0 bytes:   0 ]
  [ qlength:   0/ 50  borrows:      0  suspends:      0 ]
  [ measured:     0.0 packets/s, 0 b/s ]
queue   defaultq bandwidth 358.40Kb priority 2 cbq( borrow default )
  [ pkts:       2781  bytes:    1226812  dropped pkts:    0 bytes:   0 ]
  [ qlength:   0/ 50  borrows:    180  suspends:      0 ]
  [ measured:    17.5 packets/s, 66.61Kb/s ]
queue   lowq bandwidth 179.20Kb cbq( borrow )
  [ pkts:       5189  bytes:    6068440  dropped pkts:    0 bytes:   0 ]
  [ qlength:   0/ 50  borrows:   4341  suspends:      0 ]
  [ measured:    40.9 packets/s, 398.70Kb/s ]

Now the queue "lowq" is borrowing heavily from its parent queue
(external) and the queue is not backing up even at the default of 50
elements.

The strange thing here is that the queue "external" now shows that it is
borrowing from its parent (the root queue), although the amount of
bandwidth going through its queue is never getting close to its max of
896Kbit (about the highest I've seen it go is ~700Kbit/sec).  At any
rate, having "queue" external borrow from the root queue is a recipe for
disaster as I'll just end up saturating my upstream bandwidth.

The question is the same:  does anyone know what is happening here, or
how to solve it?

Cheers,
- -Wes
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (MingW32)

iQIVAwUBRBhdxorq8W17hxGfAQiaeQ/8DJD0Yh8bLwwx9QiwhIUx9jwr/iyDaLMP
z1wTvvGhGmE2Lksa8Y0ewhIbHHhNiAxnQ0q5bZGq6nmR1A54WntJ7DWfxKIfecSH
C45K0WYz536fo+XipkEb0OHV6YwKuaOf+cLMObiKcVvWo6fLblU/hLNg6yYwv/PS
rhZGr5Y/oul0pFgrLEWOTYmPXnn2wzUjoVZU5B4U1uNL+IZX0YBVkhlACm2yd/vc
wSVNP7abpZb8tPQ6yZ5MafFrIFDapuou2nqKhtBOGECytSCdRihR8jtBOEsTYtlh
V+Gzcwac7x4DNkbagMYuv888F+by7W1Ao9vTFU9oa0IBawtHhet1ihONRo+nbOMv
gMYs4UW50JxkflQL3OGBZw5bypRh945m1gLzwMdaBdVzvVouduiz2N/EEAxFqNMo
zcjkQiUToZAK+iZSLz5ptPVjaJnDX5NILQvu59IujpqdbDFbdORtWHpUixJnRxZh
eCm3kptfPvbNVeWlHVapraQZcdF2/grJ8gM2+bi1jOlqvBbJUCHRjmrOfs4jnv1O
5cdqex5G3gob9terfTi02iwOylnACKPxkJzFGyyjjG+4gnendLgW7EAhoIWrC4WC
20dJfLg3BbC2uyr+lJR9vP0rrS5ShKxwwXUmtKe3U8PYeN83mGAc9kwJ/SvGT+2L
GpJFrnQR4UU=
=WeJv
-----END PGP SIGNATURE-----