pf and altq bandwidth problem.

[Christopher McGee]

I apologize if this is the wrong list for this, but if it is, please let 
me know.  Basically when queue1 on my firewall starts pushing the full 
amount of bandwidth, things that use the dflt queue become unreachable 
or VERY slow.  The dflt queue NEVER uses it's full amount of bandwidth, 
generally around 3mbit/s on average.  I'm starting to think this is just 
an inherent problem in FreeBSD 5.3.  Maybe I just need to upgrade to 5.4 
when it is released, but I don't think there were many pf updates in 
that release.  I'm reluctant to post too much information about the 
firewall and it's configuration since it is a production firewall.  But 
the problem seems to be with the queues.  Here's what I think is the 
relevant information, let me know if more information is needed:

firewall# pfctl -s queue
queue root_fxp0 bandwidth 25Mb priority 0 cbq( wrr root ) {dflt, queue1}
queue  dflt bandwidth 8Mb priority 4 qlimit 150 cbq( borrow default )
queue  queue1 bandwidth 17Mb qlimit 3500

firewall# pfctl -vvsq
queue root_fxp0 bandwidth 25Mb priority 0 cbq( wrr root ) {dflt, queue1}
  [ pkts:   93469435  bytes: 57111963278  dropped pkts:      0 
bytes:      0 ]
  [ qlength:   0/ 50  borrows:      0  suspends:      0 ]
queue  dflt bandwidth 8Mb priority 4 qlimit 150 cbq( borrow default )
  [ pkts:   47160837  bytes: 20420146684  dropped pkts:    294 bytes: 
105068 ]
  [ qlength:   0/150  borrows: 2667554  suspends:    237 ]
queue  queue1 bandwidth 12Mb qlimit 3500
  [ pkts:   46308598  bytes: 36691816594  dropped pkts: 5236343 bytes: 
4887084090 ]
  [ qlength:   0/3500  borrows:      0  suspends: 13971654 ]

queue root_fxp0 bandwidth 25Mb priority 0 cbq( wrr root ) {dflt, queue1}
  [ pkts:   93472817  bytes: 57113671748  dropped pkts:      0 
bytes:      0 ]
  [ qlength:   0/ 50  borrows:      0  suspends:      0 ]
  [ measured:   676.4 packets/s, 2.73Mb/s ]
queue  dflt bandwidth 8Mb priority 4 qlimit 150 cbq( borrow default )
  [ pkts:   47163588  bytes: 20421636153  dropped pkts:    294 bytes: 
105068 ]
  [ qlength:   0/150  borrows: 2667640  suspends:    237 ]
  [ measured:   550.2 packets/s, 2.38Mb/s ]
queue  queue1 bandwidth 12Mb qlimit 3500
  [ pkts:   46309229  bytes: 36692035595  dropped pkts: 5236343 bytes: 
4887084090 ]
  [ qlength:   0/3500  borrows:      0  suspends: 13971654 ]
  [ measured:   126.2 packets/s, 350.40Kb/s ]

queue root_fxp0 bandwidth 25Mb priority 0 cbq( wrr root ) {dflt, queue1}
  [ pkts:   93475932  bytes: 57115159111  dropped pkts:      0 
bytes:      0 ]
  [ qlength:   0/ 50  borrows:      0  suspends:      0 ]
  [ measured:   649.7 packets/s, 2.56Mb/s ]
queue  dflt bandwidth 8Mb priority 4 qlimit 150 cbq( borrow default )
  [ pkts:   47166144  bytes: 20422995656  dropped pkts:    294 bytes: 
105068 ]
  [ qlength:   0/150  borrows: 2667788  suspends:    237 ]
  [ measured:   530.7 packets/s, 2.28Mb/s ]
queue  queue1 bandwidth 12Mb qlimit 3500
  [ pkts:   46309788  bytes: 36692163455  dropped pkts: 5236343 bytes: 
4887084090 ]
  [ qlength:   0/3500  borrows:      0  suspends: 13971657 ]
  [ measured:   119.0 packets/s, 277.49Kb/s ]