PF ALTQ CBQ rules

[Mel]

On Saturday 13 October 2007 16:04:10 Ovi wrote:
> Mel wrote:
> >On Saturday 13 October 2007 00:17:32 Ovi wrote:
> >>Hello guys
> >>
> >>I have this example from OpenBSD:
> >>
> >>altq on $br1_if cbq bandwidth 20Mb qlimit 100 tbrsize 1000 queue { std1,
> >>customer_1 } queue customer_1 bandwidth 1Mb cbq(red,ecn) {
> >> customer_1_bulk, customer_1_ack } queue customer_1_ack priority 7
> >>queue customer_1_bulk priority 0
> >>
> >>I want to use CBQ on FreeBSD, with similar rules still I have the
> >>following problem:
> >>On a 20Mb internet line I have 100 users. I want to limit (cap)
> >>bandwidth per user at 1 Mb and to add queues for all 100 users.
> >>The problem is that on FreeBSD this rules are not working, instead I
> >>must use this:
> >>
> >>altq on $br1_if cbq bandwidth 20Mb qlimit 100 tbrsize 1000 queue { std1,
> >>customer_1 } queue customer_1 bandwidth 1Mb cbq(red,ecn) {
> >> customer_1_bulk, customer_1_ack } queue customer_1_ack bandwidth 800Kb
> >> priority 7 queue customer_1_bulk bandwidth 128Kb priority 0
> >>
> >>
> >>This "bandwidth" option does not help me because I must not exceed 1 Mb.
> >
> >This can't be done with cbq, because in cbq the sum of child queues must
> > match the interface bandwidth or less. Use hfsc for this.
> >Also, you reserve 80% for ack and 20% for bulk, you might wanna reverse
> > that.
> >
> >>So my question is: how I do bandwidthupper limit with CBQ per user, like
> >>no more than 1 Mb, and add rules for 100 users?
> >
> >See above.
> >
> >A good resource on HFSC:
> >http://www.probsd.net/pf/index.php/HFSC
> >
> >It would look something like this:
> ># Use interface bandwidth, so your interface doesn't get limited
> >altq on $br1_if bandwidth 100Mb hfsc(upperlimit 100Mb) queue { \
> >	NO_CUSTOMER, \
> >	CUSTOMERS
> >}
> ># Any traffic not assigned to a customer comes on the NO_CUSTOMER queue.
> ># Backlogged traffic consumes a maximum of 80Mbit
> ># There's always 55Mbit available.
> ># Realtime values may not exceed 75% of root queue
> >queue NO_CUSTOMER bandwidth 80Mb hfsc(realtime 55Mb default)
> >
> ># Create a customers root queue, setting hard limits for any customers
> ># It ensures the entire internet connection is available at all times
> ># and also limits it to that ammount
> >queue CUSTOMERS bandwidth 20Mb hfsc(realtime 20Mb upperlimit 20Mb) { \
> >	customer_1, \
> >	customer_2, \
> >	..., \
> >}
> ># Assign 1% per customer (100 customers) for backlogged traffic.
> ># No realtime guarantees, let hfsc figure out how to spend the 20Mbit
> ># from the CUSTOMERS parent queue.
> ># No customer gets more then 1Mbit even if he's alone surfing the net.
> >queue customer_1 bandwidth 1% hfsc(linkshare 1% upperlimit 1Mb) \
> >	{ customer_1_bulk, customer_1_ack }
> ># default priority is 1
> >queue customer_1_bulk bandwidth 80% hfsc
> >queue customer_1_ack bandwidth 20% priority 2 hfsc
>
> Thank you for your answer!
>
> Well, I've simplified the example. My real situation is that I have 2500
> users, sharing 100 Mb fiber optic line..
> For 2500 users 1% will be too much. Can I use 0.25% ?
>
> Also regarding HFSC, I know is an linear algorytm, which means many does
> not scale well for lots of users. I know is hardcoded at 64, I've
> modified and used with success for up to 500 queues. For more queues it
> is working very slow, even with Xeon CPUs. So for 2500 users I would
> probably need 4-5 separate machines.
>
> I've used rules similar to your HFSC example, and I had to switch to
> ipfw + dummynet because of poor performance (on one machine) of HFSC
> with so many queues. (using ipfw+dummynet, multiple pass, to a limiting
> pipe then to queues to share load is working ok for browsing but I have
> 2 issues: first the traffic is not stable, it variates to much, second
> the traffic does not reach 100 Mbps which is my bandwidth (if i disable
> the firewall,when trafic goes to 100 Mbps). I have to mention that CPU
> is 95% idle.
>
> So my question is, what would be a choice to shape bandwidth with
> FreeBSD for like 2000 users, with bandwidth limiting, and sharing using
> clases. On linux it works for 1000-2000 users using HTB, (tc, traffic
> control) and scales well, but I want to use FreeBSD, there must be a way
> to get similar performance on FreeBSD too.

Hmm, have you tried freebsd-isp@ list? That many users/queues, I can't comment 
on with real-life experience and don't think I can simulate that either ;). 
Also, there's more then one factor impacting performance here (nic driver, 
kernel threading concurrency, locking, pf/ipfw implementation) - I'm sure 
folks over @current are very interested in hearing your results with 
7-RELEASE coming up.

-- 
Mel