Questioning altq (cbq) performance above 4Mb on gif or above 40 Mb on e1000

Fri Nov 5 19:32:11 UTC 2010

Thanks for the reply Ermal,

Based on your suggestion, I tried raising the qlimit, but it had no effect.

I have a GigE physical link. I set a child queue to catch my traffic and set a bandwidth limit of 1000Mb on the root and 45 Mb on the child. Through this config, I can pump and measure about 10Mb of traffic. I tried qlimits (on both root and child) of 100 and 800. Neither had any effect. I also observed with `pfctl -vvsq` that my queue depths tended to be in the 10s. I never saw a queue depth sample above 5 and by far, it was most often 0. `systat -v` shows a 99%idle cpu (an 8 core amd64).

Note: I am testing with tcp traffic, which throttles itself to the observed path speed. But I still expect to come near to my configured bandwidth limit.

I've been trying to read the altq code and grok it. On the dequeue side, I'm not sure how the token bucket regulator interacts with cbq_dequeue. Can anyone point me to an architecture or design guide document/ presentation about this?

Hopefully,
---
Ricky Charlet
Adara Networks
USA 408-433-4942

> -----Original Message-----
> From: ermal.luci at gmail.com [mailto:ermal.luci at gmail.com] On Behalf Of
> Ermal Luçi
> Sent: Friday, November 05, 2010 10:18 AM
> To: Ricky Charlet
> Cc: freebsd-pf at freebsd.org; altq at csl.sony.co.jp
> Subject: Re: Questioning altq (cbq) performance above 4Mb on gif or
> above 40 Mb on e1000
>
> On Fri, Nov 5, 2010 at 1:33 AM, Ricky  Charlet <RCharlet at adaranet.com>
> wrote:
> > Has anyone out there run altq with cbq with bandwidth limits set
> around 40 ~ 50 Mb and seen it work well (actual through put allowed to
> come near that speed)?
> >
> > Thanks
> > ---
> > Ricky Charlet
> > Adara Networks
> > USA 408-433-4942
> >
>
> I can suggest to try with higher hz and increase queue lengths in altq.
> By default they are 50 which is quite low in that regard.
>
> >
> >> -----Original Message-----
> >> From: owner-freebsd-pf at freebsd.org [mailto:owner-freebsd-
> >> pf at freebsd.org] On Behalf Of Ricky Charlet
> >> Sent: Wednesday, November 03, 2010 9:47 PM
> >> To: 'freebsd-pf at freebsd.org'; 'altq at csl.sony.co.jp'
> >> Subject: Questioning altq (cbq) performance above 4Mb on gif or
> above
> >> 40 Mb on e1000
> >>
> >> Howdy,
> >>
> >> (FreeBsd 8.0REL plus a modified e1000 driver borrowing the 8.1 altq
> >> capabilities, amd64, 8 core)
> >>
> >>         I have been measuring altq performance on my system. I set a
> >> cbq bandwith limit for a queue, match traffic to the queue with an
> >> appropriate rule and pump data.
> >>
> >>         Without altq enabled, my traffic gen tool (nuttcp) can pump
> >> 800Mb across my gig links.
> >>
> >>         With altq(cbq) setting bandwith limits (no borrow), up to
> about
> >> 40Mb on em0 altq keeps up. But from 40Mb bandwith limit and up, altq
> >> seems to slow down my traffic much slower than the bandwidth limits.
> >>
> >>         Same effect is observed on gif interfaces (where my real
> >> interest lies) but much worse. With altq(cbq) setting bandwith
> limits
> >> (no borrow), up to about 4Mb on gif0 altq keeps up. But from 4Mb
> >> bandwidth limit and up, altq seems to slow down my traffic much
> lower
> >> than the bandwidth limits.
> >>
> >>         I have a large file of sample data in excel formant which
> I'd
> >> like to share. Is there an appropriate way to share it with this
> list?
> >>
> >>         I know I'm not cpu bound (97% idle while nuttcp pushes 800Mb
> >> traffic and altq configured with a bandwidth limit of 50Mb actually
> >> passes 30Mb.)
> >>
> >>         I sort of suspect I'm mutex bound but have no idea how to
> test
> >> that.
> >>
> >>
> >>
> >>  (1) this config allows about 4Mb tcp traffic to pass:
> >> ----------cut------------------
> >> altq on em0 cbq bandwidth 100Mb queue { queue1, queue2 }
> >> queue queue1  bandwidth 1Mb priority 7 cbq  (default)
> >> queue queue2 bandwidth 50Mb priority 1 cbq
> >> pass out quick on gif10 inet proto tcp no state queue queue2
> >> pass out quick on gif10 proto icmp queue queue2
> >> ----------paste----------------
> >>
> >> (2) this config allows about 40 Mb tcp traffic to pass
> >> ----------cut------------------
> >> altq on em0 cbq bandwidth 100Mb queue { queue1, queue2 }
> >> queue queue1  bandwidth 1Mb priority 7 cbq  (default)
> >> queue queue2 bandwidth 50Mb priority 1 cbq
> >> pass out quick on em0 inet proto tcp no state queue queue2
> >> pass out quick on em0 proto icmp queue queue2
> >> ----------paste----------------
> >>
> >>         Yes, I send appropriate traffic in each case to be 'caught'
> by
> >> the tcp pass out rule wether it be found on a gif or an em.
> >>
> >>
> >>         My goal is to get 'expected' behavior (a bandwidth limit of
> 40
> >> Mb allows 40Mb to pass) at T3 link speed (45Mb). Is this a
> reasonable
> >> expectation? Any ideas about config jigs or tests to run?
> >>
> >>
> >>
> >> ---
> >> Ricky Charlet
> >> Adara Networks
> >> USA 408-433-4942
> >>
> >>
> >> _______________________________________________
> >> freebsd-pf at freebsd.org mailing list
> >> http://lists.freebsd.org/mailman/listinfo/freebsd-pf
> >> To unsubscribe, send any mail to "freebsd-pf-
> unsubscribe at freebsd.org"
> > _______________________________________________
> > freebsd-pf at freebsd.org mailing list
> > http://lists.freebsd.org/mailman/listinfo/freebsd-pf
> > To unsubscribe, send any mail to "freebsd-pf-unsubscribe at freebsd.org"
> >
>
>
>
> --
> Ermal