Traffic Shaping with pf ...

[Daniel Hartmeier]

On Wed, Nov 16, 2005 at 04:18:25PM -0600, Matthew Grooms wrote:

>      I have a couple of firewalls running freebsd 5.4 and pf and was 
> planning to use ALTQ for traffic shaping. But after doing a bit of 
> reading, it would seem that ALTQ only works on traffic passing outbound 
> on an interface. Since most of the traffic passing through my firewall 
> is http and ftp traffic, the inbound direction is the path being 
> saturated. Did I read the ALTQ documentation wrong or is there another 
> mechanism available for use with pf that could help me prioritize 
> bandwidth usage?

This question pops up frequently, if this reply is too wordy, that's
just so I can reference it in the future and safe typing. My apologies
to the poster if this is all obvious already. ;)

Rate-limiting network packets means dropping packets. It's not like a
water utility pipe where you can shut the faucet incrementally and slow
down the water running towards you from the water company, leaving
unused water in their tanks. There are no reservoirs like that in a
network (ignoring some very small buffers). If a sender is sending you
packets at a rate higher than you can receive them, packets are dropped
whereever there are gaps of decreasing bandwidth. And these gaps are on
routers at your ISP and further upstream. Many of them will drop random
packets. Some can be configured to drop based on criteria, but you don't
control those criteria, because they're not your routers.

Imagine you have a 1024 kbit/s downlink from your ISP to you. Assume
your ISP himself has a much larger downlink himself. You're downloading
a file from a web server on the Internet. Then some evil person starts
sending you a flood of pings. Let's say that person has an uplink of
2048 kbit/s. Now your ISP is receiving two kinds of packets destined for
you: a stream of TCP packets from the web server, and a stream of pings
at 2048 kbits/. He can't possibly forward all these packets to you,
since only less than half of them fit onto the link to you. So he simply
forwards as many packets as he can, randomly dropping the rest.

Obviously many of the TCP packets will get dropped randomly now. TCP is
clever and adjust to this, the sender recognizes that there is loss
between him and you, and will start to send at a lower rate. Meanwhile
the flooder continues to send you pings at a happy rate of 2048 kbit/s.

You'll notice how your download gets slower and slower, and you consider
using rate-limiting incoming packets. You identify that the http packets
are those that you prefer over the pings, and tell ALTQ to drop incoming
pings exceeding, say, a harmless rate of 64 kbit/s and reserve the rest
for the more important http packets. Fine, it could do that.

But it wouldn't change anything, because the congestion is upstream of
your ALTQ box. You can drop as many packets as you like after you
received them, that doesn't free up any bandwidth on your downlink. The
downlink will continue to carry mostly ping packets, because you
dropping packets has no influence on what happens at either sender, at
your ISP's router dropping random packets, or on your downlink. Just
like the rate of water you can draw from your water line isn't influenced
by what you do with the water that has already come out of it. Rate-limiting
is not like a faucet.

This is the reason you often get the answer "It just doesn't make sense
to rate-limit incoming packets", and I guess that's the reason why ALTQ
simply doesn't add queues for incoming packets, but only outgoing ones.

Now, if we forget about the DoS case, and assume you have only
flow-controlled TCP connections with cooperative peers, things are a little
different. If you receive two streams of TCP packets, and you start
dropping packets of one stream (after you have received them and they
have taken up bandwidth on your downlink), the corresponding peer will
detect that loss and helpfully slow down sending, freeing up bandwidth
on your downlink for the other peer. In fact, if you tell your ALTQ box
to limit one stream to, say, 128 kbit/s, and drop all excess packets,
that peer will (eventually, by trial and error) come to the conclusion
that sending you packets at a rate of about 128 kbit/s is the optimal
thing to do. But it's important to realize that you're not really
"enforcing" this limit, but it's the peer that kindly reacts in that
way.

If you want to do this with ALTQ, you can do so by limiting outgoing
packets on the "other" interface, assuming the box is forwarding all
packets between two interfaces. If a browser (on a separate local box)
is downloading a file from an external web server _through_ the ALTQ
box, you rate-limit packets going out through the internal interface.
Every packet coming in on the external interface obviously goes out
through the internal interface, hence rate-limiting outgoing packets on
the internal interface has the same effect as rate-limiting incoming
packets on the external interface.

This does not work if the client is on the ALTQ box itself, obviously
(there is no "other" interface to rate-limit on). In this case you're
facing a limitation of ALTQ itself. You might have to move ALTQ onto an
additional intermediate box, just so you do have a second interface. I
don't think there are any plans to introduce incoming queues in ALTQ.

Daniel