ipfw bandwidth shaping problems (intermittent latency)

[Andy Coates]

Luigi Rizzo (rizzo at icir.org) wrote:
> ok then it is very clear what is going on.
> 
> You have some connection which occasionally sends a burst of
> data (more than 15 pkts because you have drops, but perhaps
> less than 50 as you don't see drops with a queue of 50), which
> in turn causes the long delay.
> 
> 123ms at 2.4Mbit/s are approx 300Kbits or 32Kbytes of data + headers,
> or 22-23 full-sized packets.
> 
> Given the above numbers, I suspect (indeed, I would bet money on
> this!) that the offender is a TCP connection which has the window
> fully open and sends data intermittently -- e.g. a persistent http
> connection, or even the typical ssh connection in response to a
> command that causes a large burst of data.

Very interesting analysis.  Of the traffic I know about that consumes
the majority of bandwidth, one would be ftp transfers (usually 3 or 4
times an hour lasting 5 minutes or so) and the other would be a partial
news feed (suck).

I'm not sure how we can get this to work then.  When we're not using
these high packets/per/sec programs there doesn't seem to be a problem,
but when we are and I raise the slots to 35 (this seems to be the point
at which packets are no longer dropped) we still see the high latency:

64 bytes from xx.xx.xx.xx: icmp_seq=108 ttl=248 time=3.578 ms
64 bytes from xx.xx.xx.xx: icmp_seq=109 ttl=248 time=3.555 ms
64 bytes from xx.xx.xx.xx: icmp_seq=110 ttl=248 time=3.538 ms
64 bytes from xx.xx.xx.xx: icmp_seq=111 ttl=248 time=7.028 ms
64 bytes from xx.xx.xx.xx: icmp_seq=112 ttl=248 time=3.620 ms
64 bytes from xx.xx.xx.xx: icmp_seq=113 ttl=248 time=3.555 ms
64 bytes from xx.xx.xx.xx: icmp_seq=114 ttl=248 time=17.211 ms
64 bytes from xx.xx.xx.xx: icmp_seq=115 ttl=248 time=3.600 ms
64 bytes from xx.xx.xx.xx: icmp_seq=116 ttl=248 time=3.526 ms
64 bytes from xx.xx.xx.xx: icmp_seq=117 ttl=248 time=6.432 ms
64 bytes from xx.xx.xx.xx: icmp_seq=118 ttl=248 time=3.603 ms
64 bytes from xx.xx.xx.xx: icmp_seq=119 ttl=248 time=3.597 ms
64 bytes from xx.xx.xx.xx: icmp_seq=120 ttl=248 time=3.545 ms
64 bytes from xx.xx.xx.xx: icmp_seq=121 ttl=248 time=3.519 ms
64 bytes from xx.xx.xx.xx: icmp_seq=122 ttl=248 time=6.389 ms
64 bytes from xx.xx.xx.xx: icmp_seq=123 ttl=248 time=6.407 ms
64 bytes from xx.xx.xx.xx: icmp_seq=124 ttl=248 time=3.560 ms
64 bytes from xx.xx.xx.xx: icmp_seq=125 ttl=248 time=16.571 ms


Granted its lower than before, but its still there.  This is us
using about 900Kbit/s at the moment - so i'm lost as to what options
and parameters we should be using.  Any suggestions?

Cheers,
Andy.