kern/139581: [ipfw] "ipfw pipe" not limiting bandwidth

[alexus]

The following reply was made to PR kern/139581; it has been noted by GNATS.

From: alexus <alexus at alexus.org>
To: Ian Smith <smithi at nimnet.asn.au>
Cc: bug-followup at FreeBSD.org,
 freebsd at alexus.org
Subject: Re: kern/139581: [ipfw] "ipfw pipe" not limiting bandwidth
Date: Mon, 19 Oct 2009 11:58:41 -0400

 On Oct 19, 2009, at 10:24 AM, Ian Smith wrote:
 
 > May be a usage issue; I'll have a go.  Partial quoting, out of order.
 >
 > : I'm trying to limit my apache that runs under daemon to up 2Mbit/s
 > : when I do "ipfw pipe show" I don't see anything in my slots other  
 > then
 > : very first entry that never chage, nor does it limits my traffic, as
 > : if I look at my MRTG i see way more traffic then 2Mbit/s
 >
 > Unless you specify masks on your pipes you'll only ever see the first
 > connection that used that pipe, that's normal.
 
 ok
 
 new set of rules
 
 su-3.2# cat /etc/ipfw.rules
 flush
 pipe flush
 pipe 1 config bw 1Mbit/s mask src-port www
 pipe 2 config bw 1Mbit/s mask src-port www
 add 100 allow ip from any to any via lo0
 add 200 deny ip from any to 127.0.0.0/8
 add 300 deny ip from 127.0.0.0/8 to any
 add 8381 pipe 1 tcp from any to any dst-port www uid daemon
 add 8382 pipe 2 tcp from any to any src-port www uid daemon
 add 65000 pass all from any to any
 su-3.2# ipfw show
 00100 1476  230632 allow ip from any to any via lo0
 00200    0       0 deny ip from any to 127.0.0.0/8
 00300    0       0 deny ip from 127.0.0.0/8 to any
 08381  482   36368 pipe 1 tcp from any to any dst-port 80 uid daemon
 08382  620  743113 pipe 2 tcp from any 80 to any uid daemon
 65000 6832 5040856 allow ip from any to any
 65535    0       0 deny ip from any to any
 su-3.2# ipfw pipe show
 00001:   1.000 Mbit/s    0 ms   50 sl. 1 queues (1 buckets) droptail
      mask: 0x00 0x00000000/0x0000 -> 0x00000000/0x0000
 BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/ 
 Byte Drp
    0 tcp     64.237.55.83/49492   66.230.133.69/80     509    38156   
 0    0   0
 00002:   1.000 Mbit/s    0 ms   50 sl. 1 queues (1 buckets) droptail
      mask: 0x00 0x00000000/0x0000 -> 0x00000000/0x0000
 BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/ 
 Byte Drp
    0 tcp    66.230.133.69/80       64.237.55.83/49492  656   785292  1  
 1500   1
 su-3.2# ipfw pipe show
 00001:   1.000 Mbit/s    0 ms   50 sl. 1 queues (1 buckets) droptail
      mask: 0x00 0x00000000/0x0000 -> 0x00000000/0x0000
 BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/ 
 Byte Drp
    0 tcp     64.237.55.83/49492   66.230.133.69/80    1247    98023   
 0    0   0
 00002:   1.000 Mbit/s    0 ms   50 sl. 1 queues (1 buckets) droptail
      mask: 0x00 0x00000000/0x0000 -> 0x00000000/0x0000
 BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/ 
 Byte Drp
    0 tcp    66.230.133.69/80       64.237.55.83/49492 1475  1453606   
 0    0   1
 su-3.2#
 
 in this case i did specify mask for pipe, yet when I'm issuing ipfw  
 pipe show I still don't see anything in terms of slots that being in use
 
 su-3.2# sysctl net.inet.ip.dummynet.pipe_slot_limit
 net.inet.ip.dummynet.pipe_slot_limit: 100
 su-3.2#
 
 seems like at all time I see only 1 slot being utilized and as I  
 mention before it never changes.
 
 
 >
 > MRTG sees all traffic on an interface, and your ipfw stats indicate at
 > least 25% more traffic than that due to your webserver, so it's not
 > clear how you could tell if your pipe was exceeding 2Mbit/s or not?
 >
 
 I obviously do have other traffic then www, but majority of it is www.
 but I see why you coming with this, so let me just give you an example.
 if I at peak time shutdown my apache, my traffic drops dramatically  
 and by dramatically i mean at least 90% (and in most cases more)
 my traffic went to as much as 10mbps with supposedly limited pipe of  
 2mbps, when I set it to 1mbps it seems to be almost there...
 
 > Also, it's recommended not to run your inbound and outbound traffic
 > through the one pipe, unless simulating half-duplex connections; see
 > explanation in ipfw(8), EXAMPLES section under TRAFFIC SHAPING.
 
 i thought about that and as you suggested i did separate them into 2  
 separate pipes (see on top)
 
 >
 > : su-3.2# ipfw show
 > : 00100 1249368 205115325 allow ip from any to any via lo0
 > : 00200 0 0 deny ip from any to 127.0.0.0/8
 > : 00300 0 0 deny ip from 127.0.0.0/8 to any
 > : 08380 2838075 3586421013 pipe 1 tcp from any 80 to any uid daemon
 > : 08380 2097473 136454502 pipe 1 tcp from any to any dst-port 80 uid  
 > daemon
 > : 65000 5740679 4716157064 allow ip from any to any
 > : 65535 0 0 deny ip from any to any
 >
 > 3.586 GiB outbound from the webserver (served data)
 > 0.136 GiB inbound to the webserver (requests, acks)
 > + ---
 > 3.722 GiB through the pipe.
 > but
 > 4.716 GiB passed from any to any, either way.
 >
 > So there's about 1 Gig of extra traffic shown here, assuming you have
 > net.inet.ip.fw.one_pass=0 and all traffic eventually hits rule 65000
 > (and 4.7G extra traffic if net.inet.ip.fw.one_pass=1) but there's not
 > enough info to see whether or not it's on the interface MRTG watches?
 >
 > : su-3.2# ipfw pipe show
 > : 00001: 2.000 Mbit/s 0 ms 50 sl. 1 queues (1 buckets) droptail
 > : mask: 0x00 0x00000000/0x0000 -> 0x00000000/0x0000
 > : BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes
 > Pkt/Byte Drp
 > : 0 tcp 64.237.55.83/59388 208.80.152.3/80 4936077 3723134341 0 0  
 > 30179
 >
 > Total packets and bytes match the above, indicating that this was done
 > just after the ipfw show.  0.6% dropped packets indicates some  
 > limiting
 > happening, but with a shared in/outbound pipe, not in which direction.
 >
 > If this is still an issue, please:
 >
 > . be more precise than "way more traffic" if you have more data?
 > . say whether the extra ~25% traffic shown is on the same interface
 >  as the webserver, ie the interface MRTG monitors, or not?
 > . the value of sysctl net.inet.ip.fw.one_pass ?
 >
 > cheers, Ian
 >