Packet loss with traffic shaper and routing
John-Mark Gurney
gurney_j at resnet.uoregon.edu
Wed May 3 00:52:23 UTC 2006
Julian Elischer wrote this message on Tue, May 02, 2006 at 17:38 -0700:
> tpeixoto at widesoft.com.br wrote:
>
> ok gotcha
>
>
> probably the pipes are stored in a list or something.
>
> check the code and see if a hash table woudl be better..
Looks like it's already a hash, though a bit small:
#define HASHSIZE 16
#define HASH(num) ((((num) >> 8) ^ ((num) >> 4) ^ (num)) & 0x0f)
static __inline struct dn_pipe *
locate_pipe(int pipe_nr)
{
struct dn_pipe *pipe;
SLIST_FOREACH(pipe, &pipehash[HASH(pipe_nr)], next)
if (pipe->pipe_nr == pipe_nr)
return (pipe);
return (NULL);
}
Look at increasing HASHSIZE and changing the hash function...
> >Julian Elischer wrote:
> >
> >>tpeixoto at widesoft.com.br wrote:
> >>
> >>>Hello.
> >>>I think I should give some 'real world' examples.
> >>>
> >>>
> >>>/etc/rc.firewall:
> >>>
> >>>[Ss][Hh][Aa][Pp][Ee][Rr])
> >>>setup_loopback
> >>>
> >>>. /etc/rc.shaper
> >>>
> >>>${fwcmd} add 65000 pass all from any to any
> >>>;;
> >>>
> >>>
> >>>/etc/rc.shaper:
> >>>
> >>>${fwcmd} pipe 1 config bw 512Kbit/s
> >>>${fwcmd} pipe 2 config bw 512Kbit/s
> >>>${fwcmd} add pipe 1 all from any to any MAC any 00:11:22:33:44:55 in
> >>>${fwcmd} add pipe 2 all from any to any MAC 00:11:22:33:44:55 any out
> >>>${fwcmd} pipe 3 config bw 256Kbit/s
> >>>${fwcmd} pipe 4 config bw 256Kbit/s
> >>>${fwcmd} add pipe 3 all from any to any MAC any 66:77:88:99:aa:bb in
> >>>${fwcmd} add pipe 4 all from any to any MAC 66:77:88:99:aa:bb any out
> >>>${fwcmd} pipe 5 config bw 128Kbit/s
> >>>${fwcmd} pipe 6 config bw 128Kbit/s
> >>>${fwcmd} add pipe 5 all from any to any MAC any 00:01:02:03:04:05 in
> >>>${fwcmd} add pipe 6 all from any to any MAC 00:01:02:03:04:05 any out
> >>>${fwcmd} pipe 7 config bw 512Kbit/s
> >>>${fwcmd} pipe 8 config bw 1024Kbit/s
> >>>${fwcmd} add pipe 7 all from any to any MAC any 06:07:08:09:0a:0b in
> >>>${fwcmd} add pipe 8 all from any to any MAC 06:07:08:09:0a:0b any out
> >>>${fwcmd} pipe 9 config bw 64Kbit/s
> >>>${fwcmd} pipe 10 config bw 64Kbit/s
> >>>${fwcmd} add pipe 9 all from any to any MAC any ab:cd:ef:00:11:22 in
> >>>${fwcmd} add pipe 10 all from any to any MAC ab:cd:ef:00:11:22 any out
> >>>
> >>>
> >>OK, so, put the MACs in numerical order:
> >>
> >>00:01:02:03:04:05
> >>00:11:22:33:44:55
> >>06:07:08:09:0a:0b
> >>66:77:88:99:aa:bb
> >>ab:cd:ef:00:11:22
> >>
> >>
> >>work out MASKS that divide them into a binary set.
> >>
> >>e.g.
> >>1 skipto 10 all from any to not MAC 00:00:00:00:00:00/8
> >>2 skipto 5 all from any to not MAC 00:01:00:00:00:00/16
> >>3 pipe 1 ip from any to any
> >>5 pipe 2 ip from any to any
> >>
> >>10 skipto 12 all from any to not MAC 06:00:00:00:00:00/8
> >>11 pipe 3 all from any to any
> >>12 skipto 14 all from any to not MAC 66:00:00:00:00:00/8
> >>13 pipe 4 all from any to any
> >>14 pipe 5 all from any to any
> >>
> >>now, if you continue this on, you will run 16 rules to divide the
> >>1600 rules up to find the right pipe.
> >>
> >
> >I got your point.
> >But what I am telling is that it's not the search or it's not _only_
> >the search in the firewall rules that is making the interrupts go high.
> >Please, see below.
> >
> >
> >>>
> >>>This example is for 5 clients. We have 1600.
> >>>As you can see, there are 2 rules and 2 pipes per host, not 1600.
> >>>
> >>>
> >>>If we try rc.firewall like this...
> >>>
> >>>setup_loopback
> >>>${fwcmd} add 65000 pass all from any to any
> >>>
> >>>... we are ok. Interrupts are low.
> >>>
> >>>So, following your line of thought, I tried a simple test...
> >>>
> >>>setup_loopback
> >>>${fwcmd} skipto 65000 ip from any to any MAC any any
> >>>. /etc/rc.shaper
> >>>${fwcmd} add 65000 pass all from any to any
> >>>
> >>>This way, the packets will never pass through shaper rules, but
> >>>interrupts
> >>>still get very high.
> >>>
> >>>
> >>
> >>I don't see how that proves anything
> >>
> >
> >See, if we have just 4 rules in the kernel (3 from setup_loopback +
> >allow any to any), we don't have problems with interrupts. They are
> >low, about 15~20% with the same traffic.
> >But, if we have a 'full' set of rules, let's say 3205 (3 from
> >setup_loopback + skipto 65000 + 3200 pipes + allow any to any), where
> >only 5 of them are being matched (setup_loopback, 'skipto 65000' and
> >'allow any to any' - the skipto 65000 rule prevents any packet to
> >search through my 3200 pipes, right?), we still see interrupts go to
> >70~90%.
> >So, what I am saying is that even if we use skipto rules to create
> >'shortcuts' in the firewall stack, the system still uses lots of
> >interrupts. It seems that no matter whether the packets are being
> >checked against the rules or not, as long there are so many rules, the
> >interrupts will be generated.
> >
> >Let me know if you got my point.
> >I'll do some more tests reducing the number of pipes while keeping the
> >same amount of rules to see whether this has some effect in the
> >interrupts.
> >
> >BTW: I tested your other suggestion about splitting 'in' and 'out'
> >rules but it made no difference regarding system interrupts.
> >
> >Thanks again!
> >
> >
> >>>Basically, we need a solution to shape each MAC address with its
> >>>specifics
> >>>download e upload speeds.
> >>>Given the tests, I don't see how skipto can help, but if you believe
> >>>that
> >>>tablearg (which I am not familiar with) might help, we can try it with
> >>>7.x.
> >>>
> >>>
> >>
> >>Tablearg only works with IP addresses.
> >>
> >>>Thanks.
> >>>
> >>>
> >>>
> >>>
> >>>>oops, forgot to fix my cut-n- pastes.. corrected triage below..
> >>>>
> >>>>
> >>>>Julian Elischer wrote:
> >>>>
> >>>>
> >>>>
> >>>>>Julian Elischer wrote:
> >>>>>
> >>>>>
> >>>>>
> >>>>>>tpeixoto at widesoft.com.br wrote:
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>That would do it..
> >>>>>>>>
> >>>>>>>>In all versions of FreeBSD
> >>>>>>>>you can use the skipto rule to make sure that only a few rules are
> >>>>>>>>run for any
> >>>>>>>>address. Use it to to a binary search for the right pipe.'
> >>>>>>>>carefully using 'skipto' and 'table' can make it efficient to do
> >>>>>>>>very complex
> >>>>>>>>filters like that.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>>Sorry, but I didn't realized how to use that as we have to shape
> >>>>>>>each user individually, i.e., each MAC address on the LAN has its
> >>>>>>>own download and upload speeds.
> >>>>>>>
> >>>>>>>Could you clarify how to improve the situation with the tools you
> >>>>>>>mentioned?
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>Assuming you can not use "tablearg" yet (it will make this REALLY
> >>>>>>EASY)
> >>>>>>then if you have 30 IPs you want to shape from 1.1.1.1 to 1.1.1.30
> >>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>then, consider the following example using IP addresses.
> >>>>>
> >>>>>
> >>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>>ipfw add 1000 skipto 1110 ip from any to 1.1.1.16/28
> >>>>>ipfw add 1010 skipto 1032 ip from any to 1.1.1.8/29
> >>>>>ipfw add 1012 skipto 1021 ip from any to 1.1.1.4./30
> >>>>> ipfw add 1013 [anything] ip from any to 1.1.1.0
> >>>>> ipfw add 1014 [anything] ip from any to 1.1.1.1
> >>>>>ipfw add 1015 [anything] ip from any to 1.1.1.2
> >>>>>ipfw add 1016 [anything] ip from any to 1.1.1.3
> >>>>>
> >>>>>
> >>>>>ipfw add 1021 anything] ip from any to 1.1.1.4
> >>>>>ipfw add 1022 [anything] ip from any to 1.1.1.5
> >>>>>ipfw add 1023 [anything] ip from any to 1.1.1.6
> >>>>>ipfw add 1024 [anything] ip from any to 1.1.1.7
> >>>>>
> >>>>>
> >>>>>ipfw add 1032 skipto 1051 ip from any to 1.1.1.12./30
> >>>>>
> >>>>>ipfw add 1040 [anything] ip from any to 1.1.1.8
> >>>>>ipfw add 1041 [anything] ip from any to 1.1.1.9
> >>>>>ipfw add 1042 [anything] ip from any to 1.1.1.10
> >>>>>ipfw add 1043 [anything] ip from any to 1.1.1.11
> >>>>>
> >>>>>
> >>>>>ipfw add 1051 [anything] ip from any to 1.1.1.12
> >>>>>ipfw add 1052 [anything] ip from any to 1.1.1.13
> >>>>>ipfw add 1053 [anything] ip from any to 1.1.1.14
> >>>>>ipfw add 1054 [anything] ip from any to 1.1.1.15
> >>>>>
> >>>>>
> >>>>>ipfw add 1110 skipto 1132 ip from any to 1.1.1.24/29
> >>>>>ipfw add 1112 skipto 1121 ip from any to 1.1.1.20./30
> >>>>>ipfw add 1113 [anything] ip from any to 1.1.1.16
> >>>>>ipfw add 1114 [anything] ip from any to 1.1.1.17
> >>>>>ipfw add 1115 [anything] ip from any to 1.1.1.18
> >>>>> ipfw add 1116 [anything] ip from any to 1.1.1.19
> >>>>> ipfw add 1121 anything] ip from any to 1.1.1.20
> >>>>>ipfw add 1122 [anything] ip from any to 1.1.1.21
> >>>>>ipfw add 1123 [anything] ip from any to 1.1.1.22
> >>>>>ipfw add 1124 [anything] ip from any to 1.1.1.23
> >>>>>
> >>>>>
> >>>>>ipfw add 1132 skipto 1151 ip from any to 1.1.1.28./30
> >>>>>
> >>>>>ipfw add 1140 [anything] ip from any to 1.1.1.24
> >>>>>ipfw add 1141 [anything] ip from any to 1.1.1.25
> >>>>>ipfw add 1142 [anything] ip from any to 1.1.1.26
> >>>>>ipfw add 1143 [anything] ip from any to 1.1.1.27
> >>>>>
> >>>>>
> >>>>>ipfw add 1151 [anything] ip from any to 1.1.1.28
> >>>>>ipfw add 1152 [anything] ip from any to 1.1.1.29
> >>>>>ipfw add 1153 [anything] ip from any to 1.1.1.30
> >>>>>ipfw add 1154 [anything] ip from any to 1.1.1.31
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>now this example shows a binary search in IP space, written
> >>>>>(including
> >>>>>bugs) by hand
> >>>>>but if you are willing to write a suitable perl script, you can
> >>>>>generate a binary search in MAC address space
> >>>>>just as easily. just sort them into order and search..
> >>>>>
> >>>>>I'm not going to try it by had, but for 1600 hosts you should only
> >>>>>need to go through
> >>>>>15 rules per host on average, instead of 1600 rules per host.
> >>>>>that should cut down your ipfw cpu usage by 1/100
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>freebsd.org"
> >>>>>>
> >>>>>
> >>>
> >>>
> >>>
> >>
> _______________________________________________
> freebsd-net at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org"
--
John-Mark Gurney Voice: +1 415 225 5579
"All that I will do, has been done, All that I have, has not."
More information about the freebsd-net
mailing list