Packet loss with traffic shaper and routing

Wed May 3 00:38:14 UTC 2006

tpeixoto at widesoft.com.br wrote:

ok gotcha

probably the pipes are stored in a list or something.

check the code and see if a hash table woudl be better..

>
> Julian Elischer wrote:
>
>> tpeixoto at widesoft.com.br wrote:
>>
>>> Hello.
>>> I think I should give some 'real world' examples.
>>>
>>>
>>> /etc/rc.firewall:
>>>
>>> [Ss][Hh][Aa][Pp][Ee][Rr])
>>> setup_loopback
>>>
>>> . /etc/rc.shaper
>>>
>>> ${fwcmd} add 65000 pass all from any to any
>>> ;;
>>>
>>>
>>> /etc/rc.shaper:
>>>
>>> ${fwcmd} pipe 1 config bw 512Kbit/s
>>> ${fwcmd} pipe 2 config bw 512Kbit/s
>>> ${fwcmd} add pipe 1 all from any to any MAC any 00:11:22:33:44:55 in
>>> ${fwcmd} add pipe 2 all from any to any MAC 00:11:22:33:44:55 any out
>>> ${fwcmd} pipe 3 config bw 256Kbit/s
>>> ${fwcmd} pipe 4 config bw 256Kbit/s
>>> ${fwcmd} add pipe 3 all from any to any MAC any 66:77:88:99:aa:bb in
>>> ${fwcmd} add pipe 4 all from any to any MAC 66:77:88:99:aa:bb any out
>>> ${fwcmd} pipe 5 config bw 128Kbit/s
>>> ${fwcmd} pipe 6 config bw 128Kbit/s
>>> ${fwcmd} add pipe 5 all from any to any MAC any 00:01:02:03:04:05 in
>>> ${fwcmd} add pipe 6 all from any to any MAC 00:01:02:03:04:05 any out
>>> ${fwcmd} pipe 7 config bw 512Kbit/s
>>> ${fwcmd} pipe 8 config bw 1024Kbit/s
>>> ${fwcmd} add pipe 7 all from any to any MAC any 06:07:08:09:0a:0b in
>>> ${fwcmd} add pipe 8 all from any to any MAC 06:07:08:09:0a:0b any out
>>> ${fwcmd} pipe 9 config bw 64Kbit/s
>>> ${fwcmd} pipe 10 config bw 64Kbit/s
>>> ${fwcmd} add pipe 9 all from any to any MAC any ab:cd:ef:00:11:22 in
>>> ${fwcmd} add pipe 10 all from any to any MAC ab:cd:ef:00:11:22 any out
>>>  
>>>
>> OK, so, put the MACs in numerical order:
>>
>> 00:01:02:03:04:05
>> 00:11:22:33:44:55
>> 06:07:08:09:0a:0b
>> 66:77:88:99:aa:bb
>> ab:cd:ef:00:11:22
>>
>>
>> work out MASKS that divide them into a binary set.
>>
>> e.g.
>> 1 skipto 10 all from any to not MAC 00:00:00:00:00:00/8
>> 2 skipto 5 all from any to not MAC 00:01:00:00:00:00/16
>> 3 pipe 1 ip from any to any
>> 5 pipe 2 ip from any to any
>>
>> 10 skipto 12 all from any to not MAC 06:00:00:00:00:00/8
>> 11 pipe 3 all from any to any
>> 12 skipto 14 all from any to not MAC 66:00:00:00:00:00/8
>> 13 pipe 4 all from any to any
>> 14 pipe 5 all from any to any
>>
>> now, if you continue this on, you will run 16 rules to divide the 
>> 1600 rules up to find the right pipe.
>>
>
> I got your point.
> But what I am telling is that it's not the search or it's not _only_ 
> the search in the firewall rules that is making the interrupts go high.
> Please, see below.
>
>
>>>
>>> This example is for 5 clients. We have 1600.
>>> As you can see, there are 2 rules and 2 pipes per host, not 1600.
>>>
>>>
>>> If we try rc.firewall like this...
>>>
>>> setup_loopback
>>> ${fwcmd} add 65000 pass all from any to any
>>>
>>> ... we are ok. Interrupts are low.
>>>
>>> So, following your line of thought, I tried a simple test...
>>>
>>> setup_loopback
>>> ${fwcmd} skipto 65000 ip from any to any MAC any any
>>> . /etc/rc.shaper
>>> ${fwcmd} add 65000 pass all from any to any
>>>
>>> This way, the packets will never pass through shaper rules, but 
>>> interrupts
>>> still get very high.
>>>  
>>>
>>
>> I don't see how that proves anything
>>
>
> See, if we have just 4 rules in the kernel (3 from setup_loopback + 
> allow any to any), we don't have problems with interrupts. They are 
> low, about 15~20% with the same traffic.
> But, if we have a 'full' set of rules, let's say 3205 (3 from 
> setup_loopback + skipto 65000 + 3200 pipes + allow any to any), where 
> only 5 of them are being matched (setup_loopback, 'skipto 65000' and 
> 'allow any to any' - the skipto 65000 rule prevents any packet to 
> search through my 3200 pipes, right?), we still see interrupts go to 
> 70~90%.
> So, what I am saying is that even if we use skipto rules to create 
> 'shortcuts' in the firewall stack, the system still uses lots of 
> interrupts. It seems that no matter whether the packets are being 
> checked against the rules or not, as long there are so many rules, the 
> interrupts will be generated.
>
> Let me know if you got my point.
> I'll do some more tests reducing the number of pipes while keeping the 
> same amount of rules to see whether this has some effect in the 
> interrupts.
>
> BTW: I tested your other suggestion about splitting 'in' and 'out' 
> rules but it made no difference regarding system interrupts.
>
> Thanks again!
>
>
>>> Basically, we need a solution to shape each MAC address with its 
>>> specifics
>>> download e upload speeds.
>>> Given the tests, I don't see how skipto can help, but if you believe 
>>> that
>>> tablearg (which I am not familiar with) might help, we can try it with
>>> 7.x.
>>>  
>>>
>>
>> Tablearg only works with IP addresses.
>>
>>> Thanks.
>>>
>>>
>>>  
>>>
>>>> oops, forgot to fix my cut-n- pastes.. corrected triage below..
>>>>
>>>>
>>>> Julian Elischer wrote:
>>>>
>>>>  
>>>>
>>>>> Julian Elischer wrote:
>>>>>
>>>>>    
>>>>>
>>>>>> tpeixoto at widesoft.com.br wrote:
>>>>>>
>>>>>>      
>>>>>>
>>>>>>>> That would do it..
>>>>>>>>
>>>>>>>> In all versions of FreeBSD
>>>>>>>> you can use the skipto rule to make sure that only a few rules are
>>>>>>>> run for any
>>>>>>>> address. Use it to to a binary search for the right pipe.'
>>>>>>>> carefully using 'skipto' and 'table' can make it efficient to do
>>>>>>>> very complex
>>>>>>>> filters like that.
>>>>>>>>
>>>>>>>>           
>>>>>>>
>>>>>>> Sorry, but I didn't realized how to use that as we have to shape
>>>>>>> each user individually, i.e., each MAC address on the LAN has its
>>>>>>> own download and upload speeds.
>>>>>>>
>>>>>>> Could you clarify how to improve the situation with the tools you
>>>>>>> mentioned?
>>>>>>>         
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Assuming you can not use "tablearg" yet (it will make this REALLY 
>>>>>> EASY)
>>>>>> then if you have 30 IPs you want to shape from 1.1.1.1 to 1.1.1.30
>>>>>>       
>>>>>
>>>>>
>>>>>
>>>>> then, consider the following example using IP addresses.
>>>>>
>>>>>    
>>>>>
>>>>>>
>>>>>>       
>>>>>
>>>>> ipfw add 1000 skipto 1110 ip from any to 1.1.1.16/28
>>>>> ipfw add 1010 skipto 1032 ip from any to 1.1.1.8/29
>>>>> ipfw add 1012 skipto 1021 ip from any to 1.1.1.4./30
>>>>>     ipfw add 1013 [anything] ip from any to 1.1.1.0
>>>>>     ipfw add 1014 [anything] ip from any to 1.1.1.1
>>>>> ipfw add 1015 [anything] ip from any to 1.1.1.2
>>>>> ipfw add 1016 [anything] ip from any to 1.1.1.3
>>>>>
>>>>>
>>>>> ipfw add 1021 anything] ip from any to 1.1.1.4
>>>>> ipfw add 1022 [anything] ip from any to 1.1.1.5
>>>>> ipfw add 1023 [anything] ip from any to 1.1.1.6
>>>>> ipfw add 1024 [anything] ip from any to 1.1.1.7
>>>>>
>>>>>
>>>>> ipfw add 1032 skipto 1051 ip from any to 1.1.1.12./30
>>>>>
>>>>> ipfw add 1040 [anything] ip from any to 1.1.1.8
>>>>> ipfw add 1041 [anything] ip from any to 1.1.1.9
>>>>> ipfw add 1042 [anything] ip from any to 1.1.1.10
>>>>> ipfw add 1043 [anything] ip from any to 1.1.1.11
>>>>>
>>>>>
>>>>> ipfw add 1051 [anything] ip from any to 1.1.1.12
>>>>> ipfw add 1052 [anything] ip from any to 1.1.1.13
>>>>> ipfw add 1053 [anything] ip from any to 1.1.1.14
>>>>> ipfw add 1054 [anything] ip from any to 1.1.1.15
>>>>>
>>>>>
>>>>> ipfw add 1110 skipto 1132 ip from any to 1.1.1.24/29
>>>>> ipfw add 1112 skipto 1121 ip from any to 1.1.1.20./30
>>>>> ipfw add 1113 [anything] ip from any to 1.1.1.16
>>>>> ipfw add 1114 [anything] ip from any to 1.1.1.17
>>>>> ipfw add 1115 [anything] ip from any to 1.1.1.18
>>>>>     ipfw add 1116 [anything] ip from any to 1.1.1.19
>>>>>     ipfw add 1121 anything] ip from any to 1.1.1.20
>>>>> ipfw add 1122 [anything] ip from any to 1.1.1.21
>>>>> ipfw add 1123 [anything] ip from any to 1.1.1.22
>>>>> ipfw add 1124 [anything] ip from any to 1.1.1.23
>>>>>
>>>>>
>>>>> ipfw add 1132 skipto 1151 ip from any to 1.1.1.28./30
>>>>>
>>>>> ipfw add 1140 [anything] ip from any to 1.1.1.24
>>>>> ipfw add 1141 [anything] ip from any to 1.1.1.25
>>>>> ipfw add 1142 [anything] ip from any to 1.1.1.26
>>>>> ipfw add 1143 [anything] ip from any to 1.1.1.27
>>>>>
>>>>>
>>>>> ipfw add 1151 [anything] ip from any to 1.1.1.28
>>>>> ipfw add 1152 [anything] ip from any to 1.1.1.29
>>>>> ipfw add 1153 [anything] ip from any to 1.1.1.30
>>>>> ipfw add 1154 [anything] ip from any to 1.1.1.31
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> now this example shows a binary search in IP space, written 
>>>>> (including
>>>>> bugs) by hand
>>>>> but if you are willing to write a suitable perl script, you can
>>>>> generate a binary search in MAC address space
>>>>> just as easily. just sort them into order and search..
>>>>>
>>>>> I'm not going to try it by had, but for 1600 hosts you should only
>>>>> need to go through
>>>>> 15 rules per host on average, instead of 1600 rules per host.
>>>>> that should cut down your ipfw cpu usage by 1/100
>>>>>
>>>>>
>>>>>
>>>>>    
>>>>>
>>>>>> freebsd.org"
>>>>>>       
>>>>>
>>>
>>>  
>>>
>>