FreeBSD 7, bridge, PF and syn flood = very bad performance

Stefan Lambrev stefan.lambrev at moneybookers.com
Sun Jan 27 01:30:35 PST 2008


Goo Day,

Max Laier wrote:
> On Saturday 26 January 2008, Stefan Lambrev wrote:
>   
>> Max Laier wrote:
>>     
>>> On Friday 25 January 2008, Stefan Lambrev wrote:
>>>       
>>>> Greetings,
>>>>
>>>> Does anyone try to see PF with "keep state" in action when under syn
>>>> flood attack?
>>>> I tried to get some help in freebsd-pf@, because the test firewall,
>>>> that I build hardly can handle 2-5MB/s syn flood.
>>>> Unfortunately I do not saw useful advice.
>>>> The problem is that a quad core bridge firewall running freebsd 7
>>>> amd64 with PF is near useless and can't handle "small" SYN ddos.
>>>>
>>>> Here is the schema that I'm testing:
>>>> web server (freebsd) - freebsd (bridged interfaces) - gigabit switch
>>>> - clients + flooders
>>>> In this configuration ~25MB/s syn flood (and I think this limit is
>>>> because of my switch) is not a problem and the web server responds
>>>> without a problem.
>>>> With this configuration netperf -l 610 -p 10303 -H 10.3.3.1 shows
>>>> 116MB/s stable speed , so I guess there are no problems with cables,
>>>> hardware and etc :)
>>>>
>>>> But when I start pf (see below the config file) the traffic drops to
>>>> 2-3MB/s and the web server is hardly accessible.
>>>> It seems that device polling helps a lot in this situation, and at
>>>> least the bridge firewall is accessible. Without "polling" the
>>>> firewall is so heavily loaded
>>>> that even commands like "date" take few seconds to finish, with 2
>>>> cores at ~100% idle at same time.
>>>>
>>>> I have "flat profiles" from hwpmc, and I think it indicates a
>>>> problem:
>>>>
>>>> (bridge, pf enabled, polling enabled, sched_ule - I have profiles
>>>> and for other combinations too if needed)
>>>>   %   cumulative   self              self     total
>>>>  time   seconds   seconds    calls  ms/call  ms/call  name
>>>>  24.0  268416.00 268416.00        0  100.00%          
>>>> _mtx_lock_sleep
>>>>         
>>> Can you build a kernel with LOCK_PROFILING and try to figure out
>>> which lock is causing this?
>>>       
>> Yes I can build kernel with LOCK_PROFILING.
>> But I have no idea how to use it :)
>> Can you point me to some documentation?
>>     
>
> man LOCK_PROFILING
>
> basically:
> # sysctl debug.lock.prof.enable=1 && sleep 60 && \
>   sysctl debug.lock.prof.enable=0 && \
>   sysctl debug.lock.prof.stats > log
>
> while under attack to sample one minute of lock statistics.
>
>   
Well I think the interesting lines from this experiment are:
max              total            wait_total       count   avg 
wait_avg     cnt_hold     cnt_lock name
    39            25328476     70950955     9015860     2     7      
5854948      6309848 /usr/src/sys/contrib/pf/net/pf.c:6729 (sleep 
mutex:pf task mtx)
936935        10645209          350          50 212904     7          
110           47 /usr/src/sys/contrib/pf/net/pf.c:980 (sleep mutex:pf 
task mtx)
    41            10528492      1422891     1492295     7     0       
155627       216812 /usr/src/sys/dev/em/if_em.c:980 (sleep mutex:em1)
    26            5894103      2275517     2254004     2     1       
427066       715901 /usr/src/sys/net/if_bridge.c:2082 (sleep 
mutex:if_bridge)
    34            5466679       118638      761766     7     0         
1198         5794 /usr/src/sys/dev/em/if_em.c:980 (sleep mutex:em0)
    24            4274965      1952823     2253930     1     0       
201352       691434 /usr/src/sys/net/if_bridge.c:1991 (sleep 
mutex:if_bridge)
    28            3067953       800284     1492265     2     0       
113423       294092 /usr/src/sys/net/if_bridge.c:1674 (sleep mutex:em1)
776401        1972047            0          69 28580     0            
0            0 /usr/src/sys/kern/uipc_sockbuf.c:145 (sx:so_snd_sx)
775844        1970701            0          69 28560     0            
1            0 /usr/src/sys/netinet/tcp_usrreq.c:779 (sleep mutex:inp)
    22            1552808          922      761744     2     
0            6          405 /usr/src/sys/dev/em/if_em.c:949 (sleep 
mutex:em0)
    19            1508717           94      761736     1     0           
51           24 /usr/src/sys/net/if_bridge.c:1674 (sleep mutex:em0)
    15            713930         7045      590468     1     0         
1778         3364 /usr/src/sys/kern/kern_timeout.c:419 (spin mutex:callout)
     9             693209         4395      579397     1     0         
1305         2129 /usr/src/sys/kern/kern_timeout.c:500 (spin mutex:callout)
    23            569860          423       88509     6     0           
51          100 /usr/src/sys/kern/subr_taskqueue.c:71 (spin 
mutex:fast_taskqueue)
    46            489089          188       90306     5     0            
6            7 /usr/src/sys/kern/subr_sleepqueue.c:232 (spin 
mutex:sleepq chain)
   102           488839        28464       19935    24     1        
15840         5849 /usr/src/sys/dev/em/if_em.c:1563 (sleep mutex:em1)
 70692         443077            0          24 18461     0            
0            0 /usr/src/sys/sys/buf.h:280 (lockmgr:bufwait)
    61            291437         6501        8148    35     0         
5664         1610 /usr/src/sys/dev/em/if_em.c:1563 (sleep mutex:em0)
    27            2760115       474506     1346693     2     0       
102015       137670 /usr/src/sys/dev/em/if_em.c:949 (sleep mutex:em1)
246691        246691            0           1 246691     0            
0            0 /usr/src/sys/netinet/tcp_timer.c:423 (sleep mutex:tcp)
    13            121639           10       60134     2     0            
0            2 /usr/src/sys/kern/kern_clock.c:224 (spin mutex:sched lock 0)
    13            119466            1       60135     1     0            
0            1 /usr/src/sys/kern/kern_clock.c:224 (spin mutex:sched lock 3)
     9             111044            5       60134     1     
0            0            1 /usr/src/sys/kern/kern_clock.c:224 (spin 
mutex:sched lock 1)
   107           107       246687           1   107 246687            
0            1 /usr/src/sys/netinet/tcp_timer.c:438 (sleep mutex:inp)

you can see the whole file here - http://89.186.204.158/profiling.txt

-- 

Best Wishes,
Stefan Lambrev
ICQ# 24134177



More information about the freebsd-current mailing list