FreeBSD 7, bridge, PF and syn flood = very bad performance
Stefan Lambrev
stefan.lambrev at moneybookers.com
Sun Jan 27 01:30:35 PST 2008
Goo Day,
Max Laier wrote:
> On Saturday 26 January 2008, Stefan Lambrev wrote:
>
>> Max Laier wrote:
>>
>>> On Friday 25 January 2008, Stefan Lambrev wrote:
>>>
>>>> Greetings,
>>>>
>>>> Does anyone try to see PF with "keep state" in action when under syn
>>>> flood attack?
>>>> I tried to get some help in freebsd-pf@, because the test firewall,
>>>> that I build hardly can handle 2-5MB/s syn flood.
>>>> Unfortunately I do not saw useful advice.
>>>> The problem is that a quad core bridge firewall running freebsd 7
>>>> amd64 with PF is near useless and can't handle "small" SYN ddos.
>>>>
>>>> Here is the schema that I'm testing:
>>>> web server (freebsd) - freebsd (bridged interfaces) - gigabit switch
>>>> - clients + flooders
>>>> In this configuration ~25MB/s syn flood (and I think this limit is
>>>> because of my switch) is not a problem and the web server responds
>>>> without a problem.
>>>> With this configuration netperf -l 610 -p 10303 -H 10.3.3.1 shows
>>>> 116MB/s stable speed , so I guess there are no problems with cables,
>>>> hardware and etc :)
>>>>
>>>> But when I start pf (see below the config file) the traffic drops to
>>>> 2-3MB/s and the web server is hardly accessible.
>>>> It seems that device polling helps a lot in this situation, and at
>>>> least the bridge firewall is accessible. Without "polling" the
>>>> firewall is so heavily loaded
>>>> that even commands like "date" take few seconds to finish, with 2
>>>> cores at ~100% idle at same time.
>>>>
>>>> I have "flat profiles" from hwpmc, and I think it indicates a
>>>> problem:
>>>>
>>>> (bridge, pf enabled, polling enabled, sched_ule - I have profiles
>>>> and for other combinations too if needed)
>>>> % cumulative self self total
>>>> time seconds seconds calls ms/call ms/call name
>>>> 24.0 268416.00 268416.00 0 100.00%
>>>> _mtx_lock_sleep
>>>>
>>> Can you build a kernel with LOCK_PROFILING and try to figure out
>>> which lock is causing this?
>>>
>> Yes I can build kernel with LOCK_PROFILING.
>> But I have no idea how to use it :)
>> Can you point me to some documentation?
>>
>
> man LOCK_PROFILING
>
> basically:
> # sysctl debug.lock.prof.enable=1 && sleep 60 && \
> sysctl debug.lock.prof.enable=0 && \
> sysctl debug.lock.prof.stats > log
>
> while under attack to sample one minute of lock statistics.
>
>
Well I think the interesting lines from this experiment are:
max total wait_total count avg
wait_avg cnt_hold cnt_lock name
39 25328476 70950955 9015860 2 7
5854948 6309848 /usr/src/sys/contrib/pf/net/pf.c:6729 (sleep
mutex:pf task mtx)
936935 10645209 350 50 212904 7
110 47 /usr/src/sys/contrib/pf/net/pf.c:980 (sleep mutex:pf
task mtx)
41 10528492 1422891 1492295 7 0
155627 216812 /usr/src/sys/dev/em/if_em.c:980 (sleep mutex:em1)
26 5894103 2275517 2254004 2 1
427066 715901 /usr/src/sys/net/if_bridge.c:2082 (sleep
mutex:if_bridge)
34 5466679 118638 761766 7 0
1198 5794 /usr/src/sys/dev/em/if_em.c:980 (sleep mutex:em0)
24 4274965 1952823 2253930 1 0
201352 691434 /usr/src/sys/net/if_bridge.c:1991 (sleep
mutex:if_bridge)
28 3067953 800284 1492265 2 0
113423 294092 /usr/src/sys/net/if_bridge.c:1674 (sleep mutex:em1)
776401 1972047 0 69 28580 0
0 0 /usr/src/sys/kern/uipc_sockbuf.c:145 (sx:so_snd_sx)
775844 1970701 0 69 28560 0
1 0 /usr/src/sys/netinet/tcp_usrreq.c:779 (sleep mutex:inp)
22 1552808 922 761744 2
0 6 405 /usr/src/sys/dev/em/if_em.c:949 (sleep
mutex:em0)
19 1508717 94 761736 1 0
51 24 /usr/src/sys/net/if_bridge.c:1674 (sleep mutex:em0)
15 713930 7045 590468 1 0
1778 3364 /usr/src/sys/kern/kern_timeout.c:419 (spin mutex:callout)
9 693209 4395 579397 1 0
1305 2129 /usr/src/sys/kern/kern_timeout.c:500 (spin mutex:callout)
23 569860 423 88509 6 0
51 100 /usr/src/sys/kern/subr_taskqueue.c:71 (spin
mutex:fast_taskqueue)
46 489089 188 90306 5 0
6 7 /usr/src/sys/kern/subr_sleepqueue.c:232 (spin
mutex:sleepq chain)
102 488839 28464 19935 24 1
15840 5849 /usr/src/sys/dev/em/if_em.c:1563 (sleep mutex:em1)
70692 443077 0 24 18461 0
0 0 /usr/src/sys/sys/buf.h:280 (lockmgr:bufwait)
61 291437 6501 8148 35 0
5664 1610 /usr/src/sys/dev/em/if_em.c:1563 (sleep mutex:em0)
27 2760115 474506 1346693 2 0
102015 137670 /usr/src/sys/dev/em/if_em.c:949 (sleep mutex:em1)
246691 246691 0 1 246691 0
0 0 /usr/src/sys/netinet/tcp_timer.c:423 (sleep mutex:tcp)
13 121639 10 60134 2 0
0 2 /usr/src/sys/kern/kern_clock.c:224 (spin mutex:sched lock 0)
13 119466 1 60135 1 0
0 1 /usr/src/sys/kern/kern_clock.c:224 (spin mutex:sched lock 3)
9 111044 5 60134 1
0 0 1 /usr/src/sys/kern/kern_clock.c:224 (spin
mutex:sched lock 1)
107 107 246687 1 107 246687
0 1 /usr/src/sys/netinet/tcp_timer.c:438 (sleep mutex:inp)
you can see the whole file here - http://89.186.204.158/profiling.txt
--
Best Wishes,
Stefan Lambrev
ICQ# 24134177
More information about the freebsd-current
mailing list