FreeBSD 7, bridge, PF and syn flood = very bad performance
Max Laier
max at love2party.net
Sun Jan 27 05:24:39 PST 2008
On Sunday 27 January 2008, Stefan Lambrev wrote:
> Goo Day,
>
> Max Laier wrote:
> > On Saturday 26 January 2008, Stefan Lambrev wrote:
> >> Max Laier wrote:
> >>> On Friday 25 January 2008, Stefan Lambrev wrote:
> >>>> Greetings,
> >>>>
> >>>> Does anyone try to see PF with "keep state" in action when under
> >>>> syn flood attack?
> >>>> I tried to get some help in freebsd-pf@, because the test
> >>>> firewall, that I build hardly can handle 2-5MB/s syn flood.
> >>>> Unfortunately I do not saw useful advice.
> >>>> The problem is that a quad core bridge firewall running freebsd 7
> >>>> amd64 with PF is near useless and can't handle "small" SYN ddos.
> >>>>
> >>>> Here is the schema that I'm testing:
> >>>> web server (freebsd) - freebsd (bridged interfaces) - gigabit
> >>>> switch - clients + flooders
> >>>> In this configuration ~25MB/s syn flood (and I think this limit is
> >>>> because of my switch) is not a problem and the web server responds
> >>>> without a problem.
> >>>> With this configuration netperf -l 610 -p 10303 -H 10.3.3.1 shows
> >>>> 116MB/s stable speed , so I guess there are no problems with
> >>>> cables, hardware and etc :)
> >>>>
> >>>> But when I start pf (see below the config file) the traffic drops
> >>>> to 2-3MB/s and the web server is hardly accessible.
> >>>> It seems that device polling helps a lot in this situation, and at
> >>>> least the bridge firewall is accessible. Without "polling" the
> >>>> firewall is so heavily loaded
> >>>> that even commands like "date" take few seconds to finish, with 2
> >>>> cores at ~100% idle at same time.
> >>>>
> >>>> I have "flat profiles" from hwpmc, and I think it indicates a
> >>>> problem:
> >>>>
> >>>> (bridge, pf enabled, polling enabled, sched_ule - I have profiles
> >>>> and for other combinations too if needed)
> >>>> % cumulative self self total
> >>>> time seconds seconds calls ms/call ms/call name
> >>>> 24.0 268416.00 268416.00 0 100.00%
> >>>> _mtx_lock_sleep
> >>>
> >>> Can you build a kernel with LOCK_PROFILING and try to figure out
> >>> which lock is causing this?
> >>
> >> Yes I can build kernel with LOCK_PROFILING.
> >> But I have no idea how to use it :)
> >> Can you point me to some documentation?
> >
> > man LOCK_PROFILING
> >
> > basically:
> > # sysctl debug.lock.prof.enable=1 && sleep 60 && \
> > sysctl debug.lock.prof.enable=0 && \
> > sysctl debug.lock.prof.stats > log
> >
> > while under attack to sample one minute of lock statistics.
>
> Well I think the interesting lines from this experiment are:
> max total wait_total count avg
> wait_avg cnt_hold cnt_lock name
> 39 25328476 70950955 9015860 2 7
> 5854948 6309848 /usr/src/sys/contrib/pf/net/pf.c:6729 (sleep
> mutex:pf task mtx)
> 936935 10645209 350 50 212904 7
> 110 47 /usr/src/sys/contrib/pf/net/pf.c:980 (sleep mutex:pf
> task mtx)
Yeah, those two mostly are the culprit, but a quick fix is not really
available. You can try to "set timeout interval" to something bigger
(e.g. 60 seconds) which will decrease the average hold time of the second
lock instance at the cost of increased peak memory usage.
I have the ideas how to fix this, but it will take much much more time
than I currently have for FreeBSD :-\ In general this requires a bottom
up redesign of pf locking and some data structures involved in the state
tree handling.
The first(=main) lock instance is also far from optimal (i.e. pf is a
congestion point in the bridge forwarding path). For this I have also a
plan how to make at least state table lookups run in parallel to some
extend, but again the lack of free time to spend coding prevents me from
doing it at the moment :-\
--
/"\ Best regards, | mlaier at freebsd.org
\ / Max Laier | ICQ #67774661
X http://pf4freebsd.love2party.net/ | mlaier at EFnet
/ \ ASCII Ribbon Campaign | Against HTML Mail and News
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 187 bytes
Desc: This is a digitally signed message part.
Url : http://lists.freebsd.org/pipermail/freebsd-current/attachments/20080127/679f392d/attachment.pgp
More information about the freebsd-current
mailing list