network performance

Mon Jan 21 10:58:53 PST 2008

Greetings,

I'm trying test a bridge firewall under FreeBSD 7.

What I have as configuration is:

Freebsd7 (web server) - bridge (FreeBSD7) - gigabit switch - flooders.

Both FreeBSD servers are using  FreeBSD 7.0-RC1 amd64
With netperf -l 60 -p 10303 -H 10.3.3.1 I have no problems to reach 116MB/s
with and without pf enabled.

But what I want to test is how well will perform the firewall during syn 
floods.
For this I'm using hping3 (hping-devel in ports) to generate traffic 
from flooders
to the web server.

First think, that I notice is, that hping running on linux generate 
twice more traffic compared to freebsd.
So I plan to separate a server with dual bootable linux and fbsd and to 
see what's the real difference.

Second problem that I encountered is, that when running hping from freebsd.
It exits after few seconds/minutes with this error message:
[send_ip] sendto: No buffer space available
And this happens on FreeBSD_7 and FreeBSD 6.2-p8 too amd64)

Can I increase those buffers ?

I'm able to generate 24MB/s SYN flood and during my test I can see this 
on the bridge firewall:
netstat -w 1 -I em0 -d - external network
            input          (em0)           output
   packets  errs      bytes    packets  errs      bytes colls drops
    427613  1757   25656852     233604     0   14016924     0     0
    428089  1274   25685358     233794     0   14025174     0     0
    427433  1167   25645998     234775     0   14088834     0     0
    438270  2300   26296218     233384     0   14004474     0     0
    438425  2009   26305518     233858     0   14034114     0     0

and from the internal network:
            input          (em1)           output
   packets  errs      bytes    packets  errs      bytes colls drops
    232912     0   13974838     425796     0   25549446     0  1334
    234487     0   14069338     423986     0   25432026     0  1631
    233951     0   14037178     431330     0   25880286     0  3888
    233509     0   14010658     436496     0   26191986     0  1437
    234181     0   14050978     430291     0   25816806     0  4001
    234144     0   14048870     430208     0   25810206     0  1621
    234176     0   14050678     430292     0   25828926     0  3001

And here is top -S

last pid: 21830;  load averages:  1.01,  0.50,  
0.72                                                                                         
up 3+04:59:43  20:27:49
84 processes:  7 running, 60 sleeping, 17 waiting
CPU states:  0.0% user,  0.0% nice, 38.2% system,  0.0% interrupt, 61.8% 
idle
Mem: 17M Active, 159M Inact, 252M Wired, 120K Cache, 213M Buf, 1548M Free
Swap: 4056M Total, 4056M Free

  PID USERNAME  THR PRI NICE   SIZE    RES STATE  C   TIME   WCPU COMMAND
   14 root        1 171 ki31     0K    16K CPU0   0  76.8H 100.00% idle: 
cpu0
   11 root        1 171 ki31     0K    16K RUN    3  76.0H 100.00% idle: 
cpu3
   25 root        1 -68    -     0K    16K CPU1   1  54:26 86.28% em0 taskq
   26 root        1 -68    -     0K    16K CPU2   2  39:13 66.70% em1 taskq
   12 root        1 171 ki31     0K    16K RUN    2  76.0H 37.50% idle: cpu2
   13 root        1 171 ki31     0K    16K RUN    1  75.9H 16.89% idle: cpu1
   16 root        1 -32    -     0K    16K WAIT   0   7:00  0.00% swi4: 
clock sio
   51 root        1  20    -     0K    16K syncer 3   4:30  0.00% syncer

vmstat -i
interrupt                          total       rate
irq1: atkbd0                         544          0
irq4: sio0                         10641          0
irq14: ata0                            1          0
irq19: uhci1+                     123697          0
cpu0: timer                    553887702       1997
irq256: em0                     48227501        173
irq257: em1                     46331164        167
cpu1: timer                    553887682       1997
cpu3: timer                    553887701       1997
cpu2: timer                    553887701       1997
Total                         2310244334       8333

netstat -m
594/2361/2955 mbufs in use (current/cache/total)
592/1854/2446/204800 mbuf clusters in use (current/cache/total/max)
592/1328 mbuf+clusters out of packet secondary zone in use (current/cache)
0/183/183/12800 4k (page size) jumbo clusters in use 
(current/cache/total/max)
0/0/0/6400 9k jumbo clusters in use (current/cache/total/max)
0/0/0/3200 16k jumbo clusters in use (current/cache/total/max)
1332K/5030K/6362K bytes allocated to network (current/cache/total)

systat -ifstat
      Interface           Traffic               Peak                Total
        bridge0  in     38.704 MB/s         38.704 MB/s          185.924 GB
                 out    38.058 MB/s         38.058 MB/s          189.855 GB

            em1  in     13.336 MB/s         13.402 MB/s           51.475 GB
                 out    24.722 MB/s         24.722 MB/s          137.396 GB

            em0  in     24.882 MB/s         24.882 MB/s          138.918 GB
                 out    13.336 MB/s         13.403 MB/s           45.886 GB

Both FreeBSD servers have quad port intel network card, 2GB memory
em0 at pci0:3:0:0: class=0x020000 card=0x10bc8086 chip=0x10bc8086 rev=0x06 
hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82571EB Gigabit Ethernet Controller (Copper)'
    class      = network
    subclass   = ethernet

Firewall server is running on CPU: Intel(R) Xeon(R) X3220  @ 2.40GHz 
(quad core)
Web server is running on Intel(R) Xeon(R) CPU 3070  @ 2.66GHz (dual core)

So in brief how can I get rid of "No buffer space available",
increase the sent rate of hping in FreeBSD and get rid of dropped 
packets on rates like 24MB/s :)
What other tests can I run (switching on of cpu cores and etc)?
Anyone interested?

P.S. I'm using custom kernel, with SCHED_ULE, both freebsds build from 
source with CPUTYPE?=core2
and net.inet.icmp.icmplim_output=0

-- 

Best Wishes,
Stefan Lambrev
ICQ# 24134177