FreeBSD 7.0 bridge tuning
Stefan Lambrev
stefan.lambrev at moneybookers.com
Mon Mar 17 08:14:21 UTC 2008
Greetings,
hugoboy at inbox.lv wrote:
> Hello!
>
> I'm trying to tune FreeBSD 7.0 bridge.
>
You may want to check this thread -
http://lists.freebsd.org/pipermail/freebsd-current/2008-January/082751.html
> Environment:
> Server - 2 x Xeon 3GHz, 2 x Gb LAN(em driver) + 1 LAN for management,
> 1GB RAM.
>
Can you tell us the exact CPU model? Is it dual core Xeon? It's not
clear how many cores you have ...
> Testers -2 x Sunrise Telecom 100Mbit Ethernet testers for traffic
> generation.
>
> What I have intended to achieve is to substitute proprietary traffic
> shaper Allot with FreeBSD traffic shaper(Bridge + PF + ALTQ).
> The minimum task is to make FreeBSD shaper to perform perfectly with
> 100Mbit traffic in all spectrum of packet lengths (from 64 bytes to
> at least 1518 bytes)
>
> The situation now:
> with pf turned off - there is no problem, bridge throughput is
> 100Mbit/s no packet loss (starting from 64 byte packets)
>
> With pf on I have statistics:
> packet lengt -> Mbit/s without packet loss
> 64 -> 46
> 100 -> 66
> 150 -> 94
>
>> 200 -> 100
>>
>
>
How many packets per second do you transmit? PF have some known
limitations, hopefully they will be addressed in 8-current
and back-ported someday to 7-STABLE.
> Lower configuration of kernel/sysctl is displayed.
>
> I don't know what else can I tune?
>
> It seems to me that bottleneck is somewhere around pf/kernel buffers
> of packet headers. I read somewhere that in bridging packet payload
> does not travel through all stack - just header is evaluated.
> In case of 64 byte packets in the same time unit there are more
> packets for the same bandwith on interfaces and as plain layer2
> bridge performs 100Mbit/s with no problem
> the problem is above layer2 :)
>
> btw: kern.polling.enable=1 does not help - at packetlength 64 bytes
> performance is 2x worse than with interrupts.
>
I noticed this too - polling is not very helpful with em driver. It
reduce the load but dropped packets are more with polling.
In my situation increasing kern.hz to 3000 yielded best results, you can
try to tune this.
> kernel:
> ---------------------------
>
> cpu I686_CPU
> ident ALLOT
>
> # To statically compile in device wiring instead of
> /boot/device.hints
> #hints "GENERIC.hints" # Default places to look for
> devices.
>
> makeoptions DEBUG=-g # Build kernel with gdb(1)
> debug symbols
>
> options SCHED_ULE # ULE scheduler
> #options SCHED_4BSD # 4BSD scheduler
> options PREEMPTION # Enable kernel thread
> preemption
> options INET # InterNETworking
> #options INET6 # IPv6 communications
> protocols
> #options SCTP # Stream Control Transmission
> Protocol
> options FFS # Berkeley Fast Filesystem
> options SOFTUPDATES # Enable FFS soft updates
> support
> options UFS_ACL # Support for access control
> lists
> options UFS_DIRHASH # Improve performance on big
> directories
> options UFS_GJOURNAL # Enable gjournal-based UFS
> journaling
> options MD_ROOT # MD is a potential root
> device
> options NFSCLIENT # Network Filesystem Client
> options NFSSERVER # Network Filesystem Server
> options NFS_ROOT # NFS usable as /, requires
> NFSCLIENT
> options MSDOSFS # MSDOS Filesystem
> options CD9660 # ISO 9660 Filesystem
> options PROCFS # Process filesystem
> (requires PSEUDOFS)
> options PSEUDOFS # Pseudo-filesystem framework
> options GEOM_PART_GPT # GUID Partition Tables.
> options GEOM_LABEL # Provides labelization
> options COMPAT_43TTY # BSD 4.3 TTY compat [KEEP
> THIS!]
> options COMPAT_FREEBSD4 # Compatible with FreeBSD4
> options COMPAT_FREEBSD5 # Compatible with FreeBSD5
> options COMPAT_FREEBSD6 # Compatible with FreeBSD6
> options SCSI_DELAY=5000 # Delay (in ms) before
> probing SCSI
> options KTRACE # ktrace(1) support
> options SYSVSHM # SYSV-style shared memory
> options SYSVMSG # SYSV-style message queues
> options SYSVSEM # SYSV-style semaphores
> options _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B
> real-time extensions
> options KBD_INSTALL_CDEV # install a CDEV entry in
> /dev
> options ADAPTIVE_GIANT # Giant mutex is adaptive.
> options STOP_NMI # Stop CPUS using NMI instead
> of IPI
> options AUDIT # Security event auditing
>
> options ALTQ
> options ALTQ_CBQ
> options ALTQ_RED
> options ALTQ_RIO
> options ALTQ_HFSC
> options ALTQ_CDNR
> options ALTQ_PRIQ
> options ALTQ_NOPCC
> options HZ=1000
> options DEVICE_POLLING
> options IPSTEALTH
> options ZERO_COPY_SOCKETS
> options MPTABLE_FORCE_HTT # Enable HTT CPUs with the MP Table
> options IPI_PREEMPTION
>
> # To make an SMP kernel, the next two lines are needed
> options SMP # Symmetric MultiProcessor
> Kernel
> device apic # I/O APIC
> --------------------------------
>
> /etc/sysctl.conf
> #kern.polling.enable=1
> kern.ipc.nmbcluster=32768
> kern.ipc.maxsockbufs=2097152
> kern.ipc.somaxconn=8192
> kern.maxfiles=65536
> kern.maxfilesperproc=32768
> net.inet.tcp.delayed_ack=0
> net.inet.tcp.sendspace=65535
> net.inet.udp.recvspace=65535
> net.inet.udp.maxdgram=57344
> net.local.stream.recvspace=65535
> net.local.stream.sendspace=65535
> kern.polling.user_frac=20
> net.isr.direct=0
> net.inet.ip.forwarding=1
> -------------------------------
>
> P.S. I tried pfSense, but as we have used Allot before - we need to
> see queue statistics in graphs per queue, pfSense just offers
> numbers..
> Seems to me that pFsense is good for many things but not for
> bridge+traffic shapeing - correct me if I'm wrong.
>
> Best regards,
> Ugis
>
>
> _______________________________________________
> freebsd-performance at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-performance
> To unsubscribe, send any mail to "freebsd-performance-unsubscribe at freebsd.org"
>
More information about the freebsd-performance
mailing list