Network sysctl tuning [was Re: ZFSKnownProblems - needs
revision?]
Freddie Cash
fjwcash at gmail.com
Thu Apr 9 23:10:14 UTC 2009
On Wed, Apr 8, 2009 at 3:55 PM, Antony Mawer <fbsd-stable at mawer.org> wrote:
> Freddie Cash wrote:
> ...
>> We've also heavily modified /etc/sysctl.conf and upped a bunch of the
>> network-related sysctls. Doing so increased our SSH throughput from ~30
>> Mbits/sec across all connections to over 90 Mbits/sec per SSH connection.
>
> Are you able to share any of these with the list? It would be useful to
> compare as a lot of these tunings people do individually and it would be
> good to allow others to test in their environments to see if they help, as
> well as potentially adding them to the tuning man-page.
They're all taken from the HPN-SSH website and various google searches
related to HPN-enabled OpenSSH.
I don't know exactly what all the different, individual sysctls do,
nor whether this is the most optimal setup, but here's the sysctl.conf
that we use. This is on 2 systems using a quad-port gigabit NIC where
the top two ports are connected via lagg(4) and the bottom two ports
are connected via lagg(4), with the two laggX interfaces on separate
networks.
I did a bunch of scp/sftp transfers of 100 MB files filled with random
data pulled from /dev/random between these two boxes tweaking the
options one at a time, but didn't do too much in the way of
scientific/empirical measurements and comparisons beyond the
throughput data that scp/sftp shows.
If there are any glaring errors, gotchas, or "why would you ever do
that"s, let me know. :)
# General network settings
net.isr.direct=1 # Whether to enable Direct
Dispatch for netisr
# IP options
net.inet.ip.forwarding=0 # Whether to enable packet
forwarding for NAT/routing
net.inet.ip.process_options=0 # Disable processing of IP
options (nothing uses this field)
net.inet.ip.random_id=1 # Randomise the IP header ID number
net.inet.ip.redirect=0 # Whether to allow redirect packets
#net.inet.ip.stealth=0 # Whether to appear in traceroute output
# ICMP options
net.inet.icmp.icmplim=200 # Limit ICMP packets to this
many per second
net.inet.icmp.drop_redirect=1 # Drop ICMP redirect packets
net.inet.icmp.log_redirect=0 # Don't log ICMP redirect packets
# TCP options
net.inet.tcp.blackhole=1 # Drop packets destined to unused ports
net.inet.tcp.inflight.enable=0 # Use automatic TCP window-scaling
net.inet.tcp.log_in_vain=0 # Don't log the blackholed packets
net.inet.tcp.path_mtu_discovery=1 # Use ICMP type 3 to find the MTU to use
net.inet.tcp.recvbuf_max=16777216 # The max size of the receive
buffer (16 MB)
net.inet.tcp.recvspace=131072 # The initial size in bytes of
the receive buffer
net.inet.tcp.sack.enable=1 # Enable Selective ACKs
net.inet.tcp.sendbuf_max=16777216 # The max size of the send buffer
net.inet.tcp.sendspace=131072 # The initial size in bytes of
the send buffer
net.inet.tcp.syncookies=1 # Enable SYN cookie protection
net.inet.tcp.rfc1323=1 # Enable RFC1323 extensions
(TCP window scaling)
# UDP options
net.inet.udp.blackhole=1 # Drop packets destined to unused ports
net.inet.udp.checksum=1 # Enable UDP checksums
net.inet.udp.log_in_vain=0 # Don't log the blackholed packets
net.inet.udp.recvspace=65536 # Size in bytes of the receive buffer
# Debug options
debug.minidump=1 # Disable the small kernel
core dump (only mem in use)
debug.mpsafevfs=1 # Enable threaded VFS subsystem
# Kernel options
kern.coredump=0 # Disable kernel core dumps
kern.ipc.maxsockbuf=4194304 # Set the max size of the
socket buffers (4 MB)
kern.ipc.somaxconn=1024 # Expand the IP listen queue
kern.maxvnodes=250000 # Bump up the max number of vnodes
# PCI bus options
hw.pci.enable_msix=1 # Enable Message Signalled
Interrupts - Extended
hw.pci.enable_msi=1 # Enable Message Signalled Interrupts
hw.pci.enable_io_modes=1 # Enable alternate I/O access modes
--
Freddie Cash
fjwcash at gmail.com
More information about the freebsd-stable
mailing list