Tor on FreeBSD Performance issues

Julian Wissmann julianwissmann at gmail.com
Mon Feb 6 21:09:59 UTC 2012


Hi,

I'm an admin for a non-profit that runs Tor exit nodes, most of them on Linux currently, but due to problems on our high bandwidth nodes, we decided to migrate one of them to FreeBSD to do some testing. 
I've been using FreeBSD for quite some years now, longer than Linux, so I figured this would probably be a no-brainer, but turns out, that it isn't.

On FreeBSD I currently manage to push 150-200Mbits with some heavy tuning applied already, on Linux it is roughly 500Mbits.

Therefor I'm wondering if I'm really running into some limitations here or if I'm actually doing something wrong.

ifconfig bge0
bge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=c01db<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,POLLING,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>

As you can see I've compiled polling and currently I'm running on kern.hz=16000 as that's given me the best performance, so far.

This is my netstat output for tcp on ipv4:

netstat -s
tcp:
	265086752 packets sent
		84255022 data packets (155791599035 bytes)
		2244601 data packets (2698410010 bytes) retransmitted
		73011 data packets unnecessarily retransmitted
		684 resends initiated by MTU discovery
		153151729 ack-only packets (0 delayed)
		0 URG only packets
		19864 window probe packets
		16854982 window update packets
		8551532 control packets
	236720260 packets received
		86600847 acks (for 155842386836 bytes)
		5062568 duplicate acks
		76267 acks for unsent data
		138258588 packets (150041170335 bytes) received in-sequence
		1502804 completely duplicate packets (562243206 bytes)
		44193 old duplicate packets
		91821 packets with some dup. data (23693391 bytes duped)
		6598536 out-of-order packets (7168950882 bytes)
		32074 packets (20848106 bytes) of data after window
		7806 window probes
		1457705 window update packets
		159860 packets received after close
		5219 discarded for bad checksums
		3 discarded for bad header offset fields
		0 discarded because packet too short
		1468 discarded due to memory problems
	5665849 connection requests
	694088 connection accepts
	0 bad connection attempts
	129 listen queue overflows
	9308 ignored RSTs in the windows
	3289250 connections established (including accepts)
	6334698 connections closed (including 449721 drops)
		916420 connections updated cached RTT on close
		923786 connections updated cached RTT variance on close
		273103 connections updated cached ssthresh on close
	354989 embryonic connections dropped
	81015541 segments updated rtt (of 56772442 attempts)
	9127304 retransmit timeouts
		19875 connections dropped by rexmit timeout
	21274 persist timeouts
		215 connections dropped by persist timeout
	4541 Connections (fin_wait_2) dropped because of timeout
	10657 keepalive timeouts
		0 keepalive probes sent
		10657 connections dropped by keepalive
	39113689 correct ACK header predictions
	121244352 correct data packet header predictions
	698461 syncache entries added
		21576 retransmitted
		14546 dupsyn
		0 dropped
		694088 completed
		0 bucket overflow
		0 cache overflow
		1046 reset
		3338 stale
		135 aborted
		0 badack
		6 unreach
		0 zone failures
	698461 cookies sent
	232 cookies received
	173007 hostcache entries added
		0 bucket overflow
	285122 SACK recovery episodes
	584154 segment rexmits in SACK recovery episodes
	730380132 byte rexmits in SACK recovery episodes
	3053612 SACK options (SACK blocks) received
	6689960 SACK options (SACK blocks) sent
	0 SACK scoreboard overflow
	8236 packets with ECN CE bit set
	26367032 packets with ECN ECT(0) bit set
	56 packets with ECN ECT(1) bit set
	312220 successful ECN handshakes
	34178 times ECN reduced the congestion window

My sysctls are roughly equivalent to these: http://serverfault.com/questions/64356/freebsd-performance-tuning-sysctls-loader-conf-kernel

Any hints?
Do I need to provide more info?

Julian
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 841 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://lists.freebsd.org/pipermail/freebsd-performance/attachments/20120206/dbe628bb/signature.pgp


More information about the freebsd-performance mailing list