Tor on FreeBSD Performance issues

[K. Macy]

On Mon, Feb 6, 2012 at 9:38 PM, Julian Wissmann
<julianwissmann at gmail.com> wrote:
> Hi,
>
> I'm an admin for a non-profit that runs Tor exit nodes, most of them on Linux currently, but due to problems on our high bandwidth nodes, we decided to migrate one of them to FreeBSD to do some testing.
> I've been using FreeBSD for quite some years now, longer than Linux, so I figured this would probably be a no-brainer, but turns out, that it isn't.
>
> On FreeBSD I currently manage to push 150-200Mbits with some heavy tuning applied already, on Linux it is roughly 500Mbits.

Tor tends to keep open a lot of connections but that is really very
little bandwidth and you really shouldn't need to have polling on or
have hz set that high. What does CPU utilization look like on these
systems? I don't know if it is part of the problem, but TSO really
isn't very useful with large numbers of connections, it is better
suited to helping a single connection saturate an interface - could
please you turn that off.


Thanks

> Therefor I'm wondering if I'm really running into some limitations here or if I'm actually doing something wrong.
>
> ifconfig bge0
> bge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
>        options=c01db<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,POLLING,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
>
> As you can see I've compiled polling and currently I'm running on kern.hz=16000 as that's given me the best performance, so far.
>
> This is my netstat output for tcp on ipv4:
>
> netstat -s
> tcp:
>        265086752 packets sent
>                84255022 data packets (155791599035 bytes)
>                2244601 data packets (2698410010 bytes) retransmitted
>                73011 data packets unnecessarily retransmitted
>                684 resends initiated by MTU discovery
>                153151729 ack-only packets (0 delayed)
>                0 URG only packets
>                19864 window probe packets
>                16854982 window update packets
>                8551532 control packets
>        236720260 packets received
>                86600847 acks (for 155842386836 bytes)
>                5062568 duplicate acks
>                76267 acks for unsent data
>                138258588 packets (150041170335 bytes) received in-sequence
>                1502804 completely duplicate packets (562243206 bytes)
>                44193 old duplicate packets
>                91821 packets with some dup. data (23693391 bytes duped)
>                6598536 out-of-order packets (7168950882 bytes)
>                32074 packets (20848106 bytes) of data after window
>                7806 window probes
>                1457705 window update packets
>                159860 packets received after close
>                5219 discarded for bad checksums
>                3 discarded for bad header offset fields
>                0 discarded because packet too short
>                1468 discarded due to memory problems
>        5665849 connection requests
>        694088 connection accepts
>        0 bad connection attempts
>        129 listen queue overflows
>        9308 ignored RSTs in the windows
>        3289250 connections established (including accepts)
>        6334698 connections closed (including 449721 drops)
>                916420 connections updated cached RTT on close
>                923786 connections updated cached RTT variance on close
>                273103 connections updated cached ssthresh on close
>        354989 embryonic connections dropped
>        81015541 segments updated rtt (of 56772442 attempts)
>        9127304 retransmit timeouts
>                19875 connections dropped by rexmit timeout
>        21274 persist timeouts
>                215 connections dropped by persist timeout
>        4541 Connections (fin_wait_2) dropped because of timeout
>        10657 keepalive timeouts
>                0 keepalive probes sent
>                10657 connections dropped by keepalive
>        39113689 correct ACK header predictions
>        121244352 correct data packet header predictions
>        698461 syncache entries added
>                21576 retransmitted
>                14546 dupsyn
>                0 dropped
>                694088 completed
>                0 bucket overflow
>                0 cache overflow
>                1046 reset
>                3338 stale
>                135 aborted
>                0 badack
>                6 unreach
>                0 zone failures
>        698461 cookies sent
>        232 cookies received
>        173007 hostcache entries added
>                0 bucket overflow
>        285122 SACK recovery episodes
>        584154 segment rexmits in SACK recovery episodes
>        730380132 byte rexmits in SACK recovery episodes
>        3053612 SACK options (SACK blocks) received
>        6689960 SACK options (SACK blocks) sent
>        0 SACK scoreboard overflow
>        8236 packets with ECN CE bit set
>        26367032 packets with ECN ECT(0) bit set
>        56 packets with ECN ECT(1) bit set
>        312220 successful ECN handshakes
>        34178 times ECN reduced the congestion window
>
> My sysctls are roughly equivalent to these: http://serverfault.com/questions/64356/freebsd-performance-tuning-sysctls-loader-conf-kernel
>
> Any hints?
> Do I need to provide more info?
>
> Julian



-- 
   “The real damage is done by those millions who want to 'get by.'
The ordinary men who just want to be left in peace. Those who don’t
want their little lives disturbed by anything bigger than themselves.
Those with no sides and no causes. Those who won’t take measure of
their own strength, for fear of antagonizing their own weakness. Those
who don’t like to make waves—or enemies.

   Those for whom freedom, honour, truth, and principles are only
literature. Those who live small, love small, die small. It’s the
reductionist approach to life: if you keep it small, you’ll keep it
under control. If you don’t make any noise, the bogeyman won’t find
you.

   But it’s all an illusion, because they die too, those people who
roll up their spirits into tiny little balls so as to be safe. Safe?!
>From what? Life is always on the edge of death; narrow streets lead to
the same place as wide avenues, and a little candle burns itself out
just like a flaming torch does.

   I choose my own way to burn.”

   Sophie Scholl