RELENG_10 performance regression (was Re: 35-40% performance drop releng9 vs releng10 openvpn

[Mike Tancsa]

OK, I think I found where the RELENG_10 performance loss happened. It seems
https://lists.freebsd.org/pipermail/svn-src-stable-10/2015-March/004778.html
is the issue.

Testing with a kernel from r279796 I get 76-77Mb of throughput.  With
r279848 it drops to about 60Mb

	---Mike



On 3/20/2015 1:36 PM, Mike Tancsa wrote:
> OK, just to refocus,
>
> I had been tracking down what I thought was a regression between RELENG9
> and RELENG10, but looks more like an issue that cropped up somewhere
> between the beginning of March and now.  For RELENG9, I was actually
> using a kernel from sources back on Jan 29th by accident.  If I bring
> RELENG9 upto today, I get a similar performance loss.
>
> Again, I am testing a simple VPN router setup
>
> server1 --- apu --- server2
>
> where server1 connections to the apu via an OpenVPN tunnel and server1
> sends packets via netblast across the tunnel to server2.
>
>   I get the following throughput using netblast through the tunnel on 10
>
> Using
> # netblast 1.1.2.2 500 1200 15 (server1 to server2) on 10.x
>
> Kernel    Mb/s
>    rev
> r277684  76.7563
> r279978  59.3233
>
> All good at r278533, r278534, r279467
>
> But at r279978 its quite a bit slower.  So somewhere between r279467 and
> r279978.  I will keep trying to narrow it down...
>
>
>      ---Mike
>
>
>
>
>
> On 3/19/2015 8:26 PM, Mike Tancsa wrote:
>> On 3/18/2015 5:14 PM, John-Mark Gurney wrote:
>>> # dtrace -x stackframes=100 -n 'profile-997 /arg0/ { @[stack()] =
>>> count(); } tick-60s { exit(0); }' -o out.kern_stacks
>>>
>>> Also, another thing you can do is to compare the two using differential
>>> flame graphs:
>>> http://www.brendangregg.com/blog/2014-11-09/differential-flame-graphs.html
>>>
>>>
>>>
>>> Which will highlight where the performances differ...
>>
>> OK, some more data points.  It seems a performance regression happened
>> in RELENG_10 somewhere between r277684 (late January 2015) and now.
>> Using r277684 on RELENG_10, I can get about 75Mb/s of throughput on
>> OpenVPN. Still not as good as the 83-85Mb on RELENG_9, but much better
>> than the 61Mb using RELENG_10 from the start of this week,
>>
>> For the differential graph, see
>>
>> http://tancsa.com/diffgraph.svg
>>
>> and
>>
>> http://tancsa.com/10-r277684.svg
>> http://tancsa.com/10-r277684-kern.svg
>>
>>      ---Mike
>>
>>
>>
>>
>
>


-- 
-------------------
Mike Tancsa, tel +1 519 651 3400
Sentex Communications, mike at sentex.net
Providing Internet services since 1994 www.sentex.net
Cambridge, Ontario Canada   http://www.tancsa.com/