35-40% performance drop releng9 vs releng10 openvpn

[John-Mark Gurney]

Mike Tancsa wrote this message on Wed, Mar 18, 2015 at 15:49 -0400:
> On 3/16/2015 9:20 AM, John-Mark Gurney wrote:
> >
> > Since you have at test framework ready, you could generate some flame
> > graphs[1] using dtrace to help see where things might be having an
> > impact...
> >
> > These are very easy to generate, and posting them would be useful...
> >
> > [1] http://www.brendangregg.com/FlameGraphs/cpuflamegraphs.html
> 
> 
> Hi,
> 	I went through the steps to generate one. What args should I use for 
> dtrace to generate the information that is helpful / useful ? For my 
> setup, I have
> 
> server1---------apu-------------server2
> 
> server1 has an openvpn tunnel to the apu
> I route server2's IP address across the VPN tunnel, so if I ping from 
> server1 to server2's IP, it goes via the tunnel
> 
> on the
> 
>   dtrace -x ustackframes=100 -n 'profile-99 /execname == "openvpn" && 
> arg1/ {  @[ustack()] = count(); } tick-30s { exit(0); }' -o 10.stacks
> 
> which generated
> http://tancsa.com/10.svg

So, I would first identify the machine w/ the cpu limited load.. I
assume that is apu...  Then I would look at where most of the cpu time
is being spent, be it openvpn itself, or in the kernel... Most likely
it is the kernel, so getting stacks from the kernel would be more useful
than the one you generated...  Use the command:
# dtrace -x stackframes=100 -n 'profile-997 /arg0/ { @[stack()] = count(); } tick-60s { exit(0); }' -o out.kern_stacks

Also, another thing you can do is to compare the two using differential
flame graphs:
http://www.brendangregg.com/blog/2014-11-09/differential-flame-graphs.html

Which will highlight where the performances differ...

As I've never used OpenVPN before and their docs don't go into saying
what it's using.. Is OpenVPN a kernel or userland VPN?  Do they use
IPSec in the kernel? or are they just using UDP or TCP for their
connections?

-- 
  John-Mark Gurney				Voice: +1 415 225 5579

     "All that I will do, has been done, All that I have, has not."