svn commit: r365449 - head/sbin/rcorder

[John Baldwin]

On 9/17/20 10:49 AM, Olivier Cochard-Labbé wrote:
> On Tue, Sep 8, 2020 at 12:36 PM Andrey V. Elsukov <ae at freebsd.org> wrote:
> 
>> Author: ae
>> Date: Tue Sep  8 10:36:11 2020
>> New Revision: 365449
>> URL: https://svnweb.freebsd.org/changeset/base/365449
>>
>> Log:
>>   Add a few features to rcorder:
>>
>>
>>
> Hi Andrey,
> 
> I've spent some time bisecting an IPSec gateway performance regression on
> -head that points to this commit.
> 
> So my bench uses a simple static IPSec aes-gcm-16 setup, and their results
> vary a lot between those 2 revisions:
> 
>    - r365448: Estimated Equilibrium Ethernet throughput= 1413 Mb/s (maximum
>    value seen: 1419 Mb/s)
>    - r365449: Estimated Equilibrium Ethernet throughput= 469 Mb/s (maximum
>    value seen: 469 Mb/s)
> 
> How could a modification to the rcoder be the source cause of that ?
> My rc.conf contains this:
> kld_list="aesni"
> ipsec_enable="YES"
> 
> So My first thought was that the aesni module wasn't loaded anymore:
> Before upgrade:
> [root at hp]~# uname -a
> FreeBSD hp 13.0-CURRENT FreeBSD 13.0-CURRENT #15 r365448M: Thu Sep 17
> 18:17:58 CEST 2020     olivier at lame4.bsdrp.net:/usr/src/amd64.amd64/sys/amd64
>  amd64
> [root at hp]~# kldstat
> Id Refs Address                Size Name
>  1   11 0xffffffff80200000  1ee58b0 kernel
>  2    1 0xffffffff82319000     34c8 fdescfs.ko
>  3    1 0xffffffff8231d000     a240 aesni.ko
>  4    1 0xffffffff82328000     8c98 ioat.ko
>  5    1 0xffffffff82331000     e350 ipsec.ko
> 
> Then after upgrade:
> 
> [root at hp]~# uname -a
> FreeBSD hp 13.0-CURRENT FreeBSD 13.0-CURRENT #14 r365449M: Thu Sep 17
> 17:01:35 CEST 2020     olivier at lame4.bsdrp.net:/usr/src/amd64.amd64/sys/amd64
>  amd64
> [root at hp]~# kldstat
> Id Refs Address                Size Name
>  1   11 0xffffffff80200000  1ee58b0 kernel
>  2    1 0xffffffff82319000     34c8 fdescfs.ko
>  3    1 0xffffffff8231d000     e350 ipsec.ko
>  4    1 0xffffffff8232c000     a240 aesni.ko
>  5    1 0xffffffff82337000     8c98 ioat.ko
> 
> => aesni.ko is correctly loaded, so it is not the problem, but notice the
> order of the kernel modules that have changed.
> Could be this the source of the problem ? Let's try:
> 
> [root at hp]~# service ipsec stop
> Clearing ipsec manual keys/policies.
> [root at hp]~# kld
> kldconfig kldload   kldstat   kldunload kldxref
> [root at hp]~# kldunload ioat
> [root at hp]~# kldunload aesni
> [root at hp]~# kldunload ipsec
> [root at hp]~# kldload aesni
> [root at hp]~# kldload ipsec
> [root at hp]~# service ipsec start
> Installing ipsec manual keys/policies.
> [root at hp]~# kldstat
> Id Refs Address                Size Name
>  1   11 0xffffffff80200000  1ee58b0 kernel
>  2    1 0xffffffff82319000     34c8 fdescfs.ko
>  5    1 0xffffffff82337000     8c98 ioat.ko
>  6    1 0xffffffff8231d000     a240 aesni.ko
>  7    1 0xffffffff82328000     e350 ipsec.ko
> 
> And after that the IPSec bench results are back to their previous value :-)
> So rcorder needs to load aesni before ipsec.

I don't think the issue is with rcorder though.  I think the reason the
ordering matters warrants further investigation.  Is aesni not getting
used when ipsec is loaded first?  You can use dtrace with the script at
https://github.com/bsdjhb/kdbg/blob/master/dtrace/crypto_drivers.d to see
which driver is being used.

Hmm, the crypto driver gets selected when keys for SAs are set, so if
/etc/rc.d/ipsec is configuring SAs, then having it run before aesni is
loaded would indeed cause this.  However, that still isn't an issue with
this commit, per se, it just exposed the lack of an explicit ordering
requirement of /etc/rc.d/ipsec after loading aesni.  The problem though
is that kld_list is pretty generic, so it's hard to know when /etc/rc.d/kld
should run.  Possibly it should even run multiple times where subsequent
runs try to load any modules not loaded by the previous runs?  I suspect
btw that you could just do 'sh /etc/rc.d/ipsec restart' post-boot without
unloading any modules which would also fix your benchmark.

I think long term we want OCF's notions of sessions to be a bit more
fluid such that "client" sessions for things like GELI and IPSec can
be backed by one or more "driver" sessions (including "driver" sessions
coming and going as devices come and go).  That's a fair bit more work
however.

-- 
John Baldwin