GELI strangeness with gstat

[John-Mark Gurney]

Pete French wrote this message on Tue, Nov 28, 2017 at 09:37 +0000:
> Tnaks for the reply....
> 
> > If you just did a kldload aesni, but did not reattach the geli device,
> > then you are still using software encryption...  You should see something
> > like this:
> > GEOM_ELI: Device gpt/werner.eli created.
> > GEOM_ELI: Encryption: AES-XTS 256
> > GEOM_ELI:     Crypto: hardware
> > 
> > if you are using AES-NI...
> 
> Yes, thats exactly what I see - I rebooted the system (had to as I was 
> enabling AES_NI in the BIOS) and am loading the crypto and aesni modules 
> at boot time form loader.conf. I see this in dmesg:
> 
> Enter passphrase for ada0p4:
> GEOM_ELI: Device ada0p4.eli created.
> GEOM_ELI: Encryption: AES-XTS 128
> GEOM_ELI:     Crypto: hardware
> GEOM_ELI: Device ada1p4.eli created.
> GEOM_ELI: Encryption: AES-XTS 128
> GEOM_ELI:     Crypto: hardware
> 
> and 'geli list' says 'Crypto: hardware' in its output
> 
> > Also, what version of FreeBSD are you using?  If you're using pre-10.0-R,
> > the performance increase from using AES-NI is only marginal...
> 
> Am using 11.1-STABLE from mid June - is it worth getting the last few 
> weeks of updates ? I wouldnt have thought so as I havent see any crypto 
> chnages go past, but I can give it a go...

Nope.  Not much has changed in a while..

> > The above does make it look like you're disks are CPU bound by the
> > encryption...
> 
> Indeed. But the odd this is it only started happening *after* I upgraded 
> the machine to much faster CPU's. I am suspecting (hoping!) its actually 
> a statistical anomaly in gstat and not a real effect.
> 
> > To get a better idea of what is happening, you can run top -S to see
> > how much CPU the geli threads are using.
> 
> Ah, good idea.
> 
> Hmmmm... that shows up 12 threads, each using about 0.2% CPU.

That doesn't make sense then..  That is about what I would expect...

Also, have you tried to benchmark w/o geli?  do you get similar
performance?

> > Also, how many eli volumes do you have?  In my case, I have 13...  In
> > order to reduce the load on the scheduler, I have:
> > kern.geom.eli.threads="1"
> 
> This is interesting - my understanding was that with hardware encryption 
> that setting did nothing. From the manpage:
> 
> kern.geom.eli.threads: 0
>   Specifies how many kernel threads should be used for doing software 
> cryptography. Its purpose is to increase performance on SMP systems. If 
> hardware acceleration is available, only one thread will be started. If 
> set to 0, CPU-bound thread will be started for every active CPU.

This is a slight issue w/ the crypto framework.. When AES-NI was imported,
it was declared a "hardware" device even though it's really a software
device..  I believe to allow it priority over the software devices which
are often used as a last resort...  Even though it's listed as software
only, it does effect things for AES-NI...

> So my expectation would be that I only get 1 thread - but actuyally I 
> see 12 in 'top -S'. Again, is that a bug or out of date documentation ?

I don't know, as I don't see any output...  It should look something like:
   99 root          20    -     0K    16K geli:w  0  15:59   0.02% g_eli[0] gpt/stanle

and if you see numbers other than 0 in the brackets, then that setting
isn't working..

The setting needs to be set before the first attach..  As my root volume
is geli, I have to set it in loader.conf, otherwise sysctl.conf would
set it too late...

> I only have two volumes, and am runnign ZFS over the top with 
> compresison enabled, which I think only uses a single write thread, so 
> maybe reducing that would help. But theres still a very large 
> discrepancy between what all of my other jetricsare showing me and gstat.
> 
> Te underlying discs are SSD's by the way, so I am fairly sure thats not 
> a bottleneck (and gstats dosnt show the undrelying drive busy).

Yeah, it looks like you have geli setup resonably, so I'm not sure what
to say at this point..

-- 
  John-Mark Gurney				Voice: +1 415 225 5579

     "All that I will do, has been done, All that I have, has not."