GELI Only running at the speed of 1 thread

[Clayton Milos]

Hi Pawel / All

I have a system with 2 AMD Opteron 285's in it with the second cores
disabled so effectively a 2 CPU single core each setup.
I run the following:
#> kldload geom_zero
#> geli onetime -s 4096 /dev/gzero
#> dd if=/dev/gzero of=/dev/null bs=1m count=1k
1024+0 records in
1024+0 records out
1073741824 bytes transferred in 0.507832 secs (2114364037 bytes/sec)
#> dd if=/dev/gzero.eli of=/dev/null bs=1m count=1k
1024+0 records in
1024+0 records out
1073741824 bytes transferred in 15.919784 secs (67447009 bytes/sec)

67 MByte/sec is about right for the CPU. Problem is that if I run 2 of those
processes simultaneously then they both get over 60MB/s each. This looks to
me like the geli process isn't threading correctly.
Systat looks like this so you can see there are 2 processes but each is only
using about half of each CPU:
                    /0   /1   /2   /3   /4   /5   /6   /7   /8   /9   /10
     Load Average   |||||

                    /0%  /10  /20  /30  /40  /50  /60  /70  /80  /90  /100
root           idle XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
root     g_eli[0] g XXXXXXXXXXXXXXXXXXXXXXXXXXX
root           idle XXXXXXXXXXXXXXXXXXXXXXX  
root     g_eli[1] g XXXXXXXXXXXXXXXXXXXXXXX
root           geom X

Top is also showing that only half of the CPU power is being used:
last pid:  3081;  load averages:  1.36,  0.69,  0.40
up 0+00:14:13  18:45:15
53 processes:  1 running, 51 sleeping, 1 zombie
CPU:  0.2% user,  0.0% nice, 49.6% system,  0.4% interrupt, 49.8% idle
Mem: 819M Active, 91M Inact, 759M Wired, 2208K Cache, 635M Buf, 14G Free
Swap: 16G Total, 16G Free

With the tunables I've only changed the cache limit because I have the ram
for it and I want to use it for an 8 TB array.
#> sysctl kern.geom.eli
kern.geom.eli.batch: 0
kern.geom.eli.threads: 0
kern.geom.eli.overwrites: 5
kern.geom.eli.visible_passphrase: 0
kern.geom.eli.tries: 3
kern.geom.eli.debug: 0
kern.geom.eli.version: 6
kern.geom.eli.key_cache_misses: 0
kern.geom.eli.key_cache_hits: 4725333
kern.geom.eli.key_cache_limit: 32768

My /boot/loader.conf is:
ng_bpf_load="YES"
amdtemp_load="YES"
ubsec_load="YES"
vm.kmem_size="1024M"
vm.kmem_size_max="1024M"
vfs.zfs.arc_max="600M"
vfs.zfs.vdev.cache.size="8M"
vfs.zfs.txg.timeout="5"
kern.maxvnodes="250000"
kern.geom.eli.key_cache_limit="32768"
hint.lapic.1.disabled="1"
hint.lapic.2.disabled="1"

I have the ubsec in there because I have a BCM5823 Crypto Accelerator
installed. It does not support AES and is not used by GELI:
Kernel: GEOM_ELI: Device gzero.eli created.
kernel: GEOM_ELI: Encryption: AES-XTS 128
kernel: GEOM_ELI:     Crypto: software

Any advice would be greatly appreciated. As I said it looks like GELI isn't
threading properly. Is it because of a setting I am running?

//Clay