security/openssl speed issues

[Ollivier Robert]

Hello Dirk !

As I got a new machine with the AES-NI crypto extensions, I'm getting interested with it and as you may have seen, I've already merged into stable/9 two changesets for AES-NI support in GELI & cryptodev.

Now, I'm trying to measure the impact of said AES extentions, I tumbled on a very weird difference in behaviour between our base system openssl and the one in ports.

/usr/bin/openssl:
OpenSSL 0.9.8y 5 Feb 2013

/usr/local/bin/openssl:
OpenSSL 1.0.1e 11 Feb 2013

The one is base is not supposed to have cryptodev (and aesni) support at all as it was added apparently in 1.0.1.  Fine.

1. Trying to run both on a machine without the AES-NI extensions, I should have similar results in running speed tests but:

1181 [17:18] roberto at centre:/usr/ports> /usr/bin/openssl speed aes-256-cbc
...
OpenSSL 0.9.8y 5 Feb 2013			(9.1-BETA1)
built on: date not available
options:bn(64,64) md2(int) rc4(ptr,int) des(idx,cisc,16,int) aes(partial) idea(int) blowfish(idx)
compiler: cc
available timing options: USE_TOD HZ=128 [sysconf value]
timing function used: getrusage
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-256 cbc      58919.92k    62134.88k    62611.08k    62776.47k    62910.03k

and 

1182 [17:19] roberto at centre:/usr/ports> /usr/local/bin/openssl speed aes-256-cbc
...
OpenSSL 1.0.1e 11 Feb 2013			
built on: Sun Jul 28 16:36:48 CEST 2013
options:bn(64,64) md2(int) rc4(16x,int) des(idx,cisc,16,int) aes(partial) idea(int) blowfish(idx)
compiler: cc -fPIC -DOPENSSL_PIC -DZLIB_SHARED -DZLIB -DOPENSSL_THREADS -pthread -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -Wa,--noexecstack -DL_ENDIAN -DTERMIOS -O3 -DMD32_REG_T=int -Wall -O -pipe -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DMD5_ASM -DAES_ASM -DVPAES_ASM -DBSAES_ASM -DWHIRLPOOL_ASM -DGHASH_ASM
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-256 cbc      38790.95k    41415.66k    42009.00k    42257.07k    42213.38k

Wow, how would you explain the 37% (in the worng direction!) difference?  Is there something I could add/change in the port's configuration to fix that?

2. I have another machine with the AES-NI extensions, with a E3-1220 CPU.  If I load crypto, aesni and cryptodev, it is indentified as using them:

cryptosoft0: <software crypto> on motherboard
aesni0: <AES-CBC,AES-XTS> on motherboard

Results of openssl speed with the base one are better as you would expect, CPU is faster:

% /usr/bin/openssl speed aes-256-cbc
...
OpenSSL 0.9.8x 10 May 2012				(9.1-RELEASE)
built on: date not available
options:bn(64,64) md2(int) rc4(ptr,int) des(idx,cisc,16,int) aes(partial) blowfish(idx)
compiler: cc
available timing options: USE_TOD HZ=128 [sysconf value]
timing function used: getrusage
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-256 cbc     125404.07k   129849.19k   130514.37k   131242.71k   131164.72k
 
but...

% /usr/local/bin/openssl speed -engine cryptodev aes-256-cbc
engine "cryptodev" set.
...
OpenSSL 1.0.1c 10 May 2012
built on: Mon Apr  8 19:45:18 UTC 2013
options:bn(64,64) md2(int) rc4(16x,int) des(idx,cisc,16,int) aes(partial) idea(int) blowfish(idx)
compiler: cc -fPIC -DOPENSSL_PIC -DZLIB_SHARED -DZLIB -DOPENSSL_THREADS -pthread -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -Wa,--noexecstack -DL_ENDIAN -DTERMIOS -O3 -DMD32_REG_T=int -Wall -O2 -pipe -fno-strict-aliasing -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DMD5_ASM -DAES_ASM -DVPAES_ASM -DBSAES_ASM -DWHIRLPOOL_ASM -DGHASH_ASM
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-256 cbc      71203.16k    74667.39k    75631.27k    75975.34k    76090.03k

Still 42% diff and no "aesni" usage at all!?

I'm guessing we have an issue there...

Thanks,
-- 
Ollivier ROBERT -=- FreeBSD: The Power to Serve! -=- roberto at keltia.net
In memoriam to Ondine, our 2nd child: http://ondine.keltia.net/