Increasing GELI performance

[Fluffles]

Pawel Jakub Dawidek wrote:
> On Mon, Jul 30, 2007 at 10:35:32PM +0200, Fluffles wrote:
>   
>> Pawel Jakub Dawidek wrote:
>>     
>>> No matter how many cores/cpus you have if you run single-threaded
>>> application. What you do exactly is:
>>> 1. Send read of 128kB.
>>> 2. One of geli threads picks it up, decrypts and sends it back.
>>> 3. Send next read of 128kB.
>>> 4. One of geli threads picks it up, decrypts and sends it back.
>>> ...
>>>
>>> All threads will be used when there are more threads accessing provider.
>>>  
>>>       
>> But isn't it true that the UFS filesystem utilizes read-ahead and with 
>> that a multiple I/O queue depth (somewhere between 7 to 9 queued I/O's) 
>> - even when using something like dd to sequentially read a file on a 
>> mounted filesystem ? Then this read-ahead will cause multiple I/O 
>> request coming in and geom_eli can use multiple threads to maximize I/O 
>> throughput. Maybe Dominic can try playing with the "vfs.read_max" sysctl 
>> variable.
>>     
>
> You are right in general, but if you reread e-mail I was answering to,
> you will see that the author was reading from/writing to GEOM provider,
> not file system.
>   

Ah yes you're right. Though he might also have tested on a mounted 
filesystem, his email does not explicitely specify this. So he should 
re-run his experiment:

geli onetime /dev/da0
newfs /dev/da0
mkdir /test
mount /dev/da0 /test
dd if=/dev/zero of=/test/zerofile.000 bs=1m count=2000
(write score)
dd if=/test/zerofile.000 of=/dev/null bs=1m
(read score)

That *should* give him higher performance. Also, dominic might try 
increasing the block size, using "newfs -b 32768 /dev/da0". Without it, 
it seems it hits a performance roof on about ~130MB/s. I once wrote 
about this on the mailinglist where Bruce Evans questioned the 
usefulness of a blocksize higher than 16KB. I still have to investigate 
this further, it's on my to-do list. Deeplink: 
http://lists.freebsd.org/pipermail/freebsd-fs/2006-October/002298.html

I tried to recreate a test scenario myself, using 4 disks in a striping 
configuration (RAID0), first reading and writing on the raw .eli device, 
then on a mounted filesystem:

** raw device
# dd if=/dev/stripe/data.eli of=/dev/null bs=1m count=2000
2097152000 bytes transferred in 57.949793 secs (36189120 bytes/sec)
# dd if=/dev/zero of=/dev/stripe/data.eli bs=1m count=2000
1239416832 bytes transferred in 35.168374 secs (35242370 bytes/sec)

** mounted default newfs
# dd if=/dev/zero of=/test/zerofile.000 bs=1m count=2000
2097152000 bytes transferred in 47.843614 secs (43833478 bytes/sec)
# dd if=/test/zerofile.000 of=/dev/null bs=1m count=2000
2097152000 bytes transferred in 50.328749 secs (41669067 bytes/sec)

This was on a simple single core Sempron K8 CPU, but already there's a 
difference with the multiple queue depth VFS/UFS provides.

Good luck Dominic be sure to post again when you have new scores! I'm 
interested to see how far you can push GELI with a quadcore. :)

- Veronica