freebsd encrypted hard disk?

[Roland Smith]

On Thu, Jan 15, 2009 at 02:06:58AM +0000, RW wrote:
> On Thu, 15 Jan 2009 00:20:54 +0100
> Roland Smith <rsmith at xs4all.nl> wrote:
> 
> > On Wed, Jan 14, 2009 at 10:55:38PM +0000, RW wrote:
> 
> > > Not just in reduced transfer rates, but also in terms of CPU cycles
> > > used - a sustained geli to geli file copy makes things really slow
> > > for me.
> > 
> > That's probably because two geli kernel threads are competing for time
> > on a single core. I've had problems with that as well (geli-encrypted
> > USB drive stalling).
> > 
> > Since I've switched to a multi-core machine (where the number of cores
> > should be at least equal to the number of geli-encrypted devices), CPU
> > load for gele has dropped to barely noticable.
> > 
> 
> I find that puzzling; have you measured that on sustained geli to
> geli transfers (with GB size files).

Not really. My previous single core machine kept stalling the drive. So
instead of storing dumps for my biggest (/home) partition, I've turned to
rsync-ing that, and using dumps for /, /usr and /var.

I just did a simple test: copying a  5249634304 byte file from a non-encrypted
disk to an encrypted one. This took 3:22 with the CPU load hovering at
10-13% at full speed. That's around 24.8 MiB/s.

Copying the same file to another location on the unencrypted disk
takes 2:57 with the CPU at around 5% at about 3/4 speed. That comes down
to 28.3 MiB/s. 

Not a big difference IMHO. In normal usage the encrypted partition
doesn't feel slower, because there are still at least two cores fully
available for other jobs so interactivity doesn't suffer. A single-core
machine would obviously feel different.

The only thing that takes a while is compiling ports and big LaTeX jobs
(compiling 300 pages five times with makeindex and bibtex runs in
between).

> The reason I'm a bit sceptical is that dd'ing /dev/random to /dev/null
> runs at about 20MBytes/s on my single core (verses 700MBytes/s
> for /dev/zero). File copies into geli run at about 15Mbytes/s, openssl
> enc -aes-256-cbs runs at about the same ballpark figure. 

I get:
dd if=/dev/random of=/dev/null bs=64k count=10000
10000+0 records in
10000+0 records out
655360000 bytes transferred in 10.693598 secs (61285266 bytes/sec)

say 58 MiB/s

dd if=/dev/zero of=/dev/null bs=64k count=10000
10000+0 records in
10000+0 records out
655360000 bytes transferred in 0.281247 secs (2330194568 bytes/sec)

or around 2200 MiB/s

And that is obviously running on _one_ core (CPU at approx. 25%).

> Even if I had
> multi-cores I would still be cpu-limited to 20MB/s, and that would fully
> occupy two cores on geli to geli transfers. Your cores are probably
> faster, but I'd expect a factor of two or so would be swallowed-up by
> faster transfers. I don't see how cpu usage would be negligible unless
> your individual cores are an order of magnitude faster than that.

It turns out that on a multi-core machine a geli thread is started on
each core for each disk (4 cores, two disks):

ps -xa | grep g_eli
   92  ??  DL     5:31.75 [g_eli[0] ad4s1g]
   93  ??  DL     6:33.76 [g_eli[1] ad4s1g]
   94  ??  DL     6:26.02 [g_eli[2] ad4s1g]
   95  ??  DL     7:41.18 [g_eli[3] ad4s1g]
   98  ??  DL     1:49.60 [g_eli[0] ad6s1g]
   99  ??  DL     1:30.30 [g_eli[1] ad6s1g]
  100  ??  DL     1:53.87 [g_eli[2] ad6s1g]
  101  ??  DL     1:50.64 [g_eli[3] ad6s1g]

This can be tuned with the sysctl 'kern.geom.eli.threads'.

> Just out of curiosity what rate do you get on  
> dd if=/dev/random of=/dev/null bs=64k count=10000

See above.
 
Roland
-- 
R.F.Smith                                   http://www.xs4all.nl/~rsmith/
[plain text _non-HTML_ PGP/GnuPG encrypted/signed email much appreciated]
pgp: 1A2B 477F 9970 BA3C 2914  B7CE 1277 EFB0 C321 A725 (KeyID: C321A725)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 195 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-questions/attachments/20090115/3ce5e666/attachment.pgp