Terabyte harddisks, GELI, AMD64, Samba and Zen...

[Solon Luigi Lutz]

Hello freebsd-questions,

I've setup a Freebsd 6.2 server with a Areca 1280 24-port SATA2-Raid
controller and some 24 500gb harddisks. CPU is a AMD64 Sempron 3000+.

---
FreeBSD rad.net 6.2-STABLE FreeBSD 6.2-STABLE #0: Sun Apr  8 12:39:46 UTC 2007
root at rad.net:/usr/obj/usr/src/sys/RADIUM  amd64
---

---
Filesystem      Size    Used   Avail Capacity  Mounted on
/dev/ad10s1a    496M     70M    386M    15%    /
devfs           1.0K    1.0K      0B   100%    /dev
/dev/ad10s1e    496M     18K    456M     0%    /tmp
/dev/ad10s1f     67G    3.9G     58G     6%    /usr
/dev/ad10s1d    1.9G    1.0G    761M    58%    /var
/dev/da0.eli    9.9T    4.0T    5.1T    44%    /mnt
---

The raid contains one 10 terabyte volume, encrypted with GELI

---
Geom name: da0.eli
EncryptionAlgorithm: AES-CBC
KeyLength: 128
Crypto: software
UsedKey: 0
Flags: NONE
Providers:
1. Name: da0.eli
   Mediasize: 10999996342272 (10T)
   Sectorsize: 8192
   Mode: r1w1e1
Consumers:
1. Name: da0
   Mediasize: 10999996350464 (10T)
   Sectorsize: 512
   Mode: r1w1e1
---

As I started to fill up the volume, I experienced random crashes -
always while doing high-speed transfers over gigabit ethernet between
this and another server. Mostly 1-2 per day - might be a hardware
problem or not.

Due to the slow CPU and software encryption, the maximum speed is somewhat
limited (raw speed is somewhat around 600 MB/s):

---
File './Bonnie.2117', size: 1048576000
Writing with putc()...done
Rewriting...done
Writing intelligently...done
Reading with getc()...done
Reading intelligently...done
Seeker 1...Seeker 2...Seeker 3...start 'em...done...done...done...
              -------Sequential Output-------- ---Sequential Input-- --Random--
              -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks---
Machine    MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU  /sec %CPU
         1000 37922 31.5 48829 12.0 23827  5.0 30054 36.0 43666  6.0 1120.0  2.5
---

But I have not been able to get more than 17 MB/s when using Samba to
transfer data - FTP maxes out around 27 MB/s. I also tried that on
i386 32-bit and found it to be 8 MB/s and 17 MB/s - not good, but
nothing to worry.

What made me feel really uncomfortable was the fact, that just some
minutes ago some 3000 1GB files suddenly disappeared while working
in a directory. They where gone, but the filesystem did not report
some additional 3TB to be free and after unmounting and remounting the
filesystem the files were back where the belonged...

This just happened some minutes later again, now with only 2500 files
dis- and -reappearing again.

Questions until now:

1. 10TB as a single volume, too big for good? (fsck time: 30 min with softupdates)
2. GELI unstable on big disks and/or AMD64?
3. Why is Samba so slow?
4. Does the crypto-framwork gain speed advantages from dual-core CPUs?
5. Will the GPT-stuff change over the next releases in a way I need to
do DUMP/RESTORE?

Thanks for your patience...

Best regards,

 Solon Lutz