Terabyte harddisks, GELI, AMD64, Samba and Zen...
Solon Luigi Lutz
solon at pyro.de
Wed Apr 11 03:09:45 UTC 2007
I've setup a Freebsd 6.2 server with a Areca 1280 24-port SATA2-Raid
controller and some 24 500gb harddisks. CPU is a AMD64 Sempron 3000+.
FreeBSD rad.net 6.2-STABLE FreeBSD 6.2-STABLE #0: Sun Apr 8 12:39:46 UTC 2007
root at rad.net:/usr/obj/usr/src/sys/RADIUM amd64
Filesystem Size Used Avail Capacity Mounted on
/dev/ad10s1a 496M 70M 386M 15% /
devfs 1.0K 1.0K 0B 100% /dev
/dev/ad10s1e 496M 18K 456M 0% /tmp
/dev/ad10s1f 67G 3.9G 58G 6% /usr
/dev/ad10s1d 1.9G 1.0G 761M 58% /var
/dev/da0.eli 9.9T 4.0T 5.1T 44% /mnt
The raid contains one 10 terabyte volume, encrypted with GELI
Geom name: da0.eli
1. Name: da0.eli
Mediasize: 10999996342272 (10T)
1. Name: da0
Mediasize: 10999996350464 (10T)
As I started to fill up the volume, I experienced random crashes -
always while doing high-speed transfers over gigabit ethernet between
this and another server. Mostly 1-2 per day - might be a hardware
problem or not.
Due to the slow CPU and software encryption, the maximum speed is somewhat
limited (raw speed is somewhat around 600 MB/s):
File './Bonnie.2117', size: 1048576000
Writing with putc()...done
Reading with getc()...done
Seeker 1...Seeker 2...Seeker 3...start 'em...done...done...done...
-------Sequential Output-------- ---Sequential Input-- --Random--
-Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks---
Machine MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU /sec %CPU
1000 37922 31.5 48829 12.0 23827 5.0 30054 36.0 43666 6.0 1120.0 2.5
But I have not been able to get more than 17 MB/s when using Samba to
transfer data - FTP maxes out around 27 MB/s. I also tried that on
i386 32-bit and found it to be 8 MB/s and 17 MB/s - not good, but
nothing to worry.
What made me feel really uncomfortable was the fact, that just some
minutes ago some 3000 1GB files suddenly disappeared while working
in a directory. They where gone, but the filesystem did not report
some additional 3TB to be free and after unmounting and remounting the
filesystem the files were back where the belonged...
This just happened some minutes later again, now with only 2500 files
dis- and -reappearing again.
Questions until now:
1. 10TB as a single volume, too big for good? (fsck time: 30 min with softupdates)
2. GELI unstable on big disks and/or AMD64?
3. Why is Samba so slow?
4. Does the crypto-framwork gain speed advantages from dual-core CPUs?
5. Will the GPT-stuff change over the next releases in a way I need to
Thanks for your patience...
More information about the freebsd-questions