Terabyte harddisks, GELI, AMD64, Samba and Zen...

[Jeff Mohler]

Was your 17mb/sec a drag & drop from a windows client?

17Mb/sec is about right, as Windows deals with that as a single threaded I/O
operation.

You can stuff a GigE pipe from a windows machine to a netapp or some other
solid CIFS server if you can fire up multiple threads to copy with.

So..dont feel bad, thats just the cap of a single threaded copy from
windows.

On 4/10/07, Solon Luigi Lutz <solon at pyro.de> wrote:
>
> Hello freebsd-questions,
>
> I've setup a Freebsd 6.2 server with a Areca 1280 24-port SATA2-Raid
> controller and some 24 500gb harddisks. CPU is a AMD64 Sempron 3000+.
>
> ---
> FreeBSD rad.net 6.2-STABLE FreeBSD 6.2-STABLE #0: Sun Apr  8 12:39:46 UTC
> 2007
> root at rad.net:/usr/obj/usr/src/sys/RADIUM  amd64
> ---
>
> ---
> Filesystem      Size    Used   Avail Capacity  Mounted on
> /dev/ad10s1a    496M     70M    386M    15%    /
> devfs           1.0K    1.0K      0B   100%    /dev
> /dev/ad10s1e    496M     18K    456M     0%    /tmp
> /dev/ad10s1f     67G    3.9G     58G     6%    /usr
> /dev/ad10s1d    1.9G    1.0G    761M    58%    /var
> /dev/da0.eli    9.9T    4.0T    5.1T    44%    /mnt
> ---
>
> The raid contains one 10 terabyte volume, encrypted with GELI
>
> ---
> Geom name: da0.eli
> EncryptionAlgorithm: AES-CBC
> KeyLength: 128
> Crypto: software
> UsedKey: 0
> Flags: NONE
> Providers:
> 1. Name: da0.eli
>    Mediasize: 10999996342272 (10T)
>    Sectorsize: 8192
>    Mode: r1w1e1
> Consumers:
> 1. Name: da0
>    Mediasize: 10999996350464 (10T)
>    Sectorsize: 512
>    Mode: r1w1e1
> ---
>
> As I started to fill up the volume, I experienced random crashes -
> always while doing high-speed transfers over gigabit ethernet between
> this and another server. Mostly 1-2 per day - might be a hardware
> problem or not.
>
> Due to the slow CPU and software encryption, the maximum speed is somewhat
> limited (raw speed is somewhat around 600 MB/s):
>
> ---
> File './Bonnie.2117', size: 1048576000
> Writing with putc()...done
> Rewriting...done
> Writing intelligently...done
> Reading with getc()...done
> Reading intelligently...done
> Seeker 1...Seeker 2...Seeker 3...start 'em...done...done...done...
>               -------Sequential Output-------- ---Sequential Input--
> --Random--
>               -Per Char- --Block--- -Rewrite-- -Per Char- --Block---
> --Seeks---
> Machine    MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU  /sec
> %CPU
>          1000 37922 31.5 48829 12.0 23827  5.0 30054 36.0 43666  6.0
> 1120.0  2.5
> ---
>
> But I have not been able to get more than 17 MB/s when using Samba to
> transfer data - FTP maxes out around 27 MB/s. I also tried that on
> i386 32-bit and found it to be 8 MB/s and 17 MB/s - not good, but
> nothing to worry.
>
> What made me feel really uncomfortable was the fact, that just some
> minutes ago some 3000 1GB files suddenly disappeared while working
> in a directory. They where gone, but the filesystem did not report
> some additional 3TB to be free and after unmounting and remounting the
> filesystem the files were back where the belonged...
>
> This just happened some minutes later again, now with only 2500 files
> dis- and -reappearing again.
>
> Questions until now:
>
> 1. 10TB as a single volume, too big for good? (fsck time: 30 min with
> softupdates)
> 2. GELI unstable on big disks and/or AMD64?
> 3. Why is Samba so slow?
> 4. Does the crypto-framwork gain speed advantages from dual-core CPUs?
> 5. Will the GPT-stuff change over the next releases in a way I need to
> do DUMP/RESTORE?
>
> Thanks for your patience...
>
> Best regards,
>
> Solon Lutz
>
>
> _______________________________________________
> freebsd-questions at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-questions
> To unsubscribe, send any mail to "
> freebsd-questions-unsubscribe at freebsd.org"
>