Terabyte harddisks, GELI, AMD64, Samba and Zen...

[Solon Luigi Lutz]

Hi again,

after some troubleshooting and some hours of memory tests, it
finaly seems to be a hardware problem...
The machine is based on an ASUS M2N4-SLI (Nforce4) and since the
heat-sink on the north/southbridge is rather small and passive,
the chip seems to get too hot. I manufactured a massive one from
a IGBT heat-sink and since 20 hours the machine is doing ftp-transfers
without any reboots - I keep my fingers crossed...

BTW a "fsck_ufs -y -f /dev/da0.eli" without sofupdates on this 10 TB
volume takes only 3 hours to complete.

Thanks for your help.

Solon

P.S. Does anybody know a way to do some performance-tuning on GELI?

>>               -------Sequential Output-------- ---Sequential Input-- --Random--
>>               -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks---
>> Machine    MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU  /sec %CPU
>>          1000 37922 31.5 48829 12.0 23827  5.0 30054 36.0 43666 6.0 1120.0  2.5

IV> Interesting - your CPU doesn't look overwhelmed much.

>> But I have not been able to get more than 17 MB/s when using Samba to
>> transfer data - FTP maxes out around 27 MB/s. I also tried that on
>> i386 32-bit and found it to be 8 MB/s and 17 MB/s - not good, but
>> nothing to worry.
>> 
>> What made me feel really uncomfortable was the fact, that just some
>> minutes ago some 3000 1GB files suddenly disappeared while working
>> in a directory. They where gone, but the filesystem did not report
>> some additional 3TB to be free and after unmounting and remounting the
>> filesystem the files were back where the belonged...


>> This just happened some minutes later again, now with only 2500 files
>> dis- and -reappearing again.

IV> This can mean either file system corruption (which fsck fixed on boot?),
IV> a bug (read cache bug, where the memory representation of the directory
IV> doesn't agree with on-disk state) or a hardware memory error. Of these,
IV> hardware errors are easiest to check in your case. Download a memtest86
IV> boot CD ISO, burn it and let it run for a few hours. Next, you can try a
IV> "full" fsck, which would probably a few last days on such a big array
IV> (big arrays are inconvenient to have without journaling). If both fail,
IV> we may look for a bug somewhere.

>> Questions until now:
>> 
>> 1. 10TB as a single volume, too big for good? (fsck time: 30 min with softupdates)

IV> Yes, too big. Softupdates doesn't even do a full fsck - if you tried a
IV> full fsck it will require about a dozen GB of memory (or memory+swap)
IV> and take a really long time. If you're not scared of it, you should run
IV> 7-current and re-create the file system with gjournal, or even ZFS.

>> 2. GELI unstable on big disks and/or AMD64?

IV> You're the first to complain :)

>> 3. Why is Samba so slow?

IV> Search Google... Samba is notoriously slow on FreeBSD, but there are few
IV> ways to tune it which will help.

>> 4. Does the crypto-framwork gain speed advantages from dual-core CPUs?

IV> No, and the same goes for most GEOM classes.

>> 5. Will the GPT-stuff change over the next releases in a way I need to
>> do DUMP/RESTORE?

IV> I don't think so, except if someone discovers an incompatibility in the
IV> way FreeBSD handles GPT wrt other OSs. Shouldn't happen.