Terabyte harddisks, GELI, AMD64, Samba and Zen...
Solon Luigi Lutz
solon at pyro.de
Mon Apr 16 21:28:52 UTC 2007
after some troubleshooting and some hours of memory tests, it
finaly seems to be a hardware problem...
The machine is based on an ASUS M2N4-SLI (Nforce4) and since the
heat-sink on the north/southbridge is rather small and passive,
the chip seems to get too hot. I manufactured a massive one from
a IGBT heat-sink and since 20 hours the machine is doing ftp-transfers
without any reboots - I keep my fingers crossed...
BTW a "fsck_ufs -y -f /dev/da0.eli" without sofupdates on this 10 TB
volume takes only 3 hours to complete.
Thanks for your help.
P.S. Does anybody know a way to do some performance-tuning on GELI?
>> -------Sequential Output-------- ---Sequential Input-- --Random--
>> -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks---
>> Machine MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU /sec %CPU
>> 1000 37922 31.5 48829 12.0 23827 5.0 30054 36.0 43666 6.0 1120.0 2.5
IV> Interesting - your CPU doesn't look overwhelmed much.
>> But I have not been able to get more than 17 MB/s when using Samba to
>> transfer data - FTP maxes out around 27 MB/s. I also tried that on
>> i386 32-bit and found it to be 8 MB/s and 17 MB/s - not good, but
>> nothing to worry.
>> What made me feel really uncomfortable was the fact, that just some
>> minutes ago some 3000 1GB files suddenly disappeared while working
>> in a directory. They where gone, but the filesystem did not report
>> some additional 3TB to be free and after unmounting and remounting the
>> filesystem the files were back where the belonged...
>> This just happened some minutes later again, now with only 2500 files
>> dis- and -reappearing again.
IV> This can mean either file system corruption (which fsck fixed on boot?),
IV> a bug (read cache bug, where the memory representation of the directory
IV> doesn't agree with on-disk state) or a hardware memory error. Of these,
IV> hardware errors are easiest to check in your case. Download a memtest86
IV> boot CD ISO, burn it and let it run for a few hours. Next, you can try a
IV> "full" fsck, which would probably a few last days on such a big array
IV> (big arrays are inconvenient to have without journaling). If both fail,
IV> we may look for a bug somewhere.
>> Questions until now:
>> 1. 10TB as a single volume, too big for good? (fsck time: 30 min with softupdates)
IV> Yes, too big. Softupdates doesn't even do a full fsck - if you tried a
IV> full fsck it will require about a dozen GB of memory (or memory+swap)
IV> and take a really long time. If you're not scared of it, you should run
IV> 7-current and re-create the file system with gjournal, or even ZFS.
>> 2. GELI unstable on big disks and/or AMD64?
IV> You're the first to complain :)
>> 3. Why is Samba so slow?
IV> Search Google... Samba is notoriously slow on FreeBSD, but there are few
IV> ways to tune it which will help.
>> 4. Does the crypto-framwork gain speed advantages from dual-core CPUs?
IV> No, and the same goes for most GEOM classes.
>> 5. Will the GPT-stuff change over the next releases in a way I need to
>> do DUMP/RESTORE?
IV> I don't think so, except if someone discovers an incompatibility in the
IV> way FreeBSD handles GPT wrt other OSs. Shouldn't happen.
More information about the freebsd-questions