3+ TB Storage... CCD, growfs, etc...

[Kevin Marcus]

> Hi all,
>
> I am looking at the promise ultratrak RM 15000
>
> (http://www.promise.com/product/product_detail_eng.asp?productId=109&familyI
> d=6) Raid appliance with a 3TB disk configuration. This box connects
> to the host with a SCSI 160 interface which is no problem, and as I
> understand it UFS2 is 64 bit so I am not constrained by a 2TB
> filesystem limit. The smallest size file on this box will be 11GB, and
> there will be lots of them.
>
> My questions are.
>
> 1) What is the maximum filesystem size with UFS2?
> Are there any special tuning parameters that I should be aware of that
> will better optimize the disk?
>
> 2) How much CPU/Ram would be suggested per TB of disk attached?

> 3) If I wanted to eventually strip two+ of these external boxes what
> would I need to do? Given this configuration would Vinum or CCD be
> better? Why?
>
> Oh... and this will be running Samba to serve these files to windows
> pc's over 1Gb copper ethernet.

I can't speak to the exact hardware you're using, although I can speak to
my experiences with large filesystems.  I primarily use Adaptec RAID card
directly connected to their disks.  This is also somewhat nice since for
the freebsd 4.x series, adaptec also has a utility ('raidutil') which
allows you to monkey around with the raid right from the os instead of the
post/bios.  These utilities do not work on freebsd 5.0 or 5.1, even if you
have the proper shared libs so that is moot.

So to try and answer these questions: 
1) You will likely have problems with fdisk and disklabel as soon as you
try to get filesystems over 2TB because of int32 overflows.  I dont even
want to think of what fsck would do on a non 64 bit system.

2) Uh, as much as possible?  There are various limits around the sizes of
a single block of memory that can be allocated although these are somewhat
easy to tweak.  Older versions of FreeBSD could only use 2-4G of RAM with
some kernel hacking *AND* disabling swapping all together.  I am not sure
of the current state of affairs there - matt dillon could probably tell
you though.

3) can't help you there

I have more recently become concerned with freebsd's ability to simply
have mroe than 2TB of disk actually attached.  I recently acquired a
system with about 2.1TB of disk and I have tried again, various versions
all of which have the same problem as shown below:
---------------------
ad0: 190782MB <WDC WD2000JB-32EVA0> [387621/16/63] at ata0-master UDMA100
da1 at asr1 bus 0 target 0 lun 0
da1: <ADAPTEC RAID-0 380E> Fixed Direct Access SCSI-2 device
da1: Tagged Queueing Enabled
da1: 840086MB (1720496128 512 byte sectors: 255H 63S/T 41559C)
da2 at asr1 bus 0 target 10 lun 0
da2: <ADAPTEC RAID-0 380E> Fixed Direct Access SCSI-2 device
da2: Tagged Queueing Enabled
da2: 560057MB (1146996736 512 byte sectors: 255H 63S/T 5861C)
da0 at asr0 bus 0 target 0 lun 0
da0: <ADAPTEC RAID-5 3A0L> Fixed Direct Access SCSI-2 device
da0: Tagged Queueing Enabled
da0: 572346MB (1172164608 512 byte sectors: 255H 63S/T 7427C)
Mounting root from ufs:/dev/ad0s1a
da2: cannot find label (no disk label)
da2s1: cannot find label (no disk label)
da2: cannot find label (no disk label)
da2s1: cannot find label (no disk label)
da2: cannot find label (no disk label)
da2s1: cannot find label (no disk label)
da2: cannot find label (no disk label)
---------------------
No matter what I do and no matter how hard I try, I am simply unable to
get the da2 filesystem to function.  Since this system is brand new I have
tried rebuilding the raid (which I originally had simply as a single large
1.3+TB filesystem which didn't work at all.  Anyway, I've rearranged
things in various orders and tried making them individual disks, smaller
systems, etc. but as soon as the actual attached storage goes over 2TB, I
start to get these types of errors.  interestingly, fdisk and disklabel
also really hose the da2 system as well.  You can't even rewrite the
partition table through fdisk -i on the system.  

So I would be very very very weary before you attempt to add storage
beyond 2tb to a single system and would be cautious on any single
filesystem between 1-2tb.