Large Filesystem Woes

[Tom Arnold]

I previously posted this on -fs but got no responce so I'm trying -hackers.

Building a box thats going to house many billions of small files.  Think
innd circa 1998 or someone trying to house AOLs mail system on cyrus or
something.  To this end I've hung a 3.3TB hardware raid off a BSD box
broken into 4 partitions.  3 1TB and 1 300GB.
Originally this was on a 4.9 box.  da0s1 and da0s2 were formatted "stock"
( -f 2048 -b 16384 -i 8192 ) da1s1 and s2 were both formatted -f 512 -b 4096
-i 512.  Needless to say fsck took a while if the box came up dirty.
Switched to 5.2.  Newfs'd the RAID for UFS2.  First issue, if the machine
came up dirty, bgfsck seemed to do its thing and the machine was online and
usable after about 20 minutes however after a few hours I get this error :

fsck: /dev/da1s1e: CANNOT CREATE SNAPSHOT /export/database/.snap/fsck_snapshot: File too large
fsck: /dev/da1s1e: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.

And the second thing I've noticed is I have lost a lot of space.
Under 4.9 with UFS da1s1e was approx 870gigs and s2e was around 180, now
I see :
Filesystem    Size   Used  Avail Capacity iused      ifree %iused  Mounted on
/dev/da0s1e   992G   4.0K   912G     0%       2  134411260    0%   /export/logs1
/dev/da0s2e   992G   4.0K   912G     0%       2  134411260    0%   /export/logs2
/dev/da1s1e   510G   1.0K   469G     0%       2 2148661228    0%   /export/database
/dev/da1s2e    94G   1.0K    86G     0%       2  395214332    0%   /export/spare


I'm not certain if I've run into some kind of weird limit here or a bug or
what and am looking for ideas to persue before I'm stuck going to an OS with
something journaled.

Thanks!

-- 
 ------------------------------------------------------------------------
 - Tom Arnold -       When I was small, I was in love,                  - 
 - Sysabend   -       In love with everything.                          -
 - CareTaker  -       And now there's only you...                       - 
 --------------         -- Thomas Dolby, "Cloudburst At Shingle Street" -