comments on newfs raw disk ? Safe ? (7 terabyte array)

[Oliver Fromme]

Arone Silimantia wrote:
 > Thank you for your detailed response - my own response is inline below:

You should indent the quoted text, so it's easier to tell
the quoted text from your own text.  Most MUAs offer a
function to indent the quoted text automatically.

I tried to fix it below.

 > Oliver Fromme <olli at lurza.secnetix.de> wrote:
 > > Arone Silimantia wrote:
 > > > Big 3ware sata raid with 16 disks.  First two disks are a mirror to 
 > > > boot off of.  I installed the system with sysinstall and created all the 
 > > > partitions on the boot mirror, etc., and just didn't even touch the 
 > > > 14-disk array that was also created.
 > > > [...]
 > > > newfs -m 0 /dev/da1
 > > 
 > > You didn't mention the size of the FS, but I guess it's at
 > > least 4 TB, probably more.
 > 
 > Well, in the subject line I mentioned 7 TB, but I have
 > since rearranged some things and it will be 5.5 TB.

Sorry, I didn't look at the subject line too closely.
I expected that all important information was included
in the mail body.

 > > You'll probably want to reduce the inode density (i.e.
 > > increase the bytes-per-inode ratio).  With the default
 > > value, an fsck will be a royal pain, no matter whether you
 > > use background fsck (with snapshots) or not.  It might even
 > > not work at all if you don't have a huge amount of RAM.
 > 
 > Well, I have 4 GB of physical RAM, and 4 GB of swap - so does that
 > total of 8 GB satisfy the "1 GB per TB" requirement, or do I really
 > need >5.5 GB of actual swap space (in addition to the physical) ?

That "1 GB per TB" requirement is just a rule of thumb.
I don't know hoe accurate it is.  Also note that it is
desirable to avoid having fsck use swap, because it will
be even slower then.  A lot slower.

 > > If you increase the ratio to 64 K, it will lower the fsck
 > > time and RAM requirement by an order of magnitude, while
 > > there are still about 15 million inodes available per TB.
 > > If possible, increase the ratio (-i option) further.  It
 > > depends on the expected average file size and the maximum
 > > number of files that you intend to store on the FS, of
 > > course.
 > 
 > Ok, I will look into this.  My data population uses a little less
 > than 5 million inodes per TB, so this may be workable to tune.  So I
 > see the default is '4' - so I could run newfs with:

The default is 4096 (one inode per 4 KB).

 > newfs -i 8
 > 
 > to do what you are suggesting ?

I think you mean 8192.  That will cut the number of inodes
in half, but I think you can go even further.  Try:

# newfs -i 65536

That will leave room for about 15 million inodes per TB,
which is plenty for your needs.

By the way, reducing the inode density like that will also
give your more space for actual file data.  In UFS2, every
inode takes 256 bytes.  Reducing the bytes-per-inode ratio
from 4 KB to 64 KB will give you additional 60 GB of space.
_And_ it will reduce the memory and time requirements of
fsck.

 > > Oh by the way, I also agree with Eric that you should have
 > > a look at gjournal.  It pratically removes the fsck issues.
 > > At the moment it's only in -current, but I think Pawel
 > > provided a port for 6.x.
 > 
 > Well, I don't mind a 24 hour fsck, and I would like to remove
 > complexity and not be so on the bleeding edge with things.  Since I
 > am only using 5mil inodes per TB anyway, that ends up being 25-30
 > million inodes in the 5 TB drive which I think could fsck in a day or
 > so.

I suggest you test it before putting it into production,
i.e. populate the file system with the expected number of
files, then run fsck.

 > I just need to know if my 4+4 GB of memory is enough, and if this
 > option in loader.conf:
 > 
 > kern.maxdsiz="2048000000"

That will limit the process size to 2 GB.  You might need
to set it higher if fsck needs more than that.  (I assume
you're running FreeBSD/amd64, or otherwise you'll run into
process size limitations anyway.) 

Best regards
   Oliver

-- 
Oliver Fromme, secnetix GmbH & Co. KG, Marktplatz 29, 85567 Grafing b. M.
Handelsregister: Registergericht Muenchen, HRA 74606, USt-Id: DE204219783
Any opinions expressed in this message are personal to the author and may
not necessarily reflect the opinions of secnetix GmbH & Co KG in any way.
FreeBSD-Dienstleistungen, -Produkte und mehr:  http://www.secnetix.de/bsd

Perl is worse than Python because people wanted it worse.
        -- Larry Wall