quotas safe on >2 TB filesystems in 6.1-RELEASE ?

Eric Anderson anderson at centtech.com
Wed Dec 20 07:43:36 PST 2006

On 12/20/06 09:15, Arone Silimantia wrote:
> Eric,
> Thanks for your comments and help - your posts on this
> list are much appreciated.  Comments in line below:
> --- Eric Anderson <anderson at centtech.com> wrote:
>>  With 9TB without 
>> any journaling, you might run into problems if you
>> crash and need to 
>> fsck - the number of files you could have on the
>> file system could well 
>> require more memory/time than you have available.
> Hmmm...the time required is more dependent on inodes
> than on size of data / size of files, right ?
> My 9 TB dataset uses about 36 million inodes.
> Any comments on that number ?  Large ?  Pedestrian ? 
> Typical ?

Sounds like you have a lot of larger files (~250k per file on average 
possibly), which helps the fsck times.  36 million inodes should be 
fsck'able with enough memory (maybe ~3gb ish? That's a wild guess).  I 
have two 10Tb file systems, one has 180million inodes.  I don't attempt 
to fsck it, because it would take a very long time, and I might possibly 
run out of memory (I have 8Gb of memory).  I use GJOURNAL on these, and 
am very happy about it (thanks Pawel!).

df snippet:
Filesystem                    1K-blocks        Used      Avail Capacity 
   iused      ifree %iused  Mounted on
/dev/label/vol10-data.journal 9925732858 8780187028  351487202    96% 
180847683 1102147515   14%   /vol10
/dev/label/vol11-data.journal 9925732858 2598987846 6532686384    28% 
49422705 1233572493    4%   /vol11

I also have several 2Tb partitions (set up prior to gjournal being 
available), that are *FULL*, and each have about 25-45million inodes on 
them.  Those fsck in about 4-7hours each, using between 1Gb and 3Gb of 
memory to do so.

> I am hoping that that could be fsck'd (modern hitachi
> SATA drives, raid-6, 3ware) in 48 hours ... or am I
> way off ?

Depends on the drives really, and maybe caching.  The geom_cache module 
(still beta probably, and not in the src tree currently I don't believe) 
is said to improve fsck times.  48 hours is a long time, and I *think* 
it should complete within that time frame, but you'd really have to test 
it to be sure.

>>> But I do absolutely need to run quotas (both user
>> and
>>> group) on this 9 GB array. I also need to
>> successfully
>>> use all quota admin tools (repquota, edquota,
>> quota,
>>> etc.)
>>> Can I get an assurance that this is totally safe,
>>> sane, and fit to run in a mission critical, data
>>> critical environment ?  Anyone doing it currently
>> ?
>>> Any comments or warnings of _any kind_ much
>> appreciated.
>> I don't think anyone will say 'I promise it will
>> work' of course, but I 
>> would start by using the latest 6-STABLE source
>> since there have been 
>> quite a number of updates to file system related
>> code since 6.1.
> Ok, but all of the CLI tools (edquota, repquota,
> quota, quotacheck, quotaon) are all known-good for
> "bigdisk" ?
> And there is no known "quotas just don't work with
> bigdisk" problems ?
> I was hoping someone out there was running quotas with
> 6.1-RELEASE on a >2TB filesystem and could report favorably...

I'm not certain.  There were some bugs in quotas, that recently were 
fixed (Kris Kennaway I think reported them and saw the fixes into the 
tree), and prior to that I saw those consistently, so I stopped using 
them.  I haven't tried since the fixes have been in place, and the fixes 
(if I recall correctly) had to do with background fsck (softupdates 
maybe) and not the size of the disk.

You could try this in a mock-up environment, by creating a sparse file 
and use it with mdconfig, then newfs it, enabling quotas, mount, and 
then use a script to create massive amounts (36million-ish) files of 
about 200k-ish in size, with random users, in a similar fashion as your 
real data, and see if all goes well.  You can also try your fsck that 
way too.


Eric Anderson        Sr. Systems Administrator        Centaur Technology
An undefined problem has an infinite number of solutions.

More information about the freebsd-fs mailing list