Improving old-fashioned UFS2 performance with lots of inodes...
gosand1982 at yahoo.com
Tue Jun 28 23:14:02 UTC 2011
> > with over 100 million inodes on the filesystem, things go slow. Overall
> > throughput is fine, and I have no complaints there, but doing any kind of
> > operations with the files is quite slow. Building a file list with rsync,
> > doing a cp, or a ln -s of a big dir tree, etc.
> > Let's assume that the architecture is not changing ... it's going to be
> > 8.x, using UFS2, and raid6 on actual spinning (7200rpm) disks.
> > What can I do to speed things up ?
> > Right now I have these in my loader.conf:
> > kern.maxdsiz="4096000000"# for fsck
> > vm.kmem_size="1610612736"# for big rsyncs
> > vm.kmem_size_max="1610612736"# for big rsyncs
> On what exact OS version? Please don't say "8.2", need to know
> 8.2-RELEASE, -STABLE, or what. You said "8.x" above, which is too
> vague. If 8.2-STABLE you should not be tuning vm.kmem_size_max at all,
> and you probably don't need to tune vm.kmem_size either.
Ok, right now we are on 6.4-RELEASE, but it is our intention to move to
If the kmem loader.conf options are no longer relevant in 8.2-STABLE, should I
assume that will also be the case when 8.3-RELEASE comes along ?
> I also do not understand how vm.kmem_size would affect rsync, since
> rsync is a userland application. I imagine you'd want to adjust
> kern.maxdsiz and kern.dfldsiz (default dsiz).
Well, a huge rsync with 20+ million files dies with memory related errors, and
continued to do so until we upped the kmem values that high. We don't know
why, but we know it "fixed it".
> > and I also set:
> > vfs.ufs.dirhash_maxmem=64000000
> This tunable uses memory for a single directorie that has a huge amount
> of files in it; AFAIK it does not apply to "large directory structures"
> (as in directories within directories within directories). It's obvious
> you're just tinkering with random sysctls hoping to gain performance
> without really understanding what the sysctls do. :-) To see if you
> even need to increase that, try "sysctl -a | grep vfs.ufs.dirhash" and
> look at dirhash_mem vs. dirhash_maxmem, as well as dirhash_lowmemcount.
No, we actually ALSO have huge directories, and we do indeed need this value.
This is the one setting that we actually understand and have empirically
> The only thing I can think of on short notice is to have multiple
> filesystems (volumes) instead of one large 12TB one. This is pretty
> common in the commercial filer world.
Ok, that is interesting - are you saying create multiple, smaller UFS
on the single large 12TB raid6 array ?
Or are you saying create a handful of smaller arrays ? We have to burn two
for every raid6 array we make, as I am sure you know, so we really can't split
into multiple arrays.
We could, however, split the single raid6 array into multiple, formatted UFS2
filesystems, but I don't understand how that would help with our performance ?
Certainly fsck time would be much shorter, and we could bring up each filesystem
after it fsck'd, and then move to the next one ... but in terms of live
how does splitting the array into multiple filesystems help ? The nature of a
array (as I understand it) would have us beating all 12 disks regardless of
UFS filesystems were being used.
Can you elaborate ?
> Regarding system RAM and UFS2: I have no idea, Kirk might have to
> comment on that.
> You could "make use" of system RAM for cache (ZFS ARC) if you were using
> ZFS instead of native UFS2. However, if the system has 64GB of RAM, you
> need to ask yourself why the system has that amount of RAM in the first
> place. For example, if the machine runs mysqld and is tuned to use a
> large amount of memory, you really don't ""have"" 64GB of RAM to play
> with, and thus wouldn't want mysqld and some filesystem caching model
> fighting over memory (e.g. paging/swapping).
Actually, the system RAM is there for the purpose of someday using ZFS - and
for no other reason. However, it is realistically a few years away on our
unfortunately, so for now we will use UFS2, and as I said ... it seems a shame
that UFS2 cannot use system RAM for any good purpose...
Or can it ? Anyone ?
More information about the freebsd-fs