Improving old-fashioned UFS2 performance with lots of inodes...

[George Sanders]

Hello Jeremy,



> > with over 100  million inodes on the filesystem, things go slow.  Overall 
> >  throughput is fine, and I have no complaints there, but doing any kind of 
> > operations with the files is quite slow.  Building a file list  with rsync, 
>or 
>
> > doing a cp, or a ln -s of a big dir tree, etc.
> > 
> > Let's assume that the architecture is not changing ... it's going to be  
>FreeBSD 
>
> > 8.x, using UFS2, and raid6 on actual spinning (7200rpm)  disks.
> > 
> > What can I do to speed things up ?
> > 
> >  Right now I have these in my loader.conf:
> > 
> >  kern.maxdsiz="4096000000"# for fsck
> > vm.kmem_size="1610612736"# for big  rsyncs
> > vm.kmem_size_max="1610612736"# for big rsyncs
> 
> On what  exact OS version?  Please don't say "8.2", need to know
> 8.2-RELEASE,  -STABLE, or what.  You said "8.x" above, which is too
> vague.  If  8.2-STABLE you should not be tuning vm.kmem_size_max at all,
> and you probably  don't need to tune vm.kmem_size either.



Ok, right now we are on 6.4-RELEASE, but it is our intention to move to 
8.2-RELEASE.

If the kmem loader.conf options are no longer relevant in 8.2-STABLE, should I
assume that will also be the case when 8.3-RELEASE comes along ?


> I also do not understand how  vm.kmem_size would affect rsync, since
> rsync is a userland application.   I imagine you'd want to adjust
> kern.maxdsiz and kern.dfldsiz (default  dsiz).



Well, a huge rsync with 20+ million files dies with memory related errors, and
continued to do so until we upped the kmem values that high.  We don't know
why, but we know it "fixed it".


> > and I also set:
> > 
> >  vfs.ufs.dirhash_maxmem=64000000
> 
> This tunable uses memory for a single  directorie that has a huge amount
> of files in it; AFAIK it does not apply to  "large directory structures"
> (as in directories within directories within  directories).  It's obvious
> you're just tinkering with random sysctls  hoping to gain performance
> without really understanding what the sysctls  do.  :-)  To see if you
> even need to increase that, try "sysctl -a  | grep vfs.ufs.dirhash" and
> look at dirhash_mem vs. dirhash_maxmem, as well  as dirhash_lowmemcount.



No, we actually ALSO have huge directories, and we do indeed need this value.

This is the one setting that we actually understand and have empirically 
measured.



> The only thing I can think of on short notice  is to have multiple
> filesystems (volumes) instead of one large 12TB  one.  This is pretty
> common in the commercial filer  world.



Ok, that is interesting - are you saying create multiple, smaller UFS 
filesystems
on the single large 12TB raid6 array ?

Or are you saying create a handful of smaller arrays ?  We have to burn two 
disks
for every raid6 array we make, as I am sure you know, so we really can't split 
it up
into multiple arrays.

We could, however, split the single raid6 array into multiple, formatted UFS2
filesystems, but I don't understand how that would help with our performance ?

Certainly fsck time would be much shorter, and we could bring up each filesystem
after it fsck'd, and then move to the next one ... but in terms of live 
performance,
how does splitting the array into multiple filesystems help ?  The nature of a 
raid
array (as I understand it) would have us beating all 12 disks regardless of 
which
UFS filesystems were being used.

Can you elaborate ?


> Regarding system RAM and UFS2: I have no idea, Kirk might have  to
> comment on that.
> 
> You could "make use" of system RAM for cache (ZFS  ARC) if you were using
> ZFS instead of native UFS2.  However, if the  system has 64GB of RAM, you
> need to ask yourself why the system has that  amount of RAM in the first
> place.  For example, if the machine runs  mysqld and is tuned to use a
> large amount of memory, you really don't  ""have"" 64GB of RAM to play
> with, and thus wouldn't want mysqld and some  filesystem caching model
> fighting over memory (e.g.  paging/swapping).



Actually, the system RAM is there for the purpose of someday using ZFS - and
for no other reason.  However, it is realistically a few years away on our 
timeline,
unfortunately, so for now we will use UFS2, and as I said ... it seems a shame
that UFS2 cannot use system RAM for any good purpose...

Or can it ?  Anyone ?