VERY frustrated with FreeBSD/UFS stability - please help or comment...

[Gore Jarold]

--- Brooks Davis <brooks at freebsd.org> wrote:


> > a) am I really the only person in the world that
> moves
> > around millions of inodes throughout the day ?  Am
> I
> > the only person in the world that has ever filled
> up a
> > snapshotted FS (or a quota'd FS, for that matter)
> ? 
> > Am I the only person in the world that does a mass
> > deletion of several hundred thousand inodes
> several
> > times per day ?
> > 
> > OR:
> > 
> > b) am I just stupid ?  Is everyone doing this, and
> > there is 3 pages of sysctls and kernel tunes that
> > everyone does to their system when they are going
> to
> > use it this way ?  Am I just naive for taking a
> > release and paring down GENERIC and attempting to
> run
> > as-is out of the box without major tuning ?
> > 
> > If so, can I see those tunes/sysctls ?
> > 
> > I am _really_ hoping that it is (b) ... I would
> much
> > rather look back on all of this frustration as my
> own
> > fault than have the burden of proving all of this
> (as
> > I will no doubt be called upon to do). (1)
> > 
> > Thanks.  Please add your comments...
> 
> I'd say it's certaintly (a).  Consider that a full
> source tree contains
> a few under 85K files so that's a reasionable bound
> on average
> workloads.  Deliberatly producing a kernel that
> required tuning to just
> us the APIs without crashing would be stupid and we
> wouldn't go it
> without a very good reason and very large warnings
> all over the place.
> Lousy performance might be expected, but crashing
> wouldn't be.


Ok - your initial comments / impression are
reassuring.  It's hard to believe that the simple file
movements I do are so alien to mainstream use, but
I'll accept your judgement.



> > (1) just load up 6.2 and cp/rm a few million
> inodes
> > around.  Or turn on quotas and fill your
> filesystem
> > up.  Kaboom.
> 
> It's not clear to me what you mean by "cp/rm a few
> million inodes
> around."  The organization of those inodes into
> files and directories
> could conceviably have a major impact on the
> problem.  If you could
> provide a script that fails for you, that would
> really help.


Specifically, I have private departmental fileservers
that other fileservers rsync to using Mike Rubel-style
rsync snapshots:

http://www.mikerubel.org/computers/rsync_snapshots/

This means that the remote system runs a script like
this:

ssh user at host rm -rf backup.2
ssh user at host mv backup.1 backup.2
ssh user at host cp -al backup.0 backup.1
rsync /files user at host:/backup.0

The /files in question range from .2 to 2.2 million
files, all told. This means that when this script
runs, it first either deletes OR unlinks up to 2
million items.  Then it does a (presumably) zero cost
move operation.  Then it does a hard-link-creating cp
of the same (up to 2 million) items.

As I write this, I realize this isn't _totally_
generic, since I am using GNU cp rather than the
built-in FreeBSD cp, but that is _truly_ the extent of
customization on this system.

So that's that.  Run that a few times from a few
servers and 6.2 locks up.  Can't ping.  For trivias
sake, it does the same thing if you fill up one of its
filesystems with quotas turned on.

For further trivias sake, 6.1 is (seemingly) not
susceptible to this, but it is certainly susceptible
to other situations that I regularly produce.

What do you think of this (more specific) workload,
and are there any tunings that immediately jump to
mind ?


       
____________________________________________________________________________________Yahoo! oneSearch: Finally, mobile search 
that gives answers, not web links. 
http://mobile.yahoo.com/mobileweb/onesearch?refer=1ONXIC