VERY frustrated with FreeBSD/UFS stability - please help or comment...

[Brooks Davis]

On Mon, May 21, 2007 at 09:19:02AM -0700, Gore Jarold wrote:
> 
> I have been extremely dissatisfied with the stability
> of the FreeBSD UFS/UFS2 implementation throughout all
> of 5.x and 6.x.  I can take any random release from
> this period and halt/lock/crash it with basic and
> uninteresting filesystem operations.  In early 5.x it
> was simply big inode movements from disk to disk. 
> Later it was multiple snapshots.  In 6.x any number of
> seemingly benign operations (filling a disk, using
> quotas, dense (lots of inodes) deletes and copies) and
> each time some or all of these problems are solved in
> one release, slightly different versions of the same
> problem show up in the next one.  For instance,
> snapshot stability got a lot better from 6.0 to 6.2,
> but now 6.2 has problems just moving and deleting lots
> of inodes.
> 
> I am running _nothing interesting_. I don't even run
> snapshots anymore ... and as you can see from other
> posts, I am crashing/halting/etc. all over the place. 
> This is on systems that do nothing but TCP (scp, ftp)
> file service and some big (rm, cp) movements of inodes
> once in a while.  In other words, my setup is as
> vanilla as it gets.
> 
> It's not the hardware (happens on 3ware, adaptec,
> etc.) (different systems).
> 
> It's not my esoteric config (I take generic kernel and
> just delete the devices I don't use) ... also I have
> no custom sysctls/loader.conf
> 
> So I am at my wits end.  Since early 2004 there has
> not been a single release version of FreeBSD (well,
> except _perhaps_ 6.1-RELEASE ...) that I don't
> regularly knock over with _simple, generic movements
> of files_.
> 
> So my plea for help is as follows:
> 
> a) am I really the only person in the world that moves
> around millions of inodes throughout the day ?  Am I
> the only person in the world that has ever filled up a
> snapshotted FS (or a quota'd FS, for that matter) ? 
> Am I the only person in the world that does a mass
> deletion of several hundred thousand inodes several
> times per day ?
> 
> OR:
> 
> b) am I just stupid ?  Is everyone doing this, and
> there is 3 pages of sysctls and kernel tunes that
> everyone does to their system when they are going to
> use it this way ?  Am I just naive for taking a
> release and paring down GENERIC and attempting to run
> as-is out of the box without major tuning ?
> 
> If so, can I see those tunes/sysctls ?
> 
> I am _really_ hoping that it is (b) ... I would much
> rather look back on all of this frustration as my own
> fault than have the burden of proving all of this (as
> I will no doubt be called upon to do). (1)
> 
> Thanks.  Please add your comments...

I'd say it's certaintly (a).  Consider that a full source tree contains
a few under 85K files so that's a reasionable bound on average
workloads.  Deliberatly producing a kernel that required tuning to just
us the APIs without crashing would be stupid and we wouldn't go it
without a very good reason and very large warnings all over the place.
Lousy performance might be expected, but crashing wouldn't be.

> (1) just load up 6.2 and cp/rm a few million inodes
> around.  Or turn on quotas and fill your filesystem
> up.  Kaboom.

It's not clear to me what you mean by "cp/rm a few million inodes
around."  The organization of those inodes into files and directories
could conceviably have a major impact on the problem.  If you could
provide a script that fails for you, that would really help.

-- Brooks
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 187 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-fs/attachments/20070521/022f2346/attachment.pgp