UFS Crash and directories now missing

[Polytropon]

On Sat, 28 Apr 2012 13:52:02 -0400, Alejandro Imass wrote:
> On Sat, Apr 28, 2012 at 1:31 PM, Robert Bonomi <bonomi at mail.r-bonomi.com> wrote:
> >
> > Alejandro Imass <aimass at yabarana.com> wrote:
> >> On Sat, Apr 28, 2012 at 11:39 AM, Robert Bonomi
> >> <bonomi at mail.r-bonomi.com> wrote:
> >> >  Alejandro Imass <aimass at yabarana.com> wrote:
> >> >> After a little more research, ___it it NOT unlikely at all___ that
> >> >> under high distress and a hard boot, UFS could have somehow corrupted
> >> >> the directory structure, whilst maintaining the data intact.
> >> >
> >> > This is techically accurate, *BUT* the specifics of the quote "corruption"
> >> > unquote in the case under discussion make it *EXTREMELY* unlikely that this
> >> > is what happened.
> >> >
> >> > 99.99+++% of all UFS filesystem "corruption' issues are the result of a
> >> > system crash _between_ the time cached 'meta-data' is updated in memory
> >> > and that data is flushed to disk (a deferred write).
> >> >
> >> > The second most common (and vanishingly rare) failure mode is a powerfail
> >> > _as_ a sector of disk is being written -- resulting in 'garbage data'
> >> > being written to disk.
> >> >
> >> > The next possibility is 'cosmic rays'.  If running on 'cheap' hardware
> >> > (i.e., without 'ECC' memory), this can cause a *SINGLE-BIT* error in
> >> > data being output.
> >> >
> >> > The fact that the 'corrupted' filesystem passed fsck -without- any reported
> >> > errors shows that everything in the filesystem meta-data was consistent
> >> >
> >> [...]
> >>
> >> > I think it is safe to conclude that the probabilities -greatly- favor
> >> > alternative #1.
> >> >
> >>
> >> OK. So after your comments and further research I concur with you on
> >> the mv but if it wasn't a human, then this might be exposing a serious
> >> security flaw in the jail system or the way EzJail implements it.
> >
> > BOGON ALERT!!!
> >
> 
> I admit my ignorance on how the filesystem works but I don't think
> your condescending remarks add a lot of value. The issue here is this
> actually happened and there is a flaw somewhere other than "the stupid
> administrator did it".

If you search the archives of this list, you'll find my _first_
post to that list: I've had a similar problem, df shows data
must be there after crash (panic -> reboot -> fsck trouble), but
files aren't there (even _not_ in lost+found). It's quite possible
that in _exceptional_ moments this can happen. The fsck program
is intended to repair the most typical file system faults, but
nothing "complicated" will be done without interaction: Altering
data on disk will _always_ involve the responsible (!) admin to
check if it is really intended "to do so".

There can be many reasons. I've never found out what was the
reason for the trouble I've had. Some years ago, I found a "make"
failing because "/uss/src/blah... something not found", and
a quick memtest revealed the secret: defective RAM module that
caused a "bit error", and the difference between "r" and "s"
is just one bit. Replaced the module - everything worked.
Mean soldering rays from outer space. :-)

You'll find many useful forensic tools in the ports collection
that might help locate "lost" data (quotes intended as long as
the data is still on the disk). The more complex your setting
is (e. g. striped disks, or ZFS), this can be nearly impossible.
"Plain old UFS" can sometimes be your saviour (but BACKUP should
be your real friend).





-- 
Polytropon
Magdeburg, Germany
Happy FreeBSD user since 4.0
Andra moi ennepe, Mousa, ...