UFS Crash and directories now missing

Robert Bonomi bonomi at mail.r-bonomi.com
Thu May 3 13:34:36 UTC 2012

Alejandro Imass <ait at p2ee.org> wrote:

[ megasnip ]

> > Things to investigate :
> > - When was the last time this box was rebooted normally ? Did it went fine ?
> After I moved the jails to the right place I archived the jails with
> ezjail-admin and rebooted the server several times, and everything
> worked as expected.

Rephrasing -- when was the last time _before_the_problem_was_discovered_
that the machine was re-booted?

> > Were the jails created at this time ?
> No. Most of these jails have been operational for over a year on this
> server without any incidents.

Clarifying the question -- were the jails created at the time of the last
_prior_ reboot?  i.e., had the machine been re-booted successfully _after_
the jails were installed, or was this the _first_ such reboot?  

It appears you misunderstood the 'at this time' reference -- it did ot
mean 'at the time of the incident', but  'at the time of the last prior
reboot'.  If English is not your primary language, it is an understandable

> As I told you earlier, this server has been running for over a year
> and we have rebooted many times.

I don't believe you ever mentioed that particular point (multiple 
successful reboots after istallation) before.  Repeating a prior
question, _how_long_ before the problem showed up was the most recent
re-boot?  (Doesn't have to be exact -- an 'order of magnitude' estimate
[a day, a week, a month, several months] is sufficient.) 

>                                  If there are such problems they exist
> by using the EzJail commands and I find this unlikely.

What you 'find unlikely' is irrelevant.  The entire situation is 'unlikely',
yet it happened.  So one -has- to look at unlikely things.  <wry grin>

> here is the mount output is that's of any help:

[ first disk, and 'fdescfs', and 'procfs' references removed, for clarity ]

> /dev/ad6s1.journal on /usr/jails (ufs, asynchronous, local, gjournal)
> /usr/jails/basejail on /usr/jails/yabarana-php53/basejail (nullfs,
> local, read-only)
> /usr/jails/basejail on /usr/jails/yabarana-php52/basejail (nullfs,
> local, read-only)
> /usr/jails/basejail on /usr/jails/yabarana-cat58/basejail (nullfs,
> local, read-only)
> /usr/jails/basejail on /usr/jails/testbed/basejail (nullfs, local, read-only)
> /usr/jails/basejail on /usr/jails/pyugmao/basejail (nullfs, local, read-only)
> /usr/jails/basejail on /usr/jails/php53base/basejail (nullfs, local, read-only)
> /usr/jails/basejail on /usr/jails/php52base/basejail (nullfs, local, read-only)
> /usr/jails/basejail on /usr/jails/mcs-cat58/basejail (nullfs, local, read-only)
> /usr/jails/basejail on /usr/jails/http-proxy/basejail (nullfs, local, read-only)
> /usr/jails/basejail on /usr/jails/corcaribe-php53/basejail (nullfs,
> local, read-only)
> /usr/jails/basejail on /usr/jails/cm-website/basejail (nullfs, local, read-only)
> /usr/jails/basejail on /usr/jails/cm-idvida/basejail (nullfs, local, read-only)
> /usr/jails/basejail on /usr/jails/cat58base/basejail (nullfs, local, read-only)
> /usr/jails/basejail on /usr/jails/cmm-php52-1/basejail (nullfs, local,
> read-only)

Yes, that is a good start at useful detail.  It is, presumably, _after_
the problem, and _after_ you had restored things to their proper places.

Is it safe to  assume that you do -not- have such a 'mount' output from
some time 'before' the problem?  ( There's no rational reason why you
-would- have such, but _if_ it existed, and there were any differences
between 'then' and 'now', it could be very informative.)

Aother critical piece of information is what diretories -- by full path
name -- disappeared from 'where they were', and where -- by full path name,
again -- did you find them, and _with_what_names_?   If everything was
moved from the same source point to the same destination, it's not necessary
to itemize each one, but the details of _one_ 'typicaal' migration is needed.
It is also significant if there was 'anything else' in the 'where they 
belonged' directory that was -not- moved.  *OR* if there was anything else
(something other than the '/' of a jail) there, that was _also_ moved.

"Narrative" descriptions, as previously provided, and while clear to someone 
familiar with the machcine in question, are not sufficiently precise to allow 
an 'outsider' to follow the events without 'logically' replicating the setup, 
and then guessing at the meaning of any shorthands employed.

One comment: for 'defensive' purposes it would be useful to break ad6 up
into two slices, putting 'basejail' in it's own slice.  Then, for production
use, that slice can be mounted RO, and with the 'system immutable' flag
set on everything in that filesystem.

More information about the freebsd-questions mailing list