UFS Crash and directories now missing

[Alejandro Imass]

On Sun, Apr 29, 2012 at 11:49 PM, Erich Dollansky
<erich at alogreentechnologies.com> wrote:
> Hi,
>
> On Monday 30 April 2012 02:02:41 jb wrote:
>> Alejandro Imass <ait <at> p2ee.org> writes:
>>
>> > ...
>> > > What you should do right now is to get some recent general or security cd/dvd
>> > > with chkrootkit and rkhunter and run them from that external read-only media.
>> > > I would also suggest that you look over config files of all packages
>> > > involved.
>> > > jb
>> > >
>> >
>> > Thanks! Will do, but I don't know of any FreeBSD and/or derived
>> > distros for security. Or can I use any Linux security distro? I
>> > remember reading about some trouble of Linux chkrootkit on FBSD....
>>
>> It looks like you have only one choice with prebuilt rkhunter package only:
>> http://www.freebsd.org/releases/9.0R/announce.html
>>
>> dvd1
>> This contains everything necessary to install the base FreeBSD operating system,
>> a collection of pre-built packages aimed at getting a graphical workstation up
>> and running. It also supports booting into a "livefs" based rescue mode. This
>> should be all you need if you can burn and use DVD-sized media.
>>
>> ftp://ftp.freebsd.org/pub/FreeBSD/ports/packages/security/
>> rkhunter-1.3.8_1.tbz          04/18/12        18:56:00
>>
>> With regard to verification of config  files - you said you got backups (those
>> pre-incident would be best) and you have the incident-time files, so do a diff
>> on dirs (in particular /etc and /usr/local/etc)
>>
> I would burn the backup of these files to an optical disk, start the system and do a diff as the first step. The system can be started from an USB drive (take the 9.0 installation image) or DVD.
>
> Of course, rkhunter can be started in the second step.

ran both, found nothing

Back to theory on how the http-proxy jail 'swallowed' all the other
jails including the basejail. I noticed that jail had a not so old bug
in 2010 FBSD 8.0 which

<quote>
The jail(8) utility does not change the current working directory while
imprisoning.  The current working directory can be accessed by its
descendants.
</quote>

Reference: http://security.freebsd.org/advisories/FreeBSD-SA-10:04.jail.asc

Given that EzJail uses a single basejail and links/mounts stuff in the
child jails it would seem plausible (regression?) that somehow any
jail could access other jails' files, or that _maybe_ in an event of
crash the nullsfs mounts confuse the system somehow when fsck restores
or the journal is recovered.

Whatever the cause, it actually happened and I have already ruled out
just about anything. It doesn't seem to have been an attack, it surely
wasn't me, and EzJail author agrees it was not the EzJail scripts. So
maybe nullfs and journaling, or crash + nullfs + journaling, could
cause something like this to happen? Maybe journal has some confusion
on restoring the nullfs view of the directories or something after bad
crash like this one??