UFS Crash and directories now missing

[Alejandro Imass]

On Sun, Apr 29, 2012 at 4:37 AM, Polytropon <freebsd at edvax.de> wrote:
> On Sun, 29 Apr 2012 00:26:50 -0700, perryh at pluto.rain.com wrote:
>> Alejandro Imass <ait at p2ee.org> wrote:
>>
>> > 3) the directories were moved at reboot by journal recovery,
>> > fsck or something else
>>
>> I think it's *extremely* unlikely that fsck was involved, because
>> it just doesn't do things like that.
>
> The point is: fsck moving directories "looks different". In
> case inodes get "de-connected" (their reference entries on
> level n-1 are gone, their data on level n is still present),
> fsck will access the lost+found/ directory in the corresponding
> partition's root directory (or create it, if not present) and
> write _new_ directory entries with the inode as their name,
> because that's the only naming information possible (as the
> original names on n-1 aren't accessible anymore). So those
> directories will have names like #177628676/ and they _can_
> contain subtrees full of data, _including_ names from levels
> n+1 and onward. Files also are named #4767667892 and their
> names can _maybe_ identified from their content (the "file"
> command is helpful, and if they are textfiles containing
> a CVS or other revision control system data tag, it's possible
> to find out what they've been in their previous life).
>
> However, as it has been explained, fsck will _not_ do so
> unless being _allowed explicitely_ to do that kind of
> MODIFICATION to the file system. Flags like -yf can do
> that, but they are _not_ the default. This is due to the
> fact that _any_ critical modification of file systems
> requires the _responsible administrator_ to give permission.
>

OK, so fsck couldn't have done this. Besides fsck reported the fs as
clean so I have to conclude as others have commented that it must have
been a mv

I've been looking at the logs very carefully and trying to make sense
of this. There is a possibility that it could have been an attack
because we enabled ftp.proxy so that some clients could upload stuff
to their jails using ftp. So I was initially wrong in my assessment
because on this particular server we are running a service outside of
jails and it's this ftp.proxy that was suppose to be a temporary
solution but I guess we never got around to fixing this.

The ftp.proxy is started via inetd like so:
ftp    stream tcp      nowait nobody /usr/local/sbin/ftp.proxy ftp.proxy -e

And there was a log of a couple of ftp connections the same day this
happened, the ONLY 3 messages before the reboot at about 6 pm and they
were NOT from any of our customers. Here are the log entries:

Apr 27 05:54:37 nune ftp.proxy[2726]: connected to client:
host-46-50-183-5.bbcustomer.zsttk.net, interface= 207.158.52.74:21
Apr 27 05:54:37 nune ftp.proxy[2726]: info: monitor mode: off, ccp: <unset>
Apr 27 05:54:38 nune ftp.proxy[2726]: -ERR: missing hostname
Apr 27 18:55:42 nune syslogd: kernel boot file is /boot/kernel/kernel

OK. So let's suppose ftp.proxy is the culprit is there any way the
could have done the mv by cracking ftp and ftp.proxy ??

I have of course disabled the ftp and I am now thinking that another
possibility or combination by also using the ftp proxy on the
http-proxy jail, that is, the jail that swallowed the other jails. The
http-proxy jails was also running apache ftp proxy.

So the question now becomes: could a break in ftp.proxy coupled with
Apache ftp proxy have caused the http-proxy jails to have swallowed
all the other jails into it's configuration directory??

-- 
Alejandro Imass