fsck is failing to clean a filesystem

[Ian Smith]

On Wed, 10 Feb 2016 10:22:36 +0100, Polytropon wrote:
 > On Wed, 10 Feb 2016 20:11:03 +1100 (EST), Ian Smith wrote:
 > > On Tue, 9 Feb 2016 22:10:14 -0800, Paul Beard wrote:
 > >  > > On Feb 9, 2016, at 9:14 PM, Ian Smith <smithi at nimnet.asn.au> wrote:
 > >  > > 
 > >  > > I know your problem is with /usr, but I 
 > >  > > find the fact that /var is full or too nearly so rather concerning, and 
 > >  > > wonder whether that might have contributed to your problem in some way, 
 > >  > > and whether freeing up some space there might yet help?
 > >  > 
 > >  > Yeah, itÿÿs a mess, running a full system in a 64Gb virtual disk is 
 > >  > probably asking for trouble. I think there is some cruft in /var 
 > >  > (databases that are no longer in use) that can be pitched.
 > > 
 > > 64GB should be plenty, depending on usage of course.  A full /var is a 
 > > worry, especially if it runs short of room for logging.

I did makethe latter point, at least, from experience :)

 > When fsck is running, it usually happens in a mode where only /
 > is mounted read-only, and all other file systems (such as /usr
 > or /var) are not. So I'd say that fsck doesn't do logging
 > somewhere into /var/log because it doesn't exist at that time.
 > Single user mode is a state of heavily reduced system functionality,
 > but usually sufficient for solving file system problems.

It's true that /var/log/* can't be written then, but it's all buffered 
in dmesg, saved to messages [& console.log] once syslogd starts.  From 
single user you can say, '# dmesg >/root/dmesg.save' or a scratch disk.

 > >  > > Also, does 'du /usr/lost+found' reveal anything?
 > >  > 
 > >  > It was full of stuff /usr/src, best I could make out. Not sure why it 
 > >  > all ended up in there.
 > > 
 > > Well at least /usr/src is easily replaced.  Might be worth just deleting 
 > > all that, though of course you need a read-write mount first .. perhaps
 > > after booting from a memstick or live CD?
 > 
 > Which is still risky, assuming that the file system has not
 > been marked clean.

Absolutely.  I'm assuming that at this stage the choice is to newfs /usr 
and restore backups, unless some magic spell turns up.  Once apparently 
losing or damaging '..' from anywhere, you're pretty much in trouble.

Paul, another question: with /usr unmounted, is there anything in /usr ?

 > > You might also check (before and after deleting anything) that /usr 
 > > isn't running short of inodes (df -hi)?
 > 
 > Good suggestion.

Only what I read; newfs defaults have always been generous for my usage.

 > > Just stabbing in the dark .. scrambled filesystems are the pits!
 > 
 > And a good occassion to read more about UFS (McKusick et al.) - to
 > develop a better understanding of what's happening. :-)

Good suggestion :)  I envy people who've got the time these days ..

cheers, Ian