Strange case of root filesystem corruption

Ulf Magnusson ulfma629 at student.liu.se
Sat Sep 17 12:54:19 PDT 2005


First of all, all low level diagnostic tools I've used on my drive show
no signs of hardware error. The shutdown prior to when the problems
started went smooth, without any error messages.


Yesterday GRUB would suddenly not display the boot menu anymore. After
some investigation I discovered that all files in /boot/grub had turned
into symlinks to locale files, as seen in the following ufs2tools dump:

Jun 22,2005  01:16:24      <DIR> .
Jul 16,2005  00:41:53      <DIR> ..
May 11,2005  18:23:10         30 menu.lst -> ../it_CH.ISO8859-1/LC_MONETARY
Jun 22,2005  01:16:04        201 menu.lst~
May 11,2005  18:23:10         30 stage1 -> ../it_IT.ISO8859-1/LC_MONETARY
May 11,2005  18:23:10         30 stage2 -> ../nl_BE.ISO8859-1/LC_MONETARY
May 11,2005  18:23:10         30 ufs2_stage1_5 ->
../nl_NL.ISO8859-1/LC_MONETARY

I could no longer log in via any of my accounts, so I tried adding a new
account in single-user mode. This gave an error message about "vnodes"
(or perhaps inodes, my system won't boot anymore, so I can't reproduce
the error) and a kernel panic followed by a reboot. 

I then ran fsck, which reported a whole bunch of errors: wrong ref
wounts, something about things being bad with the superblock, and about
bad/missing things in general. Perhaps the strangest errors were about
extraneous links to files directly under / (more on this later). I
answered 'yes' to all fix/salvage questions from fsck.

After that FreeBSD would not boot at all, causing a reboot before the
beastie menu.

Looking at filesystems on the disk with the free ufs2tools program, only
the root filesystem seems to be affected by the corruption. Here's what
/ looks like (seems fsck was right about those extraneous links):

Jan 01,1970  01:00:00          0 .
Sep 16,2005  01:07:25      <DIR> .
Jan 01,1970  01:00:00          0 ..
Sep 16,2005  01:07:25      <DIR> ..
Jan 01,1970  01:00:00          0 
Jan 01,1970  01:00:00          0 
Jan 01,1970  01:00:00          0 
Jan 01,1970  01:00:00          0 
Jan 01,1970  01:00:00          0 
Jan 01,1970  01:00:00          0 
Jan 01,1970  01:00:00          0 
Jan 01,1970  01:00:00          0 
Jan 01,1970  01:00:00          0 
Jan 01,1970  01:00:00          0 
Jan 01,1970  01:00:00          0 
Jan 01,1970  01:00:00          0 
Jan 01,1970  01:00:00          0 
Jan 01,1970  01:00:00          0 
Jan 01,1970  01:00:00          0 
Jan 01,1970  01:00:00          0 
Jan 01,1970  01:00:00          0 
Jan 01,1970  01:00:00          0 
Jan 01,1970  01:00:00          0 
Jan 01,1970  01:00:00          0 
Jan 01,1970  01:00:00          0 
Jan 01,1970  01:00:00          0 
Jan 01,1970  01:00:00          0 
Jan 01,1970  01:00:00          0 
Jan 01,1970  01:00:00          0 
Jan 01,1970  01:00:00          0 
Jan 01,1970  01:00:00          0 
Jan 01,1970  01:00:00          0 
Jan 01,1970  01:00:00          0 
Jan 01,1970  01:00:00          0 
Nov 05,2004  02:27:17        801 .cshrc
Nov 05,2004  02:27:17        801 .cshrc
Nov 05,2004  02:27:17        251 .profile
Nov 05,2004  02:27:17        251 .profile
Feb 17,2005  18:21:41      <DIR> .snap
Jan 01,1970  01:00:00          0 .snap
May 11,2005  18:31:27       6192 COPYRIGHT
May 11,2005  18:31:27       6192 COPYRIGHT
May 11,2005  18:21:44      <DIR> bin
Jan 01,1970  01:00:00          0 bin
Jan 01,1970  01:00:00          0 boot
Jul 16,2005  00:41:53      <DIR> boot
Feb 17,2005  18:21:48      <DIR> cdrom
Jan 01,1970  01:00:00          0 cdrom
Feb 17,2005  18:29:51         10 compat -> usr/compat
Feb 17,2005  18:29:51         10 compat -> usr/compat
Jan 01,1970  01:00:00          0 dev
Feb 17,2005  18:21:41      <DIR> dev
Jan 01,1970  01:00:00          0 dist
Feb 17,2005  18:21:50      <DIR> dist
Sep 16,2005  01:07:22       4096 entropy
Sep 15,2005  21:27:31      <DIR> etc
Jan 01,1970  01:00:00          0 etc
Mar 10,2005  21:53:07     655360 festival-synthesis-.core
Mar 10,2005  21:53:07     655360 festival-synthesis-.core
Feb 17,2005  18:21:42      <DIR> home
Jan 01,1970  01:00:00          0 home
Mar 23,2005  22:03:23      <DIR> ipod
Jan 01,1970  01:00:00          0 ipod
Jan 01,1970  01:00:00          0 lib
Jul 16,2005  04:50:21      <DIR> lib
May 11,2005  18:21:39      <DIR> libexec
Jan 01,1970  01:00:00          0 libexec
Sep 16,2005  00:05:35      <DIR> lost+found
Nov 05,2004  02:23:32      <DIR> mnt
Jan 01,1970  01:00:00          0 mnt
Jan 01,1970  01:00:00          0 pappas
May 22,2005  14:50:23      <DIR> pappas
Apr 10,2005  21:10:07      <DIR> proc
Jan 01,1970  01:00:00          0 proc
May 11,2005  18:22:29      <DIR> rescue
Jan 01,1970  01:00:00          0 rescue
Sep 10,2005  13:08:14      <DIR> root
Jan 01,1970  01:00:00          0 root
May 29,2005  13:26:54      <DIR> s
Jan 01,1970  01:00:00          0 s
Jan 01,1970  01:00:00          0 sbin
May 11,2005  18:23:56      <DIR> sbin
Feb 17,2005  18:21:48      <DIR> stand
Jan 01,1970  01:00:00          0 stand
Jun 15,2005  18:10:54         11 sys -> usr/src/sys
Jun 15,2005  18:10:54         11 sys -> usr/src/sys
Sep 15,2005  21:27:32      <DIR> tmp
Jan 01,1970  01:00:00          0 tmp
Jan 01,1970  01:00:00          0 usr
Feb 17,2005  18:21:45      <DIR> usr
Jan 01,1970  01:00:00          0 var
Feb 17,2005  18:21:46      <DIR> var
Mar 06,2005  15:16:20      <DIR> vcdrom
Jan 01,1970  01:00:00          0 vcdrom
Jan 01,1970  01:00:00          0 win
Aug 27,2005  12:12:03      <DIR> win

Subfolders of / on the same filesystem are affected as well. I can't see
any corruption inside files though.

My problems wouldn't be so alarming if not for the fact that two friends
of mine have had very similar problems (but unfortunately never told
anyone about it). Their root filesystem suddenly went haywire on
seemingly fine disks for no apparent reason.


Some general info:

I had other filesystems (stored on the same and other disks) mounted on
/usr, /var, /home, and /win.

I'm running FreeBSD 5.4. My friends can't remember which version they
ran at the time, but they both think it was 5.x. 

In case they might be helpful, I'll include the contents of /etc and
/boot (both show signs of corruption), as well as the output of the
bsdlabel program (part of the ufs2tools package) when ran on the disk in
question:


/etc:

Sep 15,2005  21:27:31      <DIR> .
Sep 16,2005  01:07:25      <DIR> ..
Jun 06,2005  12:49:27          0 COPYRIGHT
Aug 16,2005  13:54:44      <DIR> X11
Feb 17,2005  18:22:04         12 aliases -> mail/aliases
Mar 30,2005  15:01:01      65536 aliases.db
Nov 05,2004  02:27:16        209 amd.map
Nov 05,2004  02:27:16       1234 apmd.conf
Mar 12,2005  16:33:49        225 auth.conf
Nov 05,2004  02:27:16        231 auth.conf~
Mar 12,2005  22:12:43          0 blocks.fb
Feb 17,2005  18:22:02      <DIR> bluetooth
Apr 28,2005  21:09:42          7 console
Nov 05,2004  02:27:16        737 crontab
Nov 05,2004  02:27:16        108 csh.cshrc
Nov 05,2004  02:27:16        481 csh.login
Nov 05,2004  02:27:16        110 csh.logout
May 19,2005  18:25:36      <DIR> defaults
May 11,2005  18:29:51       4846 devd.conf
Nov 05,2004  02:27:16       2071 devfs.conf
Aug 20,2005  18:55:00         23 dhclient-enter-hooks
Aug 20,2005  18:54:57         23 dhclient-enter-hooks~
Nov 05,2004  02:27:16        267 dhclient.conf
Nov 05,2004  02:27:16       6621 disktab
Nov 05,2004  02:27:17          0 dumpdates
Nov 05,2004  02:27:16        142 fbtab
Sep 03,2005  22:43:21        509 fstab
Aug 27,2005  11:26:13        511 fstab~
Nov 05,2004  02:27:16        245 ftpusers
Nov 05,2004  02:27:16       5904 gettytab
May 11,2005  18:26:29      <DIR> gnats
Aug 04,2005  16:42:42        487 group
Feb 23,2005  18:52:40        443 group~
May 07,2005  20:08:15         25 host.aliases
May 07,2005  20:04:04         21 host.aliases~
Feb 17,2005  19:57:39         60 host.conf
Feb 18,2005  20:07:35       1107 hosts
Nov 05,2004  02:27:16       3186 hosts.allow
Nov 05,2004  02:27:16        111 hosts.equiv
Nov 05,2004  02:27:16         99 hosts.lpd
Feb 17,2005  22:51:26       1106 hosts~
Apr 10,2005  22:15:42       5378 inetd.conf
Apr 10,2005  22:12:07       5380 inetd.conf~
Feb 17,2005  18:22:02      <DIR> isdn
May 11,2005  18:23:10         30 libfstab -> ../en_AU.ISO8859-1/LC_MONETARY
Aug 27,2005  23:08:06        511 libfstab~
Feb 27,2005  22:23:18        390 libmap.conf
Feb 17,2005  20:24:09        675 localtime
Nov 05,2004  02:27:16        619 locate.rc
Nov 05,2004  02:27:16       1847 login.access
Mar 12,2005  16:29:14       6522 login.conf
Mar 12,2005  16:30:16      65536 login.conf.db
Nov 05,2004  02:27:16       6522 login.conf~
Nov 05,2004  02:27:16        564 mac.conf
Jul 31,2005  00:36:29      <DIR> mail
Nov 05,2004  02:27:16        106 mail.rc
Mar 23,2005  12:59:50        117 mailcap
Mar 23,2005  12:57:00        117 mailcap~
Jul 20,2005  13:19:42      13206 make.conf
Jul 16,2005  04:08:09      13132 make.conf.bak
Mar 19,2005  11:40:53      13132 make.conf2~
Jul 20,2005  13:19:14      13206 make.conf~
Jul 20,2005  13:19:14       1087 manpath.config
Jul 16,2005  04:08:09        946 manpath.config.bak
Aug 04,2005  16:42:42       1884 master.passwd
Mar 12,2005  17:07:09       1750 master.passwd~
Jun 06,2005  12:45:59          0 motd
May 11,2005  18:16:36       1111 motd.old
May 11,2005  18:27:42      <DIR> mtree
May 02,2005  19:44:26        153 named.conf~
Sep 15,2005  21:27:31         21 namedb -> /var/named/etc/namedb
Nov 05,2004  02:27:16        783 netconfig
Nov 05,2004  02:27:16       2362 netstart
May 11,2005  18:30:09      13696 network.subr
Nov 05,2004  02:27:16        365 networks
Mar 09,2005  21:33:05       1932 newsyslog.conf
Mar 09,2005  21:28:46       1935 newsyslog.conf~
Nov 05,2004  02:27:16       1701 nsmb.conf
Feb 17,2005  19:57:39        113 nsswitch.conf
Nov 05,2004  02:23:32      <DIR> ntp
Nov 05,2004  02:27:16        432 opieaccess
Feb 17,2005  20:01:00          0 opiekeys
Feb 17,2005  18:22:02      <DIR> pam.d
Aug 04,2005  16:42:42       1582 passwd
Feb 22,2005  19:50:54       1394 passwd~
May 11,2005  18:30:56       5814 pccard_ether
Feb 17,2005  18:22:03      <DIR> periodic
Mar 21,2005  11:50:51        137 periodic.conf
Nov 05,2004  02:27:16       3567 pf.conf
Nov 05,2004  02:27:16      22556 pf.os
Nov 05,2004  02:27:16        293 phones
Feb 17,2005  18:22:03      <DIR> ppp
Apr 23,2005  16:34:38       2152 printcap
Apr 23,2005  16:32:27       2152 printcap~
May 07,2005  20:07:03        842 profile
May 07,2005  20:06:58        841 profile~
May 11,2005  18:30:11       5807 protocols
Aug 04,2005  16:42:42      40960 pwd.db
Nov 05,2004  02:27:16       2716 rc
May 11,2005  18:30:16       5298 rc.bsdextended
Aug 20,2005  19:09:38        710 rc.conf
Jun 18,2005  19:06:23        693 rc.conf.orig
Aug 20,2005  19:09:34        710 rc.conf~
May 11,2005  18:29:47      <DIR> rc.d
Nov 05,2004  02:27:16       9569 rc.firewall
Nov 05,2004  02:27:16       8905 rc.firewall6
Nov 05,2004  02:27:16       2165 rc.resume
Nov 05,2004  02:27:16       5785 rc.sendmail
May 11,2005  18:30:27       3211 rc.shutdown
May 11,2005  18:30:31      30905 rc.subr
Nov 05,2004  02:27:16       2209 rc.suspend
Nov 05,2004  02:27:16       2392 remote
Sep 15,2005  21:27:31         52 resolv.conf
Aug 20,2005  18:32:44         23 resolv.conf.no
Jul 15,2005  21:07:06         69 resolv.conf~
Feb 17,2005  18:22:04         13 rmt -> /usr/sbin/rmt
Nov 05,2004  02:27:16       1674 rpc
Nov 05,2004  02:27:16      73410 services
Jun 02,2005  12:40:40        276 shells
Nov 05,2004  02:23:32      <DIR> skel
May 11,2005  18:23:10         30 spwd.db -> ../pt_PT.ISO8859-1/LC_MONETARY
Feb 17,2005  20:00:07      <DIR> ssh
May 11,2005  18:29:49      <DIR> ssl
Nov 05,2004  02:27:16        367 sysctl.conf
Jun 06,2005  12:37:53       1352 syslog.conf
May 02,2005  20:24:16       1344 syslog.conf~
Feb 17,2005  18:22:04         23 termcap -> /usr/share/misc/termcap
Apr 28,2005  21:09:29          7 tty
Mar 12,2005  17:18:17       7670 ttys
Nov 05,2004  02:27:16       7658 ttys~
May 11,2005  18:30:52       2032 usbd.conf
Jun 16,2005  22:31:19          0 wall_cmos_clock


/boot:

ul 16,2005  00:41:53      <DIR> .
Sep 16,2005  01:07:25      <DIR> ..
May 11,2005  18:23:16       7007 beastie.4th
Mar 11,2005  23:53:02       6757 beastie.4th~
May 11,2005  18:23:15       8192 boot
May 11,2005  18:23:15        512 boot0
May 11,2005  18:23:15        512 boot0sio
May 11,2005  18:23:15        512 boot1
May 11,2005  18:23:15       7680 boot2
May 11,2005  18:23:15       1184 cdboot
May 11,2005  18:23:16      <DIR> defaults
Nov 05,2004  02:27:17       1982 device.hints
May 11,2005  18:23:16       2249 frames.4th
Jun 22,2005  01:16:24      <DIR> grub
Jun 15,2005  18:11:05      <DIR> kernel
Jun 15,2005  15:59:52      <DIR> kernel.old
May 11,2005  18:23:16     262144 loader
May 11,2005  18:23:16       7772 loader.4th
Jul 16,2005  00:41:53         58 loader.conf
May 11,2005  18:23:10         30 loader.conf.orig ->
../no_NO.ISO8859-1/LC_MONETARY
Feb 17,2005  18:29:51          0 loader.conf~
May 11,2005  18:23:16      13893 loader.help
Nov 05,2004  02:26:31     212992 loader.old
Nov 05,2004  02:26:31        370 loader.rc
May 11,2005  18:23:15        512 mbr
Jul 16,2005  00:41:48      <DIR> modules
May 11,2005  18:23:16     264192 pxeboot
May 11,2005  18:23:16        692 screen.4th
May 11,2005  18:23:16      36440 support.4th


bsdlabel output:

# Drive 0, Slice 1:
type: ESDI
disk: 
label: 
flags:
bytes/sector: 512
sectors/track: 63
tracks/cylinder: 255
sectors/cylinder: 16065
cylinders: 4865
sectors/unit: 78165360
rpm: 3600
interleave: 1
trackskew: 0
cylinderskew: 0
headswitch: 0		# milliseconds
track-to-track seek: 0	# milliseconds
drivedata: 0 

8 partitions:
#        size   offset    fstype   [fsize bsize bps/cpg]
  a:  1048576        0    4.2BSD        0     0     0 
  b:  3145728  1048576      swap                    
  c: 78164624        0    unused        0     0         # "raw" part,
don't edit
  d:  2097152  4194304    4.2BSD        0     0     0 
  e: 62914560  6291456    4.2BSD        0     0     0 
  f:  8957952 69206016    4.2BSD        0     0     0 



I haven't lost any important data because of these problems, but others
might..
/Ulf


More information about the freebsd-questions mailing list