soft updates / background fsck directory link count bug

Don Lewis truckman at FreeBSD.org
Fri Sep 23 14:47:16 PDT 2005


Background info:

    One of the file system inconsistencies that is "expected" by
    soft updates and that can be repaired by background fsck is a link
    count that is too high.

    Historically, rmdir would fail on directories with a link count
    greater than 2 even if they were empty.  This was changed a few
    months ago in ufs_vnops.c 1.269.

It appears that there is some sort of interaction between soft updates
and background fsck that results in the link count of the parent of one
of these directories being double decremented, resulting in the file
system being put into an invalid state.

The following transcript shows the link count of a leaf directory
(/mnt/tmp/a/b) being adjusted upwards, the file system being mounted,
the leaf directory being removed, the file system unmounted, and then
being successfully repaired by a foreground fsck.

# fsck -fp /dev/da0s2a
/dev/da0s2a: 374158 files, 2551020 used, 2894085 free (25677 frags, 358551 block
s, 0.5% fragmentation)
# mount /dev/da0s2a /mnt
# mkdir /mnt/tmp/a /mnt/tmp/a/b
# ls -lid /mnt/tmp/a /mnt/tmp/b
ls: /mnt/tmp/b: No such file or directory
320920 drwxr-xr-x  3 root  wheel  512 Sep 23 13:31 /mnt/tmp/a
# ls -lid /mnt/tmp/a /mnt/tmp/a/b
320920 drwxr-xr-x  3 root  wheel  512 Sep 23 13:31 /mnt/tmp/a
320921 drwxr-xr-x  2 root  wheel  512 Sep 23 13:31 /mnt/tmp/a/b
# umount /mnt
# fsdb /dev/da0s2a
** /dev/da0s2a
Editing file system `/dev/da0s2a'
Last Mounted on /mnt  
current inode: directory
I=2 MODE=40755 SIZE=512
        MTIME=Sep 20 11:54:46 2005 [0 nsec]
        CTIME=Sep 20 11:54:46 2005 [0 nsec]   
        ATIME=Sep 22 21:36:58 2005 [0 nsec]
OWNER=root GRP=wheel LINKCNT=8 FLAGS=0 BLKCNT=4 GEN=53557245
fsdb (inum: 2)> inode 320921
current inode: directory
I=320921 MODE=40755 SIZE=512
        MTIME=Sep 23 13:31:30 2005 [0 nsec]
        CTIME=Sep 23 13:31:30 2005 [0 nsec]
        ATIME=Sep 23 13:31:30 2005 [0 nsec]
OWNER=root GRP=wheel LINKCNT=2 FLAGS=0 BLKCNT=4 GEN=1292b1d5
fsdb (inum: 320921)> uplink
inode 320921 link count now 3
fsdb (inum: 320921)> q

***** FILE SYSTEM STILL DIRTY *****
*** FILE SYSTEM MARKED DIRTY
*** BE SURE TO RUN FSCK TO CLEAN UP ANY DAMAGE
*** IF IT WAS MOUNTED, RE-MOUNT WITH -u -o reload
# mount /dev/da0s2a /mnt
WARNING: /mnt was not properly dismounted
# ls -lid /mnt/tmp/a /mnt/tmp/a/b
320920 drwxr-xr-x  3 root  wheel  512 Sep 23 13:31 /mnt/tmp/a
320921 drwxr-xr-x  3 root  wheel  512 Sep 23 13:31 /mnt/tmp/a/b
# rmdir /mnt/tmp/a/b
# umount /mnt
# fsdb /dev/da0s2a
** /dev/da0s2a
Editing file system `/dev/da0s2a'
Last Mounted on /mnt
current inode: directory
I=2 MODE=40755 SIZE=512
        MTIME=Sep 20 11:54:46 2005 [0 nsec]
        CTIME=Sep 20 11:54:46 2005 [0 nsec]
        ATIME=Sep 22 21:36:58 2005 [0 nsec]
OWNER=root GRP=wheel LINKCNT=8 FLAGS=0 BLKCNT=4 GEN=53557245
fsdb (inum: 2)> inode 320920
current inode: directory
I=320920 MODE=40755 SIZE=512
        MTIME=Sep 23 13:33:56 2005 [0 nsec]
        CTIME=Sep 23 13:33:56 2005 [0 nsec]
        ATIME=Sep 23 13:31:30 2005 [0 nsec]
OWNER=root GRP=wheel LINKCNT=3 FLAGS=0 BLKCNT=4 GEN=33d3740d
fsdb (inum: 320920)> inode 320921
current inode: directory
I=320921 MODE=40755 SIZE=0
        MTIME=Sep 23 13:34:05 2005 [0 nsec]
        CTIME=Sep 23 13:34:05 2005 [0 nsec]
        ATIME=Sep 23 13:33:56 2005 [0 nsec]
OWNER=root GRP=wheel LINKCNT=1 FLAGS=0 BLKCNT=0 GEN=1292b1d5
fsdb (inum: 320921)> q

***** FILE SYSTEM STILL DIRTY *****
*** FILE SYSTEM MARKED DIRTY
*** BE SURE TO RUN FSCK TO CLEAN UP ANY DAMAGE
*** IF IT WAS MOUNTED, RE-MOUNT WITH -u -o reload
# fsck -fp /dev/da0s2a
/dev/da0s2a: LINK COUNT DIR I=320920  OWNER=root MODE=40755
/dev/da0s2a: SIZE=512 MTIME=Sep 23 13:33 2005  COUNT 3 SHOULD BE 2 (ADJUSTED)
/dev/da0s2a: ZERO LENGTH DIR I=320921  OWNER=root MODE=40755
/dev/da0s2a: SIZE=0 MTIME=Sep 23 13:34 2005  (CLEARED)
/dev/da0s2a: FREE BLK COUNT(S) WRONG IN SUPERBLK (SALVAGED)  
/dev/da0s2a: SUMMARY INFORMATION BAD (SALVAGED)
/dev/da0s2a: BLK(S) MISSING IN BIT MAPS (SALVAGED)
/dev/da0s2a: 374159 files, 2551021 used, 2894084 free (25676 frags, 358551 block
s, 0.5% fragmentation)
# fsck -fp /dev/da0s2a
/dev/da0s2a: 374159 files, 2551021 used, 2894084 free (25676 frags, 358551 block
s, 0.5% fragmentation)



The following transcript demonstrates what happens if a background fsck
is run after the leaf directory is removed.  What is interesting is that
after the directory the leaf directory has been removed, the effective
link count of the parent directory (displayed by ls) has been
decremented from 3 to 2, whereas the on-disk link count shown by fsdb is
still 3.  The background fsck appears to detect the link count as 3, and
executes the sysctl call to decrement the link count, causing both the
effective and actual link counts to be decremented to 1.

My suspicion is that the physical update of the parent directory's link
count after the rmdir of the leaf directory has been deferred until the
leaf directory's inode is zeroed, which turns out to be an indefinite
wait because the inode doesn't get zeroed until fsck is run.

# fsck -fp /dev/da0s2a
/dev/da0s2a: 374158 files, 2551020 used, 2894085 free (25677 frags, 358551 block
s, 0.5% fragmentation)
# mount /dev/da0s2a /mnt
# mkdir /mnt/tmp/a /mnt/tmp/a/b
# ls -lid /mnt/tmp/a /mnt/tmp/a/b
320920 drwxr-xr-x  3 root  wheel  512 Sep 23 13:58 /mnt/tmp/a
320921 drwxr-xr-x  2 root  wheel  512 Sep 23 13:58 /mnt/tmp/a/b
# umount /mnt
# fsdb /dev/da0s2a
** /dev/da0s2a
Editing file system `/dev/da0s2a'
Last Mounted on /mnt
current inode: directory
I=2 MODE=40755 SIZE=512
        MTIME=Sep 20 11:54:46 2005 [0 nsec]
        CTIME=Sep 20 11:54:46 2005 [0 nsec]
        ATIME=Sep 22 21:36:58 2005 [0 nsec]
OWNER=root GRP=wheel LINKCNT=8 FLAGS=0 BLKCNT=4 GEN=53557245
fsdb (inum: 2)> inode 320921
current inode: directory
I=320921 MODE=40755 SIZE=512
        MTIME=Sep 23 13:58:46 2005 [0 nsec]
        CTIME=Sep 23 13:58:46 2005 [0 nsec]
        ATIME=Sep 23 13:58:46 2005 [0 nsec]
OWNER=root GRP=wheel LINKCNT=2 FLAGS=0 BLKCNT=4 GEN=10aac654
fsdb (inum: 320921)> uplink
inode 320921 link count now 3
fsdb (inum: 320921)> q

***** FILE SYSTEM STILL DIRTY *****
*** FILE SYSTEM MARKED DIRTY
*** BE SURE TO RUN FSCK TO CLEAN UP ANY DAMAGE
*** IF IT WAS MOUNTED, RE-MOUNT WITH -u -o reload
# mount /dev/da0s2a /mnt
WARNING: /mnt was not properly dismounted
# ls -lid /mnt/tmp/a /mnt/tmp/a/b
320920 drwxr-xr-x  3 root  wheel  512 Sep 23 13:58 /mnt/tmp/a
320921 drwxr-xr-x  3 root  wheel  512 Sep 23 13:58 /mnt/tmp/a/b
# rmdir /mnt/tmp/a/b
# sync
# sleep 60
# ls -lid /mnt/tmp/a /mnt/tmp/a/b
ls: /mnt/tmp/a/b: No such file or directory
320920 drwxr-xr-x  2 root  wheel  512 Sep 23 14:00 /mnt/tmp/a
# fsdb /dev/da0s2a
** /dev/da0s2a (NO WRITE)
Editing file system `/dev/da0s2a'
Last Mounted on /mnt
current inode: directory
I=2 MODE=40755 SIZE=512
        MTIME=Sep 20 11:54:46 2005 [0 nsec]
        CTIME=Sep 20 11:54:46 2005 [0 nsec]
        ATIME=Sep 22 21:36:58 2005 [0 nsec]
OWNER=root GRP=wheel LINKCNT=8 FLAGS=0 BLKCNT=4 GEN=53557245
fsdb (inum: 2)> inode 320920
current inode: directory
I=320920 MODE=40755 SIZE=512
        MTIME=Sep 23 14:00:00 2005 [0 nsec]
        CTIME=Sep 23 14:00:00 2005 [0 nsec]
        ATIME=Sep 23 13:58:46 2005 [0 nsec]
OWNER=root GRP=wheel LINKCNT=3 FLAGS=0 BLKCNT=4 GEN=33d3740f
fsdb (inum: 320920)> q
SETTING DIRTY FLAG IN READ_ONLY MODE

UNEXPECTED SOFT UPDATE INCONSISTENCY
*** FILE SYSTEM MARKED DIRTY
*** BE SURE TO RUN FSCK TO CLEAN UP ANY DAMAGE
*** IF IT WAS MOUNTED, RE-MOUNT WITH -u -o reload
# fsck -B -p /dev/da0s2a
/dev/da0s2a: LINK COUNT DIR I=320920  OWNER=root MODE=40755
/dev/da0s2a: SIZE=512 MTIME=Sep 23 14:00 2005  COUNT 3 SHOULD BE 2 (ADJUSTED)
/dev/da0s2a: ZERO LENGTH DIR I=320921  OWNER=root MODE=40755
/dev/da0s2a: SIZE=0 MTIME=Sep 23 14:00 2005  (CLEARED)
/dev/da0s2a: Reclaimed: 0 directories, 1 files, 0 fragments
/dev/da0s2a: 374160 files, 2551021 used, 2894084 free (25676 frags, 358551 block
s, 0.5% fragmentation)
# ls -lid /mnt/tmp/a  
320920 drwxr-xr-x  1 root  wheel  512 Sep 23 14:00 /mnt/tmp/a
# fsdb /dev/da0s2a
** /dev/da0s2a (NO WRITE)
Editing file system `/dev/da0s2a'
Last Mounted on /mnt
current inode: directory
I=2 MODE=40755 SIZE=512
        MTIME=Sep 20 11:54:46 2005 [0 nsec]
        CTIME=Sep 20 11:54:46 2005 [0 nsec]
        ATIME=Sep 22 21:36:58 2005 [0 nsec]
OWNER=root GRP=wheel LINKCNT=8 FLAGS=0 BLKCNT=4 GEN=53557245
fsdb (inum: 2)> inode 320920
current inode: directory
I=320920 MODE=40755 SIZE=512
        MTIME=Sep 23 14:00:00 2005 [0 nsec]
        CTIME=Sep 23 14:05:21 2005 [0 nsec]
        ATIME=Sep 23 13:58:46 2005 [0 nsec]
OWNER=root GRP=wheel LINKCNT=1 FLAGS=0 BLKCNT=4 GEN=33d3740f
fsdb (inum: 320920)> q
SETTING DIRTY FLAG IN READ_ONLY MODE

UNEXPECTED SOFT UPDATE INCONSISTENCY
*** FILE SYSTEM MARKED DIRTY
*** BE SURE TO RUN FSCK TO CLEAN UP ANY DAMAGE
*** IF IT WAS MOUNTED, RE-MOUNT WITH -u -o reload
# umount /mnt
# fsck -fp /dev/da0s2a
/dev/da0s2a: LINK COUNT DIR I=320920  OWNER=root MODE=40755
/dev/da0s2a: SIZE=512 MTIME=Sep 23 14:00 2005  COUNT 1 SHOULD BE 2
/dev/da0s2a: LINK COUNT INCREASING
/dev/da0s2a: UNEXPECTED SOFT UPDATE INCONSISTENCY; RUN fsck MANUALLY.





More information about the freebsd-current mailing list