Unrecoverable UFS error but only with gmirror
Stefan Bethke
stb at lassitu.de
Sat Jul 11 12:54:04 UTC 2009
gmirror and/or ufs got into an odd state after a panic, where fsck
could not fix errors on the mirror device, but each member filesystem
was fine. Even after destroying the mirror, fixing all three member
file systems with fsck, and recreating the mirror with just a single
member fsck found the same errors on the mirror device.
I ended up newfs'ing the mirror and copying all data over from one of
the old mirror members (and then reattaching the two other members),
and all seems to be OK now, but it's still puzzling to me where those
errors originated from.
Blow by blow account below.
Stefan
I'm running my root fs (UFS with softupdates) on a three disk gmirror:
# gmirror status
Name Status Components
mirror/diesel_root COMPLETE ad6p2
ad8p2
ad4p2
mirror/diesel_swap DEGRADED ad6p3
ad8p3
ad4p3
After a panic, fsck could not fix the root fs:
Trying to mount root from ufs:/dev/mirror/diesel_root
WARNING: / was not properly dismounted
/: mount pending error: blocks 52 files 4
Entropy harvesting: interrupts ethernet point_to_point kickstart.
/dev/mirror/diesel_root: CANNOT READ BLK: 8388576
/dev/mirror/diesel_root: UNEXPECTED SOFT UPDATE INCONSISTENCY; RUN
fsck MANUALLY.
Automatic file system check failed; help!
ERRORJOul 11 03:10:47 init: /bin/sh on /etc/rc terminated abnormally,
ng to single user mode
Enter full pathname of shell or RETURN for /bin/sh: GEOM_MIRROR:
Device diesel_root: rebuilding provider ad6p2 finished.
GEOM_MIRROR: Device diesel_root: rebuilding provider ad4p2 finished.
# fsck -y
** /dev/mirror/diesel_root
** Last Mounted on /
** Root file system
** Phase 1 - Check Blocks and Sizes
CANNOT READ BLK: 8388576
UNEXPECTED SOFT UPDATE INCONSISTENCY
CONTINUE? yes
THE FOLLOWING DISK SECTORS COULD NOT BE READ: 8388607,
** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
UNREF FILE I=424146 OWNER=root MODE=140666
SIZE=0 MTIME=Jul 10 22:00 2009
CLEAR? yes
** Phase 5 - Check Cyl groups
FREE BLK COUNT(S) WRONG IN SUPERBLK
SALVAGE? yes
SUMMARY INFORMATION BAD
SALVAGE? yes
BLK(S) MISSING IN BIT MAPS
SALVAGE? yes
118582 files, 1357018 used, 672013 free (31549 frags, 80058 blocks,
1.6% fragmentation)
***** FILE SYSTEM STILL DIRTY *****
***** FILE SYSTEM WAS MODIFIED *****
***** PL/: reload pending error: blocks 52 files 4
EASE RERUN FSCK *****
# fsck -y
** /dev/mirror/diesel_root
** Last Mounted on /
** Root file system
** Phase 1 - Check Blocks and Sizes
CANNOT READ BLK: 8388576
UNEXPECTED SOFT UPDATE INCONSISTENCY
CONTINUE? yes
THE FOLLOWING DISK SECTORS COULD NOT BE READ: 8388607,
** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
** Phase 5 - Check Cyl groups
1a18582 files, 1357018 used, 672013 free (31549 frags, 80058 bloc6%
fragmen/: reload pending error: blocks 52 files 4
tation)
***** FILE SYSTEM STILL DIRTY *****
***** PLEASE RERUN FSCK *****
# dd if=/dev/mirror/diesel_root bs=1m of=/dev/null
4095+1 records in
4095+1 records out
4294966784 bytes transferred in 36.500572 secs (117668479 bytes/sec)
However, I could read the whole mirror device with dd without
problems. Also note the absence of any i/o errors.
So I detached one of the mirrors and ran fsck on the raw partition:
# gmirror remove diesel_root ad8p2
# GEOM_MIRROR: Device diesel_root: provider ad8p2 destroyed.
# fsck -t ufs -y /dev/ad8p2
** /dev/ad8p2
** Last Mounted on /
** Phase 1 - Check Blocks and Sizes
** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
** Phase 5 - Check Cyl groups
118582 files, 1357018 used, 672013 free (31549 frags, 80058 blocks,
1.6% fragmentation)
***** FILE SYSTEM MARKED CLEAN *****
Odd. So I decided to boot off that partition, and try and fix the
filesystem by recreating the mirror based from a single, good partition:
# gmirror remove diesel_root ad4p2
# gmirror remove diesel_root ad6p2
# gmirror remove diesel_root ad8p2
# fsck -t ufs -y /dev/ad4p2
# gmirror label diesel_root ad4p2
# fsck -t ufs -y /dev/mirror/diesel_root
** /dev/mirror/diesel_root
** Last Mounted on /
** Phase 1 - Check Blocks and Sizes
CANNOT READ BLK: 8388576
UNEXPECTED SOFT UPDATE INCONSISTENCY
CONTINUE? yes
THE FOLLOWING DISK SECTORS COULD NOT BE READ: 8388607,
** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
** Phase 5 - Check Cyl groups
118582 files, 1357018 used, 672013 free (31549 frags, 80058 blocks,
1.6% fragmentation)
***** FILE SYSTEM MARKED DIRTY *****
***** PLEASE RERUN FSCK *****
# gmirror status
Name Status Components
mirror/diesel_swap COMPLETE ad4p3
ad6p3
ad8p3
mirror/diesel_root COMPLETE ad4p2
At which point I decided to newfs the mirror and copy all data over
from one of the other (former) mirror members. That turned out to
work without problems.
--
Stefan Bethke <stb at lassitu.de> Fon +49 151 14070811
More information about the freebsd-current
mailing list