bin/115174: growfs needs zero-writing for safe filesystem expansion

[Veronica]

>Number:         115174
>Category:       bin
>Synopsis:       growfs needs zero-writing for safe filesystem expansion
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Fri Aug 03 16:30:01 GMT 2007
>Closed-Date:
>Last-Modified:
>Originator:     Veronica
>Release:        FreeBSD 6.2-RELEASE
>Organization:
Fluffles.net
>Environment:
FreeBSD phlox.fluffles.net 6.2-RELEASE FreeBSD 6.2-RELEASE #1: Mon Jul 30 19:08:02 CEST 2007     root@:/usr/obj/usr/src/sys/PHLOX  amd64
>Description:
I am writing capacity expansion support for geom_raid5, created by Arne Woerner. This expansion works great, yet i also need growfs to expand the filesystem on it afterwards. Through testing Arne and i discovered a bug or weakness in growfs: it will wrongfully assume the expanded disk area is full of zero's.

Growfs itself does not show any error and seems to work fine. It is the fsck afterwards that *can* kill a lot of data. It's safe to assume this is caused by 'stale' data on the expanded disk area, which confuses fsck because it thinks its part of the filesystem where in fact it is not 'active' filesystem data. In most cases, fsck will report a ton of errors, clear bogus files and leave you with a healthy filesystem where only your original files remained.

In some cases, though, i managed to kill off about 40% of my original files after expanding using growfs. I did a fsck before using growfs to verify the filesystem was clean, after using growfs i ran fsck again and it did an 'extra step' (2b i think) to find more DUPs (duplicate inodes i assume?). After fsck finished, it left me with a crippled filesystem where lots of files were gone. Also fsck was not able to resolve all issues, no matter how much i ran fsck there remained some errors. I suspect this is because the bogus data in the expanded area was part of a real UFS filesystem in the past, and as such many UFS file structures remained that would have confused fsck.

This behavior is not acceptable! Simply referring to the "you should have made a backup" is no excuse for having a flawed growfs. Growfs is required to make a proper RAID5 expansion possible - a feature that will be quite hot when geom_raid5 is integrated into FreeBSD sourcetree. Please help resolve this issue and pave the way for the "Just Works" principle that makes software popular and sexy.
>How-To-Repeat:
* create a filesystem shorter than the device, using the -s argument and populate with data:

newfs -s 30000 /dev/ad4
mount /dev/ad4 /mnt
(copy data onto /mnt until its full)
umount /dev/ad4

* now write random data to the area which will be expanded
dd if=/dev/urandom of=/dev/ad4 oseek=30000 bs=512

* check filesystem for errors (should be clean)
fsck -t ufs /dev/ad4

* grow filesystem
growfs /dev/ad4

* check again for errors
fsck -t ufs -y /dev/ad4

Result:
You will find tons if not millions of errors. Most will be about the bogus data in the expanded area. But in a few cases fsck after growfs destroyed 'original' files as well, deleting over 40% of the original filesystem.
>Fix:
The solution is rather simple: growfs should write zeroes to the expanding area before writing the metadata structures. This will ensure no bogus filesystem structures exist which may lead to fsck mutilating the filesystem. Until such feature exists, the manpage should make it very clear that a zero-write procedure has to be performed manually, using dd.

Workaround:
before using growfs, blank the area using dd:
dd if=/dev/zero of=<disk> bs=512 oseek=<sector starting with expansion>

Contact via email for details. - Veronica

>Release-Note:
>Audit-Trail:
>Unformatted: