kern/92272: Filling a filesystem while creating a snapshot on it locks the system

[John Kozubik]

>Number:         92272
>Category:       kern
>Synopsis:       Filling a filesystem while creating a snapshot on it locks the system
>Confidential:   no
>Severity:       serious
>Priority:       high
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Tue Jan 24 16:40:05 GMT 2006
>Closed-Date:
>Last-Modified:
>Originator:     John Kozubik
>Release:        FreeBSD 6.0-RELEASE
>Organization:
www.kozubik.com
>Environment:
FreeBSD test.kozubik.com 6.0-RELEASE FreeBSD 6.0-RELEASE #0: Tue Nov 29 16:44:37 PST 2005     user at test.kozubik.com:/usr/src/sys/i386/compile/test  i386
>Description:
If a filesystem is completely full (approaching 110% in `df` output on most systems), mksnap_ffs will refuse to begin a snapshot for that reason.  There seem to be no negative consequences.

However, if the filesystem is not quite completely full, mksnap_ffs will (correctly) begin creating the snapshot as expected.  If, during the course of snapshot creation, the filesystem being snapshotted becomes full, mksnap_ffs will exit with an error exactly as it does if it is started on an already full filesystem.

However, several hours later, the system will lock up completely.  It still responds to pings, and open connections to and from it remain in that state, but they cannot be used and new connections cannot be established.
>How-To-Repeat:
dd, or otherwise fill a filesystem, until it is not quite full - 108% or so, as shown in the output of `df`.  Then, in one terminal, prepare a mksnap_ffs command line that will create a snapshot on the filesystem, and in another terminal create a dd line that will more than fill up the filesystem.

First, start the mksnap_ffs process, and then immediately start the dd process.  Both processes will eventually exit complaining of a full filesystem.

If the filesystem became full after the snapshot was initiated, but before it could complete successfully, the system will lock up several (sometimes as long as 12-15) hours later.

I suggest doing this test on a large filesystem (several hundreds of gigabytes) because snapshots take longer to create on such filesystems and it will be easier to fill it to capacity with dd before the snapshot completes itself.

I have repeated this problem on two different systems, each with 380 Gigabyte filesystems.
>Fix:
Do not fill a filesystem completely while snapshots are being created on that filesystem.
>Release-Note:
>Audit-Trail:
>Unformatted: