kern/94769: Multiple file deletions on multi-snapshotted filesystems causes hang

[John Kozubik]

>Number:         94769
>Category:       kern
>Synopsis:       Multiple file deletions on multi-snapshotted filesystems causes hang
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Tue Mar 21 07:10:15 GMT 2006
>Closed-Date:
>Last-Modified:
>Originator:     John Kozubik
>Release:        FreeBSD 6.1-BETA4
>Organization:
www.kozubik.com
>Environment:
FreeBSD test.kozubik.com 6.1-BETA4 FreeBSD 6.1-BETA4 #0: Tue Mar 14 13:59:38 UTC 2006     root at wv1u.samsco.home:/usr/obj/usr/src/sys/GENERIC  i386
>Description:
If you have a UFS2 filesystem with multiple snapshots on it, and commence several processes of rapid file deletions on that filesystem, the system will hang.

I first produced this problem on a system that had a filesystem with six snapshots enabled on it.  I proceeded to start three concurrent `rm` processes of directories with large numbers of files in them (on that filesystem).  The system hung as a result.

Further testing showed that in the exact same environment, three such `rm` processes did not hang a system when the filesystem in question only had two snapshots.  I incremented the number of snapshots to 3, and the system hung when the same three `rm` processes were performed.

It is possible that any ratio of a large number of snapshots to a large number of intensive file deletions on the filesystem that is snapshotted will cause the system to hang.  For instance, one snapshot and ten file deletion processes, or perhaps ten snapshots and two file deletion processes.  I have not tested any further combinations than the ones listed above.

I am highly confident that this behavior exists and is reproducible.  I have reproduced this behavior on both 6.0 and 6.1.
>How-To-Repeat:

(assume a filesystem mounted on /mnt/data1)

cp -R /usr/src /mnt/data1/test1
cp -R /usr/src /mnt/data1/test2
cp -R /usr/src /mnt/data1/test3

mksnap_ffs /mnt/data1 /mnt/data1/.sync/snap1
mksnap_ffs /mnt/data1 /mnt/data1/.sync/snap2
mksnap_ffs /mnt/data1 /mnt/data1/.sync/snap3

rm -rf /mnt/data1/test1 &
rm -rf /mnt/data1/test2 &
rm -rf /mnt/data1/test3 &

(system will hang)

>Fix:
Do not use UFS2 snapshots.  It is unreasonable to assume that concurrent rapid file deletions will not occur on a filesystem that has snapshots on it.
>Release-Note:
>Audit-Trail:
>Unformatted: