kern/142872: ZFS ZVOL Lockmgr Deadlock

[Justin Head]

>Number:         142872
>Category:       kern
>Synopsis:       ZFS ZVOL Lockmgr Deadlock
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Fri Jan 15 21:50:00 UTC 2010
>Closed-Date:
>Last-Modified:
>Originator:     Justin Head
>Release:        8.0
>Organization:
>Environment:
FreeBSD test1.ord.encarnate.net 8.0-RELEASE FreeBSD 8.0-RELEASE #0: Fri Jan 15 17:33:51 UTC 2010     root at test1.ord.encarnate.net:/usr/obj/usr/src/sys/GENERIC  amd64
FreeBSD test1.ord.encarnate.net 8.0-STABLE FreeBSD 8.0-STABLE #2: Fri Jan 15 13:18:15 UTC 2010     root at test1.ord.encarnate.net:/usr/obj/usr/src/sys/DEBUG  amd64
>Description:
ZFS's zvols appear to not be very robust at handling a large stream of data coming into them.  I was attempting test one that was exported via ggated but continued to have odd lockups with it.  Tracing it back down it appears to be a general problem with storing data to zvols that happens even on the local system.

When running the repeatable steps you will eventually notice that dd is no longer returning data back.  If you check your other console window you will notice that iostat reports climbing system cpu usage.  The system will not panic and you may be able to still do some things that do not require disk access but as soon as you do your console will lock.

The locking will occur immediately in the GENERIC kernel.  After you enable debugging options such as WITNESS and the like it will be much harder to do but still occur after a handful of times running the dd loop.

The issue does not occur if you run the dd loop directly against a zfs file system under zroot.

I have tried this on 8.0-Release with zfs v13 and 8.0-Stable with zfs v14.  The issue occurs on both.  I would attach ddb debug output but the file is over 100k and will not attach.  Email me for it.

I should note that I am running a ZFS on root setup so the whole system is ZFS only.  It is also a fresh load so no tuning at all has been done.

>How-To-Repeat:
Open two consoles.

Console 1: Run 'ioscan 1'.

Console 2: Create a zvol and then base a zpool on it.  Give yourself enough space to play around with.

# zfs create -V 25g zroot/test
# zpool create ztest /dev/zvol/zroot/test 
# rm /ztest/*; size=0; for i in 0 1 2 3 4 5 6 7 8 9; do size=$(($size + 250)); dd if=/dev/random of=/ztest/file$i bs=1m count=$size; done

>Fix:


>Release-Note:
>Audit-Trail:
>Unformatted: