kern/162036: [geom] Fatal trap 12: page fault while in kernel mode -- Stopped at atomic_subtract_int+0x4

[Fabian Keil]

>Number:         162036
>Category:       kern
>Synopsis:       [geom] Fatal trap 12: page fault while in kernel mode -- Stopped at atomic_subtract_int+0x4
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Wed Oct 26 17:40:08 UTC 2011
>Closed-Date:
>Last-Modified:
>Originator:     Fabian Keil
>Release:        HEAD
>Organization:
>Environment:
FreeBSD r500.local 10.0-CURRENT FreeBSD 10.0-CURRENT #6 r+3084a02: Mon Oct 24 15:55:58 CEST 2011 fk at r500.local:/usr/obj/usr/src/sys/GENERIC amd64
>Description:
I reproducible get the following (handtranscribed) panic
when sending an zfs snapshot to a geli provider based on a
labeled USB stick that disappears (due to a bug, or because
it's unplugged): 

Fatal trap 12: page fault while in kernel mode
cpuid = 0: apic id = 00
fault virtual address = 0x288
fault code	      = supervisor write data, page not present
instruction pointer   = 0x20:0xffffffff808e2984
stack pointer         = 0x28:0xffffff800023fba0
frame pointer         = 0x28:0xffffff800023fbb0
code segment          = base 0x0, limit 0xfffff, type 0x1b
                      = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags      = interrupt enabled, resume, IOPL = 0
current process       = 13 (g_up)
[ thread pid 13 tid 100010 ]
Stopped at    atomic_subtract_int+0x4: lock subl %esi,(%rdi)
db> where  
Tracing pid 13 tid 100010 td 0xfffffe00027998c0
atomic_subtract_int() at atomic_subtract_int+0x4
g_io_schdule_up() at g_io_schedule_up+0xa6
g_up_procbody() at g_up_procbody+0x5c
fork_exit() at fork_exit+0x11f
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff800023fd00, rbp 0 ---

It seems to be important that ZFS is actually writing to the stick.
If the stick is unplugged while the operation is stalled for other
reasons, this particular panic doesn't seem to occur.

In case the zpool layout isn't clear, it's the same as used in:
http://www.freebsd.org/cgi/query-pr.cgi?pr=162010

While I end up in the debugger, dumping core doesn't work
and produces a double fault.

The panic above is from a custom kernel without WITNESS and
INVARIANTS (a bit older than the uname -a output above),
but enabling them doesn't seem to affect the trace
before the double fault.

I wasn't able to reproduce the panic by unplugging the stick
while writing to the pool using dd (but only tried once).
>How-To-Repeat:
Either use a flaky USB stick as geli-encrypted vdev and wait
for it to disappear by itself while a zfs receive is in progress
and actually writing to the device, or use a reliable stick and
unplug it manually while zpool iostat is showing writes.
>Fix:


>Release-Note:
>Audit-Trail:
>Unformatted: