panic: Memory modified after free in zio_create, passthru in use [Was: 11.1-pre runtime Undefined symbol "xdr_accepted_reply" /lib/libc.so.7]

Sun Jun 11 10:37:32 UTC 2017

 Bezüglich Harry Schmalzbauer's Nachricht vom 06.06.2017 14:03 (localtime):
>  Hello,
>
> suddenly, I'm getting this error:
> /lib/libc.so.7: Undefined symbol "xdr_accepted_reply"
>
> Very mysterious: It showed up on a running system, which worked
> flawlessly for some hours. And that host has root-fs (/) mounted
> readonly from a memorydisk. So to my understanding, it's completely
> impossible that /lib/libc.so.7 is corrupted since last boot.
>
> I'm completely out of ideas what could cause this strange error during
> "normal" operation.
>
> Normal operation in this case is serving as a bhyve test machine.
> I first noticed that error after one guest - with passthru device
> attached - was shut down.
>
> My suspicion is some undiscovered passthru interference... Since I
> noticed one other _very_ strange passthru-effect:
> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=215740

Hello,

this time I caught a panic with a debuging kernel under 11.1-BETA1,
which again occured after shuting down a VM which had ppt in use:
cpuid = 5
KDB: stack backtrace:
#0 0xffffffff805bf327 at kdb_backtrace+0x67
#1 0xffffffff8057f266 at vpanic+0x186
#2 0xffffffff8057f2e3 at panic+0x43
#3 0xffffffff8082eaeb at trash_ctor+0x4b
#4 0xffffffff8082aaec at uma_zalloc_arg+0x52c
#5 0xffffffff813b54a6 at zio_add_child+0x26
#6 0xffffffff813b5a05 at zio_create+0x385
#7 0xffffffff813b6de2 at zio_vdev_child_io+0x232
#8 0xffffffff81396be0 at vdev_mirror_io_start+0x370
#9 0xffffffff813bc629 at zio_vdev_io_start+0x4a9
#10 0xffffffff813b76bc at zio_execute+0x36c
#11 0xffffffff813b6868 at zio_nowait+0xb8
#12 0xffffffff81396bec at vdev_mirror_io_start+0x37c
#13 0xffffffff813bc383 at zio_vdev_io_start+0x203
#14 0xffffffff813b76bc at zio_execute+0x36c
#15 0xffffffff805d10dd at taskqueue_run_locked+0x13d
#16 0xffffffff805d1e78 at taskqueue_thread_loop+0x88
#17 0xffffffff80543844 at fork_exit+0x84

#0  doadump (textdump=<value optimized out>) at pcpu.h:222
#1  0xffffffff8057ece0 in kern_reboot (howto=260) at
/usr/local/share/deploy-tools/RELENG_11/src/sys/kern/kern_shutdown.c:366
#2  0xffffffff8057f2a0 in vpanic (fmt=<value optimized out>, ap=<value
optimized out>)
    at
/usr/local/share/deploy-tools/RELENG_11/src/sys/kern/kern_shutdown.c:759
#3  0xffffffff8057f2e3 in panic (fmt=<value optimized out>) at
/usr/local/share/deploy-tools/RELENG_11/src/sys/kern/kern_shutdown.c:690
#4  0xffffffff8082eaeb in trash_ctor (mem=<value optimized out>,
size=<value optimized out>, arg=<value optimized out>, flags=<value
optimized out>)
    at /usr/local/share/deploy-tools/RELENG_11/src/sys/vm/uma_dbg.c:80
#5  0xffffffff8082aaec in uma_zalloc_arg (zone=0xfffff8001febc680,
udata=0xfffff8001ad5f340, flags=<value optimized out>)
    at /usr/local/share/deploy-tools/RELENG_11/src/sys/vm/uma_core.c:2152
#6  0xffffffff813b54a6 in zio_add_child (pio=0xfffff8026f350b88,
cio=0xfffff8002478b7b0)
    at
/usr/local/share/deploy-tools/RELENG_11/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:460
#7  0xffffffff813b5a05 in zio_create (pio=0xfffff8026f350b88, spa=<value
optimized out>, txg=433989, bp=<value optimized out>,
data=0xfffffe0058afa000,
    size=1024, type=<value optimized out>,
priority=ZIO_PRIORITY_ASYNC_WRITE, flags=<value optimized out>,
vd=<value optimized out>,
    offset=<value optimized out>, zb=<value optimized out>,
pipeline=<value optimized out>)
    at
/usr/local/share/deploy-tools/RELENG_11/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:690
#8  0xffffffff813b6de2 in zio_vdev_child_io (pio=0xfffff8026f350b88,
bp=<value optimized out>, vd=<value optimized out>, offset=325398016,
    data=<value optimized out>, size=1024, type=<value optimized out>,
flags=1048704, done=<value optimized out>)
    at
/usr/local/share/deploy-tools/RELENG_11/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1141
#9  0xffffffff81396be0 in vdev_mirror_io_start (zio=0xfffff8026f350b88)
    at
/usr/local/share/deploy-tools/RELENG_11/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_mirror.c:488
#10 0xffffffff813bc629 in zio_vdev_io_start (zio=0xfffff8026f350b88)
    at
/usr/local/share/deploy-tools/RELENG_11/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3143
#11 0xffffffff813b76bc in zio_execute (zio=<value optimized out>)
    at
/usr/local/share/deploy-tools/RELENG_11/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1681
#12 0xffffffff813b6868 in zio_nowait (zio=0xfffff8026f350b88)
    at
/usr/local/share/deploy-tools/RELENG_11/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1739
#13 0xffffffff81396bec in vdev_mirror_io_start (zio=0xfffff8026f7a7b88)
    at
/usr/local/share/deploy-tools/RELENG_11/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_mirror.c:488
#14 0xffffffff813bc383 in zio_vdev_io_start (zio=0xfffff8026f7a7b88)
    at
/usr/local/share/deploy-tools/RELENG_11/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3021
#15 0xffffffff813b76bc in zio_execute (zio=<value optimized out>)
    at
/usr/local/share/deploy-tools/RELENG_11/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1681
#16 0xffffffff805d10dd in taskqueue_run_locked
(queue=0xfffff8001ab5a700) at
/usr/local/share/deploy-tools/RELENG_11/src/sys/kern/subr_taskqueue.c:454
#17 0xffffffff805d1e78 in taskqueue_thread_loop (arg=<value optimized
out>) at
/usr/local/share/deploy-tools/RELENG_11/src/sys/kern/subr_taskqueue.c:741
#18 0xffffffff80543844 in fork_exit (callout=0xffffffff805d1df0
<taskqueue_thread_loop>, arg=0xfffff8001aa90720, frame=0xfffffe043f609ac0)
    at /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/kern_fork.c:1042
#19 0xffffffff808598ae in fork_trampoline () at
/usr/local/share/deploy-tools/RELENG_11/src/sys/amd64/amd64/exception.S:611
#20 0x0000000000000000 in ?? ()

This machine is usually rock solid, but shows most strange errors each
time I utilize "passthru" with bhyve.

Besides runngin a debug kernel, I don't know how to help tracking this down.

I can imagine that above quoted PR and the unexplainable »lib/libc.so.7:
Undefined symbol "xdr_accepted_reply"« error all have the same
undiscovered cause, which shows up as soon as byhve and passtrhu are
involved.

Please, can anybody of the xperts add a comment?

Thanks,

-harry