[ZFS][PANIC] Solaris Assert/zio.c:2548

[Steven Hartland]

----- Original Message ----- 
From: "Dan Mack" <mack at macktronics.com>
To: "Steven Hartland" <killing at multiplay.co.uk>
Cc: <freebsd-fs at freebsd.org>; <freebsd-current at freebsd.org>; "Larry Rosenman" <ler at lerctr.org>
Sent: Monday, July 21, 2014 2:29 AM
Subject: Re: [ZFS][PANIC] Solaris Assert/zio.c:2548


> On Mon, 21 Jul 2014, Steven Hartland wrote:
> 
>>> I just updated to I think 268921 earlier today and this is the first
>>> time I've had a panic (HEAD-268921 that is)
>>> 
>>> I'll try to get some more data if I can get it back up and running.
>>
>> That doesn't look like a related trace tbh.
>>
>>   Regards
>>   Steve
> 
> After rebooting with a dumpdev; I got this :
> 
> kbd2 at ukbd0
> Trying to mount root from zfs:tank []...
> panic: deadlkres: possible deadlock detected for 0xfffff8000e089000, blocked for 1801216 ticks
> 
> cpuid = 6
> KDB: stack backtrace:
> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe085ef1d8d0
> kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe085ef1d980
> vpanic() at vpanic+0x126/frame 0xfffffe085ef1d9c0
> panic() at panic+0x43/frame 0xfffffe085ef1da20
> deadlkres() at deadlkres+0x35c/frame 0xfffffe085ef1da70
> fork_exit() at fork_exit+0x84/frame 0xfffffe085ef1dab0
> fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe085ef1dab0
> --- trap 0, rip = 0, rsp = 0xfffffe085ef1db70, rbp = 0 ---
> KDB: enter: panic
> [ thread pid 0 tid 100070 ]
> Stopped at      kdb_enter+0x3e: movq    $0,kdb_why
> 
> I cannot seem to get past this yet so I'm open to suggestions.  I'm
> still at the db> prompt if you'd like me to attempt to collect more
> info.

Just spotted an interesting message on a recent commit which may be
relavent:

> URL: http://svnweb.freebsd.org/changeset/base/268855
> This specific commit makes boot hang just before mounting the root 
> dataset for me when vfs.zfs.vdev.cache.size tunable is set. Unsetting 
> this tunable or reverting this commit (currently running r268933 minus 
> r268855) fixes the boot for me.
>
> Please let me know if I can provide any more information.
>
> - Nikolai Lifanov

The current code disables vdev caching by default so this will only
occur if manually enabled.

The code details the reason for this as:-
 * TODO: Note that with the current ZFS code, it turns out that the
 * vdev cache is not helpful, and in some cases actually harmful.  It
 * is better if we disable this.  Once some time has passed, we should
 * actually remove this to simplify the code.  For now we just disable
 * it by setting the zfs_vdev_cache_size to zero.  Note that Solaris 11
 * has made these same changes.

    Regards
    Steve