zio_done panic in 10.3

[Andriy Gapon]

On 21/11/2017 21:30, Shiva Bhanujan wrote:
> it did get compressed to 0.5G - still too big to send via email.  I did send some more debug information by running kgdb on the core file to Andriy, and I'm waiting for any analysis that he might provide.

Yes, kgdb-over-email turned out to be a far more efficient compression :-)
I already have an analysis based on the information provided by Shiva and by
another user who has the same problem and contacted me privately.
I am discussing possible ways to fix the problem with George Wilson who was very
kind to double-check the analysis, complete it and suggest possible fixes.

A short version is that dbuf_prefetch and dbuf_prefetch_indirect_done functions
chain new zio-s under the same parent zio (a completion of one child zio may
create another child zio).  They do it using arc_read which can create either a
logical zio in most cases or a vdev zio for a read from a cache device (2arc).
zio_done() has a check for the completion of a parent zio's children but that
check is not completely safe and can be broken by the pattern that dbuf_prefetch
can create.  So, under some specific circumstances the parent zio may complete
and get destroyed while there is a child zio.

I believe this problem to be rather rare, but there could be configurations and
workloads where it's triggered more often.
The problem does not happen if there are no cache devices.

> From: Conrad Meyer [cem at freebsd.org]
> 
> Sent: Tuesday, November 21, 2017 9:04 AM
> 
> To: Shiva Bhanujan
> 
> Cc: Andriy Gapon; freebsd-fs at freebsd.org
> 
> Subject: Re: zio_done panic in 10.3
> 
> 
> 
> 
> 
> 
> 
> Have you tried compressing it with e.g. xz or zstd?
> 
> -- 
Andriy Gapon