Re: Debugging a (potentially?) ZFS-related panic, and discussion about large patchsets

On Tue, Jan 11, 2022 at 12:43:06AM +0100, Mateusz Guzik wrote:
> On 1/11/22, Mark Johnston <markj@freebsd.org> wrote:
> > On Mon, Jan 10, 2022 at 05:11:16PM -0500, Shawn Webb wrote:
> >> Hey all,
> >>
> >> So I'm getting an interesting ZFS-related kernel panic. I've uploaded
> >> the core.txt at [0]. I suspect it's related to FreeBSD commit
> >> 681ce946f33e75c590e97c53076e86dff1fe8f4a (zfs: merge
> >> openzfs/zfs@f291fa658 (master) into main).
> >>
> >> I'm able to reproduce it on a single system with some level of
> >> determinism: I'm building the security appliance firmware at ${DAYJOB}
> >> in a bhyve VM that's backed by a zvol. The host is a Dell Precision
> >> 7540 laptop with a single NVMe drive in it. The VM is configured with
> >> a single zvol, booting with UEFI.
> >>
> >> Looking at the commit email sent to dev-commits-src-all@, I see this:
> >> 146 files changed, 4933 insertions(+), 1572 deletions(-)
> >>
> >> Strangely, when I run `git show
> >> 681ce946f33e75c590e97c53076e86dff1fe8f4a`, I only see a small subset
> >> of those changes.
> >
> > That is a merge commit.  You need to specify that you want a diff
> > against the first parent (the preceding FreeBSD), so something
> > equivalent to "git diff --stat 681ce946f^ 681ce946f".  Use
> > "git log 681ce946f^2" to see the merged OpenZFS commits.
> >
> >> As a downstream consumer of 14-CURRENT, how am I supposed to even
> >> start debugging such a large patchset in any manner that respects my
> >> time?
> >>
> >> It seems to me that breaking up commits into smaller, bite-size chunks
> >> would make life easier for those experiencing bugs, especially ones
> >> that result in kernel panics.
> >
> > That's up to the upstream project, in this case OpenZFS.
> >
> >> ZFS in and of itself is a beast, and I've yet to study any of its
> >> code, so when there's a commit that large, even thinking about
> >> debugging it is a daunting task.
> >>
> >> Needless to say, I'm going to need some hand holding here for
> >> debugging this. Anyone have any idea what's going on?
> >
> > To start, you'll need to look at the stack trace for the thread with tid
> > 100061.
> >
> 
> imo the kernel should be patched to obtain the trace on its own. As
> the target has interrupts disabled it will have to do it with NMI, but
> support for that got scrapped in
> 
> commit 1c29da02798d968eb874b86221333a56393a94c3
> Author: Mark Johnston <markj@FreeBSD.org>
> Date:   Fri Jan 31 15:43:33 2020 +0000
> 
>     Reimplement stack capture of running threads on i386 and amd64.

More general and useful, to me at least, is having "acttrace" output
available in core.txt.  So I propose https://reviews.freebsd.org/D33817

I don't think the NMI-based stack(9) machinery to capture stacks is
really very useful here anyway.  We already raise NMIs on all CPUs
during a panic, so just reuse that and add some handler which can call
kdb_backtrace() on the target CPU in ipi_nmi_handler().