[Bug 296269] zfs panic: NULL deref in zfs_acl_release_nodes during znode reclaim (15.1-RELEASE)

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=296269

            Bug ID: 296269
           Summary: zfs panic: NULL deref in zfs_acl_release_nodes during
                    znode reclaim (15.1-RELEASE)
           Product: Base System
           Version: 15.1-RELEASE
          Hardware: Any
                OS: Any
            Status: New
          Severity: Affects Only Me
          Priority: ---
         Component: kern
          Assignee: bugs@FreeBSD.org
          Reporter: fax@nohik.ee

Created attachment 272118
  --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=272118&action=edit
full symbolized "bt full" + registers

After upgrading from 14.3-RELEASE to 15.1-RELEASE (a "zpool upgrade" was
done as part of the upgrade), the system suffers recurring kernel panics:
"page fault while in kernel mode", a supervisor WRITE to virtual address
0x8 in the ZFS ACL teardown path, while a znode is being reclaimed.
Crash-dump inspection shows the affected znode's in-memory cached ACL
(aclnode list) is internally inconsistent, while the on-disk object is
fully intact (verified with zdb). This points to an in-memory defect, not
on-disk corruption.


Environment
-----------
- FreeBSD 15.1-RELEASE releng/15.1-n283562-96841ea08dcf GENERIC amd64
  (non-debug kernel; some locals show <optimized out>)
- Upgraded from 14.3-RELEASE; zpool upgrade performed during the upgrade
- Default NFSv4 ACLs:
    # zfs get acltype,aclmode,aclinherit tank/<dataset>
    acltype     nfsv4        default
    aclmode     discard      default
    aclinherit  restricted   default
  (live zfsvfs in the dump confirms z_acl_type=2, z_acl_mode=0,
  z_acl_inherit=4)
- Affected object carries the trivial three-entry ACL
  (owner@/group@/everyone@; z_acl_count=3, z_acl_bytes=24).
  No NFS exports or services involved.
- Vnode state at crash time: kern.maxvnodes=8388608,
  vfs.numvnodes=8388608 (vnode cache at its limit)


Panic signature (Host B, python3.11)
-------------------------------------
Fatal trap 12: page fault while in kernel mode
fault virtual address  = 0x8
fault code             = supervisor write data, page not present
instruction pointer    = 0x20:0xffffffff82682d1f
current process        = 77899 (python3.11)
panic: page fault

Key registers: rax=0xfffff84432c87000 (head node), rbx=0xfffff83234800300
(aclp), rdi=r14=0xfffff83234800330 (&aclp->z_acl), fault va=0x8.
Full register dumps for both hosts are in the attached backtrace.


Backtrace (key frames; full "bt full" attached)
------------------------------------------------
#8  list_remove_head (list=0xfffff83234800330)   spl/list.c:138
                                                 [head=0xfffff84432c87000]
#9  zfs_acl_release_nodes (aclp=0xfffff83234800300)   zfs/zfs_acl.c:499
#10 zfs_acl_free (aclp=0xfffff83234800300)            zfs/zfs_acl.c:508
#11 zfs_znode_free (zp=0xfffffe0aaab93728)            zfs/zfs_znode_os.c:1321
#12 zfs_zinactive -> #13 zfs_freebsd_reclaim
        zfs/zfs_znode_os.c:1297 / zfs_vnops_os.c:5738
#14 VOP_RECLAIM_APV -> vgonel -> vtryrecycle -> vnlru_free_impl
        -> vn_alloc_hard -> getnewvnode_reserve
#22 zfs_zget                                          zfs/zfs_znode_os.c:959
#23 zfs_dirent_lookup -> zfs_dirlook -> zfs_lookup -> zfs_freebsd_lookup
        -> vfs_cache_lookup -> namei
#32 kern_statat (fstatat(2))


What the crash dump shows
-------------------------
The node passed to list_remove_node() is a valid, linked node
(head = 0xfffff84432c87000) whose forward pointer is NULL:

    head: { list_next = 0x0,
            list_prev = 0xfffff83234800338 (= &z_acl.list_head) }

list_remove_node() runs:

    node->list_prev->list_next = node->list_next;  // (A) sets
                                                   //     list_head.list_next =
NULL
    node->list_next->list_prev = node->list_prev;  // (B) NULL->list_prev
                                                   //     => write to va 0x8

Line (B) dereferences the NULL list_next and writes at offset 0x8
(list_prev), which is exactly the faulting address. (The
list_head.list_next = 0x0 visible in the post-mortem dump of z_acl is a
side effect of line (A) during the crash, not the pre-existing state.)

The list is internally inconsistent: the forward chain
(list_head.list_next -> head, head->list_next = NULL) and the back pointer
(list_head.list_prev = 0xfffff832e5483bc0) disagree, and z_curr_node
(0xfffff832e5483bc0) points to a node already in the post-unlink
{list_next=0, list_prev=0} state while it is still referenced and
z_acl_count still reports 3. In other words, the cached aclnode list
appears to have been (partially) torn down already. Full kgdb structure
dumps are attached.


On-disk object is intact
------------------------
zdb -dddd tank/<dataset> 84644447 reads cleanly: valid metadata,
100% full, no errors, ACL stored as SA in the 168-byte bonus buffer;
pflags 40800000004 match the in-core znode z_pflags. gen 2718637 means the
object (and its ACL) predates the 14.3 -> 15.1 upgrade. See attachment.


Reproducibility
---------------
Process-independent and reproduces across hosts: captured on two
independent machines under unrelated processes (du and python3.11), each
time with the identical fault signature (write to va 0x8; module-relative
offset zfs_acl_free+0x45; the two hosts differ only by zfs.ko load base).
It is reached via the standard vnode-reclaim path: with the vnode cache at
its limit (vfs.numvnodes == kern.maxvnodes), a lookup drives
getnewvnode_reserve() -> vnlru -> vgonel -> VOP_RECLAIM ->
zfs_freebsd_reclaim -> ... -> zfs_acl_release_nodes -> list_remove_head.


Hypothesis (for maintainers to confirm)
---------------------------------------
The on-disk object is clean and predates the upgrade, so this looks like
an in-memory defect - most plausibly a double-free / use-after-free of the
cached aclnode list, leaving a linked node with a NULL forward pointer
that the reclaim-time teardown then walks into. A second, independent
anomaly in the same znode is consistent with memory corruption rather than
a pure list-logic bug: z_zfsvfs = 0xfffff80157aec001, i.e. the live
(intact) zfsvfs_t at 0xfffff80157aec000 with bit 0 set. Since the ACL was
written by 14.3-era OpenZFS and the crash only appears after the
14.3 -> 15.1 upgrade, the 15.1 code path that decodes/caches/frees a
14.3-written ACL seems a reasonable starting point. Offered as a lead, not
a conclusion.


Notes
-----
- Non-debug GENERIC kernel, so the list/teardown ASSERTs are compiled out
  (which is why it faults rather than asserting).
- vmcore / core.txt can't be shared publicly due to size and privacy, but
  I can provide them privately on request.
- Dataset name anonymized as <dataset>; the file name 'beiruti' is itself
  already a pseudonym (length preserved).

-- 
You are receiving this mail because:
You are the assignee for the bug.