Re: "(ada0:ahcich0:0:0:0): Periph destroyed", followed by panic
Date: Sun, 18 Jan 2026 15:46:35 UTC
On Sun, Jan 18, 2026 at 4:22 AM Fabian Keil <freebsd-listen@fabiankeil.de>
wrote:
> Warner Losh <imp@bsdimp.com> wrote on 2026-01-17 at 10:29:44:
>
> > On Sat, Jan 17, 2026 at 10:03 AM Fabian Keil <
> freebsd-listen@fabiankeil.de>
> > wrote:
> >
> > > On an ElectroBSD system based on stable/14 215d16e4102b I got this
> panic:
> > >
> > > fk@elektrobier4 /dpool/crash $kgdb151 /boot/kernel/kernel vmcore.0
> > > [...]
>
> > > Fatal trap 12: page fault while in kernel mode
> > > [...]
> > > Reading symbols from /usr/lib/debug//boot/kernel/fdescfs.ko.debug...
> > > __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:57
> > > warning: Source file is more recent than executable.
> > > 57 __asm("movq %%gs:%P1,%0" : "=r" (td) : "n"
> > > (offsetof(struct pcpu,
> > > (kgdb) where
> > > #0 __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:57
> > > #1 doadump (textdump=textdump@entry=1) at
> > > /usr/src/sys/kern/kern_shutdown.c:405
> > > #2 0xffffffff80a201c0 in kern_reboot (howto=260) at
> > > /usr/src/sys/kern/kern_shutdown.c:523
> > > #3 0xffffffff80a206c9 in vpanic (fmt=0xffffffff80f8236f "%s",
> ap=ap@entry=0xfffffe00c4d46d40)
> > > at /usr/src/sys/kern/kern_shutdown.c:967
> > > #4 0xffffffff80a20503 in panic (fmt=<unavailable>) at
> > > /usr/src/sys/kern/kern_shutdown.c:891
> > > #5 0xffffffff80ec27c0 in trap_fatal (frame=<optimized out>,
> > > eva=<optimized out>) at /usr/src/sys/amd64/amd64/trap.c:1000
> > > #6 0xffffffff80ec27c0 in trap_pfault (frame=0xfffffe00c4d46dc0,
> > > usermode=false, signo=<optimized out>, ucode=<optimized out>)
> > > #7 <signal handler called>
> > > #8 0xffffffff8036c567 in atomic_fcmpset_long
> (src=18446735277681260352,
> > > dst=<optimized out>, expect=<optimized out>) at
> > > /usr/src/sys/amd64/include/atomic.h:184
> > > #9 xpt_done_process (ccb_h=0xfffff800041d0dd0) at
> > > /usr/src/sys/cam/cam_xpt.c:5331
> > > #10 0xffffffff8036e6e5 in xpt_done_td (arg=arg@entry
> =0xffffffff8157b240
> > > <cam_doneqs>) at /usr/src/sys/cam/cam_xpt.c:5434
> > > #11 0xffffffff809d7321 in fork_exit (callout=0xffffffff8036e5f0
> > > <xpt_done_td>, arg=0xffffffff8157b240 <cam_doneqs>,
> > > frame=0xfffffe00c4d46f40) at /usr/src/sys/kern/kern_fork.c:1153
> > > #12 <signal handler called>
> > > (kgdb) f 9
> > > #9 xpt_done_process (ccb_h=0xfffff800041d0dd0) at
> > > /usr/src/sys/cam/cam_xpt.c:5331
> > > warning: Source file is more recent than executable.
> > >
> >
> > That's concerning...
>
> That's the result of me copying the sources to the system that panicked.
> My remote-git-update script first fetches the FreeBSD sources and then
> applies the ElectroBSD patches with "git am", thereby updating the
> timestamps.
>
> > > 5331 mtx_lock(&devq->send_mtx);
> > > (kgdb) l
> > > 5326 if (sim)
> > > 5327 devq = sim->devq;
> > > 5328 KASSERT(devq, ("Periph disappeared with CCB %p
> %s
> > > request pending.",
> > > 5329 ccb_h,
> xpt_action_name(ccb_h->func_code)));
> > >
> >
> > so if devq is NULL here, it's quite likely there's corruption and/or the
> > sim has already been freed. That suggests that there's a "ref leak" bug
> > around tracking the lifetimes of the sim. If you can reproduce this, you
> > may be able to find where. They are tricky things to go after, though.
> > dtrace can help, but lately it's ability to track fbt stuff is wobbly due
> > to clang's aggressive inlining. The sim can be non-null here when
> multiple
> > different cam_path objects refer to the same path and one is released,
> and
> > the other isn't so we don't null out the bus, so sim is a stale value,
> > likely read from free'd memory. This can only happen, though, in the
> > ref-leak case where not enough refs are taken to the bus, IIRC.
> >
> > This all happens after the periph is destroyed. generically, that's
> > perfectly fine (we have about 100 systems a week at work have this happen
> > due to drive failure in our population of tens of thousands of hard
> disks).
> > We don't use ahci, though, anymore (we have several sata SSDs in the
> fleet
> > still, though). ahci when we did use it did have a fair amount of
> problems.
> > I chased a lot of bugs there, though they were often that some part of
> ahci
> > decided the device was back, and we tried to create a new periph while
> the
> > old one was being destroyed and hit issues there. I fixed all the ones
> that
> > were hitting our fleet a few years ago.
>
> Thanks a lot for the detailed explanation and the quick response.
>
> > > I have a couple of local patches including one in sys/cam/cam_xpt.c
> [0].
> [...]
> > > [0] <
> > >
> https://www.fabiankeil.de/sourcecode/electrobsd/ElectroBSD-20220822-d9391cfeef5b/0183-cam-Fix-nullpointer-dereference.diff
> >
> >
> > You should never pass NULL as the path to xpt_async(). It's a required
> > element of the call: the path you are sending the async event to. And the
> > path must have either a device or a bus, which that traceback didn't
> have.
> > Now, how did it come to pass that the bus and/or device was NULL here?
> > That's where the bug is... xpt_create_path always fills in the bus (via
> > xpt_compile_path). One common way for it to become NULL like this is
> > xpt_release_path is called on it. Maybe you can use dtrace to find where
> > that's coming to pass? I'm not seeing anything super quick in the code,
> but
> > I'm just doing a quick analysis to answer this email + past experience...
> >
> > So I hope that helps get to the next level of 'hey, this is wrong' but I
> > know climbing that hill is hard. I'll help when I can, but at this stage
> I
> > can only offer generic advice that I'm not feeling is very helpful.
>
> While I have not yet been able to reproduce the exact same panic on the
> same system, I ran into a related issue while trying to reproduce the
> problem in a byhve vm.
>
> Running ahcich0-detacher.sh:
>
> | #!/bin/sh
> |
> | main() {
> | while true; do
> | date
> | echo "Detaching ahcich0"
> | sudo devctl detach ahcich0
> | echo "Attaching ahcich0"
> | sudo devctl attach ahcich0
> | sleep 0.1
> | sudo dd if=/dev/random bs=1m of=/dev/ada0 &
> | sleep 0.1
> | done
> | }
> |
> | main
>
> in two terminals at the same time within seconds reliably results in:
>
This script may find races in newbus. If anything in the attach routine
sleeps, it may allow the detach code to run on incompletely constructed
devices... you might add some code to see if you are hitting that case.
Warner
> fk@test-vm ~ $kgdb151 /boot/kernel/kernel /var/crash/vmcore.3
> GNU gdb (GDB) 15.1 [GDB v15.1 for FreeBSD]
> [...]
> Reading symbols from /boot/kernel/kernel...
> Reading symbols from /usr/lib/debug//boot/kernel/kernel.debug...
>
> Unread portion of the kernel message buffer:
> trap number = 12
> panic: page fault
> cpuid = 1
> time = 1768731529
> KDB: stack backtrace:
> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame
> 0xfffffe00b5a727c0
> vpanic() at vpanic+0x161/frame 0xfffffe00b5a728f0
> panic() at panic+0x43/frame 0xfffffe00b5a72950
> trap_pfault() at trap_pfault+0x3e0/frame 0xfffffe00b5a729a0
> calltrap() at calltrap+0x8/frame 0xfffffe00b5a729a0
> --- trap 0xc, rip = 0xffffffff80369a66, rsp = 0xfffffe00b5a72a70, rbp =
> 0xfffffe00b5a72aa0 ---
> xpt_async() at xpt_async+0x2d6/frame 0xfffffe00b5a72aa0
> ahci_ch_detach() at ahci_ch_detach+0x63/frame 0xfffffe00b5a72ae0
> device_detach() at device_detach+0x180/frame 0xfffffe00b5a72b20
> devctl2_ioctl() at devctl2_ioctl+0x412/frame 0xfffffe00b5a72bf0
> devfs_ioctl() at devfs_ioctl+0xcb/frame 0xfffffe00b5a72c40
> vn_ioctl() at vn_ioctl+0xc8/frame 0xfffffe00b5a72cb0
> devfs_ioctl_f() at devfs_ioctl_f+0x1e/frame 0xfffffe00b5a72cd0
> kern_ioctl() at kern_ioctl+0x286/frame 0xfffffe00b5a72d40
> sys_ioctl() at sys_ioctl+0x117/frame 0xfffffe00b5a72e00
> amd64_syscall() at amd64_syscall+0x117/frame 0xfffffe00b5a72f30
> fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe00b5a72f30
> --- syscall (54, FreeBSD ELF64, ioctl), rip = 0x13788352457a, rsp =
> 0x1378821f4e68, rbp = 0x1378821f4f20 ---
> Uptime: 4m34s
> Dumping 533 out of 10203
> MB:..3%..12%..21%..33%..42%..51%..63%..72%..81%..93%
>
> Reading symbols from /boot/kernel/geom_eli.ko...
> Reading symbols from /usr/lib/debug//boot/kernel/geom_eli.ko.debug...
> Reading symbols from /boot/kernel/zfs.ko...
> Reading symbols from /usr/lib/debug//boot/kernel/zfs.ko.debug...
> __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:57
> 57 __asm("movq %%gs:%P1,%0" : "=r" (td) : "n"
> (offsetof(struct pcpu,
> (kgdb) where
> #0 __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:57
> #1 doadump (textdump=textdump@entry=1) at
> /usr/src/sys/kern/kern_shutdown.c:405
> #2 0xffffffff80a20360 in kern_reboot (howto=260) at
> /usr/src/sys/kern/kern_shutdown.c:523
> #3 0xffffffff80a20869 in vpanic (fmt=0xffffffff80f8238a "%s", ap=ap@entry=0xfffffe00a2de6930)
> at /usr/src/sys/kern/kern_shutdown.c:967
> #4 0xffffffff80a206a3 in panic (fmt=<unavailable>) at
> /usr/src/sys/kern/kern_shutdown.c:891
> #5 0xffffffff80ec27c0 in trap_fatal (frame=<optimized out>,
> eva=<optimized out>) at /usr/src/sys/amd64/amd64/trap.c:1000
> #6 0xffffffff80ec27c0 in trap_pfault (frame=0xfffffe00a2de69b0,
> usermode=false, signo=<optimized out>, ucode=<optimized out>)
> #7 <signal handler called>
> #8 0xffffffff80369a66 in xpt_async (async_code=256,
> path=0xfffff8000606eb60, async_arg=0x0) at /usr/src/sys/cam/cam_xpt.c:4321
> #9 0xffffffff804b8d23 in ahci_ch_detach (dev=0xfffff800039d7800) at
> /usr/src/sys/dev/ahci/ahci.c:952
> #10 0xffffffff80a5d050 in DEVICE_DETACH (dev=0xfffff800039d7800) at
> ./device_if.h:234
> #11 device_detach (dev=dev@entry=0xfffff800039d7800) at
> /usr/src/sys/kern/subr_bus.c:2712
> #12 0xffffffff80a64cc2 in devctl2_ioctl (cdev=<optimized out>,
> cmd=<optimized out>, data=0xfffff80108617700 "ahcich0", fflag=<optimized
> out>, td=<optimized out>) at /usr/src/sys/kern/subr_bus.c:5843
> #13 0xffffffff808cd19b in devfs_ioctl (ap=0xfffffe00a2de6c58) at
> /usr/src/sys/fs/devfs/devfs_vnops.c:963
> #14 0xffffffff80b25348 in VOP_IOCTL (vp=<optimized out>,
> command=<optimized out>, data=<optimized out>, fflag=<optimized out>,
> cred=<optimized out>, td=<optimized out>) at ./vnode_if.h:633
> #15 0xffffffff80b25348 in vn_ioctl (fp=<optimized out>, com=0, data=0x2,
> active_cred=0x0, td=0xfffffe0012063d00)
> #16 0xffffffff808cd80e in devfs_ioctl_f (fp=0xfffff8000606eb60, com=0,
> data=0x2, cred=0x0, td=0xfffffe0012063d00) at
> /usr/src/sys/fs/devfs/devfs_vnops.c:894
> #17 0xffffffff80a93556 in fo_ioctl (fp=0xfffff8000cd2feb0, com=0,
> data=0xfffff80108617700, active_cred=0x0, td=0xfffff8000cdc4740) at
> /usr/src/sys/sys/file.h:379
> #18 kern_ioctl (td=td@entry=0xfffff8000cdc4740, fd=<optimized out>,
> com=0, com@entry=2157462530, data=data@entry=0xfffff80108617700
> "ahcich0") at /usr/src/sys/kern/sys_generic.c:810
> #19 0xffffffff80a93267 in sys_ioctl (td=<optimized out>, uap=<optimized
> out>) at /usr/src/sys/kern/sys_generic.c:715
> #20 0xffffffff80ec3107 in syscallenter (td=0xfffff8000cdc4740) at
> /usr/src/sys/amd64/amd64/../../kern/subr_syscall.c:193
> #21 amd64_syscall (td=0xfffff8000cdc4740, traced=0) at
> /usr/src/sys/amd64/amd64/trap.c:1241
> #22 <signal handler called>
> #23 0x00000047b214757a in ?? ()
> Backtrace stopped: Cannot access memory at address 0x47af775a58
> (kgdb) f 8
> #8 0xffffffff80369a66 in xpt_async (async_code=256,
> path=0xfffff8000606eb60, async_arg=0x0) at /usr/src/sys/cam/cam_xpt.c:4321
> 4321 xpt_freeze_simq(path->bus->sim, 1);
> (kgdb) l
> 4316 }
> 4317 if (path != NULL) {
> 4318 if (path->device != NULL && path->device->lun_id
> != CAM_LUN_WILDCARD)
> 4319 xpt_freeze_devq(path, 1);
> 4320 else
> 4321 xpt_freeze_simq(path->bus->sim, 1);
> 4322 }
> 4323 xpt_action(ccb);
> 4324 }
> 4325
> (kgdb) p *path
> $1 = {periph = 0x0, bus = 0x0, target = 0x0, device = 0x0}
>
> So this actually reaches the code I previously patched in [0]
> but the patch isn't sufficient to hide the problem.
>
> I also once managed to get devctl sleeping "forever" by running only
> one ahcich0-detacher.sh instance, but haven't been able to reproduce
> this yet. After triggering a panic through debug.kdb.panic I got:
>
> (kgdb) where
> #0 sched_switch (td=td@entry=0xfffff8002f485000, flags=flags@entry=259)
> at /usr/src/sys/kern/sched_ule.c:2437
> #1 0xffffffff80a2e18d in mi_switch (flags=flags@entry=259) at
> /usr/src/sys/kern/kern_synch.c:548
> #2 0xffffffff80a80799 in sleepq_switch (wchan=<optimized out>,
> pri=<optimized out>) at /usr/src/sys/kern/subr_sleepqueue.c:608
> #3 0xffffffff80a80685 in sleepq_wait (wchan=<unavailable>, wchan@entry=0xfffff802341c7980,
> pri=<unavailable>) at /usr/src/sys/kern/subr_sleepqueue.c:659
> #4 0xffffffff80a2d8e3 in _sleep (ident=ident@entry=0xfffff802341c7980,
> lock=lock@entry=0xfffffe00b78c8400, priority=priority@entry=43,
> wmesg=<optimized out>, sbt=sbt@entry=0, pr=pr@entry=0, flags=256)
> at /usr/src/sys/kern/kern_synch.c:223
> #5 0xffffffff80366607 in cam_sim_free (sim=0xfffff802341c7980,
> free_devq=free_devq@entry=1) at /usr/src/sys/cam/cam_sim.c:157
> #6 0xffffffff804b8d67 in ahci_ch_detach (dev=0xfffff800039d7800, dev@entry=<error
> reading variable: value is not available>) at
> /usr/src/sys/dev/ahci/ahci.c:960
> #7 0xffffffff80a5d050 in DEVICE_DETACH (dev=0xfffff800039d7800) at
> ./device_if.h:234
> #8 device_detach (dev=dev@entry=0xfffff800039d7800) at
> /usr/src/sys/kern/subr_bus.c:2712
> #9 0xffffffff80a64cc2 in devctl2_ioctl (cdev=<optimized out>,
> cmd=<unavailable>, cmd@entry=<error reading variable: value is not
> available>, data=0xfffff80090a01000 "ahcich0",
> data@entry=<error reading variable: value is not available>,
> fflag=<optimized out>, td=<unavailable>, td@entry=<error reading
> variable: value is not available>) at /usr/src/sys/kern/subr_bus.c:5843
> #10 0xffffffff808cd19b in devfs_ioctl (ap=0xfffffe00b5c4fc58, ap@entry=<error
> reading variable: value is not available>) at
> /usr/src/sys/fs/devfs/devfs_vnops.c:963
> #11 0xffffffff80b25348 in VOP_IOCTL (vp=<optimized out>,
> command=<optimized out>, data=<optimized out>, fflag=<optimized out>,
> cred=<optimized out>, td=<optimized out>) at ./vnode_if.h:633
> #12 0xffffffff80b25348 in vn_ioctl (fp=<optimized out>, com=<unavailable>,
> data=<unavailable>, active_cred=<unavailable>, td=<unavailable>)
> #13 0xffffffff808cd80e in devfs_ioctl_f (fp=<unavailable>, fp@entry=<error
> reading variable: value is not available>, com=<unavailable>, com@entry=<error
> reading variable: value is not available>, data=<unavailable>,
> data@entry=<error reading variable: value is not available>,
> cred=<unavailable>, cred@entry=<error reading variable: value is not
> available>, td=<unavailable>, td@entry=<error reading variable: value is
> not available>)
> at /usr/src/sys/fs/devfs/devfs_vnops.c:894
> #14 0xffffffff80a93556 in fo_ioctl (fp=0xfffff8000cd6d5f0,
> com=<unavailable>, data=0xfffff80090a01000, active_cred=<unavailable>,
> td=0xfffff8002f485000) at /usr/src/sys/sys/file.h:379
> #15 kern_ioctl (td=td@entry=0xfffff8002f485000, fd=<optimized out>,
> com=<unavailable>, com@entry=2157462530, data=data@entry=0xfffff80090a01000
> "ahcich0") at /usr/src/sys/kern/sys_generic.c:810
> #16 0xffffffff80a93267 in sys_ioctl (td=<optimized out>, uap=<optimized
> out>) at /usr/src/sys/kern/sys_generic.c:715
> #17 0xffffffff80ec3107 in syscallenter (td=0xfffff8002f485000) at
> /usr/src/sys/amd64/amd64/../../kern/subr_syscall.c:193
> #18 amd64_syscall (td=0xfffff8002f485000, traced=0) at
> /usr/src/sys/amd64/amd64/trap.c:1241
> #19 <signal handler called>
> #20 0x0000314d6884457a in ?? ()
> Backtrace stopped: Cannot access memory at address 0x314d656097a8
> (kgdb) f 5
> #5 0xffffffff80366607 in cam_sim_free (sim=0xfffff802341c7980,
> free_devq=free_devq@entry=1) at /usr/src/sys/cam/cam_sim.c:157
> 157 error = msleep(sim, mtx, PRIBIO, "simfree", 0);
> (kgdb) l
> 152 mtx_assert(mtx, MA_OWNED);
> 153 }
> 154 KASSERT(sim->refcount >= 1, ("sim->refcount >= 1"));
> 155 sim->refcount--;
> 156 if (sim->refcount > 0) {
> 157 error = msleep(sim, mtx, PRIBIO, "simfree", 0);
> 158 KASSERT(error == 0, ("invalid error value for
> msleep(9)"));
> 159 }
> 160 KASSERT(sim->refcount == 0, ("sim->refcount == 0"));
> 161 if (mtx == &cam_sim_free_mtx) /* sim->mtx == NULL */
> (kgdb) p *sim
> $1 = {sim_action = 0xffffffff804b93d0 <ahciaction>, sim_poll =
> 0xffffffff804b9a60 <ahcipoll>, sim_name = 0xffffffff80fd9fa8 "ahcich",
> softc = 0xfffffe00b78c7000, mtx = 0xfffffe00b78c8400, links = {tqe_next =
> 0x0,
> tqe_prev = 0x0}, path_id = 0, unit_number = 0, bus_id = 0,
> max_tagged_dev_openings = 32, max_dev_openings = 2, flags = 0, devq =
> 0xfffff80006070040, refcount = 1}
>
> Fabian
>
>