Re: "(ada0:ahcich0:0:0:0): Periph destroyed", followed by panic
- Reply: Fabian Keil : "Re: "(ada0:ahcich0:0:0:0): Periph destroyed", followed by panic"
- Reply: Warner Losh : "Re: "(ada0:ahcich0:0:0:0): Periph destroyed", followed by panic"
- Reply: Fabian Keil : "Re: "(ada0:ahcich0:0:0:0): Periph destroyed", followed by panic"
- In reply to: Warner Losh : "Re: "(ada0:ahcich0:0:0:0): Periph destroyed", followed by panic"
- Go to: [ bottom of page ] [ top of archives ] [ this month ]
Date: Sun, 18 Jan 2026 11:20:05 UTC
Warner Losh <imp@bsdimp.com> wrote on 2026-01-17 at 10:29:44:
> On Sat, Jan 17, 2026 at 10:03 AM Fabian Keil <freebsd-listen@fabiankeil.de>
> wrote:
>
> > On an ElectroBSD system based on stable/14 215d16e4102b I got this panic:
> >
> > fk@elektrobier4 /dpool/crash $kgdb151 /boot/kernel/kernel vmcore.0
> > [...]
> > Fatal trap 12: page fault while in kernel mode
> > [...]
> > Reading symbols from /usr/lib/debug//boot/kernel/fdescfs.ko.debug...
> > __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:57
> > warning: Source file is more recent than executable.
> > 57 __asm("movq %%gs:%P1,%0" : "=r" (td) : "n"
> > (offsetof(struct pcpu,
> > (kgdb) where
> > #0 __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:57
> > #1 doadump (textdump=textdump@entry=1) at
> > /usr/src/sys/kern/kern_shutdown.c:405
> > #2 0xffffffff80a201c0 in kern_reboot (howto=260) at
> > /usr/src/sys/kern/kern_shutdown.c:523
> > #3 0xffffffff80a206c9 in vpanic (fmt=0xffffffff80f8236f "%s", ap=ap@entry=0xfffffe00c4d46d40)
> > at /usr/src/sys/kern/kern_shutdown.c:967
> > #4 0xffffffff80a20503 in panic (fmt=<unavailable>) at
> > /usr/src/sys/kern/kern_shutdown.c:891
> > #5 0xffffffff80ec27c0 in trap_fatal (frame=<optimized out>,
> > eva=<optimized out>) at /usr/src/sys/amd64/amd64/trap.c:1000
> > #6 0xffffffff80ec27c0 in trap_pfault (frame=0xfffffe00c4d46dc0,
> > usermode=false, signo=<optimized out>, ucode=<optimized out>)
> > #7 <signal handler called>
> > #8 0xffffffff8036c567 in atomic_fcmpset_long (src=18446735277681260352,
> > dst=<optimized out>, expect=<optimized out>) at
> > /usr/src/sys/amd64/include/atomic.h:184
> > #9 xpt_done_process (ccb_h=0xfffff800041d0dd0) at
> > /usr/src/sys/cam/cam_xpt.c:5331
> > #10 0xffffffff8036e6e5 in xpt_done_td (arg=arg@entry=0xffffffff8157b240
> > <cam_doneqs>) at /usr/src/sys/cam/cam_xpt.c:5434
> > #11 0xffffffff809d7321 in fork_exit (callout=0xffffffff8036e5f0
> > <xpt_done_td>, arg=0xffffffff8157b240 <cam_doneqs>,
> > frame=0xfffffe00c4d46f40) at /usr/src/sys/kern/kern_fork.c:1153
> > #12 <signal handler called>
> > (kgdb) f 9
> > #9 xpt_done_process (ccb_h=0xfffff800041d0dd0) at
> > /usr/src/sys/cam/cam_xpt.c:5331
> > warning: Source file is more recent than executable.
> >
>
> That's concerning...
That's the result of me copying the sources to the system that panicked.
My remote-git-update script first fetches the FreeBSD sources and then
applies the ElectroBSD patches with "git am", thereby updating the timestamps.
> > 5331 mtx_lock(&devq->send_mtx);
> > (kgdb) l
> > 5326 if (sim)
> > 5327 devq = sim->devq;
> > 5328 KASSERT(devq, ("Periph disappeared with CCB %p %s
> > request pending.",
> > 5329 ccb_h, xpt_action_name(ccb_h->func_code)));
> >
>
> so if devq is NULL here, it's quite likely there's corruption and/or the
> sim has already been freed. That suggests that there's a "ref leak" bug
> around tracking the lifetimes of the sim. If you can reproduce this, you
> may be able to find where. They are tricky things to go after, though.
> dtrace can help, but lately it's ability to track fbt stuff is wobbly due
> to clang's aggressive inlining. The sim can be non-null here when multiple
> different cam_path objects refer to the same path and one is released, and
> the other isn't so we don't null out the bus, so sim is a stale value,
> likely read from free'd memory. This can only happen, though, in the
> ref-leak case where not enough refs are taken to the bus, IIRC.
>
> This all happens after the periph is destroyed. generically, that's
> perfectly fine (we have about 100 systems a week at work have this happen
> due to drive failure in our population of tens of thousands of hard disks).
> We don't use ahci, though, anymore (we have several sata SSDs in the fleet
> still, though). ahci when we did use it did have a fair amount of problems.
> I chased a lot of bugs there, though they were often that some part of ahci
> decided the device was back, and we tried to create a new periph while the
> old one was being destroyed and hit issues there. I fixed all the ones that
> were hitting our fleet a few years ago.
Thanks a lot for the detailed explanation and the quick response.
> > I have a couple of local patches including one in sys/cam/cam_xpt.c [0].
[...]
> > [0] <
> > https://www.fabiankeil.de/sourcecode/electrobsd/ElectroBSD-20220822-d9391cfeef5b/0183-cam-Fix-nullpointer-dereference.diff
>
>
> You should never pass NULL as the path to xpt_async(). It's a required
> element of the call: the path you are sending the async event to. And the
> path must have either a device or a bus, which that traceback didn't have.
> Now, how did it come to pass that the bus and/or device was NULL here?
> That's where the bug is... xpt_create_path always fills in the bus (via
> xpt_compile_path). One common way for it to become NULL like this is
> xpt_release_path is called on it. Maybe you can use dtrace to find where
> that's coming to pass? I'm not seeing anything super quick in the code, but
> I'm just doing a quick analysis to answer this email + past experience...
>
> So I hope that helps get to the next level of 'hey, this is wrong' but I
> know climbing that hill is hard. I'll help when I can, but at this stage I
> can only offer generic advice that I'm not feeling is very helpful.
While I have not yet been able to reproduce the exact same panic on the
same system, I ran into a related issue while trying to reproduce the
problem in a byhve vm.
Running ahcich0-detacher.sh:
| #!/bin/sh
|
| main() {
| while true; do
| date
| echo "Detaching ahcich0"
| sudo devctl detach ahcich0
| echo "Attaching ahcich0"
| sudo devctl attach ahcich0
| sleep 0.1
| sudo dd if=/dev/random bs=1m of=/dev/ada0 &
| sleep 0.1
| done
| }
|
| main
in two terminals at the same time within seconds reliably results in:
fk@test-vm ~ $kgdb151 /boot/kernel/kernel /var/crash/vmcore.3
GNU gdb (GDB) 15.1 [GDB v15.1 for FreeBSD]
[...]
Reading symbols from /boot/kernel/kernel...
Reading symbols from /usr/lib/debug//boot/kernel/kernel.debug...
Unread portion of the kernel message buffer:
trap number = 12
panic: page fault
cpuid = 1
time = 1768731529
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe00b5a727c0
vpanic() at vpanic+0x161/frame 0xfffffe00b5a728f0
panic() at panic+0x43/frame 0xfffffe00b5a72950
trap_pfault() at trap_pfault+0x3e0/frame 0xfffffe00b5a729a0
calltrap() at calltrap+0x8/frame 0xfffffe00b5a729a0
--- trap 0xc, rip = 0xffffffff80369a66, rsp = 0xfffffe00b5a72a70, rbp = 0xfffffe00b5a72aa0 ---
xpt_async() at xpt_async+0x2d6/frame 0xfffffe00b5a72aa0
ahci_ch_detach() at ahci_ch_detach+0x63/frame 0xfffffe00b5a72ae0
device_detach() at device_detach+0x180/frame 0xfffffe00b5a72b20
devctl2_ioctl() at devctl2_ioctl+0x412/frame 0xfffffe00b5a72bf0
devfs_ioctl() at devfs_ioctl+0xcb/frame 0xfffffe00b5a72c40
vn_ioctl() at vn_ioctl+0xc8/frame 0xfffffe00b5a72cb0
devfs_ioctl_f() at devfs_ioctl_f+0x1e/frame 0xfffffe00b5a72cd0
kern_ioctl() at kern_ioctl+0x286/frame 0xfffffe00b5a72d40
sys_ioctl() at sys_ioctl+0x117/frame 0xfffffe00b5a72e00
amd64_syscall() at amd64_syscall+0x117/frame 0xfffffe00b5a72f30
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe00b5a72f30
--- syscall (54, FreeBSD ELF64, ioctl), rip = 0x13788352457a, rsp = 0x1378821f4e68, rbp = 0x1378821f4f20 ---
Uptime: 4m34s
Dumping 533 out of 10203 MB:..3%..12%..21%..33%..42%..51%..63%..72%..81%..93%
Reading symbols from /boot/kernel/geom_eli.ko...
Reading symbols from /usr/lib/debug//boot/kernel/geom_eli.ko.debug...
Reading symbols from /boot/kernel/zfs.ko...
Reading symbols from /usr/lib/debug//boot/kernel/zfs.ko.debug...
__curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:57
57 __asm("movq %%gs:%P1,%0" : "=r" (td) : "n" (offsetof(struct pcpu,
(kgdb) where
#0 __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:57
#1 doadump (textdump=textdump@entry=1) at /usr/src/sys/kern/kern_shutdown.c:405
#2 0xffffffff80a20360 in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:523
#3 0xffffffff80a20869 in vpanic (fmt=0xffffffff80f8238a "%s", ap=ap@entry=0xfffffe00a2de6930) at /usr/src/sys/kern/kern_shutdown.c:967
#4 0xffffffff80a206a3 in panic (fmt=<unavailable>) at /usr/src/sys/kern/kern_shutdown.c:891
#5 0xffffffff80ec27c0 in trap_fatal (frame=<optimized out>, eva=<optimized out>) at /usr/src/sys/amd64/amd64/trap.c:1000
#6 0xffffffff80ec27c0 in trap_pfault (frame=0xfffffe00a2de69b0, usermode=false, signo=<optimized out>, ucode=<optimized out>)
#7 <signal handler called>
#8 0xffffffff80369a66 in xpt_async (async_code=256, path=0xfffff8000606eb60, async_arg=0x0) at /usr/src/sys/cam/cam_xpt.c:4321
#9 0xffffffff804b8d23 in ahci_ch_detach (dev=0xfffff800039d7800) at /usr/src/sys/dev/ahci/ahci.c:952
#10 0xffffffff80a5d050 in DEVICE_DETACH (dev=0xfffff800039d7800) at ./device_if.h:234
#11 device_detach (dev=dev@entry=0xfffff800039d7800) at /usr/src/sys/kern/subr_bus.c:2712
#12 0xffffffff80a64cc2 in devctl2_ioctl (cdev=<optimized out>, cmd=<optimized out>, data=0xfffff80108617700 "ahcich0", fflag=<optimized out>, td=<optimized out>) at /usr/src/sys/kern/subr_bus.c:5843
#13 0xffffffff808cd19b in devfs_ioctl (ap=0xfffffe00a2de6c58) at /usr/src/sys/fs/devfs/devfs_vnops.c:963
#14 0xffffffff80b25348 in VOP_IOCTL (vp=<optimized out>, command=<optimized out>, data=<optimized out>, fflag=<optimized out>, cred=<optimized out>, td=<optimized out>) at ./vnode_if.h:633
#15 0xffffffff80b25348 in vn_ioctl (fp=<optimized out>, com=0, data=0x2, active_cred=0x0, td=0xfffffe0012063d00)
#16 0xffffffff808cd80e in devfs_ioctl_f (fp=0xfffff8000606eb60, com=0, data=0x2, cred=0x0, td=0xfffffe0012063d00) at /usr/src/sys/fs/devfs/devfs_vnops.c:894
#17 0xffffffff80a93556 in fo_ioctl (fp=0xfffff8000cd2feb0, com=0, data=0xfffff80108617700, active_cred=0x0, td=0xfffff8000cdc4740) at /usr/src/sys/sys/file.h:379
#18 kern_ioctl (td=td@entry=0xfffff8000cdc4740, fd=<optimized out>, com=0, com@entry=2157462530, data=data@entry=0xfffff80108617700 "ahcich0") at /usr/src/sys/kern/sys_generic.c:810
#19 0xffffffff80a93267 in sys_ioctl (td=<optimized out>, uap=<optimized out>) at /usr/src/sys/kern/sys_generic.c:715
#20 0xffffffff80ec3107 in syscallenter (td=0xfffff8000cdc4740) at /usr/src/sys/amd64/amd64/../../kern/subr_syscall.c:193
#21 amd64_syscall (td=0xfffff8000cdc4740, traced=0) at /usr/src/sys/amd64/amd64/trap.c:1241
#22 <signal handler called>
#23 0x00000047b214757a in ?? ()
Backtrace stopped: Cannot access memory at address 0x47af775a58
(kgdb) f 8
#8 0xffffffff80369a66 in xpt_async (async_code=256, path=0xfffff8000606eb60, async_arg=0x0) at /usr/src/sys/cam/cam_xpt.c:4321
4321 xpt_freeze_simq(path->bus->sim, 1);
(kgdb) l
4316 }
4317 if (path != NULL) {
4318 if (path->device != NULL && path->device->lun_id != CAM_LUN_WILDCARD)
4319 xpt_freeze_devq(path, 1);
4320 else
4321 xpt_freeze_simq(path->bus->sim, 1);
4322 }
4323 xpt_action(ccb);
4324 }
4325
(kgdb) p *path
$1 = {periph = 0x0, bus = 0x0, target = 0x0, device = 0x0}
So this actually reaches the code I previously patched in [0]
but the patch isn't sufficient to hide the problem.
I also once managed to get devctl sleeping "forever" by running only
one ahcich0-detacher.sh instance, but haven't been able to reproduce
this yet. After triggering a panic through debug.kdb.panic I got:
(kgdb) where
#0 sched_switch (td=td@entry=0xfffff8002f485000, flags=flags@entry=259) at /usr/src/sys/kern/sched_ule.c:2437
#1 0xffffffff80a2e18d in mi_switch (flags=flags@entry=259) at /usr/src/sys/kern/kern_synch.c:548
#2 0xffffffff80a80799 in sleepq_switch (wchan=<optimized out>, pri=<optimized out>) at /usr/src/sys/kern/subr_sleepqueue.c:608
#3 0xffffffff80a80685 in sleepq_wait (wchan=<unavailable>, wchan@entry=0xfffff802341c7980, pri=<unavailable>) at /usr/src/sys/kern/subr_sleepqueue.c:659
#4 0xffffffff80a2d8e3 in _sleep (ident=ident@entry=0xfffff802341c7980, lock=lock@entry=0xfffffe00b78c8400, priority=priority@entry=43, wmesg=<optimized out>, sbt=sbt@entry=0, pr=pr@entry=0, flags=256)
at /usr/src/sys/kern/kern_synch.c:223
#5 0xffffffff80366607 in cam_sim_free (sim=0xfffff802341c7980, free_devq=free_devq@entry=1) at /usr/src/sys/cam/cam_sim.c:157
#6 0xffffffff804b8d67 in ahci_ch_detach (dev=0xfffff800039d7800, dev@entry=<error reading variable: value is not available>) at /usr/src/sys/dev/ahci/ahci.c:960
#7 0xffffffff80a5d050 in DEVICE_DETACH (dev=0xfffff800039d7800) at ./device_if.h:234
#8 device_detach (dev=dev@entry=0xfffff800039d7800) at /usr/src/sys/kern/subr_bus.c:2712
#9 0xffffffff80a64cc2 in devctl2_ioctl (cdev=<optimized out>, cmd=<unavailable>, cmd@entry=<error reading variable: value is not available>, data=0xfffff80090a01000 "ahcich0",
data@entry=<error reading variable: value is not available>, fflag=<optimized out>, td=<unavailable>, td@entry=<error reading variable: value is not available>) at /usr/src/sys/kern/subr_bus.c:5843
#10 0xffffffff808cd19b in devfs_ioctl (ap=0xfffffe00b5c4fc58, ap@entry=<error reading variable: value is not available>) at /usr/src/sys/fs/devfs/devfs_vnops.c:963
#11 0xffffffff80b25348 in VOP_IOCTL (vp=<optimized out>, command=<optimized out>, data=<optimized out>, fflag=<optimized out>, cred=<optimized out>, td=<optimized out>) at ./vnode_if.h:633
#12 0xffffffff80b25348 in vn_ioctl (fp=<optimized out>, com=<unavailable>, data=<unavailable>, active_cred=<unavailable>, td=<unavailable>)
#13 0xffffffff808cd80e in devfs_ioctl_f (fp=<unavailable>, fp@entry=<error reading variable: value is not available>, com=<unavailable>, com@entry=<error reading variable: value is not available>, data=<unavailable>,
data@entry=<error reading variable: value is not available>, cred=<unavailable>, cred@entry=<error reading variable: value is not available>, td=<unavailable>, td@entry=<error reading variable: value is not available>)
at /usr/src/sys/fs/devfs/devfs_vnops.c:894
#14 0xffffffff80a93556 in fo_ioctl (fp=0xfffff8000cd6d5f0, com=<unavailable>, data=0xfffff80090a01000, active_cred=<unavailable>, td=0xfffff8002f485000) at /usr/src/sys/sys/file.h:379
#15 kern_ioctl (td=td@entry=0xfffff8002f485000, fd=<optimized out>, com=<unavailable>, com@entry=2157462530, data=data@entry=0xfffff80090a01000 "ahcich0") at /usr/src/sys/kern/sys_generic.c:810
#16 0xffffffff80a93267 in sys_ioctl (td=<optimized out>, uap=<optimized out>) at /usr/src/sys/kern/sys_generic.c:715
#17 0xffffffff80ec3107 in syscallenter (td=0xfffff8002f485000) at /usr/src/sys/amd64/amd64/../../kern/subr_syscall.c:193
#18 amd64_syscall (td=0xfffff8002f485000, traced=0) at /usr/src/sys/amd64/amd64/trap.c:1241
#19 <signal handler called>
#20 0x0000314d6884457a in ?? ()
Backtrace stopped: Cannot access memory at address 0x314d656097a8
(kgdb) f 5
#5 0xffffffff80366607 in cam_sim_free (sim=0xfffff802341c7980, free_devq=free_devq@entry=1) at /usr/src/sys/cam/cam_sim.c:157
157 error = msleep(sim, mtx, PRIBIO, "simfree", 0);
(kgdb) l
152 mtx_assert(mtx, MA_OWNED);
153 }
154 KASSERT(sim->refcount >= 1, ("sim->refcount >= 1"));
155 sim->refcount--;
156 if (sim->refcount > 0) {
157 error = msleep(sim, mtx, PRIBIO, "simfree", 0);
158 KASSERT(error == 0, ("invalid error value for msleep(9)"));
159 }
160 KASSERT(sim->refcount == 0, ("sim->refcount == 0"));
161 if (mtx == &cam_sim_free_mtx) /* sim->mtx == NULL */
(kgdb) p *sim
$1 = {sim_action = 0xffffffff804b93d0 <ahciaction>, sim_poll = 0xffffffff804b9a60 <ahcipoll>, sim_name = 0xffffffff80fd9fa8 "ahcich", softc = 0xfffffe00b78c7000, mtx = 0xfffffe00b78c8400, links = {tqe_next = 0x0,
tqe_prev = 0x0}, path_id = 0, unit_number = 0, bus_id = 0, max_tagged_dev_openings = 32, max_dev_openings = 2, flags = 0, devq = 0xfffff80006070040, refcount = 1}
Fabian