Re: "(ada0:ahcich0:0:0:0): Periph destroyed", followed by panic
Date: Thu, 22 Jan 2026 15:05:03 UTC
Fabian Keil <freebsd-listen@fabiankeil.de> wrote on 2026-01-21 at 15:02:30:
> In the meantime the system with the real flaky disk (elektrobier4)
> panicked again in xpt_done_process() and I noticed that a similar
> problem has been reported in PR/277499 "panic in doneq0 xpt_done_td
> xpt_done_process after HDD falling off the bus (Periph destroyed)" [0]
> which references PR 263704 "Panic because device went away with
> XPT_ATA_IO pending" [1]. Both are "Closed FIXED" at the moment.
>
> For now I've set kern.cam.iosched.dynamic=0 on elektrobier4 as well
> to see how the behaviour differs when ada0 acts up again which
> I expect to happen within days.
With kern.cam.iosched.dynamic=0 the system tolerated the device loss:
2026-01-22T03:02:42.884803+01:00 elektrobier4 kernel <2>1 - - - (ada0:ahcich0:0:0:0): READ_FPDMA_QUEUED. ACB: 60 08 28 88 a2 40 01 00 00 00 00 00
2026-01-22T03:02:42.885493+01:00 elektrobier4 kernel <2>1 - - - (ada0:ahcich0:0:0:0): CAM status: Auto-Sense Retrieval Failed
2026-01-22T03:02:42.885508+01:00 elektrobier4 kernel <2>1 - - - (ada0:ahcich0:0:0:0): Error 5, Unretryable error
2026-01-22T03:02:42.885520+01:00 elektrobier4 kernel <2>1 - - - GEOM_ELI: g_eli_read_done() failed (error=5) gpt/rpool-ada2.eli[READ(offset=13832835072, length=4096)]
2026-01-22T03:02:44.141421+01:00 elektrobier4 kernel <2>1 - - - ahcich0: Timeout on slot 30 port 0
2026-01-22T03:02:44.141486+01:00 elektrobier4 kernel <2>1 - - - ahcich0: is 04000000 cs 40000000 ss 00000000 rs 40000000 tfd 451 serr 00400000 cmd 0000fe17
2026-01-22T03:02:44.141501+01:00 elektrobier4 kernel <2>1 - - - (ada0:ahcich0:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 08 a0 a9 35 40 18 00 00 00 00 00
2026-01-22T03:02:44.141514+01:00 elektrobier4 kernel <2>1 - - - (ada0:ahcich0:0:0:0): CAM status: Auto-Sense Retrieval Failed
2026-01-22T03:02:44.141527+01:00 elektrobier4 kernel <2>1 - - - (ada0:ahcich0:0:0:0): Error 5, Unretryable error
2026-01-22T03:02:44.141539+01:00 elektrobier4 kernel <2>1 - - - GEOM_ELI: g_eli_write_done() failed (error=5) gpt/dpool-ada2.eli[WRITE(offset=147618742272, length=4096)]
2026-01-22T03:02:44.141552+01:00 elektrobier4 kernel <2>1 - - - (ada0:ahcich0:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 08 e0 8b 36 40 1d 00 00 00 00 00
2026-01-22T03:02:44.141565+01:00 elektrobier4 kernel <2>1 - - - (ada0:ahcich0:0:0:0): CAM status: ATA Status Error
2026-01-22T03:02:44.141577+01:00 elektrobier4 kernel <2>1 - - - (ada0:ahcich0:0:0:0): ATA status: 00 ()
2026-01-22T03:02:44.141590+01:00 elektrobier4 kernel <2>1 - - - (ada0:ahcich0:0:0:0): RES: 00 00 00 00 00 00 00 00 00 00 00
2026-01-22T03:02:44.141602+01:00 elektrobier4 kernel <2>1 - - - (ada0:ahcich0:0:0:0): Retrying command, 3 more tries remain
2026-01-22T03:02:44.141615+01:00 elektrobier4 kernel <2>1 - - - (ada0:ahcich0:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 08 b0 b9 00 40 01 00 00 00 00 00
2026-01-22T03:02:44.141629+01:00 elektrobier4 kernel <2>1 - - - (ada0:ahcich0:0:0:0): CAM status: ATA Status Error
2026-01-22T03:02:44.141641+01:00 elektrobier4 kernel <2>1 - - - (ada0:ahcich0:0:0:0): ATA status: 00 ()
2026-01-22T03:02:44.141655+01:00 elektrobier4 kernel <2>1 - - - (ada0:ahcich0:0:0:0): RES: 00 00 00 00 00 00 00 00 00 00 00
2026-01-22T03:02:44.141669+01:00 elektrobier4 kernel <2>1 - - - (ada0:ahcich0:0:0:0): Retrying command, 3 more tries remain
2026-01-22T03:02:44.141682+01:00 elektrobier4 kernel <2>1 - - - (ada0:ahcich0:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 08 50 fa 4a 40 00 00 00 00 00 00
2026-01-22T03:02:44.141694+01:00 elektrobier4 kernel <2>1 - - - (ada0:ahcich0:0:0:0): CAM status: ATA Status Error
2026-01-22T03:02:44.141706+01:00 elektrobier4 kernel <2>1 - - - (ada0:ahcich0:0:0:0): ATA status: 00 ()
2026-01-22T03:02:44.141720+01:00 elektrobier4 kernel <2>1 - - - (ada0:ahcich0:0:0:0): RES: 00 00 00 00 00 00 00 00 00 00 00
2026-01-22T03:02:44.141733+01:00 elektrobier4 kernel <2>1 - - - (ada0:ahcich0:0:0:0): Retrying command, 3 more tries remain
2026-01-22T03:02:44.306685+01:00 elektrobier4 ZFS <188>1 92995 - - vdev I/O failure, zpool=dpool path=/dev/gpt/dpool-ada2.eli offset=147618742272 size=4096 error=5
2026-01-22T03:02:44.323799+01:00 elektrobier4 ZFS <188>1 23854 - - vdev I/O failure, zpool=rpool path=/dev/gpt/rpool-ada2.eli offset=13832835072 size=4096 error=5
2026-01-22T03:50:05.395476+01:00 elektrobier4 kernel <2>1 - - - (ada0:ahcich0:0:0:0): READ_FPDMA_QUEUED. ACB: 60 28 f8 6a 76 40 00 00 00 00 00 00
2026-01-22T03:50:05.396193+01:00 elektrobier4 kernel <2>1 - - - (ada0:ahcich0:0:0:0): CAM status: Auto-Sense Retrieval Failed
2026-01-22T03:50:05.396208+01:00 elektrobier4 kernel <2>1 - - - (ada0:ahcich0:0:0:0): Error 5, Unretryable error
2026-01-22T03:50:05.396220+01:00 elektrobier4 kernel <2>1 - - - GEOM_ELI: g_eli_read_done() failed (error=5) gpt/rpool-ada2.eli[READ(offset=3762679808, length=20480)]
2026-01-22T03:50:06.571682+01:00 elektrobier4 kernel <2>1 - - - ahcich0: Timeout on slot 27 port 0
2026-01-22T03:50:06.571730+01:00 elektrobier4 kernel <2>1 - - - ahcich0: is 04000000 cs 08000000 ss 00000000 rs 08000000 tfd 451 serr 00400000 cmd 0000fb17
2026-01-22T03:50:06.571743+01:00 elektrobier4 kernel <2>1 - - - (ada0:ahcich0:0:0:0): READ_FPDMA_QUEUED. ACB: 60 28 40 2f 7b 40 00 00 00 00 00 00
2026-01-22T03:50:06.571755+01:00 elektrobier4 kernel <2>1 - - - (ada0:ahcich0:0:0:0): CAM status: Auto-Sense Retrieval Failed
2026-01-22T03:50:06.571767+01:00 elektrobier4 kernel <2>1 - - - (ada0:ahcich0:0:0:0): Error 5, Unretryable error
2026-01-22T03:50:06.571778+01:00 elektrobier4 kernel <2>1 - - - GEOM_ELI: g_eli_read_done() failed (error=5) gpt/rpool-ada2.eli[READ(offset=3922624512, length=20480)]
2026-01-22T03:50:06.571789+01:00 elektrobier4 kernel <2>1 - - - (ada0:ahcich0:0:0:0): READ_FPDMA_QUEUED. ACB: 60 20 68 2f 7b 40 00 00 00 00 00 00
2026-01-22T03:50:06.571800+01:00 elektrobier4 kernel <2>1 - - - (ada0:ahcich0:0:0:0): CAM status: ATA Status Error
2026-01-22T03:50:06.571815+01:00 elektrobier4 kernel <2>1 - - - (ada0:ahcich0:0:0:0): ATA status: 00 ()
2026-01-22T03:50:06.571825+01:00 elektrobier4 kernel <2>1 - - - (ada0:ahcich0:0:0:0): RES: 00 00 00 00 00 00 00 00 00 00 00
2026-01-22T03:50:06.571837+01:00 elektrobier4 kernel <2>1 - - - (ada0:ahcich0:0:0:0): Retrying command, 3 more tries remain
2026-01-22T03:50:06.579364+01:00 elektrobier4 ZFS <188>1 97887 - - vdev I/O failure, zpool=rpool path=/dev/gpt/rpool-ada2.eli offset=3922624512 size=20480 error=5
2026-01-22T07:49:33.995687+01:00 elektrobier4 kernel <2>1 - - - ahcich0: Timeout on slot 31 port 0
2026-01-22T07:49:33.995756+01:00 elektrobier4 kernel <2>1 - - - ahcich0: is 00000000 cs 80000000 ss 00000000 rs 80000000 tfd c0 serr 00000801 cmd 0000ff17
2026-01-22T07:49:33.995771+01:00 elektrobier4 kernel <2>1 - - - (ada0:ahcich0:0:0:0): FLUSHCACHE48. ACB: ea 00 00 00 00 40 00 00 00 00 00 00
2026-01-22T07:49:33.995786+01:00 elektrobier4 kernel <2>1 - - - (ada0:ahcich0:0:0:0): CAM status: Command timeout
2026-01-22T07:49:33.995798+01:00 elektrobier4 kernel <2>1 - - - (ada0:ahcich0:0:0:0): Retrying command, 0 more tries remain
2026-01-22T07:55:36.487677+01:00 elektrobier4 kernel <2>1 - - - ahcich0: Timeout on slot 0 port 0
2026-01-22T07:55:36.487748+01:00 elektrobier4 kernel <2>1 - - - ahcich0: is 00000000 cs 00000001 ss 00000000 rs 00000001 tfd c0 serr 00000000 cmd 0000e017
2026-01-22T07:55:36.487762+01:00 elektrobier4 kernel <2>1 - - - (ada0:ahcich0:0:0:0): FLUSHCACHE48. ACB: ea 00 00 00 00 40 00 00 00 00 00 00
2026-01-22T07:55:36.487777+01:00 elektrobier4 kernel <2>1 - - - (ada0:ahcich0:0:0:0): CAM status: Command timeout
2026-01-22T07:55:36.487791+01:00 elektrobier4 kernel <2>1 - - - (ada0:ahcich0:0:0:0): Retrying command, 0 more tries remain
2026-01-22T08:05:41.028119+01:00 elektrobier4 kernel <2>1 - - - ahcich0: Timeout on slot 7 port 0
2026-01-22T08:05:41.028189+01:00 elektrobier4 kernel <2>1 - - - ahcich0: is 00000000 cs 00000080 ss 00000000 rs 00000080 tfd c0 serr 00000000 cmd 0000e717
2026-01-22T08:05:41.028205+01:00 elektrobier4 kernel <2>1 - - - (ada0:ahcich0:0:0:0): FLUSHCACHE48. ACB: ea 00 00 00 00 40 00 00 00 00 00 00
2026-01-22T08:05:41.028220+01:00 elektrobier4 kernel <2>1 - - - (ada0:ahcich0:0:0:0): CAM status: Command timeout
2026-01-22T08:05:41.028233+01:00 elektrobier4 kernel <2>1 - - - (ada0:ahcich0:0:0:0): Retrying command, 0 more tries remain
2026-01-22T08:06:36.232073+01:00 elektrobier4 kernel <2>1 - - - ahcich0: AHCI reset: device not ready after 31000ms (tfd = 00000080)
2026-01-22T08:06:36.232140+01:00 elektrobier4 kernel <2>1 - - - ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
2026-01-22T08:06:36.232156+01:00 elektrobier4 kernel <2>1 - - - ada0: <WDC WD20EFRX-68EUZN0 80.00A80> s/n WD-WMC4M2924273 detached
2026-01-22T08:06:36.232170+01:00 elektrobier4 kernel <2>1 - - - GEOM_ELI: g_eli_write_done() failed (error=6) gpt/rpool-ada2.eli[WRITE(offset=28995383296, length=4096)]
2026-01-22T08:06:36.232184+01:00 elektrobier4 kernel <2>1 - - - GEOM_ELI: g_eli_read_done() failed (error=6) gpt/rpool-ada2.eli[READ(offset=270336, length=8192)]
2026-01-22T08:06:36.232197+01:00 elektrobier4 kernel <2>1 - - - GEOM_ELI: g_eli_write_done() failed (error=6) gpt/dpool-ada2.eli[WRITE(offset=842082689024, length=4096)]
2026-01-22T08:06:36.232215+01:00 elektrobier4 kernel <2>1 - - - GEOM_ELI: g_eli_write_done() failed (error=6) gpt/dpool-ada2.eli[WRITE(offset=623058358272, length=4096)]
2026-01-22T08:06:36.435045+01:00 elektrobier4 kernel <2>1 - - - GEOM_ELI: Device gpt/rpool-ada2.eli destroyed.
2026-01-22T08:06:36.435154+01:00 elektrobier4 kernel <2>1 - - - GEOM_ELI: Detached gpt/rpool-ada2.eli on last close.
2026-01-22T08:06:36.857089+01:00 elektrobier4 ZFS <189>1 73396 - - vdev state changed, pool_guid=8709909639177779425 vdev_guid=10409952866552169513
2026-01-22T08:06:36.860947+01:00 elektrobier4 ZFS <189>1 91138 - - vdev is removed, pool_guid=8709909639177779425 vdev_guid=10409952866552169513
2026-01-22T08:06:36.864851+01:00 elektrobier4 ZFS <189>1 20 - - vdev state changed, pool_guid=8709909639177779425 vdev_guid=10409952866552169513
2026-01-22T08:06:37.358376+01:00 elektrobier4 kernel <2>1 - - - GEOM_ELI: Device gpt/dpool-ada2.eli destroyed.
2026-01-22T08:06:37.358426+01:00 elektrobier4 kernel <2>1 - - - GEOM_ELI: Detached gpt/dpool-ada2.eli on last close.
2026-01-22T08:06:37.358442+01:00 elektrobier4 kernel <2>1 - - - (ada0:ahcich0:0:0:0): Periph destroyed
2026-01-22T08:06:37.361938+01:00 elektrobier4 ZFS <189>1 28764 - - vdev state changed, pool_guid=7956782197117635111 vdev_guid=2600988480923001126
2026-01-22T08:06:41.638686+01:00 elektrobier4 kernel <2>1 - - - (aprobe0:ahcich0:0:0:0): NOP FLUSHQUEUE. ACB: 00 00 00 00 00 00 00 00 00 00 00 00
2026-01-22T08:06:41.638749+01:00 elektrobier4 kernel <2>1 - - - (aprobe0:ahcich0:0:0:0): CAM status: ATA Status Error
2026-01-22T08:06:41.638764+01:00 elektrobier4 kernel <2>1 - - - (aprobe0:ahcich0:0:0:0): ATA status: d1 (BSY DRDY SERV ERR), error: 04 (ABRT )
2026-01-22T08:06:41.638777+01:00 elektrobier4 kernel <2>1 - - - (aprobe0:ahcich0:0:0:0): RES: d1 04 ff ff ff ff ff ff ff ff ff
2026-01-22T08:06:41.638791+01:00 elektrobier4 kernel <2>1 - - - (aprobe0:ahcich0:0:0:0): Error 5, Retries exhausted
2026-01-22T08:06:41.638804+01:00 elektrobier4 kernel <2>1 - - - (aprobe1:ahcich0:0:15:0): NOP FLUSHQUEUE. ACB: 00 00 00 00 00 00 00 00 00 00 00 00
2026-01-22T08:06:41.638816+01:00 elektrobier4 kernel <2>1 - - - (aprobe1:ahcich0:0:15:0): CAM status: ATA Status Error
2026-01-22T08:06:41.638831+01:00 elektrobier4 kernel <2>1 - - - (aprobe1:ahcich0:0:15:0): ATA status: d1 (BSY DRDY SERV ERR), error: 04 (ABRT )
2026-01-22T08:06:41.638843+01:00 elektrobier4 kernel <2>1 - - - (aprobe1:ahcich0:0:15:0): RES: d1 04 ff ff ff ff ff ff ff ff ff
2026-01-22T08:06:41.638856+01:00 elektrobier4 kernel <2>1 - - - (aprobe1:ahcich0:0:15:0): Error 5, Retries exhausted
2026-01-22T08:06:41.638869+01:00 elektrobier4 kernel <2>1 - - - (aprobe0:ahcich0:0:0:0): NOP FLUSHQUEUE. ACB: 00 00 00 00 00 00 00 00 00 00 00 00
2026-01-22T08:06:41.638882+01:00 elektrobier4 kernel <2>1 - - - (aprobe0:ahcich0:0:0:0): CAM status: ATA Status Error
2026-01-22T08:06:41.638895+01:00 elektrobier4 kernel <2>1 - - - (aprobe0:ahcich0:0:0:0): ATA status: d1 (BSY DRDY SERV ERR), error: 04 (ABRT )
2026-01-22T08:06:41.638909+01:00 elektrobier4 kernel <2>1 - - - (aprobe0:ahcich0:0:0:0): RES: d1 04 ff ff ff ff ff ff ff ff ff
2026-01-22T08:06:41.638921+01:00 elektrobier4 kernel <2>1 - - - (aprobe0:ahcich0:0:0:0): Error 5, Retries exhausted
fk@elektrobier4 ~ $zpool status
pool: dpool
state: DEGRADED
status: One or more devices has been removed by the administrator.
Sufficient replicas exist for the pool to continue functioning in a
degraded state.
action: Online the device using zpool online' or replace the device with
'zpool replace'.
scan: scrub repaired 0B in 04:53:11 with 0 errors on 2026-01-18 00:39:36
config:
NAME STATE READ WRITE CKSUM
dpool DEGRADED 0 0 0
mirror-0 DEGRADED 0 0 0
gpt/dpool-ada1.eli ONLINE 0 0 0
gpt/dpool-ada2.eli REMOVED 0 0 0
errors: No known data errors
pool: rpool
state: DEGRADED
status: One or more devices are faulted in response to persistent errors.
Sufficient replicas exist for the pool to continue functioning in a
degraded state.
action: Replace the faulted device, or use 'zpool clear' to mark the device
repaired.
scan: scrub repaired 0B in 00:02:27 with 0 errors on 2026-01-16 09:37:26
config:
NAME STATE READ WRITE CKSUM
rpool DEGRADED 0 0 0
mirror-0 DEGRADED 0 0 0
gpt/rpool-ada2.eli FAULTED 0 0 0 too many errors
gpt/rpool-ada1.eli ONLINE 0 0 0
errors: No known data errors
Fabian