Re: nvme controller reset failures on recent -CURRENT
Date: Sun, 25 Oct 2026 21:19:26 UTC
You are running 002C firmware. Please update to latest . You should have nda0: <INTEL SSDPEKNW512G8 **005C** BTNH940617WE512A> not nda0: <INTEL SSDPEKNW512G8 *002C* BTNH940617WE512A>. This looks like a nvme firmware problem. Abdelkader On Sun, 25 Oct 2026, at 21:02, Don Lewis wrote: > This problem is back. > > Oct 24 19:23:53 zipper kernel: nvme0: Resetting controller due to a timeout and possible hot unplug. > Oct 24 19:23:53 zipper syslogd: last message repeated 1 times > Oct 24 19:23:53 zipper kernel: nvme0: event="start" > Oct 24 19:23:54 zipper kernel: nvme0: Waiting for reset to complete > Oct 24 19:23:54 zipper kernel: nvme0: event="timed_out" > Oct 24 19:23:54 zipper kernel: nvme0: Waiting for reset to complete > Oct 24 19:23:54 zipper kernel: nvme0: failing queued i/o > Oct 24 19:23:54 zipper kernel: nvme0: READ (02) sqid:8 cid:0 nsid:1 lba:1180280 len:72 > Oct 24 19:23:54 zipper kernel: nvme0: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:8 cid:0 cdw0:0 > Oct 24 19:23:54 zipper kernel: nvme0: WRITE (01) sqid:8 cid:0 nsid:1 lba:243296104 len:16 > Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): READ (02). NCB: opc=2 fuse=0 nsid=1 prp1=0 prp2=0 cdw=120278 0 47 0 0 0 > Oct 24 19:23:54 zipper kernel: nvme0: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:8 cid:0 cdw0:0 > Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): CAM status: NVME Status Error > Oct 24 19:23:54 zipper kernel: nvme0: WRITE (01) sqid:8 cid:0 nsid:1 lba:243296120 len:16 > Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): NVMe status: ABORTED_BY_REQUEST (00/07) DNR > Oct 24 19:23:54 zipper kernel: nvme0: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:8 cid:0 cdw0:0 > Oct 24 19:23:54 zipper kernel: nvme0: WRITE (01) sqid:8 cid:0 nsid:1 lba:243296136 len:16 > Oct 24 19:23:54 zipper kernel: nvme0: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:8 cid:0 cdw0:0 > Oct 24 19:23:54 zipper kernel: nvme0: WRITE (01) sqid:8 cid:0 nsid:1 lba:243296152 len:16 > Oct 24 19:23:54 zipper kernel: nvme0: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:8 cid:0 cdw0:0 > Oct 24 19:23:54 zipper kernel: nvme0: WRITE (01) sqid:8 cid:0 nsid:1 lba:243296168 len:8 > Oct 24 19:23:54 zipper kernel: nvme0: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:8 cid:0 cdw0:0 > Oct 24 19:23:54 zipper kernel: nvme0: WRITE (01) sqid:8 cid:0 nsid:1 lba:243296176 len:16 > Oct 24 19:23:54 zipper kernel: nvme0: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:8 cid:0 cdw0:0 > Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): Error 5, Retries exhausted > Oct 24 19:23:54 zipper kernel: nvme0: WRITE (01) sqid:8 cid:0 nsid:1 lba:243296192 len:16 > Oct 24 19:23:54 zipper kernel: nvme0: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:8 cid:0 cdw0:0 > Oct 24 19:23:54 zipper kernel: nvme0: WRITE (01) sqid:8 cid:0 nsid:1 lba:243296208 len:16 > Oct 24 19:23:54 zipper kernel: nvme0: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:8 cid:0 cdw0:0 > Oct 24 19:23:54 zipper kernel: nvme0: WRITE (01) sqid:8 cid:0 nsid:1 lba:243296224 len:8 > Oct 24 19:23:54 zipper kernel: nvme0: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:8 cid:0 cdw0:0 > Oct 24 19:23:54 zipper kernel: nvme0: WRITE (01) sqid:8 cid:0 nsid:1 lba:243296232 len:16 > Oct 24 19:23:54 zipper kernel: nvme0: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:8 cid:0 cdw0:0 > Oct 24 19:23:54 zipper kernel: swap_pager: I/O error - pagein failed; blkno 147279,size 36864, error 5 > Oct 24 19:23:54 zipper kernel: vm_fault: pager read error, pid 1916 (devd) > Oct 24 19:23:54 zipper kernel: nda0 at nvme0 bus 0 scbus9 target 0 lun 1 > Oct 24 19:23:54 zipper kernel: nda0: <INTEL SSDPEKNW512G8 002C BTNH940617WE512A> s/n BTNH940617WE512A detached > Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): WRITE (01). NCB: opc=1 fuse=0 nsid=1 prp1=0 prp2=0 cdw=e806768 0 f 0 0 0 > Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): CAM status: NVME Status Error > Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): NVMe status: ABORTED_BY_REQUEST (00/07) DNR > Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): Error 6, Periph was invalidated > Oct 24 19:23:54 zipper kernel: swap_pager: I/O error - pagein failed; blkno 1286155,size 8192, error 6 > Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): WRITE (01). NCB: opc=1 fuse=0 nsid=1 prp1=0 prp2=0 cdw=e806778 0 f 0 0 0 > Oct 24 19:23:54 zipper kernel: vm_fault: pager read error, pid 1916 (devd) > Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): CAM status: NVME Status Error > Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): NVMe status: ABORTED_BY_REQUEST (00/07) DNR > Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): Error 6, Periph was invalidated > Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): WRITE (01). NCB: opc=1 fuse=0 nsid=1 prp1=0 prp2=0 cdw=e806788 0 f 0 0 0 > Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): CAM status: NVME Status Error > Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): NVMe status: ABORTED_BY_REQUEST (00/07) DNR > Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): Error 6, Periph was invalidated > Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): WRITE (01). NCB: opc=1 fuse=0 nsid=1 prp1=0 prp2=0 cdw=e806798 0 f 0 0 0 > Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): CAM status: NVME Status Error > Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): NVMe status: ABORTED_BY_REQUEST (00/07) DNR > Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): Error 6, Periph was invalidated > Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): WRITE (01). NCB: opc=1 fuse=0 nsid=1 prp1=0 prp2=0 cdw=e8067a8 0 7 0 0 0 > Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): CAM status: NVME Status Error > Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): NVMe status: ABORTED_BY_REQUEST (00/07) DNR > Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): Error 6, Periph was invalidated > Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): WRITE (01). NCB: opc=1 fuse=0 nsid=1 prp1=0 prp2=0 cdw=e8067b0 0 f 0 0 0 > Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): CAM status: NVME Status Error > Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): NVMe status: ABORTED_BY_REQUEST (00/07) DNR > Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): Error 6, Periph was invalidated > Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): WRITE (01). NCB: opc=1 fuse=0 nsid=1 prp1=0 prp2=0 cdw=e8067c0 0 f 0 0 0 > Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): CAM status: NVME Status Error > Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): NVMe status: ABORTED_BY_REQUEST (00/07) DNR > Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): Error 6, Periph was invalidated > Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): WRITE (01). NCB: opc=1 fuse=0 nsid=1 prp1=0 prp2=0 cdw=e8067d0 0 f 0 0 0 > Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): CAM status: NVME Status Error > > The machine never recovered until I did a reset. > > FreeBSD zipper.catspoiler.org 16.0-CURRENT FreeBSD 16.0-CURRENT #52 main-n290318-a3d1d4e88ae7: Sat Oct 24 10:54:51 PDT 2026 dl@zipper.catspoiler.org:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64 > > nda0 at nvme0 bus 0 scbus9 target 0 lun 1 > nda0: <INTEL SSDPEKNW512G8 002C BTNH940617WE512A> > nda0: Serial Number BTNH940617WE512A > nda0: nvme version 1.3 > nda0: 488386MB (1000215216 512 byte sectors) > > # nvmecontrol logpage -p 0x1 nvme0 > Error Information Log > ===================== > No error entries found > # nvmecontrol logpage -p 0x2 nvme0 > SMART/Health Information Log > ============================ > Critical Warning State: 0x00 > Available spare: 0 > Temperature: 0 > Device reliability: 0 > Read only: 0 > Volatile memory backup: 0 > Temperature: 306 K, 32.85 C, 91.13 F > Available spare: 100 > Available spare threshold: 10 > Percentage used: 11 > Data units (512,000 byte) read: 45037975 > Data units written: 80040521 > Host read commands: 2359001540 > Host write commands: 1489208174 > Controller busy time (minutes): 92832 > Power cycles: 131 > Power on hours: 42510 > Unsafe shutdowns: 16 > Media errors: 0 > No. error info log entries: 0 > Warning Temp Composite Time: 0 > Error Temp Composite Time: 0 > Temperature 1 Transition Count: 5231 > Temperature 2 Transition Count: 0 > Total Time For Temperature 1: 41213 > Total Time For Temperature 2: 0 > > The Temperature 1 Count and Time are identical to the values from 2024 > (see below), now that the drive has a finned heatsink. > > In both cases, the failures have occurred at low ambient temperatures. > Yesterday's failure happened with the room at about 66 F. I had no > problems under the same load in July with the room at about 85 F. > > > > On 13 Feb, Don Lewis wrote: > > On 12 Feb, Warner Losh wrote: > >> On Mon, Feb 12, 2024 at 9:15 PM Don Lewis <truckman@freebsd.org> wrote: > >> > >>> On 12 Feb, Maxim Sobolev wrote: > >>> > Might be an overheating. Today's nvme drives are notoriously flaky if you > >>> > run them without proper heat sink attached to it. > >>> > >>> I don't think it is a thermal problem. According to the drive health > >>> page, the device temperature has never reached Temperature 2, whatever > >>> that is. The room temperature is around 65F. The system was stable > >>> last summer when the room temperature spent a lot of time in the 80-85F > >>> range. The device temperature depends a lot on the I/O rate, and the > >>> last panic happened when the I/O rate had been below 40tps for quite a > >>> while. > >>> > >> > >> It did reach temperature 1, though. That's the 'Warning this drive is too > >> hot' temperature. It has spent 41213 minutes of your 19297 hours of up > >> time, or an average of 2 minutes per hour. That's too much. Temperature > >> 2 is critical error: we are about to shut down completely due to it > >> being too hot. It's only a couple degrees below hardware power off > >> due to temperature in many drives. Some really cheap ones don't really > >> implement it at all. On my card with the bad heat sink, Warning temp is > >> 70C while critical is 75C while IIRC thermal shutdown is 78C or 80C. > >> > >> I don't think we report these values in nvmecontrol identify. But you can > >> do a raw dump with -x look at bytes 266:267 for warning and 268:269 > >> for critical. > >> > >> In contrast, the few dozen drives that I have, all of which have been > >> abused in various ways, And only one of them has any heat issues, > >> and that one is an engineering special / sample with what I think is > >> a damaged heat sink. If your card has no heat sink, this could well > >> be what's going on. > >> > >> This panic means "the nvme card lost its mind and stopped talking > >> to the host". Its status registers read 0xff's, which means that the card > >> isn't decoding bus signals. Usually this means that the firmware on the > >> card has faulted and rebooted. If the card is overheating, then this could > >> well be what's happening. > >> > >> There's a tiny chance that this could be something more exotic, > >> but my money is on hardware gone bad after 2 years of service. I don't think > >> this is 'wear out' of the NAND (it's only 15TB written, but it could be if > >> this > >> drive is really really crappy nand: first generation QLC maybe, but it seems > >> too new). It might also be a connector problem that's developed over time. > >> There might be a few other things too, but I don't think this is a U.2 drive > >> with funky cables. > > > > The system was probably idle the majority of those two years of power on > > time. > > > > It's one of these: > > https://www.techpowerup.com/ssd-specs/intel-660p-512-gb.d437 > > I've seen comments that these generally don't need cooling. > > > > I just ordered a heatsink with some nice big fins, but it will take a > > week or more to arrive. > > > >> > >>> > On Mon, Feb 12, 2024, 4:28 PM Don Lewis <truckman@freebsd.org> wrote: > >>> > > >>> >> I just upgraded my package build machine to: > >>> >> FreeBSD 15.0-CURRENT #110 main-n268161-4015c064200e > >>> >> from: > >>> >> FreeBSD 15.0-CURRENT #106 main-n265953-a5ed6a815e38 > >>> >> and I've had two nvme-triggered panics in the last day. > >>> >> > >>> >> nvme is being used for swap and L2ARC. I'm not able to get a crash > >>> >> dump, probably because the nvme device has gone away and I get an error > >>> >> about not having a dump device. It looks like a low-memory panic > >>> >> because free memory is low and zfs is calling malloc(). > >>> >> > >>> >> This shows up in the log leading up to the panic: > >>> >> Feb 12 10:07:41 zipper kernel: nvme0: Resetting controller due to a > >>> >> timeout a > >>> >> nd possible hot unplug. > >>> >> Feb 12 10:07:41 zipper syslogd: last message repeated 1 times > >>> >> Feb 12 10:07:41 zipper kernel: nvme0: resetting controller > >>> >> Feb 12 10:07:41 zipper kernel: nvme0: Resetting controller due to a > >>> >> timeout a > >>> >> nd possible hot unplug. > >>> >> Feb 12 10:07:41 zipper syslogd: last message repeated 1 times > >>> >> Feb 12 10:07:41 zipper kernel: nvme0: Waiting for reset to complete > >>> >> Feb 12 10:07:41 zipper syslogd: last message repeated 2 times > >>> >> Feb 12 10:07:41 zipper kernel: nvme0: failing queued i/o > >>> >> Feb 12 10:07:41 zipper kernel: nvme0: Failed controller, stopping > >>> watchdog > >>> >> ti > >>> >> meout. > >>> >> > >>> >> The device looks healthy to me: > >>> >> SMART/Health Information Log > >>> >> ============================ > >>> >> Critical Warning State: 0x00 > >>> >> Available spare: 0 > >>> >> Temperature: 0 > >>> >> Device reliability: 0 > >>> >> Read only: 0 > >>> >> Volatile memory backup: 0 > >>> >> Temperature: 312 K, 38.85 C, 101.93 F > >>> >> Available spare: 100 > >>> >> Available spare threshold: 10 > >>> >> Percentage used: 3 > >>> >> Data units (512,000 byte) read: 5761183 > >>> >> Data units written: 29911502 > >>> >> Host read commands: 471921188 > >>> >> Host write commands: 605394753 > >>> >> Controller busy time (minutes): 32359 > >>> >> Power cycles: 110 > >>> >> Power on hours: 19297 > >>> >> Unsafe shutdowns: 14 > >>> >> Media errors: 0 > >>> >> No. error info log entries: 0 > >>> >> Warning Temp Composite Time: 0 > >>> >> Error Temp Composite Time: 0 > >>> >> Temperature 1 Transition Count: 5231 > >>> >> Temperature 2 Transition Count: 0 > >>> >> Total Time For Temperature 1: 41213 > >>> >> Total Time For Temperature 2: 0 > >>> >> > >>> >> > >>> >> > >>> > >>> > >>> > > > > > > >