Re: nvme controller reset failures on recent -CURRENT
Date: Sun, 25 Oct 2026 21:02:59 UTC
This problem is back. Oct 24 19:23:53 zipper kernel: nvme0: Resetting controller due to a timeout= and possible hot unplug. Oct 24 19:23:53 zipper syslogd: last message repeated 1 times Oct 24 19:23:53 zipper kernel: nvme0: event=3D"start" Oct 24 19:23:54 zipper kernel: nvme0: Waiting for reset to complete Oct 24 19:23:54 zipper kernel: nvme0: event=3D"timed_out" Oct 24 19:23:54 zipper kernel: nvme0: Waiting for reset to complete Oct 24 19:23:54 zipper kernel: nvme0: failing queued i/o Oct 24 19:23:54 zipper kernel: nvme0: READ (02) sqid:8 cid:0 nsid:1 lba:118= 0280 len:72 Oct 24 19:23:54 zipper kernel: nvme0: ABORTED_BY_REQUEST (00/07) crd:0 m:0 = dnr:1 p:0 sqid:8 cid:0 cdw0:0 Oct 24 19:23:54 zipper kernel: nvme0: WRITE (01) sqid:8 cid:0 nsid:1 lba:24= 3296104 len:16 Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): READ (02). NCB: opc=3D2 = fuse=3D0 nsid=3D1 prp1=3D0 prp2=3D0 cdw=3D120278 0 47 0 0 0 Oct 24 19:23:54 zipper kernel: nvme0: ABORTED_BY_REQUEST (00/07) crd:0 m:0 = dnr:1 p:0 sqid:8 cid:0 cdw0:0 Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): CAM status: NVME Status = Error Oct 24 19:23:54 zipper kernel: nvme0: WRITE (01) sqid:8 cid:0 nsid:1 lba:24= 3296120 len:16 Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): NVMe status: ABORTED_BY_= REQUEST (00/07) DNR Oct 24 19:23:54 zipper kernel: nvme0: ABORTED_BY_REQUEST (00/07) crd:0 m:0 = dnr:1 p:0 sqid:8 cid:0 cdw0:0 Oct 24 19:23:54 zipper kernel: nvme0: WRITE (01) sqid:8 cid:0 nsid:1 lba:24= 3296136 len:16 Oct 24 19:23:54 zipper kernel: nvme0: ABORTED_BY_REQUEST (00/07) crd:0 m:0 = dnr:1 p:0 sqid:8 cid:0 cdw0:0 Oct 24 19:23:54 zipper kernel: nvme0: WRITE (01) sqid:8 cid:0 nsid:1 lba:24= 3296152 len:16 Oct 24 19:23:54 zipper kernel: nvme0: ABORTED_BY_REQUEST (00/07) crd:0 m:0 = dnr:1 p:0 sqid:8 cid:0 cdw0:0 Oct 24 19:23:54 zipper kernel: nvme0: WRITE (01) sqid:8 cid:0 nsid:1 lba:24= 3296168 len:8 Oct 24 19:23:54 zipper kernel: nvme0: ABORTED_BY_REQUEST (00/07) crd:0 m:0 = dnr:1 p:0 sqid:8 cid:0 cdw0:0 Oct 24 19:23:54 zipper kernel: nvme0: WRITE (01) sqid:8 cid:0 nsid:1 lba:24= 3296176 len:16 Oct 24 19:23:54 zipper kernel: nvme0: ABORTED_BY_REQUEST (00/07) crd:0 m:0 = dnr:1 p:0 sqid:8 cid:0 cdw0:0 Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): Error 5, Retries exhaust= ed Oct 24 19:23:54 zipper kernel: nvme0: WRITE (01) sqid:8 cid:0 nsid:1 lba:24= 3296192 len:16 Oct 24 19:23:54 zipper kernel: nvme0: ABORTED_BY_REQUEST (00/07) crd:0 m:0 = dnr:1 p:0 sqid:8 cid:0 cdw0:0 Oct 24 19:23:54 zipper kernel: nvme0: WRITE (01) sqid:8 cid:0 nsid:1 lba:24= 3296208 len:16 Oct 24 19:23:54 zipper kernel: nvme0: ABORTED_BY_REQUEST (00/07) crd:0 m:0 = dnr:1 p:0 sqid:8 cid:0 cdw0:0 Oct 24 19:23:54 zipper kernel: nvme0: WRITE (01) sqid:8 cid:0 nsid:1 lba:24= 3296224 len:8 Oct 24 19:23:54 zipper kernel: nvme0: ABORTED_BY_REQUEST (00/07) crd:0 m:0 = dnr:1 p:0 sqid:8 cid:0 cdw0:0 Oct 24 19:23:54 zipper kernel: nvme0: WRITE (01) sqid:8 cid:0 nsid:1 lba:24= 3296232 len:16 Oct 24 19:23:54 zipper kernel: nvme0: ABORTED_BY_REQUEST (00/07) crd:0 m:0 = dnr:1 p:0 sqid:8 cid:0 cdw0:0 Oct 24 19:23:54 zipper kernel: swap_pager: I/O error - pagein failed; blkno= 147279,size 36864, error 5 Oct 24 19:23:54 zipper kernel: vm_fault: pager read error, pid 1916 (devd) Oct 24 19:23:54 zipper kernel: nda0 at nvme0 bus 0 scbus9 target 0 lun 1 Oct 24 19:23:54 zipper kernel: nda0: <INTEL SSDPEKNW512G8 002C BTNH940617WE= 512A> s/n BTNH940617WE512A detached Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): WRITE (01). NCB: opc=3D1= fuse=3D0 nsid=3D1 prp1=3D0 prp2=3D0 cdw=3De806768 0 f 0 0 0 Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): CAM status: NVME Status = Error Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): NVMe status: ABORTED_BY_= REQUEST (00/07) DNR Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): Error 6, Periph was inva= lidated Oct 24 19:23:54 zipper kernel: swap_pager: I/O error - pagein failed; blkno= 1286155,size 8192, error 6 Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): WRITE (01). NCB: opc=3D1= fuse=3D0 nsid=3D1 prp1=3D0 prp2=3D0 cdw=3De806778 0 f 0 0 0 Oct 24 19:23:54 zipper kernel: vm_fault: pager read error, pid 1916 (devd) Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): CAM status: NVME Status = Error Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): NVMe status: ABORTED_BY_= REQUEST (00/07) DNR Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): Error 6, Periph was inva= lidated Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): WRITE (01). NCB: opc=3D1= fuse=3D0 nsid=3D1 prp1=3D0 prp2=3D0 cdw=3De806788 0 f 0 0 0 Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): CAM status: NVME Status = Error Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): NVMe status: ABORTED_BY_= REQUEST (00/07) DNR Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): Error 6, Periph was inva= lidated Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): WRITE (01). NCB: opc=3D1= fuse=3D0 nsid=3D1 prp1=3D0 prp2=3D0 cdw=3De806798 0 f 0 0 0 Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): CAM status: NVME Status = Error Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): NVMe status: ABORTED_BY_= REQUEST (00/07) DNR Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): Error 6, Periph was inva= lidated Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): WRITE (01). NCB: opc=3D1= fuse=3D0 nsid=3D1 prp1=3D0 prp2=3D0 cdw=3De8067a8 0 7 0 0 0 Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): CAM status: NVME Status = Error Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): NVMe status: ABORTED_BY_= REQUEST (00/07) DNR Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): Error 6, Periph was inva= lidated Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): WRITE (01). NCB: opc=3D1= fuse=3D0 nsid=3D1 prp1=3D0 prp2=3D0 cdw=3De8067b0 0 f 0 0 0 Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): CAM status: NVME Status = Error Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): NVMe status: ABORTED_BY_= REQUEST (00/07) DNR Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): Error 6, Periph was inva= lidated Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): WRITE (01). NCB: opc=3D1= fuse=3D0 nsid=3D1 prp1=3D0 prp2=3D0 cdw=3De8067c0 0 f 0 0 0 Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): CAM status: NVME Status = Error Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): NVMe status: ABORTED_BY_= REQUEST (00/07) DNR Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): Error 6, Periph was inva= lidated Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): WRITE (01). NCB: opc=3D1= fuse=3D0 nsid=3D1 prp1=3D0 prp2=3D0 cdw=3De8067d0 0 f 0 0 0 Oct 24 19:23:54 zipper kernel: (nda0:nvme0:0:0:1): CAM status: NVME Status = Error The machine never recovered until I did a reset. FreeBSD zipper.catspoiler.org 16.0-CURRENT FreeBSD 16.0-CURRENT #52 main-n2= 90318-a3d1d4e88ae7: Sat Oct 24 10:54:51 PDT 2026 dl@zipper.catspoiler.o= rg:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64 nda0 at nvme0 bus 0 scbus9 target 0 lun 1 nda0: <INTEL SSDPEKNW512G8 002C BTNH940617WE512A> nda0: Serial Number BTNH940617WE512A nda0: nvme version 1.3 nda0: 488386MB (1000215216 512 byte sectors) # nvmecontrol logpage -p 0x1 nvme0 Error Information Log =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D No error entries found # nvmecontrol logpage -p 0x2 nvme0 SMART/Health Information Log =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D Critical Warning State: 0x00 Available spare: 0 Temperature: 0 Device reliability: 0 Read only: 0 Volatile memory backup: 0 Temperature: 306 K, 32.85 C, 91.13 F Available spare: 100 Available spare threshold: 10 Percentage used: 11 Data units (512,000 byte) read: 45037975 Data units written: 80040521 Host read commands: 2359001540 Host write commands: 1489208174 Controller busy time (minutes): 92832 Power cycles: 131 Power on hours: 42510 Unsafe shutdowns: 16 Media errors: 0 No. error info log entries: 0 Warning Temp Composite Time: 0 Error Temp Composite Time: 0 Temperature 1 Transition Count: 5231 Temperature 2 Transition Count: 0 Total Time For Temperature 1: 41213 Total Time For Temperature 2: 0 The Temperature 1 Count and Time are identical to the values from 2024 (see below), now that the drive has a finned heatsink. In both cases, the failures have occurred at low ambient temperatures. Yesterday's failure happened with the room at about 66 F. I had no problems under the same load in July with the room at about 85 F. On 13 Feb, Don Lewis wrote: > On 12 Feb, Warner Losh wrote: >> On Mon, Feb 12, 2024 at 9:15=E2=80=AFPM Don Lewis <truckman@freebsd.org>= wrote: >>=20 >>> On 12 Feb, Maxim Sobolev wrote: >>> > Might be an overheating. Today's nvme drives are notoriously flaky if= you >>> > run them without proper heat sink attached to it. >>> >>> I don't think it is a thermal problem. According to the drive health >>> page, the device temperature has never reached Temperature 2, whatever >>> that is. The room temperature is around 65F. The system was stable >>> last summer when the room temperature spent a lot of time in the 80-85F >>> range. The device temperature depends a lot on the I/O rate, and the >>> last panic happened when the I/O rate had been below 40tps for quite a >>> while. >>> >>=20 >> It did reach temperature 1, though. That's the 'Warning this drive is to= o >> hot' temperature. It has spent 41213 minutes of your 19297 hours of up >> time, or an average of 2 minutes per hour. That's too much. Temperature >> 2 is critical error: we are about to shut down completely due to it >> being too hot. It's only a couple degrees below hardware power off >> due to temperature in many drives. Some really cheap ones don't really >> implement it at all. On my card with the bad heat sink, Warning temp is >> 70C while critical is 75C while IIRC thermal shutdown is 78C or 80C. >>=20 >> I don't think we report these values in nvmecontrol identify. But you ca= n >> do a raw dump with -x look at bytes 266:267 for warning and 268:269 >> for critical. >>=20 >> In contrast, the few dozen drives that I have, all of which have been >> abused in various ways, And only one of them has any heat issues, >> and that one is an engineering special / sample with what I think is >> a damaged heat sink. If your card has no heat sink, this could well >> be what's going on. >>=20 >> This panic means "the nvme card lost its mind and stopped talking >> to the host". Its status registers read 0xff's, which means that the car= d >> isn't decoding bus signals. Usually this means that the firmware on the >> card has faulted and rebooted. If the card is overheating, then this cou= ld >> well be what's happening. >>=20 >> There's a tiny chance that this could be something more exotic, >> but my money is on hardware gone bad after 2 years of service. I don't t= hink >> this is 'wear out' of the NAND (it's only 15TB written, but it could be = if >> this >> drive is really really crappy nand: first generation QLC maybe, but it s= eems >> too new). It might also be a connector problem that's developed over tim= e. >> There might be a few other things too, but I don't think this is a U.2 d= rive >> with funky cables. >=20 > The system was probably idle the majority of those two years of power on > time. >=20 > It's one of these: > https://www.techpowerup.com/ssd-specs/intel-660p-512-gb.d437 > I've seen comments that these generally don't need cooling. >=20 > I just ordered a heatsink with some nice big fins, but it will take a > week or more to arrive. >=20 >>=20 >>> > On Mon, Feb 12, 2024, 4:28=E2=80=AFPM Don Lewis <truckman@freebsd.org= > wrote: >>> > >>> >> I just upgraded my package build machine to: >>> >> FreeBSD 15.0-CURRENT #110 main-n268161-4015c064200e >>> >> from: >>> >> FreeBSD 15.0-CURRENT #106 main-n265953-a5ed6a815e38 >>> >> and I've had two nvme-triggered panics in the last day. >>> >> >>> >> nvme is being used for swap and L2ARC. I'm not able to get a crash >>> >> dump, probably because the nvme device has gone away and I get an er= ror >>> >> about not having a dump device. It looks like a low-memory panic >>> >> because free memory is low and zfs is calling malloc(). >>> >> >>> >> This shows up in the log leading up to the panic: >>> >> Feb 12 10:07:41 zipper kernel: nvme0: Resetting controller due to a >>> >> timeout a >>> >> nd possible hot unplug. >>> >> Feb 12 10:07:41 zipper syslogd: last message repeated 1 times >>> >> Feb 12 10:07:41 zipper kernel: nvme0: resetting controller >>> >> Feb 12 10:07:41 zipper kernel: nvme0: Resetting controller due to a >>> >> timeout a >>> >> nd possible hot unplug. >>> >> Feb 12 10:07:41 zipper syslogd: last message repeated 1 times >>> >> Feb 12 10:07:41 zipper kernel: nvme0: Waiting for reset to complete >>> >> Feb 12 10:07:41 zipper syslogd: last message repeated 2 times >>> >> Feb 12 10:07:41 zipper kernel: nvme0: failing queued i/o >>> >> Feb 12 10:07:41 zipper kernel: nvme0: Failed controller, stopping >>> watchdog >>> >> ti >>> >> meout. >>> >> >>> >> The device looks healthy to me: >>> >> SMART/Health Information Log >>> >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D >>> >> Critical Warning State: 0x00 >>> >> Available spare: 0 >>> >> Temperature: 0 >>> >> Device reliability: 0 >>> >> Read only: 0 >>> >> Volatile memory backup: 0 >>> >> Temperature: 312 K, 38.85 C, 101.93 F >>> >> Available spare: 100 >>> >> Available spare threshold: 10 >>> >> Percentage used: 3 >>> >> Data units (512,000 byte) read: 5761183 >>> >> Data units written: 29911502 >>> >> Host read commands: 471921188 >>> >> Host write commands: 605394753 >>> >> Controller busy time (minutes): 32359 >>> >> Power cycles: 110 >>> >> Power on hours: 19297 >>> >> Unsafe shutdowns: 14 >>> >> Media errors: 0 >>> >> No. error info log entries: 0 >>> >> Warning Temp Composite Time: 0 >>> >> Error Temp Composite Time: 0 >>> >> Temperature 1 Transition Count: 5231 >>> >> Temperature 2 Transition Count: 0 >>> >> Total Time For Temperature 1: 41213 >>> >> Total Time For Temperature 2: 0 >>> >> >>> >> >>> >> >>> >>> >>> >=20 >=20