Re: [External] Re: ZFS - reboot during resilver doesn't work

On 08/09/2025 00:23, Doron Beit-Halahmi wrote:
> On Mon, 8 Sep 2025 00:15:06 +0100
> Frank Leonhardt<freebsd-doc@fjl.co.uk> wrote:
>
>
>> If you have a ZFS mirror you're supposed to be able to reboot while
>> it's resilvering. It's a background job and should just continue
>> where it left off, more or less. I'm pretty sure I've done this in
>> the past, with the system on the pool being resilvered too.
>>
>> But I've noticed it doesn't quite work (14.3-RELEASE) as it won't
>> shut down. The disk activity LED is jammed on. It syncs the buffers,
>> but carries accessing the disks with an unresponsive terminal. The
>> only unusual thing is that at least one of the drives is SMR (don't
>> ask!)
>>
>> If you do shut it down hard (kill the power) it restarts the
>> resilvering, but having taken several steps backwards.
>>
>> Has anyone else noticed this?
>>
>> Why do I care? I don't like to leave a distant data centre without
>> restarting something a few times to make sure I can reboot safely
>> later on.
>>
>> Thanks, Frank.
>>
>>
>>
> Frank,
>
> You’re not imagining things — I’ve hit this before too. In theory a
> resilver is just a background job, and you should be able to reboot
> mid-stream. In practice, two things get in the way:
>
> SMR drives really don’t like resilvers. The random write pattern makes
> their firmware go off into la-la land doing band rewrites, which is
> usually when the disk light gets stuck on and the box won’t shut down
> cleanly.
>
> On 14.x the shutdown path isn’t great about killing off ZFS resilver
> threads, so it just sits there waiting on I/O that the drive may never
> finish.
>
> The “backwards progress” after a hard reset is just ZFS being
> conservative — it widens the DTL and re-walks more of the tree because
> it can’t be sure what really finished. Annoying, but expected.
>
> Easiest workarounds: don’t mix SMR into pools you care about, or if you
> have to, tune down resilver speed (vfs.zfs.resilver_min_time_ms) to
> give the disks a fighting chance. With CMR drives you’ll likely see
> reboots behave as you remember.
>
> If you can reproduce it, might be worth a PR so the FreeBSD/OpenZFS
> folks can look at the shutdown side.
>
> — A fellow who’s also been burned by “cheap” SMRs 🙂
>
Thanks, I was starting to question my sanity here.  I did suspect SMR 
had something to do with it, which I why I threw that in.

As for reproducing it, I'm sitting at the data centre watching it now 
(it's well past midnight). I've shut everything else down apart from 
sshd to give it the best chance.

A certain drive manufacturer added SMR as a feature to a drive without 
changing the model name/number (only the generation). I got suspicious 
when it was considerably thinner and lighter, but I'm stuck with it.

I don't really see how increasing vfs.zfs.resilver_min_time_ms is going 
to help, as I'm keeping the system idle anyway. Or would decreasing it 
help in some way. I guess I'm just going to have to leave it running. 
pool: zr state: DEGRADED status: One or more devices is currently being 
resilvered. The pool will continue to function, possibly in a degraded 
state. action: Wait for the resilver to complete. scan: resilver in 
progress since Sun Sep 7 21:30:58 2025 322G / 922G scanned at 196B/s, 
206G / 922G issued at 125B/s 206G resilvered, 22.32% done, no estimated 
completion time config: NAME STATE READ WRITE CKSUM zr DEGRADED 0 0 0 
mirror-0 DEGRADED 0 0 0 replacing-0 DEGRADED 3 0 0 ada0p3/old UNAVAIL 0 
0 0 cannot open ada0p3 ONLINE 0 0 5 (resilvering) ada1p3 ONLINE 0 0 0 
This after a hard reset two hours ago - before this it was estimating 
five hours. Oh, hang on, six hours. No, wait, seven hours...

The SMR drive is now at 37C (the other is at 30C). I'm thinking of 
pulling it and putting a spare six-year-old Constellation in its place, 
which I brought along just-in-case. Regards, Frank.