Re: ZFS - reboot during resilver doesn't work
Date: Mon, 08 Sep 2025 18:26:51 UTC
On 08/09/2025 05:08, David Christensen wrote: > On 9/7/25 17:00, Frank Leonhardt wrote: >> ... I did suspect SMR had something to do with it, which I why I >> threw that in. >> >> ... I'm sitting at the data centre watching it now (it's well past >> midnight). I've shut everything else down apart from sshd to give it >> the best chance. >> >> <snip> >> >> pool: zr >> state: DEGRADED >> status: One or more devices is currently being resilvered. The pool >> will continue to function, possibly in a degraded state. >> action: Wait for the resilver to complete. >> scan: resilver in progress since Sun Sep 7 21:30:58 2025 322G / 922G >> scanned at 196B/s, 206G / 922G issued at 125B/s 206G resilvered, >> 22.32% done, no estimated completion time config: NAME STATE READ >> WRITE CKSUM >> zr DEGRADED 0 0 0 mirror-0 DEGRADED 0 0 0 replacing-0 DEGRADED 3 0 0 >> ada0p3/old UNAVAIL 0 0 0 >> cannot open ada0p3 ONLINE 0 0 5 (resilvering) >> ada1p3 ONLINE 0 0 0 >> This after a hard reset two hours ago - before this it was estimating >> five hours. Oh, hang on, six hours. No, wait, seven hours... >> >> The SMR drive is now at 37C (the other is at 30C). I'm thinking of >> pulling it and putting a spare six-year-old Constellation in its >> place, which I brought along just-in-case. Regards, Frank. > > > The few times I have replaced and resilvered a HDD, the process took > about 4 hours per TB of data. The disks were 3 TB SATA Seagate > Barracuda or Constellation ES.2. > > > If you are wondering if the existing drive and/or the replacement > drive are having issues, perhaps running `smartctl -x ...` could > confirm or refute. And, look at /var/log/messages for errors. > > > David Yes, that would be about right on the speed. For everyone's amusement, it re-silvered 900Gb of data in about nine hours, which could be worse. I'm convinced, as Doron concurred, it was down to the replacement being SMR. These days I test new drives thoroughly before heading to a data centre with one, and interestingly it was pretty good on sequential read/write - 250MB/s sustained. No flaws. Re-silvering it managed about 20Mb/s according to iostat. According to zfs status it was doing about 150B/s (that's bytes). There are clearly a few bugs here - both in what zfs status reports and in shutting down the re-silver to reboot. I'm investigating offline. SMR drives are bad news. Incidentally, if you do a zpool replace with a -s sequential option it should go a lot quicker as it's normally doing the equivalent of a scrub, rebuilding the mirror with a load of random writes. This is an enhancement from OpenZFS and the reason for it is now very apparent. I shall be writing it up on my blog as usual. Now, do I dare reboot it? Regards, Frank. Here's the progress log, just for fun. Note the crazy issue speed until 05:47:02 - after that it matched what iostat said it was doing. 00:31:59 261G / 922G scanned at 159B/s, 160G / 922G issued at 97B/s 160G resilvered, 17.35% done, no estimated completion time 00:36:59 267G / 922G scanned at 163B/s, 176G / 922G issued at 107B/s 176G resilvered, 19.08% done, no estimated completion time 00:41:59 322G / 922G scanned at 196B/s, 185G / 922G issued at 112B/s 185G resilvered, 20.04% done, no estimated completion time 00:47:00 322G / 922G scanned at 196B/s, 194G / 922G issued at 118B/s 194G resilvered, 21.03% done, no estimated completion time 00:52:00 322G / 922G scanned at 196B/s, 215G / 922G issued at 131B/s 216G resilvered, 23.31% done, no estimated completion time 00:57:00 322G / 922G scanned at 196B/s, 225G / 922G issued at 137B/s 226G resilvered, 24.39% done, no estimated completion time 01:02:00 347G / 922G scanned at 212B/s, 239G / 922G issued at 146B/s 240G resilvered, 25.91% done, no estimated completion time 01:07:00 347G / 922G scanned at 212B/s, 251G / 922G issued at 153B/s 251G resilvered, 27.18% done, no estimated completion time 01:12:00 347G / 922G scanned at 212B/s, 268G / 922G issued at 163B/s 269G resilvered, 29.07% done, no estimated completion time 01:17:00 347G / 922G scanned at 212B/s, 281G / 922G issued at 171B/s 281G resilvered, 30.41% done, no estimated completion time 01:22:00 347G / 922G scanned at 212B/s, 293G / 922G issued at 179B/s 294G resilvered, 31.78% done, no estimated completion time 01:27:00 347G / 922G scanned at 212B/s, 302G / 922G issued at 184B/s 303G resilvered, 32.75% done, no estimated completion time 01:32:00 347G / 922G scanned at 212B/s, 311G / 922G issued at 189B/s 312G resilvered, 33.70% done, no estimated completion time 01:37:00 347G / 922G scanned at 212B/s, 322G / 922G issued at 196B/s 323G resilvered, 34.90% done, no estimated completion time 01:42:00 347G / 922G scanned at 212B/s, 328G / 922G issued at 200B/s 329G resilvered, 35.56% done, no estimated completion time 01:47:00 347G / 922G scanned at 212B/s, 334G / 922G issued at 204B/s 335G resilvered, 36.21% done, no estimated completion time 01:52:00 347G / 922G scanned at 212B/s, 344G / 922G issued at 210B/s 346G resilvered, 37.31% done, no estimated completion time 01:57:00 450G / 922G scanned at 274B/s, 347G / 922G issued at 212B/s 349G resilvered, 37.63% done, no estimated completion time 02:02:00 626G / 922G scanned at 382B/s, 364G / 922G issued at 222B/s 366G resilvered, 39.45% done, no estimated completion time 02:07:00 626G / 922G scanned at 382B/s, 383G / 922G issued at 234B/s 385G resilvered, 41.52% done, no estimated completion time 02:12:00 626G / 922G scanned at 382B/s, 404G / 922G issued at 247B/s 406G resilvered, 43.83% done, no estimated completion time 02:17:00 626G / 922G scanned at 382B/s, 417G / 922G issued at 254B/s 418G resilvered, 45.16% done, no estimated completion time 02:22:00 626G / 922G scanned at 382B/s, 432G / 922G issued at 264B/s 434G resilvered, 46.85% done, no estimated completion time 02:27:00 626G / 922G scanned at 382B/s, 443G / 922G issued at 270B/s 445G resilvered, 48.07% done, no estimated completion time 02:32:00 626G / 922G scanned at 382B/s, 456G / 922G issued at 278B/s 458G resilvered, 49.40% done, no estimated completion time 02:37:00 626G / 922G scanned at 382B/s, 461G / 922G issued at 281B/s 463G resilvered, 49.94% done, no estimated completion time 02:42:01 626G / 922G scanned at 382B/s, 471G / 922G issued at 287B/s 473G resilvered, 51.02% done, no estimated completion time 02:47:01 626G / 922G scanned at 382B/s, 473G / 922G issued at 289B/s 475G resilvered, 51.29% done, no estimated completion time 02:52:01 626G / 922G scanned at 382B/s, 481G / 922G issued at 293B/s 482G resilvered, 52.09% done, no estimated completion time 02:57:01 626G / 922G scanned at 382B/s, 493G / 922G issued at 301B/s 495G resilvered, 53.45% done, no estimated completion time 03:02:01 721G / 922G scanned at 440B/s, 506G / 922G issued at 308B/s 508G resilvered, 54.82% done, no estimated completion time 03:07:01 721G / 922G scanned at 440B/s, 515G / 922G issued at 314B/s 517G resilvered, 55.81% done, no estimated completion time 03:12:01 721G / 922G scanned at 440B/s, 528G / 922G issued at 322B/s 530G resilvered, 57.20% done, no estimated completion time 03:17:01 721G / 922G scanned at 440B/s, 534G / 922G issued at 326B/s 536G resilvered, 57.88% done, no estimated completion time 03:22:01 721G / 922G scanned at 440B/s, 538G / 922G issued at 328B/s 540G resilvered, 58.31% done, no estimated completion time 03:27:01 721G / 922G scanned at 440B/s, 542G / 922G issued at 330B/s 544G resilvered, 58.72% done, no estimated completion time 03:32:01 721G / 922G scanned at 440B/s, 561G / 922G issued at 342B/s 563G resilvered, 60.77% done, no estimated completion time 03:37:01 721G / 922G scanned at 440B/s, 567G / 922G issued at 346B/s 569G resilvered, 61.49% done, no estimated completion time 03:42:01 721G / 922G scanned at 440B/s, 584G / 922G issued at 356B/s 586G resilvered, 63.26% done, no estimated completion time 03:47:01 724G / 922G scanned at 442B/s, 597G / 922G issued at 365B/s 599G resilvered, 64.76% done, no estimated completion time 03:52:01 739G / 922G scanned at 451B/s, 616G / 922G issued at 376B/s 618G resilvered, 66.81% done, no estimated completion time 03:57:01 739G / 922G scanned at 451B/s, 618G / 922G issued at 377B/s 620G resilvered, 66.99% done, no estimated completion time 04:02:01 739G / 922G scanned at 451B/s, 631G / 922G issued at 385B/s 633G resilvered, 68.37% done, no estimated completion time 04:07:02 739G / 922G scanned at 451B/s, 643G / 922G issued at 392B/s 645G resilvered, 69.69% done, no estimated completion time 04:12:02 739G / 922G scanned at 451B/s, 653G / 922G issued at 399B/s 656G resilvered, 70.84% done, no estimated completion time 04:17:02 739G / 922G scanned at 451B/s, 655G / 922G issued at 400B/s 657G resilvered, 71.04% done, no estimated completion time 04:22:02 739G / 922G scanned at 451B/s, 658G / 922G issued at 401B/s 660G resilvered, 71.31% done, no estimated completion time 04:27:02 739G / 922G scanned at 451B/s, 659G / 922G issued at 402B/s 661G resilvered, 71.43% done, no estimated completion time 04:32:02 739G / 922G scanned at 451B/s, 662G / 922G issued at 404B/s 664G resilvered, 71.76% done, no estimated completion time 04:37:02 739G / 922G scanned at 451B/s, 662G / 922G issued at 404B/s 664G resilvered, 71.78% done, no estimated completion time 04:42:02 739G / 922G scanned at 451B/s, 663G / 922G issued at 404B/s 665G resilvered, 71.83% done, no estimated completion time 04:47:02 739G / 922G scanned at 451B/s, 677G / 922G issued at 413B/s 679G resilvered, 73.43% done, no estimated completion time 04:52:02 739G / 922G scanned at 451B/s, 693G / 922G issued at 423B/s 695G resilvered, 75.09% done, no estimated completion time 04:57:02 739G / 922G scanned at 451B/s, 704G / 922G issued at 430B/s 706G resilvered, 76.30% done, no estimated completion time 05:02:02 739G / 922G scanned at 451B/s, 712G / 922G issued at 435B/s 715G resilvered, 77.21% done, no estimated completion time 05:07:02 739G / 922G scanned at 451B/s, 721G / 922G issued at 440B/s 723G resilvered, 78.14% done, no estimated completion time 05:12:02 739G / 922G scanned at 451B/s, 727G / 922G issued at 443B/s 729G resilvered, 78.77% done, no estimated completion time 05:17:02 739G / 922G scanned at 451B/s, 735G / 922G issued at 449B/s 737G resilvered, 79.67% done, no estimated completion time 05:22:02 877G / 922G scanned at 535B/s, 745G / 922G issued at 455B/s 748G resilvered, 80.80% done, no estimated completion time 05:27:02 877G / 922G scanned at 535B/s, 766G / 922G issued at 467B/s 768G resilvered, 83.00% done, no estimated completion time 05:32:02 904G / 922G scanned at 552B/s, 778G / 922G issued at 475B/s 780G resilvered, 84.32% done, no estimated completion time 05:37:02 904G / 922G scanned at 552B/s, 786G / 922G issued at 480B/s 789G resilvered, 85.26% done, no estimated completion time 05:42:02 904G / 922G scanned at 552B/s, 793G / 922G issued at 484B/s 795G resilvered, 85.93% done, no estimated completion time 05:47:02 922G / 922G scanned, 799G / 922G issued at 20.6M/s 802G resilvered, 86.63% done, 01:42:23 to go 05:52:02 922G / 922G scanned, 805G / 922G issued at 20.1M/s 808G resilvered, 87.26% done, 01:40:08 to go 05:57:03 922G / 922G scanned, 809G / 922G issued at 18.4M/s 812G resilvered, 87.74% done, 01:44:36 to go 06:02:03 922G / 922G scanned, 817G / 922G issued at 20.7M/s 820G resilvered, 88.61% done, 01:26:46 to go 06:07:03 922G / 922G scanned, 820G / 922G issued at 18.2M/s 823G resilvered, 88.88% done, 01:36:09 to go 06:12:03 922G / 922G scanned, 823G / 922G issued at 16.8M/s 826G resilvered, 89.20% done, 01:41:04 to go 06:17:03 922G / 922G scanned, 831G / 922G issued at 18.4M/s 834G resilvered, 90.08% done, 01:24:43 to go 06:22:03 922G / 922G scanned, 834G / 922G issued at 17.4M/s 837G resilvered, 90.42% done, 01:26:37 to go 06:27:03 922G / 922G scanned, 837G / 922G issued at 16.7M/s 840G resilvered, 90.77% done, 01:26:48 to go 06:32:03 922G / 922G scanned, 839G / 922G issued at 15.7M/s 842G resilvered, 90.99% done, 01:30:04 to go 06:37:03 922G / 922G scanned, 842G / 922G issued at 15.1M/s 845G resilvered, 91.26% done, 01:31:15 to go 06:42:03 922G / 922G scanned, 852G / 922G issued at 16.6M/s 854G resilvered, 92.31% done, 01:12:59 to go 06:47:03 922G / 922G scanned, 858G / 922G issued at 17.0M/s 861G resilvered, 93.03% done, 01:04:27 to go 06:52:03 922G / 922G scanned, 861G / 922G issued at 16.4M/s 863G resilvered, 93.29% done, 01:04:21 to go 06:57:03 922G / 922G scanned, 868G / 922G issued at 17.0M/s 871G resilvered, 94.11% done, 00:54:25 to go 07:02:03 922G / 922G scanned, 872G / 922G issued at 16.8M/s 875G resilvered, 94.52% done, 00:51:20 to go 07:07:03 922G / 922G scanned, 881G / 922G issued at 17.6M/s 884G resilvered, 95.48% done, 00:40:26 to go 07:12:03 922G / 922G scanned, 892G / 922G issued at 18.7M/s 895G resilvered, 96.69% done, 00:27:49 to go 07:17:03 922G / 922G scanned, 906G / 922G issued at 20.2M/s 909G resilvered, 98.17% done, 00:14:15 to go 07:22:03 922G / 922G scanned, 914G / 922G issued at 20.6M/s 917G resilvered, 99.06% done, 00:07:09 to go 07:27:04 922G / 922G scanned, 922G / 922G issued at 21.0M/s 926G resilvered, 100.00% done, 00:00:00 to go