From nobody Thu Aug 21 00:55:57 2025 X-Original-To: questions@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4c6lHq63Gzz65TXV for ; Thu, 21 Aug 2025 00:56:07 +0000 (UTC) (envelope-from robert@webtent.org) Received: from www1.webtent.net (www1.webtent.net [208.38.145.35]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4c6lHp1v39z43cH for ; Thu, 21 Aug 2025 00:56:06 +0000 (UTC) (envelope-from robert@webtent.org) Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=pass (policy=reject) header.from=webtent.org; spf=pass (mx1.freebsd.org: domain of robert@webtent.org designates 208.38.145.35 as permitted sender) smtp.mailfrom=robert@webtent.org Received: from [192.168.192.50] (static-47-206-68-190.tamp.fl.frontiernet.net [47.206.68.190]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) (Authenticated sender: robert) by www1.webtent.net (Postfix) with ESMTPSA id 50B2A885886 for ; Wed, 20 Aug 2025 20:55:59 -0400 (EDT) DKIM-Filter: OpenDKIM Filter v2.11.0 www1.webtent.net 50B2A885886 X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 1.2.0 at tbwh.webtent.net Message-ID: Date: Wed, 20 Aug 2025 20:55:57 -0400 List-Id: User questions List-Archive: https://lists.freebsd.org/archives/freebsd-questions List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-questions@freebsd.org Sender: owner-freebsd-questions@FreeBSD.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Content-Language: en-US To: questions@freebsd.org From: Robert Subject: Replacing a REMOVED drive in DEGRADED zpool Autocrypt: addr=robert@webtent.com; keydata= xsFNBGaSjMABEACSkWgxP4zpsGUyogci2IWFvhrzP4Gjh4DA0Aq6RDE3rrfHh9hvqpu3pPjK AqEcH4+k4iaumo28lro3r+nIGSafrE/oCHOxn3Y+Ed8JKknKA8m0OHmiYG4QEN5cBEgO6JXh DMpQDxNobj+FnReb3VugbP8FM4buwb528zutAc/ZjNCvRfSATxq7w9WqYevTzjZR3YOliMUi yQaHrT4lwY1TDDSihAMTK4ec2hRV0qrisrBSt0uqokXHopj5Dmb2LbR7rYyLc8k41mk+OHRY 6btAbSmycdiNFRX1EUQGe6lIByNVxfF7GKDOtASOq6P24lJ6zXN2aSpABKe0IYT3K4AjRntD X21NssbWJozHr6gy4nxKVNupbtGOf+kyQ3830w9Ocv7wyggIx5kUVYxEGtomkrVyvmYXy7zF uIDRe/WixHPzIzAWS043jU7tQxNB+EfwFYYaSB/Bknvh95syWm06JKYYI71qva3hp4izzQeO /XpDxITJIozCQnZFAgHJiD3OkouYnQNbfnTR4UqMdsNQmByolhKo0XiosJD3bRHMwPEr6dg1 Nm0prBbBs5x2yfgEfFgJlZP1rKjit3c1sqV3icba7pTu0Lo9xd/D1cjPhJDjQqbwBqPKj4ss DiCHMMOs/W7F5bV4xkyYbFmHA+GDQVTqxTeQoczDyOvVJT7GTwARAQABzSNSb2JlcnRAV2Vi VGVudCA8cm9iZXJ0QHdlYnRlbnQuY29tPsLBigQQAQgAPgWCZpKMwAQLCQcICZDS94pDmMsy jAMVCAoEFgACAQIZAQKbAwIeARYhBOn5Lg3SYuhw7PdnXdL3ikOYyzKMAADv9g/9HSiJjbHy eXWYL8MexAUQ13CmSuprj3qoSqaFC3xXn0gJuufSDo/jFv17fO0OSnvF1ubVuL11QaM2b14M IZweQraM9CKZbsUQN629UDPmSnKn6UpD9v3iZuwreFNFiBplFOY5E0JgEOTc5fPTwkTcWAvs vKU23OsSX5ioxaVVfHMhWF4BcRsI92pYSAYAGhFawbM7akQ7q2M3i0xyT+c76B1yGVs1W5o3 9PvrzId9xWmO/xOnDCMpQvpVa17PzlY88WFOcCaIEJsDD4X8PFeC5LcnIrY8HRIMsxGiKlpd KlU5nUNflWpL6Z3p0z60PlaidjHrnQRts3/IEo5Bn9rmqrmgnhsFGVaW0QV54EPtlCAH8ej2 h4fd75VE941DRWgo1BU4QqC75CwUrGgwmPOGcxUlIopKZscKOwxjQj38ap2jMLUidvueSsr8 6SC2Y2nlqJB1MnEuCkcZjp7grbmHb6QLdagn3H8Keq1f3sZoCWHOiYtMBPKIR2dnAkrbayvB eG63G3jj/lpRphCRWtVQQxPejZILKdcmjEwm8MT4NNGR5ahvMCa4X2YIRZ7t04zKOpGgePa4 ADbw2QnOpR0ebExLc2tlRlkGXcTz0Mah3cDa2OjKUuhOjUUvHmxUgNF4OJa+cE65E9y8i38K VI3+eeW7TrQUDtMxuoaYo0h/qDfOwU0EZpKMwAEQAMHK4RWvTexZweo1w27FA+BJ7TsKA75i 9z8HeJPsorV6e7nzTvo3JQ4RONOtba7X3qTOzDq/RBKbSsNde6PP0zjv/tN2sRh7j4pGaB57 Hp8wUpnTbQkGxBV7LCo6pKlyr/i5rfW6ug32ayZ8/ADy0Rj1D9l6Q0TIQ3jnpFWfuktyM1R7 WwS1cCJcvixyEhR6gbJ7kENQ5lsb9m7ySHEi8lIXsRWU0JCcjccqwV6SKV+XViBwPy3LumIp CxjrbBQ2LGuwGSnz/Gb7nFbOz3pJw5R+1Ai0dZkkSe3FjQWknGpaXu7qMWSh5uJ5DdqEzWZK SyWRs/OJotmEjSSDMWjot7KO4FuUJX2YxkOrEs6RNQ5wUAOvdBDpfATqKlCV4cmKI9Lx72/m CmSIYBPqtXYMNIjdMHe0SCWlfMPKJJuQ8kMFtNmIS12drLR361YmMc6BQ5e9YsKZjUpgLS6D 1Gz94WDKfHOg1OpRjnYeCZcdJkW43YijnI8A1PoFu8PWqAN09O/mdn+j52Ffxv4YyT7+w83E GiGl4qJHdKvEe9waYNQngvCAxdBHoHoWVI2XN9yIQyXhhs4IKxwvvx8c3QOMgNdu0VXioTfy tDQqt2WwRN2vMP0LkzdYnjI+gMbj5wFrey39fPOHlIYJQHQ7QU8ekvNimlVyoosroIR2NBXh 3FlvABEBAAHCwXYEGAEIACoFgmaSjMAJkNL3ikOYyzKMApsMFiEE6fkuDdJi6HDs92dd0veK Q5jLMowAADf0D/42qQqHYHlrwRQ6JtjvY7Tqc83ePbKByhfxgUJ+X/tsAPHZlaGNnNn8epfP fYTHn0o3uYrBiE7emf8sbdCb53Ym1n5rl0tfPgly8WD/GK4SICIYo2YUW3h8y08OvpISkmsz FTe460NWlSbEfocmGis4nawIEl51xzrHSmvyraJ6ywjT1gdiCM+1P9g7y/4iaJn0qs/wjlHU iVskrTe1w0Jd3LsYhypUHMc5LPKc92fXo/ZXw/G2fHtYTzawGLgfcdx/297DGdMI/95dTMJq Z0O0tUqj3SEa5dak7vqcHqZG+JPilT3GBXM42EU3o1P/tE7u5SCuWGNJkaN0pFcShTSS5TtR cFZZH3FrWSoIPPkkUHPmFaVybY5Q32rLr+EusocMCHkiz50V+P1Yzjl2PNkDE7IMDtqvW9Sf w78R/cXXhcNSaewu5UQKEs0WajJjQSEDYyztuqh5bStDJMoFobvaeoZYs536meRBjhv/+z73 l2IGnWJdV5v3+GMHSyctM6LhvFvBNdMpchF7ft5b4WBuFYdsQutpLRNfajoTBAsCYinlkg+O iHI3HQ5j6HgRpOAMhEv1e85R8TJdJStwX9/c24kZh/tW/HMjQXWdVdFz1taqNdRkwfFm1orG iBMeuAvUBI5MfR4oQDER+HtRi7bb8/Aklm25wHOTqfAV5NzZBA== Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.6.2 (www1.webtent.net [0.0.0.0]); Wed, 20 Aug 2025 20:55:59 -0400 (EDT) X-Spamd-Bar: -- X-Spamd-Result: default: False [-2.83 / 15.00]; NEURAL_HAM_MEDIUM(-0.99)[-0.992]; NEURAL_HAM_LONG(-0.95)[-0.950]; DMARC_POLICY_ALLOW(-0.50)[webtent.org,reject]; NEURAL_HAM_SHORT(-0.29)[-0.291]; ONCE_RECEIVED(0.20)[]; R_SPF_ALLOW(-0.20)[+ip4:208.38.145.0/26]; MIME_GOOD(-0.10)[text/plain]; MIME_TRACE(0.00)[0:+]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_COUNT_ONE(0.00)[1]; RCPT_COUNT_ONE(0.00)[1]; ASN(0.00)[asn:16724, ipnet:208.38.144.0/22, country:US]; FREEFALL_USER(0.00)[robert]; MID_RHS_MATCH_FROM(0.00)[]; R_DKIM_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; TO_DN_NONE(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[questions@freebsd.org]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MLMMJ_DEST(0.00)[questions@freebsd.org]; RCVD_TLS_ALL(0.00)[] X-Rspamd-Queue-Id: 4c6lHp1v39z43cH I have my first zpool degraded on a FreeBSD 13.5 server and looking for advice on the steps I'll be taking to successfully replace the REMOVED drive in a 4 disk 2 mirror zpool. It is scrubbed monthly with last scrub August 3rd... root@db1:~ # zpool list NAME   SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP DEDUP    HEALTH  ALTROOT zdb1   262G   102G   160G        -         -    53%    38% 1.00x  DEGRADED  - root@db1:~ # zpool status   pool: zdb1  state: DEGRADED status: One or more devices has been removed by the administrator.         Sufficient replicas exist for the pool to continue functioning in a         degraded state. action: Online the device using zpool online' or replace the device with         'zpool replace'.   scan: scrub repaired 0B in 00:27:57 with 0 errors on Sun Aug 3 04:43:48 2025 config:         NAME        STATE     READ WRITE CKSUM         zdb1        DEGRADED     0     0     0           mirror-0  DEGRADED     0     0     0             ada0p3  REMOVED      0     0     0             ada1p3  ONLINE       0     0     0           mirror-1  ONLINE       0     0     0             ada2p3  ONLINE       0     0     0             ada3p3  ONLINE       0     0     0 I have data backup of any important data. I also use zfs-autobackup form an old remote FreeNAS server to take hourly snapshots ... zfs-autobackup -v --keep-source 72 --keep-target 168 --ssh-source db1 offsite1 DATA/backups/db1 So, I have the last 72 hours of snapshots on the local server with 7 days worth on the remote NAS. The disk entered the REMOVED state at 6am this morning, a little over 14 hours ago and I plan to replace on Friday night to give myself some time in case a restore needs to happen. Perhaps I should bump the local snapshot storage up to 168 hours (1 week) as well at this point or hold what is there, can I hold all snapshots with one command? Here is the disk info for the 3 drives remaining in the zpool ... root@db1:~ # camcontrol devlist    at scbus1 target 0 lun 0 (ada1,pass1)    at scbus2 target 0 lun 0 (ada2,pass2)    at scbus3 target 0 lun 0 (ada3,pass3) root@db1:~ # gpart show ada1 =>       40  293046688  ada1  GPT  (140G)          40       1024     1  freebsd-boot  (512K)        1064        984        - free -  (492K)        2048   16777216     2  freebsd-swap  (8.0G)    16779264  276267008     3  freebsd-zfs  (132G)   293046272        456        - free -  (228K) All the drive report identical layouts as ada1. I've used camcontrol with identify to get all the serial numbers of these drives, so I plan to shut the server down, pull the bad drive and insert the replacement, boot up and replace. Would these be the steps I need to take assuming the replacement drive shows up as the same ada0 device? 1. Run `zpool offline zdb1 ada0p3` 2. Shut down and pull/insert replacement 3. Boot up and run `gpart backup ada1 > gpart.ada1` then `gpart restore ada0 < gpart.ada1` 4. Run `zpool replace zdb1 ada0p3 ada0p3` I'm just not sure if this is all that is needed with a ROOT zpool or if all correct. I appreciate any guidance. Here is the full zfs list... root@db1:~ # zfs list NAME                                          USED  AVAIL  REFER  MOUNTPOINT zdb1                                          103G   152G  96K  /zdb1 zdb1/ROOT                                     101G   151G  96K  none zdb1/ROOT/13.1-RELEASE-p7_2023-05-04_200035     8K   151G  15.8G  / zdb1/ROOT/13.2-RELEASE-p1_2023-08-07_124053     8K   151G  25.4G  / zdb1/ROOT/13.2-RELEASE-p2_2023-09-09_111305     8K   151G  29.0G  / zdb1/ROOT/13.2-RELEASE-p3_2023-12-31_111612     8K   151G  33.1G  / zdb1/ROOT/13.2-RELEASE-p9_2024-04-14_121449     8K   151G  34.9G  / zdb1/ROOT/13.2-RELEASE_2023-05-04_200614        8K   151G  15.9G  / zdb1/ROOT/13.2-RELEASE_2023-08-01_151806        8K   151G  25.1G  / zdb1/ROOT/13.3-RELEASE-p1_2024-04-14_121907     8K   151G  34.9G  / zdb1/ROOT/13.3-RELEASE-p1_2024-08-04_122937     8K   151G  36.2G  / zdb1/ROOT/13.3-RELEASE-p4_2025-01-04_162341     8K   151G  36.3G  / zdb1/ROOT/13.3-RELEASE-p8_2025-01-04_164203     8K   151G  36.9G  / zdb1/ROOT/13.4-RELEASE-p1_2025-01-04_164619     8K   151G  37.0G  / zdb1/ROOT/13.4-RELEASE-p2_2025-05-10_133828     8K   151G  39.4G  / zdb1/ROOT/13.5-RELEASE-p1_2025-07-04_113332     8K   151G  39.6G  / zdb1/ROOT/13.5-RELEASE_2025-05-10_134206        8K   151G  39.4G  / zdb1/ROOT/default                             101G   151G  39.5G  / zdb1/tmp                                     1.12M   151G 200K  /tmp zdb1/usr                                     1.17G   151G  96K  /usr zdb1/usr/home                                1.31M   151G  1.30M  /usr/home zdb1/usr/ports                               1.17G   151G  1.17G  /usr/ports zdb1/usr/src                                   96K   151G  96K  /usr/src zdb1/var                                     5.35M   151G  96K  /var zdb1/var/audit                                 96K   151G  96K  /var/audit zdb1/var/crash                                 96K   151G  96K  /var/crash zdb1/var/log                                 4.78M   151G 660K  /var/log zdb1/var/mail                                 200K   151G 144K  /var/mail zdb1/var/tmp                                   96K   151G  96K  /var/tmp Thank you. -- Robert