From nobody Fri Sep 06 14:24:10 2024 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X0dlg66bgz5VB0D for ; Fri, 06 Sep 2024 14:24:31 +0000 (UTC) (envelope-from cross+freebsd@relay.distal.com) Received: from relay.wiredblade.com (relay.wiredblade.com [168.235.95.80]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4X0dlf4v2Zz4SbN for ; Fri, 6 Sep 2024 14:24:30 +0000 (UTC) (envelope-from cross+freebsd@relay.distal.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=relay.distal.com header.s=mail header.b=kcl31+z2; dmarc=none; spf=pass (mx1.freebsd.org: domain of cross+freebsd@relay.distal.com designates 168.235.95.80 as permitted sender) smtp.mailfrom=cross+freebsd@relay.distal.com dkim-signature: v=1; a=rsa-sha256; d=relay.distal.com; s=mail; c=relaxed/relaxed; q=dns/txt; h=From:Subject:Date:Message-ID:To:CC:MIME-Version:Content-Type:Content-Transfer-Encoding; bh=6Ajf7bf+sKqz2h4nzIEQibpUSL8PArGNsXdYgjPfa8w=; b=kcl31+z2d1H+vclyQ+zCikQeI2C4O8Vgrz/OYq1TW2K6kfPPaHLT5o1jR9e0iEchJYR5RbgnoVBZSaAd004+N2tM5/KchN7/5tmi0/V2VbUe3d/95yYYkBVbzeN4hm/M7tZ+PF9Dhm8f+xHcy73PL6DRXRKzPVZpZBx6OGc5D4hyYD15z5dT95oJEoCL6W+qT6vSomovSaPdqNd2DpQSxZODfECiEJtfqFgCn/tS1A/ULNOerxiiEinoF4 VxqAbN7hsOCozq9UUDYH+r1OTn5J129eG1cEfVT+p3m9hqXWlWBSHt1cuyGYtpB2mEMLW3qS8JC72CmJoi37PjkTJfJQ== Received: from mail.distal.com (pool-108-51-233-124.washdc.fios.verizon.net [108.51.233.124]) by relay.wiredblade.com with ESMTPSA (version=TLSv1.2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256) ; Fri, 6 Sep 2024 14:24:22 +0000 Received: from smtpclient.apple ( [2001:420:c0c4:1001::9f]) by tristain.distal.com (OpenSMTPD) with ESMTPSA id 52c632bb (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256:NO); Fri, 6 Sep 2024 10:24:21 -0400 (EDT) From: Chris Ross Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@FreeBSD.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3776.700.51\)) Subject: Unable to replace drive in raidz1 Message-Id: <5ED5CB56-2E2A-4D83-8CDA-6D6A0719ED19@distal.com> Date: Fri, 6 Sep 2024 10:24:10 -0400 Cc: Chris Ross To: FreeBSD Filesystems X-Mailer: Apple Mail (2.3776.700.51) X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.20 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.998]; FORGED_SENDER(0.30)[cross@distal.com,cross@relay.distal.com]; R_SPF_ALLOW(-0.20)[+a:relay.dynu.com]; R_DKIM_ALLOW(-0.20)[relay.distal.com:s=mail]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[distal.com]; RCPT_COUNT_TWO(0.00)[2]; DKIM_TRACE(0.00)[relay.distal.com:+]; RCVD_TLS_ALL(0.00)[]; TO_DN_ALL(0.00)[]; TAGGED_FROM(0.00)[freebsd]; RCVD_COUNT_TWO(0.00)[2]; RCVD_VIA_SMTP_AUTH(0.00)[]; ARC_NA(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; FROM_NEQ_ENVFROM(0.00)[cross@distal.com,cross@relay.distal.com]; FROM_HAS_DN(0.00)[]; ASN(0.00)[asn:3842, ipnet:168.235.92.0/22, country:US]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org]; APPLE_MAILER_COMMON(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; TAGGED_RCPT(0.00)[freebsd]; MIME_TRACE(0.00)[0:+] X-Rspamd-Queue-Id: 4X0dlf4v2Zz4SbN Hello. I have searched the interwebs a bit and seen tell of this and = others like it, but I haven=E2=80=99t found a solution. I have a pool with three 3-disk raidz1 vols. I want to replace the = disks in the first vdev with larger disks. I=E2=80=99ve done this before, but = may=E2=80=99ve done something wrong here. I belive I used =E2=80=9Czpool remove tank da3=E2=80=9D, but command = history doesn=E2=80=99t have that. I=E2=80=99ve used many commands since I started. I = might=E2=80=99ve =E2=80=9Czpool offline=E2=80=9Dd the device. I=E2=80=99m sorry I = don=E2=80=99t remember the original command. Then replaced the disk and rebooted. This of course renumbered the disks. :-(. But, finding the new/replacement disk (da10), I try to =E2=80=9Czpool replace tank da3 da10=E2=80=9D. This always produces: cannot replace da3 with da10: already in replacing/spare config; wait = for completion or use 'zpool detach=E2=80=99 Now, I can=E2=80=99t use =E2=80=9Czpool detach=E2=80=9D I=E2=80=99ve = learned because that doesn=E2=80=99t work on zraid. And I can=E2=80=99t tell what it _thinks_ is happening. I = even Did a scrub of the pool and let that finish, but am still seeing the same. I have now: =E2=80=948<=E2=80=948<=E2=80=948<--- >>>>>> zpool status -v tank pool: tank state: DEGRADED status: One or more devices are faulted in response to persistent = errors. Sufficient replicas exist for the pool to continue functioning = in a degraded state. action: Replace the faulted device, or use 'zpool clear' to mark the = device repaired. scan: scrub repaired 0B in 17:14:03 with 0 errors on Fri Sep 6 = 09:08:34 2024 config: NAME STATE READ WRITE CKSUM tank DEGRADED 0 0 0 raidz1-0 DEGRADED 0 0 0 da3 FAULTED 0 0 0 external = device fault da1 ONLINE 0 0 0 da2 ONLINE 0 0 0 raidz1-1 ONLINE 0 0 0 diskid/DISK-K1GMBN9D ONLINE 0 0 0 diskid/DISK-K1GMEDMD ONLINE 0 0 0 diskid/DISK-K1GMAX1D ONLINE 0 0 0 raidz1-2 ONLINE 0 0 0 diskid/DISK-3WJDHJ2J ONLINE 0 0 0 diskid/DISK-3WK3G1KJ ONLINE 0 0 0 diskid/DISK-3WJ7ZMMJ ONLINE 0 0 0 errors: No known data errors =E2=80=948<=E2=80=948<=E2=80=948<=E2=80=94 I=E2=80=99ll note that before the switcharoo, the second and third = vdev=E2=80=99s listed =E2=80=9Cda4 da5 da6=E2=80=9D and =E2=80=9Cda7 da8 da9=E2=80=9D. The = moving around of names caused the above, which again I=E2=80=99ve seen before, and am fine with. (Oh, you can see I ran a =E2=80=9Czpool offline -f=E2=80=9D on it most = recently. But that was in the list of things I=E2=80=99ve tried that haven=E2=80=99t = helped.) Please let me know if someone knows how I=E2=80=99ve gotten to this = state, and what I need to do to correct it. What is the "in replacing/spare = config=E2=80=9D meaning? - Chris