From nobody Sat Sep 25 19:28:39 2021 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 8C30517DF4CB for ; Sat, 25 Sep 2021 19:28:54 +0000 (UTC) (envelope-from zbeeble@gmail.com) Received: from mail-ed1-x529.google.com (mail-ed1-x529.google.com [IPv6:2a00:1450:4864:20::529]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4HGzTx4qgNz3KZQ for ; Sat, 25 Sep 2021 19:28:53 +0000 (UTC) (envelope-from zbeeble@gmail.com) Received: by mail-ed1-x529.google.com with SMTP id y12so14135817edo.9 for ; Sat, 25 Sep 2021 12:28:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:from:date:message-id:subject:to; bh=xVQkdG6gBoCSUXLf0OqvaxCNYN4qsrPb0pBHL0UdmO4=; b=EydMdB8vPbYGLX91LsLprL3E2jLPzv+LaPOUgz8RJoun2rELE+HiW0Jhi3G7uaYfxF cPp8w5iE9p9wpqEbGksXdULXno1kRtXiY2rPpJWVAL00Dlt825QDHQp1qHALbZm9OlQz OQkQr9YEP5afnsjrhinPwZE9oyqZwNPEFJriJX92BxTU1JbvQwaCsVxrylPKXc8ByUUO gvlQR7btalr0ZQlDT/56RGcM4dZVUeUvkFcQphtqtx7dHlPMiU43qxg+A+F/qy1SUYMC QHWOF4MNc310QSeVx5tPHtM1svF0HN8iKn0aGDFVCeTEe5dRWfVVRQT8NyShHbse3tnF TN+g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=xVQkdG6gBoCSUXLf0OqvaxCNYN4qsrPb0pBHL0UdmO4=; b=bg34d50AU8bY15ssZIo5AC3J6iVoTFCg60KnTUN2G8tD2H7FuHEM0FcdvdbgiRLjLS 9jDnlyq7E4bLD7j+oGjK+tuS5A9rYCFjohD+wwppvNJzIxIQj3uQQ7PN3Rn/IBGnm6Io SLomCTN1O4Ck4KLR1+cBrNmFIWS+vuhJtkSR02fnbePcpVzCcszRe9GthFqax1+gyPEw P/X3ctWhZ5pW+dxP7GkBzqCa6KkB0C/E9uxwvQkafReWmB3UdjnZeHxkxF5lvh8pXnuj Tkjr3AXl30BuJ9I3HZ+iDtcPRvdDHXOOzSQC/6Bag5iBMXYz9ywbR9AqJECf6ok7uXWs J71g== X-Gm-Message-State: AOAM532sPbnO8TAquEf4uTceOZ1f7Zj3pvntY5hV9f9eVmShJLZsFsCH qSeIeV1aLjyLDChDu/qSLTkR8hj+RoamP8Q5Mu2LVyQzog== X-Google-Smtp-Source: ABdhPJzyApdWsQScPwareuOUa7JaPz8aJFpZw05C4b+u5hOU+YHQGXmeLPZHZWvNAGdbi0qmyDb6uDoYUcHTY1Mk9sc= X-Received: by 2002:a17:906:1e16:: with SMTP id g22mr1749314ejj.471.1632598131539; Sat, 25 Sep 2021 12:28:51 -0700 (PDT) List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 From: Zaphod Beeblebrox Date: Sat, 25 Sep 2021 15:28:39 -0400 Message-ID: Subject: ZFS errors with two HEX numbers. To: freebsd-fs Content-Type: multipart/alternative; boundary="000000000000b1030f05ccd6df75" X-Rspamd-Queue-Id: 4HGzTx4qgNz3KZQ X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20210112 header.b=EydMdB8v; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of zbeeble@gmail.com designates 2a00:1450:4864:20::529 as permitted sender) smtp.mailfrom=zbeeble@gmail.com X-Spamd-Result: default: False [-3.90 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.999]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36:c]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; NEURAL_HAM_LONG(-1.00)[-1.000]; RCPT_COUNT_ONE(0.00)[1]; MID_RHS_MATCH_FROMTLD(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; MIME_BASE64_TEXT(0.10)[]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::529:from]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim] X-ThisMailContainsUnwantedMimeParts: Y --000000000000b1030f05ccd6df75 Content-Type: text/plain; charset="UTF-8" I've got a ZFS pool --- 3 x 8 x 4T disks in a RAIDZ2 configuration. Every few months I scrub them. This time, curiously, it came up with an error. All the disks in a vdev simultaneously give an error when this is scanned.... thusly: raidz2-3 ONLINE 0 0 0 gpt/v1-f0 ONLINE 0 0 1 gpt/v1-f1 ONLINE 0 0 1 gpt/v1-f2 ONLINE 0 0 1 gpt/v1-f3 ONLINE 0 0 1 gpt/v1-f4 ONLINE 0 0 1 gpt/v1-f5 ONLINE 0 0 1 gpt/v1-f6a ONLINE 0 0 1 gpt/v1-f7b ONLINE 0 0 1 ... and if I add -v to the status, I see the following right now: errors: Permanent errors have been detected in the following files: <0x57b2>:<0x73ab46> Now... since that happened, I have scrubbed twice more. The first number changes each time. Last scrub it was: <0x4223>:<0x73ab46>. Note the last number is not changing. I have done some googling and found that this might be something that got corrupted on delete or somesuch. My OCD would like to know what 0x4223 maps to (or now 0x57b2) s.t. I could find it and remove it. I _think_ this is a snapshot reference? Anyways... any help in tracking this down would be appreciated. For those who've read this far, here's the full zfs scrub -v output... If you're all intensely curious, the last letter following the drive number tracks the number of times a drive has been replaced. pool: vr1 state: ONLINE status: One or more devices has experienced an error resulting in data corruption. Applications may be affected. action: Restore the file in question if possible. Otherwise restore the entire pool from backup. see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-8A scan: scrub repaired 0B in 1 days 05:01:08 with 1 errors on Fri Sep 24 23:23:34 2021 config: NAME STATE READ WRITE CKSUM vr1 ONLINE 0 0 0 raidz2-0 ONLINE 0 0 0 gpt/v1-d0 ONLINE 0 0 0 gpt/v1-d1 ONLINE 0 0 0 gpt/v1-d2 ONLINE 0 0 0 gpt/v1-d3 ONLINE 0 0 0 gpt/v1-d4a ONLINE 0 0 0 gpt/v1-d5 ONLINE 0 0 0 gpt/v1-d6a ONLINE 0 0 0 gpt/v1-d7a ONLINE 0 0 0 raidz2-2 ONLINE 0 0 0 gpt/v1-e0c ONLINE 0 0 0 gpt/v1-e1c ONLINE 0 0 0 gpt/v1-e2b ONLINE 0 0 0 gpt/v1-e3b ONLINE 0 0 0 gpt/v1-e4b ONLINE 0 0 0 gpt/v1-e5a ONLINE 0 0 0 gpt/v1-e6a ONLINE 0 0 0 gpt/v1-e7d ONLINE 0 0 0 raidz2-3 ONLINE 0 0 0 gpt/v1-f0 ONLINE 0 0 1 gpt/v1-f1 ONLINE 0 0 1 gpt/v1-f2 ONLINE 0 0 1 gpt/v1-f3 ONLINE 0 0 1 gpt/v1-f4 ONLINE 0 0 1 gpt/v1-f5 ONLINE 0 0 1 gpt/v1-f6a ONLINE 0 0 1 gpt/v1-f7b ONLINE 0 0 1 logs gpt/vr1log ONLINE 0 0 0 cache gpt/vr1cache ONLINE 0 0 0 errors: Permanent errors have been detected in the following files: <0x57b2>:<0x73ab46> --000000000000b1030f05ccd6df75--