From nobody Sun Dec 12 09:20:32 2021 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 7FB1518D3362 for ; Sun, 12 Dec 2021 09:20:48 +0000 (UTC) (envelope-from freebsd@walstatt-de.de) Received: from smtp6.goneo.de (smtp6.goneo.de [IPv6:2001:1640:5::8:31]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4JBfJH4Zv8z3jZ5 for ; Sun, 12 Dec 2021 09:20:47 +0000 (UTC) (envelope-from freebsd@walstatt-de.de) Received: from hub2.goneo.de (hub2.goneo.de [IPv6:2001:1640:5::8:53]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by smtp6.goneo.de (Postfix) with ESMTPS id EE09A10A330D for ; Sun, 12 Dec 2021 10:20:38 +0100 (CET) Received: from hub2.goneo.de (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by hub2.goneo.de (Postfix) with ESMTPS id CC90F10A330B for ; Sun, 12 Dec 2021 10:20:38 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=walstatt-de.de; s=DKIM001; t=1639300838; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=RTe2bekCzSBW0kNVdFoZiadCKg03aKXHZ7dFHiHVLhE=; b=TV9Vuee8Mm+X11bO/uHVOInKYVIVzs7DM0ctyh8aMDp6KMOzWARnceh8+9YvynI00ME3+r NcmbASBsBQZ4RlcLLSgL9HC4dykhApO/g1BSzeKat9mRyoCvjSLGal7g9UYPju8Ibj1Kt2 NtfXmzxox4BsXM7L8K5PmFOMMZZjC0Eb9eSMJbjnEbcWFrsTbo5iNv99bB/leLpfZV/anK VGiBl74r3UmX/3ht1LhEv6zQrLznqsunn1V8nwS8tD73BbVH+CDMMnNsml9IKsjkmnlUX/ BlQnkzRqz/UaUE+cnRc9knQA8WXhocl4JejRN6c7YYEFxhdfPwwat9FkvqBDFg== Received: from jelly.fritz.box (dynamic-2a01-0c22-ad1c-cf00-5585-92d4-c169-1fb9.c22.pool.telefonica.de [IPv6:2a01:c22:ad1c:cf00:5585:92d4:c169:1fb9]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by hub2.goneo.de (Postfix) with ESMTPSA id 9FFDF10A3309 for ; Sun, 12 Dec 2021 10:20:38 +0100 (CET) Date: Sun, 12 Dec 2021 10:20:32 +0100 From: FreeBSD User To: freebsd-current@freebsd.org Subject: CURRENT: ZFS freezes system beyond reboot Message-ID: <20211212102032.08af9689@jelly.fritz.box> Organization: FreeBSD in der Heimstatt List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Rspamd-UID: f6ad8b X-Rspamd-UID: 42ff9c X-Rspamd-Queue-Id: 4JBfJH4Zv8z3jZ5 X-Spamd-Bar: ++ Authentication-Results: mx1.freebsd.org; dkim=pass header.d=walstatt-de.de header.s=DKIM001 header.b=TV9Vuee8; dmarc=none; spf=none (mx1.freebsd.org: domain of freebsd@walstatt-de.de has no SPF policy when checking 2001:1640:5::8:31) smtp.mailfrom=freebsd@walstatt-de.de X-Spamd-Result: default: False [2.48 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[walstatt-de.de:s=DKIM001]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_SPAM_SHORT(0.99)[0.988]; MIME_GOOD(-0.10)[text/plain]; TO_DN_NONE(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-current@freebsd.org]; NEURAL_SPAM_MEDIUM(0.90)[0.900]; RCPT_COUNT_ONE(0.00)[1]; HAS_ORG_HEADER(0.00)[]; RCVD_COUNT_THREE(0.00)[4]; DMARC_NA(0.00)[walstatt-de.de]; DKIM_TRACE(0.00)[walstatt-de.de:+]; NEURAL_SPAM_LONG(0.89)[0.891]; R_SPF_NA(0.00)[no SPF record]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:25394, ipnet:2001:1640::/32, country:DE]; RCVD_TLS_ALL(0.00)[] X-ThisMailContainsUnwantedMimeParts: N Running CURRENT (FreeBSD 14.0-CURRENT #52 main-n251260-156fbc64857: Thu Dec 2 14:45:55 CET 2021 amd64), out of the sudden the ZFS RAIDZ pool suffered from an error: Solaris: WARNING: Pool 'POOL00' has encountered an uncorrectable I/O failure and has been suspended. The system does not repsond anymore on that pool, transactions to and from that pool are frozen, the system is 99.9% idle. The most "not so funny" part is: the box doesn't even recognize a "shutdown -r now" or a brute force "reboot". I still can login via ssh, but any action regarding the ZFS pool freezes the console/terminal. ZFS very often renders the system unresponsible forever. How can this be mitigated? The system in question is on a remote site and it seems not only to be bound to CURRENT, we realised similar problems on 13-STABLE as well. What can I do to "unfreeze" the ZFS? The main OS is, luckily, on an UFS/FFS filesystem and so not affected from that problem. By the way, here some more details, as far as I can pick those up: zpool clear POOL00 cannot clear errors for POOL00: I/O error Whatever took out the ZFS pool (can not see any hardware errors, the pool is part of services and especially a poudriere build system and under heavy load all the time, the box has 16 GB RAM), it also renders the rest of the system unusable in a way which is beyond a "reboot". Kind regrads, oh