From nobody Mon Sep 04 05:06:46 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RfGp12rx2z4rMWk; Mon, 4 Sep 2023 05:06:49 +0000 (UTC) (envelope-from mavbsd@gmail.com) Received: from mail-yw1-x1132.google.com (mail-yw1-x1132.google.com [IPv6:2607:f8b0:4864:20::1132]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RfGp06dNNz4s4f; Mon, 4 Sep 2023 05:06:48 +0000 (UTC) (envelope-from mavbsd@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-yw1-x1132.google.com with SMTP id 00721157ae682-58e6c05f529so10175947b3.3; Sun, 03 Sep 2023 22:06:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1693804008; x=1694408808; darn=freebsd.org; h=content-transfer-encoding:in-reply-to:from:references:to :content-language:subject:user-agent:mime-version:date:message-id :sender:from:to:cc:subject:date:message-id:reply-to; bh=XBjPJraRZjbNo81Dg7oGngWW7szayA/9F2dZbfHQfdQ=; b=T7BbM1ZfxjtrcmU33eHqdhiYjvxJhy5MBKWSG6P6FOzOp+7CqklEVGQCTdpjeGFTdf pQyJXOI4sqnhBMMXEFpMlzMfJq6zGaancnJY/vl0xtKuqONwDyQL90MWHGPTVRqb0EKl TFc34VVzI7uZWP4DaJhlbTivyg44C2YCVpzaeRZhqMF4ExFYNsS2eDTT4MR0R7q9L8tM YS9wKpYqr5ahabO6gpabSXjFakUQM+duJV6fweH/acv87KBUF3IGm3v4l2VKJJHWh/In FeYp+eggTDZMcWwjz0ZhTny50VJ/fELvNu8DEI8c2hzTDMoFzDy26V162ZFLNIv65VZY AOTg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1693804008; x=1694408808; h=content-transfer-encoding:in-reply-to:from:references:to :content-language:subject:user-agent:mime-version:date:message-id :sender:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=XBjPJraRZjbNo81Dg7oGngWW7szayA/9F2dZbfHQfdQ=; b=d3ja6pe3sgfQZBP7Jle1tWi2T3PUgtzOuWiIno5PCYE9mRazqPl5DfioCuFgCZXcZ+ buir0WeFqRm6ZtmCWyOMSNwTAG051xXbIFlBHFbiuKZlI/E8hcdHeLq/M/ywA5OmK+fo cBe7LZydNOfF82Cn/eoVx79JNZFRFATjzf0TVAQJAqSJQQuzAoYeTFQSVkPjwseFfJVi XW1YtvkD4ooY5GVFo+A3mmu1tY7BTpizQWSYDrsvlPiqubgAd3eqHwhV7Ql8QKtWScfl V1xO/sNNpjhC9zGG31iAGL3/OuBcU8ipPsQ7za7/B5JfNTuscAf1vMjup4I1VDs9t5aD KK9Q== X-Gm-Message-State: AOJu0YzHBlVN9444aTYCE8LvlIKFg81WWDqtC+IBBF1zoov7wxnF+52A 3CrQq2lZPRZCRXi22nAHxwC6wxL/kiR3nw== X-Google-Smtp-Source: AGHT+IEURq5mJx6dliGO6vrnZDnrj+D/zboaXVPr2Zc9JF4R8HfFa+oxn30QDQ7pjGoull77rG68gg== X-Received: by 2002:a0d:eacf:0:b0:586:a689:d28a with SMTP id t198-20020a0deacf000000b00586a689d28amr10943997ywe.34.1693804007907; Sun, 03 Sep 2023 22:06:47 -0700 (PDT) Received: from [192.168.1.66] (104-55-12-234.lightspeed.knvltn.sbcglobal.net. [104.55.12.234]) by smtp.gmail.com with ESMTPSA id g185-20020a0dc4c2000000b00594e355aa78sm2455855ywd.143.2023.09.03.22.06.46 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sun, 03 Sep 2023 22:06:47 -0700 (PDT) Message-ID: Date: Mon, 4 Sep 2023 01:06:46 -0400 List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:102.0) Gecko/20100101 Thunderbird/102.12.0 Subject: Re: An attempted test of main's "git: 2ad756a6bbb3" "merge openzfs/zfs@95f71c019" that did not go as planned Content-Language: en-US To: Mark Millard , dev-commits-src-main@freebsd.org, Current FreeBSD References: From: Alexander Motin In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Queue-Id: 4RfGp06dNNz4s4f Mark, On 03.09.2023 22:54, Mark Millard wrote: > After that ^t produced the likes of: > > load: 6.39 cmd: sh 4849 [tx->tx_quiesce_done_cv] 10047.33r 0.51u 121.32s 1% 13004k So the full state is not "tx->tx", but is actually a "tx->tx_quiesce_done_cv", which means the thread is waiting for new transaction to be opened, which means some previous to be quiesced and then synced. > #0 0xffffffff80b6f103 at mi_switch+0x173 > #1 0xffffffff80bc0f24 at sleepq_switch+0x104 > #2 0xffffffff80aec4c5 at _cv_wait+0x165 > #3 0xffffffff82aba365 at txg_wait_open+0xf5 > #4 0xffffffff82a11b81 at dmu_free_long_range+0x151 Here it seems like transaction commit is waited due to large amount of delete operations, which ZFS tries to spread between separate TXGs. You should probably see some large and growing number in sysctl kstat.zfs.misc.dmu_tx.dmu_tx_dirty_frees_delay . > #5 0xffffffff829a87d2 at zfs_rmnode+0x72 > #6 0xffffffff829b658d at zfs_freebsd_reclaim+0x3d > #7 0xffffffff8113a495 at VOP_RECLAIM_APV+0x35 > #8 0xffffffff80c5a7d9 at vgonel+0x3a9 > #9 0xffffffff80c5af7f at vrecycle+0x3f > #10 0xffffffff829b643e at zfs_freebsd_inactive+0x4e > #11 0xffffffff80c598cf at vinactivef+0xbf > #12 0xffffffff80c590da at vput_final+0x2aa > #13 0xffffffff80c68886 at kern_funlinkat+0x2f6 > #14 0xffffffff80c68588 at sys_unlink+0x28 > #15 0xffffffff8106323f at amd64_syscall+0x14f > #16 0xffffffff8103512b at fast_syscall_common+0xf8 What we don't see here is what quiesce and sync threads of the pool are actually doing. Sync thread has plenty of different jobs, including async write, async destroy, scrub and others, that all may delay each other. Before you rebooted the system, depending how alive it is, could you save a number of outputs of `procstat -akk`, or at least specifically `procstat -akk | grep txg_thread_enter` if the full is hard? Or somehow else observe what they are doing. `zpool status`, `zpool get all` and `sysctl -a` would also not harm. PS: I may be wrong, but USB in "USB3 NVMe SSD storage" makes me shiver. Make sure there is no storage problems, like some huge delays, timeouts, etc, that can be seen, for example, as busy percents regularly spiking far above 100% in your `gstat -spod`. -- Alexander Motin