From nobody Mon Sep 11 05:03:03 2023 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RkZNd5P7fz4sw7x for ; Mon, 11 Sep 2023 05:03:13 +0000 (UTC) (envelope-from jonc@chen.org.nz) Received: from egress.chen.org.nz (egress.chen.org.nz [170.75.172.82]) by mx1.freebsd.org (Postfix) with ESMTP id 4RkZNc3jQJz3FNZ for ; Mon, 11 Sep 2023 05:03:12 +0000 (UTC) (envelope-from jonc@chen.org.nz) Authentication-Results: mx1.freebsd.org; dkim=none; spf=pass (mx1.freebsd.org: domain of jonc@chen.org.nz designates 170.75.172.82 as permitted sender) smtp.mailfrom=jonc@chen.org.nz; dmarc=none Received: from mail.chen.org.nz (unknown [210.54.37.164]) by egress.chen.org.nz (Postfix) with ESMTP id 733CDEAE6C; Mon, 11 Sep 2023 17:03:05 +1200 (NZST) Received: from mail.chen.org.nz (localhost [127.0.0.1]) by filter.inside.chen.org.nz (Postfix) with ESMTP id AD5F880C91; Mon, 11 Sep 2023 17:03:03 +1200 (NZST) X-Spam-Checker-Version: SpamAssassin 4.0.0 (2022-12-14) on ametrine.inside.chen.org.nz Received: from [192.168.1.10] (jade.inside.chen.org.nz [192.168.1.10]) by mail.chen.org.nz (Postfix) with ESMTPS id A24BB80C90; Mon, 11 Sep 2023 17:03:03 +1200 (NZST) Message-ID: <9c818d6e-e1d1-fae2-2dc6-6d472d837c60@chen.org.nz> Date: Mon, 11 Sep 2023 17:03:03 +1200 List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:102.0) Gecko/20100101 Thunderbird/102.15.0 Subject: Re: Did something change with ZFS and vnode caching? To: Garrett Wollman References: <25827.33600.611577.665054@hergotha.csail.mit.edu> <25831.30103.446606.733311@hergotha.csail.mit.edu> <25840.58487.468791.344785@hergotha.csail.mit.edu> <25853.10676.45028.623279@hergotha.csail.mit.edu> Content-Language: en-US Cc: freebsd-stable@freebsd.org From: Jonathan Chen In-Reply-To: <25853.10676.45028.623279@hergotha.csail.mit.edu> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Spamd-Bar: -- X-Spamd-Result: default: False [-2.19 / 15.00]; SUBJECT_ENDS_QUESTION(1.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-0.996]; NEURAL_HAM_SHORT(-0.99)[-0.990]; R_SPF_ALLOW(-0.20)[+a:egress.chen.org.nz]; MIME_GOOD(-0.10)[text/plain]; RCVD_NO_TLS_LAST(0.10)[]; FROM_EQ_ENVFROM(0.00)[]; ASN(0.00)[asn:174, ipnet:170.75.160.0/20, country:US]; MLMMJ_DEST(0.00)[freebsd-stable@freebsd.org]; R_DKIM_NA(0.00)[]; ARC_NA(0.00)[]; BLOCKLISTDE_FAIL(0.00)[170.75.172.82:server fail,210.54.37.164:server fail]; MIME_TRACE(0.00)[0:+]; TO_DN_SOME(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; DMARC_NA(0.00)[chen.org.nz]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; MID_RHS_MATCH_FROM(0.00)[] X-Rspamd-Queue-Id: 4RkZNc3jQJz3FNZ Hi, On 10/09/23 14:28, Garrett Wollman wrote: > < said: > >> zfs lock arrays are a known problem, bumping them is definitely an option. > > This is the thing I tried next. It took a few attempts (mostly I > think due to my errors) but I'm now running with 512 (instead of 64) > and plan to deploy 1024 soon, as the results are significant: while we > still see significant loads and kmem pressure during the backup > window, backups are able to complete some 5 to 8 hours sooner, and > nfsd remains responsive. How did you bump up the zfs lock array value? Is this a sysctl-tweakable value? I'm currently running a recent version of stable/13 on a 24-CPU machine. It runs Postgresql 15. If I slam the system with a "pg_restore -j 20 ...", the system will freeze more often than not. This issue does not seem to appear if I use UFS as my backing store, so I'm left to assume that it's ZFS related. I'm wondering whether your tweaks will affect my system's behaviour. Cheers. -- Jonathan Chen