From nobody Thu Nov 25 05:18:05 2021 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id E577F18A1DAB for ; Thu, 25 Nov 2021 05:18:17 +0000 (UTC) (envelope-from cross+freebsd@distal.com) Received: from relay.wiredblade.com (relay.wiredblade.com [168.235.105.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4J05kJ5YSkz3Kvw; Thu, 25 Nov 2021 05:18:16 +0000 (UTC) (envelope-from cross+freebsd@distal.com) Received: from mail.distal.com (pool-108-48-165-176.washdc.fios.verizon.net [108.48.165.176]) by relay.wiredblade.com with ESMTPSA (version=TLSv1.2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256) ; Thu, 25 Nov 2021 05:18:09 +0000 Received: from smtpclient.apple ( [2001:420:c0c4:1006::505]) by tristain.distal.com (OpenSMTPD) with ESMTPSA id 1f51dc61 (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256:NO); Thu, 25 Nov 2021 00:18:07 -0500 (EST) From: Chris Ross Message-Id: <8DA407FF-0895-4ED6-8495-86F00556DD7F@distal.com> Content-Type: multipart/alternative; boundary="Apple-Mail=_2DF50FC9-29CF-4043-B7B1-51883708619F" List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 15.0 \(3693.20.0.1.32\)) Subject: Re: swap_pager: cannot allocate bio Date: Thu, 25 Nov 2021 00:18:05 -0500 In-Reply-To: Cc: Andriy Gapon , freebsd-fs To: Mark Johnston References: <4E5511DF-B163-4928-9CC3-22755683999E@distal.com> <19A3AAF6-149B-4A3C-8C27-4CFF22382014@distal.com> <6DA63618-F0E9-48EC-AB57-3C3C102BC0C0@distal.com> <35c14795-3b1c-9315-8e9b-a8dfad575a04@FreeBSD.org> <471B80F4-B8F4-4D5A-9DEB-3F1E00F42A68@distal.com> X-Mailer: Apple Mail (2.3693.20.0.1.32) X-Rspamd-Queue-Id: 4J05kJ5YSkz3Kvw X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of cross@distal.com designates 168.235.105.136 as permitted sender) smtp.mailfrom=cross@distal.com X-Spamd-Result: default: False [-2.80 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; ARC_NA(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; MV_CASE(0.50)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; R_SPF_ALLOW(-0.20)[+a:relay.dynu.com]; DMARC_NA(0.00)[distal.com]; NEURAL_HAM_LONG(-1.00)[-0.999]; RCVD_COUNT_THREE(0.00)[3]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; TO_DN_ALL(0.00)[]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:3842, ipnet:168.235.104.0/22, country:US]; TAGGED_FROM(0.00)[freebsd]; RCVD_TLS_ALL(0.00)[]; RECEIVED_SPAMHAUS_PBL(0.00)[108.48.165.176:received] X-ThisMailContainsUnwantedMimeParts: Y --Apple-Mail=_2DF50FC9-29CF-4043-B7B1-51883708619F Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 >> On Nov 20, 2021, at 13:23, Mark Johnston > wrote: >>=20 >> Here is a patch which tries to address the proximate cause of the >> problem. It would be helpful to know if it addresses the deadlocks >> you're seeing. I tested it lightly by putting a NUMA system under >> memory pressure using postgres. >=20 > Thank you. I=E2=80=99ve updated my stable/13 tree today, and applied = this > patch. Build and installed a kernel, and I=E2=80=99ll see in a few = days if the > system has remains operational and get back to you all. The system is still cooking along, running the job that previously was causing it to get stuck after 48ish hours. It=E2=80=99s been running = more than 80 hours now, so a definite improvement. zfs-stats reports things as stable, 80% cache hit ratio, 64GB max arc, ~85% of that currently as =E2=80=9Cadaptive target=E2=80=9D. If = there=E2=80=99s anything you would like to get from the system, data-wise, let me know. Happy to share, and help get this fix, or a different better fix if needed, into = the tree. Thanks again. - Chris --Apple-Mail=_2DF50FC9-29CF-4043-B7B1-51883708619F--