From nobody Thu Nov 11 18:50:39 2021 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 83F35183D6E5 for ; Thu, 11 Nov 2021 18:50:50 +0000 (UTC) (envelope-from ronald-lists@klop.ws) Received: from smarthost1.greenhost.nl (smarthost1.greenhost.nl [195.190.28.88]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4HqrQK40FBz3sFg for ; Thu, 11 Nov 2021 18:50:49 +0000 (UTC) (envelope-from ronald-lists@klop.ws) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=klop.ws; s=mail; h=In-Reply-To:Message-ID:From:Content-Transfer-Encoding:MIME-Version: Date:References:Subject:To:Content-Type:Sender:Reply-To:Cc:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=ALPUixyQ+n6pEZvdvaOluSYxiDP8NL2EqAlewY1iP5M=; b=mNsFid2wyTxBZjc9+c2MB4zI7D 0N6jeiizYaZH2tQnI66xemvdsAIBnu+PqnjwLiP5w5RvOOKAOp+/fa37FQe7tZTyhOsDGhiYZH3l/ ticV7Wx5AMq3K/L1hjsdGjqHvgrK01LFpZay+HQ5t5FPXeVvEXCoCsiWTYQwWt20bd54=; Content-Type: text/plain; charset=utf-8; format=flowed; delsp=yes To: freebsd-fs@freebsd.org Subject: Re: swap_pager: cannot allocate bio References: <9FE99EEF-37C5-43D1-AC9D-17F3EDA19606@distal.com> <09989390-FED9-45A6-A866-4605D3766DFE@distal.com> Date: Thu, 11 Nov 2021 19:50:39 +0100 List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 Content-Transfer-Encoding: Quoted-Printable Message-ID: In-Reply-To: <09989390-FED9-45A6-A866-4605D3766DFE@distal.com> User-Agent: Opera Mail/1.0 (Win32) X-Authenticated-As-Hash: bdb49c4ff80bd276e321aade33e76e02752072e2 X-Virus-Scanned: by clamav at smarthost1.greenhost.nl X-Spam-Level: / X-Spam-Score: -0.4 X-Spam-Status: No, score=-0.4 required=5.0 tests=ALL_TRUSTED,BAYES_50,DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF autolearn=disabled version=3.4.2 X-Scan-Signature: d9b0ae15ee993d77aea4f0208a5c5b8c X-Rspamd-Queue-Id: 4HqrQK40FBz3sFg X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=pass header.d=klop.ws header.s=mail header.b=mNsFid2w; dmarc=pass (policy=quarantine) header.from=klop.ws; spf=pass (mx1.freebsd.org: domain of ronald-lists@klop.ws designates 195.190.28.88 as permitted sender) smtp.mailfrom=ronald-lists@klop.ws X-Spamd-Result: default: False [-1.38 / 15.00]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[klop.ws:s=mail]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:195.190.28.64/27]; MIME_GOOD(-0.10)[text/plain]; TO_DN_NONE(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_SPAM_MEDIUM(1.00)[1.000]; RCPT_COUNT_ONE(0.00)[1]; DKIM_TRACE(0.00)[klop.ws:+]; DMARC_POLICY_ALLOW(-0.50)[klop.ws,quarantine]; RCVD_IN_DNSWL_NONE(0.00)[195.190.28.88:from]; NEURAL_HAM_SHORT(-0.88)[-0.882]; RCVD_COUNT_ZERO(0.00)[0]; FROM_EQ_ENVFROM(0.00)[]; MID_RHS_NOT_FQDN(0.50)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:47172, ipnet:195.190.28.0/24, country:NL]; RWL_MAILSPIKE_POSSIBLE(0.00)[195.190.28.88:from] Reply-To: ronald-lists@klop.ws From: Ronald Klop via freebsd-fs X-Original-From: Ronald Klop X-ThisMailContainsUnwantedMimeParts: N On Thu, 11 Nov 2021 05:43:46 +0100, Chris Ross = wrote: > > >> On Nov 10, 2021, at 23:35, Chris Ross wrot= e: >> >> Hey all. I have a system that I=E2=80=99m trying to do some intensiv= e CPU and = >> I/O on. FreeBSD 13.0-RELEASE, amd64, 128GB RAM, hardware RAID1 OS = >> volume, and a large (40TB) zpool where most of the I/O is happening. >> >> Initially, it was failing for me because it was running out of swap = >> space. It had only the normal small (4-8G) swap partition, so I = >> resized the filesystems on the root disk and now have 400+GB swap. T= he = >> system had frozen up and I wasn=E2=80=99t able to log in. When I go = to the = >> console, I find a long list of: >> >> swap_pager: cannot allocate bio >> >> lines. I was able to log into the console as root and pstat -s shows= = >> the swap minimally used (7.5GB used). Attempting a =E2=80=9Czpool st= atus=E2=80=9D at = >> that point locked up. I don=E2=80=99t know if the problem is the mem= ory = >> subsystem, or zfs. >> >> But, based on the error, is there perhaps some kernel parameter I can= = >> tune that might prevent the swap pager from encountering that error? > > Moving to freebsd-fs. More information makes it looks more like a ZFS= = > problem than anything else. > > I am able to log into another root virtual console, and I can run ps = > (shows many things, including dozens of "cron: running job (cron)=E2=80= =9D jobs, = > in D state), and I=E2=80=99m able to wander around the root disk (3T u= fs = > filesystem) without trouble. But, as mentioned above the =E2=80=9Czpo= ol status=E2=80=9D = > is hung, and I suspect if I tried to access anything in that filesyste= m = > it would hang to. Those cron jobs, which aren=E2=80=99t anything I ad= ded, I = > assume are just system =E2=80=9Ccheck around the system=E2=80=9D cron = jobs that are = > getting stuck there. > > So, if anyone has any suggestions. I can leave this system stuck like= = > this for a little while, but I=E2=80=99ll probably want to bring it ba= ck before = > the end of the day tomorrow. (I=E2=80=99m US EST, so it=E2=80=99s alm= ost midnight = > here. I=E2=80=99ll check in on email for suggestions or ideas in the = morning.) > > Thanks all. > > - Chris > Can you press ctrl-t on the hanging process? That should print the = stacktrace indicating where it is waiting on. Ronald.