From nobody Wed Dec 13 01:25:55 2023 X-Original-To: freebsd-arch@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Sqd922RcSz53cdf for ; Wed, 13 Dec 2023 01:25:58 +0000 (UTC) (envelope-from mike@karels.net) Received: from mail2.karels.net (mail2.karels.net [3.19.118.201]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "freebsd", Issuer "freebsd" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Sqd913ywMz4ShW for ; Wed, 13 Dec 2023 01:25:57 +0000 (UTC) (envelope-from mike@karels.net) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=karels.net header.s=mail2 header.b="arYI//7b"; spf=pass (mx1.freebsd.org: domain of mike@karels.net designates 3.19.118.201 as permitted sender) smtp.mailfrom=mike@karels.net; dmarc=none Received: from mail2.karels.net (localhost [IPv6:0:0:0:0:0:0:0:1]) by mail2.karels.net (8.17.1/8.17.1) with ESMTP id 3BD1PuZf033098 for ; Tue, 12 Dec 2023 19:25:56 -0600 (CST) (envelope-from mike@karels.net) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=karels.net; s=mail2; t=1702430756; bh=UR19XMw01y3ZmApDnXxFpokWXft5eEEYT54ykP/WFLk=; h=From:To:Subject:Date; b=arYI//7bW51BC8ltgJhtTrwMZPj+ZLKa5f5gCKUM9BPVMn/2Ogez+JG3qZTyOFQgc uwhv/Zzzkz1gaAcxtlDZg+8zrVst71T/UjrynfdDtfVmwi8rA8l4x2LuvQLt9fG8M0 /KwXQEVPbh859EG7qtK9/3MKO9tZe3tn9Sau1alMkc4ZuERJUQQOfEzkkfwuIAMf0n i8ejJ1CHSL2D913eug3Fwc3Dho7rjptQIyzeaQiiQyHpQ4EqdQ9PxPNNsrBvTuBCI0 2e2ggMlB+8TIksb3icETR7mNv8HHpaH2Zt/7JZD4D++orFqgHZe0xP6PEJ1Jyizasz EpFN2xfHXxVPg== Received: from [10.0.2.130] ([73.62.165.147]) by mail2.karels.net with ESMTPSA id XEJvEyQIeWVIgQAAs/W3XQ (envelope-from ) for ; Tue, 12 Dec 2023 19:25:56 -0600 From: Mike Karels To: freebsd-arch Subject: tmpfs is overly aggressive on memory usage Date: Tue, 12 Dec 2023 19:25:55 -0600 X-Mailer: MailMate (1.14r5964) Message-ID: List-Id: Discussion related to FreeBSD architecture List-Archive: https://lists.freebsd.org/archives/freebsd-arch List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-arch@freebsd.org MIME-Version: 1.0 Content-Type: text/plain X-Spamd-Result: default: False [-3.50 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[karels.net:s=mail2]; R_SPF_ALLOW(-0.20)[+ip4:3.19.118.201]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DKIM_TRACE(0.00)[karels.net:+]; MIME_TRACE(0.00)[0:+]; MLMMJ_DEST(0.00)[freebsd-arch@freebsd.org]; TO_DN_ALL(0.00)[]; ASN(0.00)[asn:16509, ipnet:3.16.0.0/14, country:US]; FROM_EQ_ENVFROM(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; FREEFALL_USER(0.00)[mike]; ARC_NA(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_HAS_DN(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-arch@freebsd.org]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DMARC_NA(0.00)[karels.net]; MID_RHS_MATCH_FROM(0.00)[] X-Rspamd-Queue-Id: 4Sqd913ywMz4ShW X-Spamd-Bar: --- I have been looking at tmpfs and doing some experiments. I had noticed that tmpfs defaults to using a memory limit of "available amount of memory (including main memory and swap space)", which seemed overly optimistic. It isn't as bad as I first thought, as it is looking at current free memory and swap space. However, tests writing a file until something fails caused all of swap space to be exhausted, and processes started getting killed. In my test, this started with a large memory hog, but then killed root shells, nfsd, etc. This seems bad, and the system was screwed up enough that there was no way to reboot it short of a reset. One part of the problem was that with a default size, tmpfs enforced the limit when creating a file but not when writing it. I think this is an outright bug, and I have a proposed fix in https://reviews.freebsd.org/D43010. It has the disadvantage that it can fail a write with ENOSPC due to a transient memory load spike. The memory limit is aggressive enough that the system is very close to memory exhaustion, though, as well as swap space exhaustion (assuming there is swap). However, the limit is still high enough that continuous writes were likely to cause processes to be killed and even system hangs. I decided to try increasing the memory reserve; currently tmpfs has a memory reserve threshold of 4 MB. I ended up with a default reserve based on a percentage of memory plus swap, allowing 95% of available space to be used. This works fairly well in my tests, failing writes when the system is close to the edge and paging fairly heavily, but without killing processes. However, other memory load was essentially constant; these changes cannot predict changes in other memory demand, just react to the current situation. This change is in https://reviews.freebsd.org/D43011. I'm interested in any feedback, in particular other approaches that might be more general. Mike