From nobody Sat Jul 03 11:35:59 2021 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 1F14A11D7881 for ; Sat, 3 Jul 2021 11:36:15 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4GH8zK5NKTz4v9c for ; Sat, 3 Jul 2021 11:36:13 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.16.1/8.16.1) with ESMTPS id 163BZxgY062996 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Sat, 3 Jul 2021 14:36:02 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua 163BZxgY062996 Received: (from kostik@localhost) by tom.home (8.16.1/8.16.1/Submit) id 163BZx8M062995; Sat, 3 Jul 2021 14:35:59 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Sat, 3 Jul 2021 14:35:59 +0300 From: Konstantin Belousov To: Vitaliy Gusev Cc: freebsd-hackers@freebsd.org Subject: Re: madvise(MADV_FREE) doesn't work in some cases? Message-ID: References: <0A95973D-254A-4574-8DC7-9F515F60B873@gmail.com> List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <0A95973D-254A-4574-8DC7-9F515F60B873@gmail.com> X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FORGED_GMAIL_RCVD,FREEMAIL_FROM, NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.5 X-Spam-Checker-Version: SpamAssassin 3.4.5 (2021-03-20) on tom.home X-Rspamd-Queue-Id: 4GH8zK5NKTz4v9c X-Spamd-Bar: ++ Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=fail reason="No valid SPF, No valid DKIM" header.from=gmail.com (policy=none); spf=softfail (mx1.freebsd.org: 2001:470:d5e7:1::1 is neither permitted nor denied by domain of kostikbel@gmail.com) smtp.mailfrom=kostikbel@gmail.com X-Spamd-Result: default: False [2.00 / 15.00]; TO_DN_SOME(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; HAS_XAW(0.00)[]; R_SPF_SOFTFAIL(0.00)[~all:c]; RCPT_COUNT_TWO(0.00)[2]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; RBL_DBL_DONT_QUERY_IPS(0.00)[2001:470:d5e7:1::1:from]; R_DKIM_NA(0.00)[]; ASN(0.00)[asn:6939, ipnet:2001:470::/32, country:US]; ARC_NA(0.00)[]; SUBJECT_ENDS_QUESTION(1.00)[]; FROM_HAS_DN(0.00)[]; NEURAL_SPAM_SHORT(1.00)[1.000]; NEURAL_HAM_LONG(-1.00)[-0.998]; TAGGED_RCPT(0.00)[]; MIME_GOOD(-0.10)[text/plain]; NEURAL_SPAM_MEDIUM(1.00)[1.000]; SPAMHAUS_ZRD(0.00)[2001:470:d5e7:1::1:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-hackers]; DMARC_POLICY_SOFTFAIL(0.10)[gmail.com : No valid SPF, No valid DKIM,none] X-ThisMailContainsUnwantedMimeParts: N On Sat, Jul 03, 2021 at 02:32:01AM +0300, Vitaliy Gusev wrote: > Hi, > > I came across not expected behaviour with madvise() in FreeBSD. > > Attached test program mmapfork does: mmap, fork, touch memory and then madvise(MADV_FREE). > > Expected behaviour - one process can allocate memory (lazy allocation) while system is freeing previously allocated memory for a second process. > > Current behaviour - system kills one process with message in dmesg: > > pid 31314 (mmapfork), jid 0, uid 1001, was killed: out of swap space > > Running this test in Linux or illumos shows expected behaviour with a little difference in illumos - it frees memory almost immediately, w/o needs lack of memory in a system. > > If use MADV_NOTNEED - no changes. > > If modify program and do not do fork(), but run two instances - that shows expected behaviour. > > To reproduce just disable swap, and run program with argument as 1/2 RAM on a system. For instance, command below will try run and use ~ 2GB area twice. > > [vetal@bsdev ~]$ ./mmapfork 2000 > > Testing program is attached. > > Note, during testing I disabled swap on all systems: Linux, illumos and FreeBSD. > > Does it mean madvise() doesn't work well in FreeBSD or test does something wrong? Your program does not exactly what you described above. There is a generic race to consume memory, and some specific details about madvise(2) on FreeBSD. From the code, you do: - mmap anonymous private region - fork - both child and parent start touching the mmaped region. Two processes race to consume 1/2 of RAM on your system. If one of them happen to execute faster then another, you do get to the case where one of them does madvise(). But it could be that processes execute in lockstep, and try to eat all the memory before going to madvise(). Did you excluded this case? Now, about the specific of madvise(MADV_FREE) on FreeBSD. Due to the way CoW is implemented with the shadow chain of objects, we cannot drop the top of the shadow chain, otherwise instead of returning zeroed pages next time, we would return content back in the time. It was relatively recent discovery, see bf5661f4a1af6931ec4b6, PR 240061. To explain it in simplified form, when there is potential old content under the CoW copy for the mapping, we cannot drop CoW-ed pages. This is the motivation why madvise(MADV_FREE) does nothing for your program. When you run two instances without fork, there is no previous content and no Cow, so madvise() can safely remove the pages from the object, and on the next access they are zero-filled. You can read more details in the referenced commit, as well as some musings about way to make it somewhat better. I must say, that trying to allocated 1/2 + 1/2 of RAM this way, on a system without swap, is the way to ask for troubles anyway.