From nobody Thu Apr 24 02:01:49 2025 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4ZjfNv2SD1z5tGJ3 for ; Thu, 24 Apr 2025 02:02:07 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic304-25.consmr.mail.gq1.yahoo.com (sonic304-25.consmr.mail.gq1.yahoo.com [98.137.68.206]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4ZjfNs0V0wz3Dn3 for ; Thu, 24 Apr 2025 02:02:05 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=UgdEVCTp; dmarc=pass (policy=reject) header.from=yahoo.com; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.68.206 as permitted sender) smtp.mailfrom=marklmi@yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1745460123; bh=JiXRX0Yg4R/6XJwmzSTaQUvcz80kRhTAwcPhQttWe4M=; h=From:Subject:Date:To:References:From:Subject:Reply-To; b=UgdEVCTpo36HvT1HCmiR8ttYLc933/CW4+antZd2Q8p+8CcUTpfxRZ3/N+1BiDs8hDtbqb18znMrUI4OhpEt9OfOJx2Lrh6QOYvZXbKh151OBv1UsDphNOKTHv7ScqRAWStyDY9TwwAWUVyZYevTzrkVARRfpoE/4grCvD21WjNoDzLkjdQMbysJlTRH0/Dnbqat46jD4tTzCnZQKwfUZk88ja9zlh0jqb9dgrf7fmib/XG/lh3JPiSHOLno/nxp7r1LTMDKRqY6VKLoxGIh7/O6YlvRrH+ACSzze8zF30ek1uLcheHbS/XJo+aAPQzf0PTC+S4qMdMZ49QJI/w8Iw== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1745460123; bh=U+fpCmwEpoGG6dROBCB2vNyrBQTVinK2qmQZ5EgoEU+=; h=X-Sonic-MF:From:Subject:Date:To:From:Subject; b=tMOjaAwT79IEdOwThN2Pn/tw/GDnx0eAfM6ihuDvFg0u6QC6eNv00nSiLV1ocGz7swh8hbZhQP5HPkeCdKDvVsf+ETqB6YCRfIWWcTfNdDv1eJm6LPQYwOJxWR3R17eDLV+AQm8YKHn/n9hWk8wbPDXOT7xJJ24Uxh+qQfx6gesIwvYk8CSyu7E5+jIRPHlGOf3H4gwKB/fdLIey5xM3i15q3zX7M+YxO2RGzPJXe8N4DyhhHRTArhRt8Al1rI/APzWbfvDM2UKVyltDTtluRkGPuoVq+ASThXKdv03MwNE5KtT76B60ypiT8pCSRNaa4NbPcN+qNLydZTtuSUgwMQ== X-YMail-OSG: 5ta1tm4VM1k51oLn2Xo4NISb6etWIXELi6ihTkUjXQeg3oxpxU7aXthEMai7Iuh QNO_IFzLclGjyuSVJAEW5R4C3pCP0eOTN6RYMz9fUTpN7LUq0jK609xSxuf8hwh1AVVWZyZKym7l 6e8k.5IjT9BUv.YeYRuwGQEcE_ntyNxXmVSHrM1vhatC0IsnG_BGJ6rqiKGzCT2HBzhS077cWRG8 ufTWXCps27atIOtcAL5faHxb.fxv3f2rY_NEwJvCeVtedcBqAN415fp_5y8W8tb3JjNmq5zvfnic btlpEO3FA3JpGvguFguhuB7rivE9WS.R4Ua3JdFhOTic83hJVA5js2KCQmVLPmgOTs2XKPQomOJH Qj4JduBLy_NCTeA8m2KyZVCT1pYG1vSDB8_9BSEaM94_of1MLV_sqGqKt9mXz.niTS.A3XA_ANG_ eUtY2CnL.oUfQnOlHvxbuO5KiIscxmDxYPX44.dAIovI0RppUohHc1wCj5gZVlND8C4pw_SQtw7w 3HZB0pZNJqJYP5GJe_D8GLe..xgjpQRwrMrSvo0RXNECVxJUeYu.633reTSsIhIWv87_hWpVLgSC 81DWuB2u82hx.R796HFf770ysjFSHOnOKRLTQYtXij9QFjwq1AR_7HAPYNFo_i4NWfw67CvbAhWc lpzHWwNsw03MgkMbj8pqbf34gk.escS6_XFmusqb9E5l.9xM3nVR0KfrOjdWR7qZePYygB6GuYXW tBU_e7F.mTnxdGloKr.mxpmbPr_Mwj4yxXoeSVmp_JqKgJkfyjzutkM5Wt1Wp0JlzqRBhYaN1Wrq 1wlKJvaEtidaD8NQ37a3HaR7YPhDpTcHa4AvMgQYCiwDsiJ7hPrc.HjN8sUcHYbMuYguPnwEWIlp sCtfv_0E6zPb48uzQZIUaIDPQre66Da895f8LLSg7Ic0mRKzFc_tE3jRk5vPpf2QEPpcKkqk3xNi eiihkUV4RYIiftyIPiV4kcBgW12Nl6_wq.cLyNxmy.KYoXAbOTZNLzIBV8Q0rvSrtEkvxPtWZPVp dZYEhobpttEq.9m8HC6aAyqN3IYiv6H9ZjIKq3kLzQi40IGOcwcB6y0SOtf9_ZGqzBNpqzFbD3ie X0k344t8rrXAQrqgqxQcZJeCQh_rfa5TrIHfxY4nQeB6g3vcFTK34ZLFiHFvVWxtVikzrNV6kaee RCaUOGk6hpSVH1f1Oj0Rd9PjfDCMTQm4lYuCWjSyeV3s5kHT34EyZKEYNUFWHERakrsPyiDm5LGK emAXFelH0j.2ZRKMxdVjZUj.miqVEHgd5Y8x4gLjpUFG16zFmKXrihUAguzEs6zgVCdt1mt3WSGS S8PlF0sPoF_oR3C4pFyD0G84CW0ht44cJVdRL_SHqB6h6xDjUZ2l3fzYmoel1aaArZdgxw6UXjaZ .b0tIYXtwVYxWgRGqUBf6nggDGuHjzvBnrKx91LOnnksBJck2BGd7KuCLgcXAVVgz9OnqxHTLOV3 1F4P9lJ_AC3ea1k4K9SJvSVMpuzixfQILbMI0ihT_ZLgWBGzYqTsZEdTxMhAfzh7mAfqVJ8EcZ0x WorcC21Y.5V4PkcSl2PloLTAddfGhXrrnl8a8pz2gkhsLRlBtY5RLD48H2w042WYdfmVjiko_R1R .2MdwksfEr2GRbKswYyiaEReEiSXKFPnt6rIciTa.jX.04Fl8KtEtdxypBNgQsfikdtlHn2yJL2J Yzg7VUcfprieMBPE_TTSik_DMrs7jmdKVKC7vrK9eB7zaXrJr27dvX.5ZqYvRfvcYykAuwFzf4KW RyLpbnoi.4aHn.cbGhbdsValHThyLNyVLxNvjaRLUAgE1fsrOXbY9rrgc_YGqI4fmUgPGbzTjhts EsPjLNUbOea_hQ2N4.6TUMlKGqcI7kEN9PZ3ilCjLfOQhaQpcgUAu3n7ztiQGp4lFU2y..o.A2iw e_k6m81QdbJL_sLGFeUkcwPiOfPQFCfBbjQ6.0PuiXoeWJWRimSQE4K1LQioQNxgDfgJdEtCupxj Rzu_xe85QaujDBVct9fC8W6LtghWLO2mfPcx.9uYk__rFLAyUg1AwNBBXvcFIbwZHth23IUmbIOE ao9_iUTxDmFWb_I.wB0OwsZw7dT.feiIlPjhTnqIF.blAT8sgExzvmtlG5AgT7lc6N60XHO5P52Z YxWKeBBJjKejXokqKLw_W0j4M6JMnFDksh6zWZn97cbGl5f0okOa_yK0UCruPPIL48lkI5RGs6ch I7Rz28r6bQ1HNSJnJRd_zULePNNDm2FZfM9PnM2zyuNieDJMlIRf6i7OgMbHG4wIeO36uTfcA0Vl u X-Sonic-MF: X-Sonic-ID: 3874b62c-e8cd-4847-92fe-b01f647d500f Received: from sonic.gate.mail.ne1.yahoo.com by sonic304.consmr.mail.gq1.yahoo.com with HTTP; Thu, 24 Apr 2025 02:02:03 +0000 Received: by hermes--production-gq1-74d64bb7d7-s6s6l (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 8ba04b0282b67a8ae8e8774013cfcb8d; Thu, 24 Apr 2025 02:02:00 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@FreeBSD.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3826.500.181.1.5\)) Subject: Re: zfs (?) issues? Message-Id: <4330CE6C-510E-426E-A53E-AF09100ACCF0@yahoo.com> Date: Wed, 23 Apr 2025 19:01:49 -0700 To: freebsd-current-freebsd-org111@ketas.si.pri.ee, FreeBSD Current X-Mailer: Apple Mail (2.3826.500.181.1.5) References: <4330CE6C-510E-426E-A53E-AF09100ACCF0.ref@yahoo.com> X-Spamd-Result: default: False [-1.49 / 15.00]; SUBJECT_ENDS_QUESTION(1.00)[]; NEURAL_HAM_MEDIUM(-0.89)[-0.891]; NEURAL_HAM_SHORT(-0.70)[-0.698]; MV_CASE(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; NEURAL_HAM_LONG(-0.40)[-0.396]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; MIME_TRACE(0.00)[0:+]; TO_DN_SOME(0.00)[]; ARC_NA(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; FREEMAIL_FROM(0.00)[yahoo.com]; FROM_HAS_DN(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.68.206:from]; MID_RHS_MATCH_FROM(0.00)[]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; RCVD_IN_DNSWL_NONE(0.00)[98.137.68.206:from] X-Rspamd-Queue-Id: 4ZjfNs0V0wz3Dn3 X-Spamd-Bar: - Sulev-Madis Silber = wrote on Date: Wed, 23 Apr 2025 06:55:06 UTC : >=20 > On April 23, 2025 8:34:36 AM GMT+03:00, Mark Millard = wrote: > >On Apr 22, 2025, at 21:59, Mark Millard wrote: > > > >> Sulev-Madis Silber = wrote on > >> Date: Wed, 23 Apr 2025 04:31:41 UTC : > >>=20 > >> . . . > . . . >=20 > >>=20 > >>=20 > >> If FreeBSD 13.4 can still swapping out process kernel > >> stacks, you may want the likes of /etc/sysctl.conf > >> to have: > >>=20 > >> # > >> # Together this pair avoids swapping out the process kernel stacks. > >> # This avoids processes for interacting with the system from being > >> # hung-up by such. > >> vm.swap_enabled=3D0 > >> vm.swap_idle_enabled=3D0 >=20 > would it be related here? The settings prevent one way of ending up with a normal way to interact with the system failing to do so. There are other means of having such failures. The settings should not lead to problems of themselves, even if the conditions they would avoid never occur for other reasons: The settings should be safe. > >>=20 > >> (I've no clue that that is why you lost control but > >> it may be a possibility.) > >>=20 > >> (main [FreeBSD 15] no longer does such swapping out of any > >> process kernel stacks and the 2 settings have been removed.) > > > >Are you using a file system based SWAP space? Vs. a > >Partition or Slice based SWAP space? >=20 > just 2 gpt partitions Good. Are they encrypted (GELI)? Having the SWAP I/O also involve such processing by FreeBSD does add to the risk of memory management related deadlocks, as I understand anyway. Can testing without the use of GELI to see if that makes a difference be done (if GELI is in use for the SWAP space)? > > > >Quoting: > > > >https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D206048#c7 > > > >on why it should be Partition/Slice based: > > > >QUOTE > >On 2017-Feb-13, at 7:20 PM, Konstantin Belousov wrote > >on the freebsd-arm list: > > > >. . . > > > >swapfile write requires the write request to come through the = filesystem > >write path, which might require the filesystem to allocate more = memory > >and read some data. E.g. it is known that any ZFS write request > >allocates memory, and that write request on large UFS file might = require > >allocating and reading an indirect block buffer to find the block = number > >of the written block, if the indirect block was not yet read. > > > >As result, swapfile swapping is more prone to the trivial and = unavoidable > >deadlocks where the pagedaemon thread, which produces free memory, = needs > >more free memory to make a progress. Swap write on the raw partition = over > >simple partitioning scheme directly over HBA are usually safe, while = e.g. > >zfs over geli over umass is the worst construction. > >END QUOTE > > > >Note the references to ZFS and GELI. Your forum notes reference such. > > > > > >A separate tunable: in case "was killed: failed to reclaim memory" > >is involved but not reported/recorded: in /boot/loader.conf > > > ># > ># Delay when persistent low free RAM leads to > ># Out Of Memory killing of processes: > >vm.pageout_oom_seq=3D120 >=20 > helpful here? This tunable contributes to how long before one type of OOM activity starts. Larger values lead to larger times. The default is 12. More than 120 can be used. The figure is good for allowing known temporary conditions time to complete so that they do not lead to OOM activity. It used to be that FreeBSD reported "out of swap" inaccurately for some other conditions where it now produces more accurate/specific messages. There is still a "out of swap" catchall that can be inaccurate. The setting should not lead to problems of itself. >=20 > > > > > > > >Separate question: why did some forum top runs show > >qemu-system-arm threads? That could be a significant > >competition for RAM+SWAP. >=20 > it's small vm yes. but it behaves well. part of it gets swapped out. = then back in, so on Just because something appears to behave well considered in isolation, does not classify if it is contributing to system level problems. In a small RAM context, top showed 600+ MiBytes of resident (probably active?) memory for qemu-system-arm, if I remember right. In other words, possibly 600+ MiBytes not normally available to to other activities that compete for RAM, at elast for a time. > it's all funny, i might be using more things in that machine than i = have ram for. but it just gets swapped out. swapping it back in causes = delay in that thing that needs that. like always It also leads to delays in whatever had pages written out to make room for what was to be paged in, time on overhead instead of on more-directly productive work. > i still think git does something here. I've no way to collect evidence relative to such issues. As stands, I've no way to use these sorts of notes: I cannot even produce a setup known to be analogous and then test it for if I end up with the problem reproduced. (Apparently, that context would include use of qemu-system-arm in order to be analogous.) Your context may be too complicated to reasonably set up something analogous. > if i don't touch git, everything is absolutely perfect. why git? if i = restrain it from config, it seems to not cause issues. except maybe they = are hidden now >=20 > whatever happens, it happens in memory that is impossible to swap out = or is not preferred to swap out >=20 > git is not swapping and then being killed. that's not how it will = affect the system. i think i helps zfs to take all memory and either = throw unusual errors, like that single case. or just cause total = exhaustion >=20 > what's fun is it never appears elsewhere. everything else seems to = work as one would expect it to work when low rammed I've no idea how to put that description to any use or to test confirm or deny any of the hypotheses involved. > > > > > >=3D=3D=3D > >Mark Millard > >marklmi at yahoo.com > > > > >=20 > p.s: >=20 > i also found >=20 > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D231457 >=20 > where people experince (maybe) similar issues many years ago >=20 > is it active issue still? FreeBSD 11.* was a fair time ago and various issues have been addressed since then. It is not clear how that bugzilla could be of use. =3D=3D=3D Mark Millard marklmi at yahoo.com