From nobody Fri Jun 27 15:02:35 2025 X-Original-To: current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4bTK464B30z5yvNQ for ; Fri, 27 Jun 2025 15:20:06 +0000 (UTC) (envelope-from bzeeb-lists@lists.zabbadoz.net) Received: from mx-01.divo.sbone.de (mx-01.divo.sbone.de [IPv6:2003:a:140a:2200:6:594:fffe:19]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature ECDSA (prime256v1) client-digest SHA256) (Client CN "mx-01.divo.sbone.de", Issuer "E5" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4bTK436TGjz3fL3; Fri, 27 Jun 2025 15:20:03 +0000 (UTC) (envelope-from bzeeb-lists@lists.zabbadoz.net) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=zabbadoz.net header.s=20240622 header.b=bV+D4Z0g; spf=pass (mx1.freebsd.org: domain of bzeeb-lists@lists.zabbadoz.net designates 2003:a:140a:2200:6:594:fffe:19 as permitted sender) smtp.mailfrom=bzeeb-lists@lists.zabbadoz.net; dmarc=pass (policy=none) header.from=zabbadoz.net Received: from mail.sbone.de (mail.sbone.de [IPv6:fde9:577b:c1a9:4902:0:7404:2:1025]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (prime256v1) server-digest SHA256) (No client certificate requested) by mx-01.divo.sbone.de (Postfix) with ESMTPS id 2E8FAA64805; Fri, 27 Jun 2025 15:19:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=zabbadoz.net; s=20240622; t=1751037592; bh=O+yYmGz1zemn057KnmFFPYIKWtFL3JzuJN5sQl3rRvI=; h=Date:From:To:cc:Subject:In-Reply-To:References; b=bV+D4Z0gUo3ziOWTcLRiMDfhSZNgSdPsRaQwg4by2oIIepG2jgTLoETLmVfsr+DTt 6dtG8eIBCmCRqQXnYHqwDk/wxZ4WVNUgMI+qlHubgs0mw2t62roC6XoQAaNmjINEoF DYuaf9QR3B2CxMXwIGzn8nWSDySPYrt6cV1jU8ti72Txh9ytdP+XsiZDopcYLfOq9q tO72D5bCQDQThslEgtB4sgocSZ4Ibz5wDlQnahjVXe03UlCOqUgvldonKs7H2M7XAa YavCkDOgld9lS1Emfll8pTCIQy5yHtb/XyqRYnK5PWGrHAh9H4aT7DPfVlgOaBzn2/ Q12DSMJoDbnL8hCEjXp0+UUhNHlV93EZoLK/gfQA08g6zMgN2se4Ku4+DH00rftkdd DyVcsUGm3oemu2GQUoWntE4Xb5z1g1dSa9UhWxxdtsHhEdfXuulX5RZQuTvT3+xvio 7lUvOIt2UfSGFnsQAvSH2y92sArtFLXdl6qm/iZeJPsl6HJjjImLo7uJyjsWyvpvFb Eua3kFVYJ/ileVcyFtpJB8gP8DkEyCLOqUWWw0FEdFc3AB3xSQQi56BIwLmTRySiP6 UQR5OD9wTU2keOIPmPAaHRuNe0z6DhgLdHQLjWXk8IeWDPF5p3DjhFx0OniYxtNvAF fppTR35Hjk9Hsqic82rtfjaA= Received: from content-filter.t4-02.sbone.de (content-filter.t4-02.sbone.de [IPv6:fde9:577b:c1a9:4902:0:7404:2:2742]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mail.sbone.de (Postfix) with ESMTPS id 4BDAD2D029E0; Fri, 27 Jun 2025 15:02:56 +0000 (UTC) X-Virus-Scanned: amavisd-new at sbone.de Received: from mail.sbone.de ([IPv6:fde9:577b:c1a9:4902:0:7404:2:1025]) by content-filter.t4-02.sbone.de (content-filter.t4-02.sbone.de [IPv6:fde9:577b:c1a9:4902:0:7404:2:2742]) (amavisd-new, port 10024) with ESMTP id SjV_NxaQiesl; Fri, 27 Jun 2025 15:02:43 +0000 (UTC) Received: from strong-rtwn0.sbone.de (strong-rtwn0.sbone.de [IPv6:fde9:577b:c1a9:4902:3e64:cfff:fe55:bc80]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mail.sbone.de (Postfix) with ESMTPSA id 19AA52D029D8; Fri, 27 Jun 2025 15:02:36 +0000 (UTC) Date: Fri, 27 Jun 2025 15:02:35 +0000 (UTC) From: "Bjoern A. Zeeb" To: Zhenlei Huang cc: FreeBSD Current , Olivier Certner Subject: Re: regression: memory issues on main/arm64 over sched/runq changes In-Reply-To: <0A01B9F5-C49C-41D8-BAB7-4378DEDBF647@FreeBSD.org> Message-ID: <28o26o81-so5r-qq79-6q6n-0q6746o7oo79@yvfgf.mnoonqbm.arg> References: <43005447-2rq0-6nn2-pnr5-4939s112npr4@yvfgf.mnoonqbm.arg> <0A01B9F5-C49C-41D8-BAB7-4378DEDBF647@FreeBSD.org> X-OpenPGP-Key-Id: 0x14003F198FEFA3E77207EE8D2B58B8F83CCF1842 List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed X-Spamd-Result: default: False [-3.41 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-0.80)[-0.802]; NEURAL_HAM_SHORT(-0.61)[-0.611]; DMARC_POLICY_ALLOW(-0.50)[zabbadoz.net,none]; R_DKIM_ALLOW(-0.20)[zabbadoz.net:s=20240622]; R_SPF_ALLOW(-0.20)[+ip6:2003:a:140a:2200:6:594:fffe:19]; MIME_GOOD(-0.10)[text/plain]; RCVD_VIA_SMTP_AUTH(0.00)[]; MISSING_XM_UA(0.00)[]; ARC_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:3320, ipnet:2003::/19, country:DE]; MLMMJ_DEST(0.00)[current@freebsd.org]; RCVD_COUNT_THREE(0.00)[4]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCVD_TLS_LAST(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[zabbadoz.net:+] X-Rspamd-Queue-Id: 4bTK436TGjz3fL3 X-Spamd-Bar: --- On Wed, 25 Jun 2025, Zhenlei Huang wrote: Hi, I appplied olce's change from the review but it didn't make a difference on my arm64 and now on a tree with local changes (wifi bits, user sapce bits, etc). Now I netbooted that tree on X86 hardware (an old Lenovo Laptop) and ran into something else (the same tree boots in a bhyve instance on a different machine from a local disk image). At the end of if_addgroup() I had added the following for local debugging (really crude sorry): ... + atomic_thread_fence_seq_cst(); IF_ADDR_WLOCK(ifp); CK_STAILQ_INSERT_TAIL(&ifg->ifg_members, ifgm, ifgm_next); CK_STAILQ_INSERT_TAIL(&ifp->if_groups, ifgl, ifgl_next); IF_ADDR_WUNLOCK(ifp); IFNET_WUNLOCK(); // excl unlock if (new) EVENTHANDLER_INVOKE(group_attach_event, ifg); EVENTHANDLER_INVOKE(group_change_event, groupname); + IFNET_RLOCK(); // shared, panic + CK_STAILQ_FOREACH(ifgl, &ifp->if_groups, ifgl_next) { + if (bz_debug_groups) if_printf(ifp, "XXXXXXXXXXXXXXXXXXXXXXXXXXX-BZ %s:%d: ifgl %p, ifgl_group %p, ifg_group %p\n", __func__, __LINE__, ifgl, (ifgl != NULL) ? ifgl->ifgl_group : NULL, (ifgl != NULL && ifgl->ifgl_group != NULL) ? ifgl->ifgl_group->ifg_group : NULL); + } + IFNET_RUNLOCK(); + return (0); } You see the anotation //shared ? I got a panic: excl->share with that. The excl. is the IFNET_WLOCK(); // excl at the top of the function after the groupname check. But that gets unlocked before the event handler above so how can this happen? Sadly I cannot even dump or anything as the keyboard is as dead as the rest of the laptop. Have to power cycle it hard. Apart from the debugging I added I have no local changes in sys/net in that tree. sys/kern seems to have no relevant changes either (added a bus func, toggle link_elf_leak_locals default, and a printf got an extra argument to print %d error when modules fail to load). I'll try a plain main (hopefully tonight) on that machine too but I am really at a loss here now that it's also happening on X86 and only for me and always around the same code there... I'll also try to boot this tree from a USB pen drive or something; not that my problem comes in from netbooing... I'll keep you posted... /bz -- Bjoern A. Zeeb r15:7