From nobody Sat Jun 28 15:35:17 2025 X-Original-To: current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4bTxMZ56LBz608jF for ; Sat, 28 Jun 2025 15:35:38 +0000 (UTC) (envelope-from zlei@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R11" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4bTxMZ4Q76z43JY; Sat, 28 Jun 2025 15:35:38 +0000 (UTC) (envelope-from zlei@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1751124938; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=H1kRygCAelwZENWmHthKScaPDVM0bqFmu9jSgp9ufWA=; b=vtpQGhoRFlGCiGfbSae02wqrMyO1CGfBUZURAe5xRKIdqQlCCFrFCp6btbyKq2sZMAZUF3 r01Kcejr2pbzrw9muzpDqMe5RAhqS8voOFvNMkdmPD/eILqOozmGBjiKChaMxvUWaRC0jS rDykBq97u7WMdh3LqW0vqR+WpanPBQswKkb8y+x0oFCOH35zZ+T6FvSgs1nO93EBBzocrz m7358wCzXyaTqLEaRkUUCjafJTbN5HfDjKkXVzpuJAoAvVLzOsbvm0DaOqo/fNNrW1yOYQ UAhW3kFFaSIVl88nBbtPadRRJ7QSomhb44MOQK0L9x+vdyq/uj//AcxNCY6NrQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1751124938; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=H1kRygCAelwZENWmHthKScaPDVM0bqFmu9jSgp9ufWA=; b=PR/vVIMFlFtdXP3fQ0w9TJvUQSPSUpU2Ag85L1NXuUPPNXfW/XanGQCQr56QMIy+9vormf RWRKBXRGdMhRatR9ho6eHTXxVcUPwJaYinnI7zeUCh9M4YoOPHDHJwmOjqus6xMu/J3k2Z fldEcQENwTuvI/TE+8jWGRDCDNZYCx9yrY+TcXE7nqeTRZKbIgULkv9BdxQucR1u4p0q4Y ZeGYYR7pZWDFPnwTuAUqq0tBAFEorV0bLn3Kx8+Q3UgsNTRyt+QpJYOec/f4g3rsToHOHC CUhSCoJdWt311lNgMlncGynqQaG8o5ebrrSSO+mx3msZudPFrlayW9aHYCVAcg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1751124938; a=rsa-sha256; cv=none; b=AJ+SH9QxKMkZVnwkvn6eTDIBWmuhxK5HMIJDbKzS8hbCoeT2e76BoDeXAA+JTNR8Oq5quh XHHPBtpU6C2Gp702umZpOAR7L2xDPOM/nJ77TfiGqxi+FregThn/LWKm6pKO7326zpT4ZL YCSRAoRme6LbMPYYbtuDnobIX4yUIoH/tA348A3+CdJdkmjj/l5joCXPbXY1G1Si2kR66d sToVY2liRDnOdC5FP8cea7XA7VAqaeSdmlvDvDPYKI4yZPQogOnNPQFg/xhVvQFWL/Jhbf qWp6YEKbIMdEcrE7JzhhR9llttQqR7lVg2DBD4eqXSPMW0IX+ppS0dRRtF3yRg== Received: from smtpclient.apple (unknown [IPv6:2001:19f0:6001:9db:98f0:9fe0:3545:10]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) (Authenticated sender: zlei/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 4bTxMT443Qzvng; Sat, 28 Jun 2025 15:35:33 +0000 (UTC) (envelope-from zlei@FreeBSD.org) From: Zhenlei Huang Message-Id: <907D042E-AE8A-4818-A807-AD45F36354FD@FreeBSD.org> Content-Type: multipart/alternative; boundary="Apple-Mail=_BD4DCEE9-A403-4921-8050-56B9168FD726" List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@FreeBSD.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3696.120.41.1.10\)) Subject: Re: regression: memory issues on main/arm64 over sched/runq changes Date: Sat, 28 Jun 2025 23:35:17 +0800 In-Reply-To: <23n1773o-10o2-5p5o-25s4-r623rnn44649@yvfgf.mnoonqbm.arg> Cc: FreeBSD Current , Olivier Certner , John Baldwin To: "Bjoern A. Zeeb" References: <43005447-2rq0-6nn2-pnr5-4939s112npr4@yvfgf.mnoonqbm.arg> <0A01B9F5-C49C-41D8-BAB7-4378DEDBF647@FreeBSD.org> <28o26o81-so5r-qq79-6q6n-0q6746o7oo79@yvfgf.mnoonqbm.arg> <6A003013-415A-4594-AB04-AF5A9B2D660D@FreeBSD.org> <23n1773o-10o2-5p5o-25s4-r623rnn44649@yvfgf.mnoonqbm.arg> X-Mailer: Apple Mail (2.3696.120.41.1.10) --Apple-Mail=_BD4DCEE9-A403-4921-8050-56B9168FD726 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii > On Jun 28, 2025, at 4:12 AM, Bjoern A. Zeeb = wrote: >=20 > On Sat, 28 Jun 2025, Zhenlei Huang wrote: >=20 >>=20 >>=20 >>> On Jun 27, 2025, at 11:02 PM, Bjoern A. Zeeb = wrote: >>>=20 >>> On Wed, 25 Jun 2025, Zhenlei Huang wrote: >>>=20 >>> Hi, >>>=20 >>> I appplied olce's change from the review but it didn't make a = difference >>> on my arm64 and now on a tree with local changes (wifi bits, user = sapce >>> bits, etc). >>>=20 >>> Now I netbooted that tree on X86 hardware (an old Lenovo Laptop) and = ran >>> into something else (the same tree boots in a bhyve instance on a >>> different machine from a local disk image). >>>=20 >>> At the end of if_addgroup() I had added the following for local >>> debugging (really crude sorry): >>>=20 >>> ... >>>=20 >>> + atomic_thread_fence_seq_cst(); >>> IF_ADDR_WLOCK(ifp); >>> CK_STAILQ_INSERT_TAIL(&ifg->ifg_members, ifgm, ifgm_next); >>> CK_STAILQ_INSERT_TAIL(&ifp->if_groups, ifgl, ifgl_next); >>> IF_ADDR_WUNLOCK(ifp); >>>=20 >>> IFNET_WUNLOCK(); // excl unlock >>>=20 >>> if (new) >>> EVENTHANDLER_INVOKE(group_attach_event, ifg); >>> EVENTHANDLER_INVOKE(group_change_event, groupname); >>>=20 >>> + IFNET_RLOCK(); // shared, panic >>> + CK_STAILQ_FOREACH(ifgl, &ifp->if_groups, ifgl_next) { >>> + if (bz_debug_groups) if_printf(ifp, = "XXXXXXXXXXXXXXXXXXXXXXXXXXX-BZ %s:%d: ifgl %p, ifgl_group %p, ifg_group = %p\n", __func__, __LINE__, ifgl, (ifgl !=3D NULL) ? ifgl->ifgl_group : = NULL, (ifgl !=3D NULL && ifgl->ifgl_group !=3D NULL) ? = ifgl->ifgl_group->ifg_group : NULL); >>> + } >>> + IFNET_RUNLOCK(); >>> + >>> return (0); >>> } >>>=20 >>>=20 >>>=20 >>> You see the anotation //shared ? >>>=20 >>> I got a panic: excl->share with that. >>=20 >> Well, I applied identical patch with you and I can repeat that panic, = but my screen freezes and the top most stack is >=20 > I took a video of the boot at 60fps so I could "scroll" a bit = backwards. Good idea! >=20 >> ``` >> _sx_slock_int() at _sx_slock_int+0x64/frame 0xff.... >> if_addgroup() at ..... >> .... >> device_attach() at ... >> ... >> root_bus_configure() at ... >> configure() at ... >> mi_startup() at .. >> ``` >>=20 >> I've no idea what's wrong. =46rom the disassembly it appears the = panic happens just after witness_checkorder . >=20 > That is interesting. So it's not just me. >=20 > Did you do a netboot or from disk? I boot from disk. Updates on this locking issue, I think I finally figured out why. More stack trace from my video: ``` shared lock of (sx) ifnet_sx = @/usr/home/zlei/freebsd-src/sys/net/if.c:1467 while exclusively locked from = /usr/home/zlei/freebsd-src/sys/net/if.c:1416 panic: excl->share ... witness_checkorder() at ... _sx_slock_int() at _sx_slock_int+0x64/frame .... if_addgroup() at ... if_attach_internal() at ... ether_ifattach() at ... iflib_device_register() at ... iflib_device_attach() at ... device_attach() at ... ... root_bus_configure() at ... configure() at ... mi_startup() at ... ``` The ifnet_sx has flag bit SX_RECURSE then it can be recursively locked. iflib_device_register() acquired ifnet_sx exclusively and then calls = ethernet_ifattach() which will then calls if_addgroup(). It is = prohibited to re-acquire the same lock shared so the witness blames. I think the witness should show the first file location of the = exclusively lock, i.e. sys/net/iflib.c rather than the sys/net/if.c:1416 = . So that it is more straight forward to figure out how that happens. CC = John to see if that can be improved. OK, lets turn it over and focus on the original synchronization issue :) = .=20 Best regards, Zhenlei >=20 > --=20 > Bjoern A. Zeeb = r15:7 --Apple-Mail=_BD4DCEE9-A403-4921-8050-56B9168FD726 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii

On Jun 28, 2025, at 4:12 AM, Bjoern A. Zeeb <bzeeb-lists@lists.zabbadoz.net> wrote:

On Sat, 28 Jun 2025, Zhenlei Huang wrote:



On Jun = 27, 2025, at 11:02 PM, Bjoern A. Zeeb <bzeeb-lists@lists.zabbadoz.net> wrote:
On Wed, 25 Jun 2025, Zhenlei Huang wrote:

Hi,

I appplied olce's change = from the review but it didn't make a difference
on my = arm64 and now on a tree with local changes (wifi bits, user sapce
bits, etc).

Now I netbooted that = tree on X86 hardware (an old Lenovo Laptop) and ran
into = something else (the same tree boots in a bhyve instance on a
different machine from a local disk image).

At the end of if_addgroup() I had added the following for = local
debugging (really crude sorry):

...

+ =       atomic_thread_fence_seq_cst();
      IF_ADDR_WLOCK(ifp);
      CK_STAILQ_INSERT_TAIL(&= ifg->ifg_members, ifgm, ifgm_next);
      CK_STAILQ_INSERT_TAIL(&= ifp->if_groups, ifgl, ifgl_next);
      IF_ADDR_WUNLOCK(ifp);

      IFNET_WUNLOCK(); // excl = unlock

      if= (new)
          &nb= sp;   EVENTHANDLER_INVOKE(group_attach_event, ifg);
      EVENTHANDLER_INVOKE(group_c= hange_event, groupname);

+ =       IFNET_RLOCK();  // shared, = panic
+ =       CK_STAILQ_FOREACH(ifgl, = &ifp->if_groups, ifgl_next) {
+ =             &n= bsp; if (bz_debug_groups) if_printf(ifp, = "XXXXXXXXXXXXXXXXXXXXXXXXXXX-BZ %s:%d: ifgl %p, ifgl_group %p, ifg_group = %p\n", __func__, __LINE__, ifgl, (ifgl !=3D NULL) ? ifgl->ifgl_group = : NULL, (ifgl !=3D NULL && ifgl->ifgl_group !=3D NULL) ? = ifgl->ifgl_group->ifg_group : NULL);
+ =       }
+ =       IFNET_RUNLOCK();
+
      return (0);
}



You see the anotation //shared ?

I= got a panic: excl->share with that.

Well, I applied identical patch with you and I can repeat = that panic, but my screen freezes and the top most stack is

I took a video of the boot at 60fps so I could "scroll" a bit = backwards.

Good = idea!


```
_sx_slock_int() at _sx_slock_int+0x64/frame 0xff....
if_addgroup() at .....
....
device_attach() at ...
...
root_bus_configure() at ...
configure() at = ...
mi_startup() at ..
```

I've no idea what's wrong. =46rom the disassembly it appears = the panic happens just after witness_checkorder .

That is interesting.  So it's not just me.

Did you do a = netboot or from disk?

I boot from disk.

Updates on this locking issue,

I think I finally figured out why. More stack = trace from my video:

```
shared lock of (sx) ifnet_sx = @/usr/home/zlei/freebsd-src/sys/net/if.c:1467
while = exclusively locked from = /usr/home/zlei/freebsd-src/sys/net/if.c:1416
panic: = excl->share
...
witness_checkorder() at = ...
_sx_slock_int() at _sx_slock_int+0x64/frame = ....
if_addgroup() at ...
if_attach_internal() at = ...
ether_ifattach() at ...
iflib_device_register() = at ...
iflib_device_attach() at ...
device_attach() = at ...
...
root_bus_configure() at = ...
configure() at ...
mi_startup() at = ...
```

The ifnet_sx has = flag bit SX_RECURSE then it can be recursively = locked.

iflib_device_register() = acquired ifnet_sx exclusively and then calls ethernet_ifattach() which = will then calls if_addgroup(). It is prohibited to re-acquire the same = lock shared so the witness blames.

I = think the witness should show the first file location of the exclusively = lock, i.e. sys/net/iflib.c rather than the sys/net/if.c:1416 . So that = it is more straight forward to figure out how that happens. CC John to = see if that can be improved.

OK, = lets turn it over and focus on the original synchronization issue :) = . 

Best regards,
Zhenlei


-- Bjoern A. = Zeeb =             &n= bsp;           &nbs= p;            =             &n= bsp;  r15:7



= --Apple-Mail=_BD4DCEE9-A403-4921-8050-56B9168FD726--