[Bug 299220] Linuxulator: futex wakeup from native process not delivered to Linux waiter on shared memory (xshmfence), Linux GLX apps hang under Xwayland

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=299220

            Bug ID: 299220
           Summary: Linuxulator: futex wakeup from native process not
                    delivered to Linux waiter on shared memory
                    (xshmfence), Linux GLX apps hang under Xwayland
           Product: Base System
           Version: 15.1-RELEASE
          Hardware: amd64
                OS: Any
            Status: New
          Severity: Affects Only Me
          Priority: ---
         Component: kern
          Assignee: bugs@FreeBSD.org
          Reporter: kilikmatej@protonmail.com

Hello,

I'm debugging my installation and trying to get 3D hardware acceleration in
linuxulator led me to this error. 
Every Linux GLX application that presents to a window hangs on its first frame.
Example: glxinfo -B from the Ubuntu tree hangs right after "Using DRI3 for
screen 0". 
The GPU itself works in linuxulator: EGL surfaceless and device platforms
report radeonsi gfx1200 with full acceleration. Only presenting through DRI3 to
native Xwayland hangs.

Environment:
  FreeBSD 15.1-RELEASE amd64, drm-612-kmod 6.12.85
  AMD Navi 44 (1002:7590, gfx1200)
  Wayland compositor (mango) + native FreeBSD Xwayland <VERSION>
  Linux userland: Ubuntu 24.04 (Mesa 25.2.8, LLVM 20.1.2, libxshmfence 1.3)
  linsysfs with render node support (main 102adf88e6e8 backported, see bug
278363)

I am not going to pretend that I fully understand this error (while I have CS
education, I am nowhere near a kernel source hacker), in my own words it seems
that the program waits on a wake up call, which never comes, so a workaround
was tried to check the state every millisecond, which works. This is an
analysis and workaround I made with claude, so yeah, as I said, I'm not gonna
pretend I know what exactly is happening.
All the steps were done on my hardware by me, I changed the source in order to
see if it becomes unstuck. 
I don't know what is your stance on AI reports, but this is claude's analysis:
-------------------------------------------------------
Analysis (ktrace/kdump of the Linux glxinfo):
The client creates an xshmfence (memfd_create, ftruncate to 4 bytes, mmap),
creates amdgpu buffers, exports a dma-buf and passes buffer and fence fds to
Xwayland via sendmsg. All calls succeed. Then the main thread waits on the
fence page:

  CALL  linux_sys_futex(0x801091000,0,0xffffffff,0,0,0)

a shared FUTEX_WAIT (op 0, no timeout) on the xshmfence address, waiting while
the value is still -1 (untriggered). It never returns. procstat -kk shows every
thread in linux_sys_futex / umtxq_sleep; no thread is blocked in the DRM
driver.

The fence is triggered by Xwayland, a native FreeBSD process, which wakes
waiters through native umtx on the same shared page. The wakeup does not reach
the waiter in the Linux futex emulation. Possibly the Linux futex code and
native umtx use different key types for shared memory, so they never match.

Workaround that confirms the diagnosis:
Rebuilt Ubuntu's libxshmfence with a 1 ms timeout on the futex wait, retrying
on timeout:

--- src/xshmfence_futex.h (Linux branch)
 static inline int futex_wait(int32_t *addr, int32_t value) {
-       return sys_futex(addr, FUTEX_WAIT, value, NULL, NULL, 0);
+       struct timespec ts = { 0, 1000000 };
+       return sys_futex(addr, FUTEX_WAIT, value, &ts, NULL, 0);
 }
--- src/xshmfence_futex.c, xshmfence_await()
-                       if (errno != EWOULDBLOCK)
+                       if (errno != EWOULDBLOCK && errno != ETIMEDOUT && errno
!= EINTR)
                                return -1;

With this, all Linux GLX apps work with full hardware acceleration (glxinfo
reports "Accelerated: yes", glxgears ~24000 FPS with vblank_mode=0, a large
commercial CAD application runs normally). So Xwayland's write to the shared
page is visible to the Linux side; only the wakeup is lost.

How to reproduce:
1. Linux userland with Mesa new enough for the GPU, render node visible in
linsysfs (102adf88e6e8).
2. Under a Wayland compositor with native Xwayland, run a Linux GLX client:
/compat/<linux>/usr/bin/glxinfo -B
3. It hangs; kdump shows the FUTEX_WAIT above as the last call.

Expected: a native umtx wake on a shared address wakes Linux futex waiters
sleeping on the same address, as needed for xshmfence between Linux clients and
native X servers.

Minor, related: DMA_BUF_SET_NAME (ioctl 0x40086201) is not implemented in the
Linuxulator ("linux_ioctl_fallback ... not implemented" in dmesg). Mesa ignores
the failure, but it fills the log.
-------------------------------------------------------

I will test and try whatever you tell me. I can revert the change, recreate and
provide the kdump if needed.

-- 
You are receiving this mail because:
You are the assignee for the bug.