buildworld in NFS /usr/src/ vs. automount /usr/src path

From: Steve Rikli <sr_at_genyosha.net>
Date: Thu, 13 Nov 2025 22:03:40 UTC
I have small VM's for testing, and recently have been trying builds with
/usr/src/ mounted from my NFS server, read-only. /usr/obj/ is local on
the virtual system disk.

I'm simply using buildworld, nothing fancy; same routine I use on my
other -STABLE systems, with a local /usr/src/ read-write on their own
sysdisk.

Depending on how the NFS mount is done, this works fine; that is, I've
noticed an apparent difference in buildworld behavior, and it seems to
cause a failure. Here are the 2 different setups:

1) NFS mount the remote server's export directly at the VM's /usr/src/

E.g. it essentially looks like:

$ df -h /usr/src/
Filesystem                       Size    Used   Avail Capacity  Mounted on
srv:/os/freebsd/src/stable-15     14T    9.5T    4.2T    70%    /usr/src

This works fine, aside from some warnings about read-only /usr/src/, it
behaves just as if /usr/src/ was on the local system disk, and results
in successful buildworld completion.

2) /usr/src is a symlink pointing to an automount path of the remote
   NFS server's export

This setup looks like:

$ ls -la /usr/src
lrwxr-xr-x  1 root wheel 35 Oct 17 11:51 /usr/src -> /net/srv/os/freebsd/src/stable-15

While the data itself in each case's /usr/src is the same (i.e. from
the same export on the same NFS server), the 2nd scenario does not work;
it gives a buildworld failure like this:

...
awk -f /net/srv/os/freebsd/src/stable-15/sys/conf/kmod_syms.awk irdma.ko.full  export_syms | xargs -J % objcopy % irdma.ko.full
objcopy --only-keep-debug irdma.ko.full irdma.ko.debug
objcopy --strip-debug --add-gnu-debuglink=irdma.ko.debug  irdma.ko.full irdma.ko
===> isci (all)
[Creating objdir /usr/obj/net/srv/os/freebsd/src/stable-15/amd64.amd64/sys/GENERIC/modules/net/srv/os/freebsd/src/stable-15/sys/modules/isci...]
machine -> /net/srv/os/freebsd/src/stable-15/sys/amd64/include
x86 -> /net/srv/os/freebsd/src/stable-15/sys/x86/include
i386 -> /net/srv/os/freebsd/src/stable-15/sys/i386/include
make[4]: don't know how to make sci_base_memory_descriptor_list.c. Stop

make[4]: stopped making "all" in /net/srv/os/freebsd/src/stable-15/sys/modules/isci
*** Error code 2

Stop.
make[3]: stopped making "all" in /net/srv/os/freebsd/src/stable-15/sys/modules
*** Error code 1

Stop.
make[2]: stopped making "all" in /usr/obj/net/srv/os/freebsd/src/stable-15/amd64.amd64/sys/GENERIC
     3039.85 real      2180.19 user       241.05 sys
*** Error code 1

Stop.
make[1]: stopped making "buildkernel" in /net/srv/os/freebsd/src/stable-15
*** Error code 1

Stop.
make: stopped making "buildkernel" in /net/srv/os/freebsd/src/stable-15

----

I've tried a couple different symlink automount paths but it doesn't
seem to matter -- I didn't really expect it would.

For clarity, that same git src tree runs buildworld to completion OK
when it is mounted directly as /usr/src/ on the same VM.

I'm guessing that buildworld is getting confused because something
like /bin/pwd is not "/usr/src/" in the symlink automount path setup,
i.e. it will instead be something like "/net/nfsserver/path/..." and
that seems to be confusing buildworld during creation of files in the
local /usr/obj/ directory.

I can workaround the mounting setup well enough, I'm just wondering if
buildworld is expected to work okay with building in paths other than
"/usr/src". Until recently I've never tried this sort of thing.  :-)

Cheers,
sr.