Re: use of ccache 4.14 can result in a defect in the FreeBSD kernel

Am 2026-11-02 08:35, schrieb Alexander Leidinger:

> Userland programs segfault. Basically you get an unusable system at 
> boot time.

And now with a bit more time to explain...

The problem is a ccache bug, not a FreeBSD bug. ccache 4.14 can return a 
stale vDSO when it builds a FreeBSD kernel, and the kernel then crashes 
userland processes that use signals. A fresh objdir or a clean rebuild 
doesn't fix it.

A kernel built with ccache 4.14.x (e.g. via WITH_CCACHE_BUILD) can 
contain the vDSO from an earlier build while everything else is current. 
The build finishes with no error and no warning, and the cache 
statistics look normal.

When this happened, every process that returned from a signal handler 
died with SIGSEGV. sshd, sshd-session and cron crashed within seconds of 
booting.

WHY IT BREAKS

  	* The vDSO is pulled in at assembly time, not by the preprocessor. 
sys/tools/amd64_vdso.sh assembles sys/tools/vdso_wrap.S, which copies 
the already-built elf-vdso.so.1 into an object with .incbin VDSO_FILE. 
.incbin is not #include: the file never shows up in the dependency list, 
so ccache's cache key doesn't cover it. If the .so.1 changes and the .S 
file and its flags don't, the key is the same, and ccache serves the old 
object.
  	* ccache used to protect against this. Since 3.3.3 it looks for 
.incbin, skips caching that compile, and counts it as 
unsupported_code_directive. 4.9.1 still does this for our compile 
(measured).
  	* 4.14 broke that check. In direct mode, the default, 4.14 only scans 
the raw source and expects a literal " right after .incbin. FreeBSD 
writes it with a macro (.incbin VDSO_FILE, with the file name passed as 
-DVDSO_FILE=...), so the scan misses it. The preprocessed output, where 
the macro has expanded to a literal string, is no longer checked in 
direct mode. Result: the compile is cached and later served stale.
  	* FreeBSD changed the vDSO. Commit 64a73d1b6dc9 (2026-09-19, sigtramp 
CFI) changed the vDSO's contents and size. A cache entry made before 
that commit gives the old vDSO, while the kernel is compiled with the 
new VDSO_SIGCODE_OFFSET. Signal returns then jump to the wrong address.

Bye,
Alexander.

> --
> Send from a mobile device, please forgive brevity and misspellings.
> 
> Am 2. November 2026 05:04:56 schrieb "Enji Cooper (yaneurabeya)" 
> <yaneurabeya@gmail.com>:
> 
> On Nov 1, 2026, at 7:08 AM, Alexander Leidinger 
> <Alexander@Leidinger.net> wrote:
> 
> Hi,
> 
> The defect: from 4.14 on, ccache in its default direct mode looks for 
> .incbin only in the raw source text, and only before a literal quote. 
> vdso_wrap.S names the file through a macro (VDSO_FILE), so the check 
> misses it. In direct mode the preprocessed text, where the name would 
> be visible, is no longer scanned.
> 
> When it hits: when the vDSO changes.
> 
> The workaround: use CCACHE_NODIRECT=1
> 
> Hi Alexander,
> What's the net-effect of this issue? Is it a functional problem? Is it 
> a security problem? Etc. Is there a bug tracking this issue on 
> Bugzilla?
> Thank you,
> -Enji

-- 
http://www.Leidinger.net Alexander@Leidinger.net: PGP 0x8F31830F9F2772BF
http://www.FreeBSD.org    netchild@FreeBSD.org  : PGP 0x8F31830F9F2772BF