Avoiding programmer invariant violations (was: Re: FreeBSD Kernel buffer overflow)

[Robert Watson]

On Sat, 18 Sep 2004 gerarra at tin.it wrote:

> Here i report a patch different from Giorgos' one. The approch is
> completely different: working on syscall_register() function in
> kern/kern_syscalls.c file.

I'd suggest that we need to look at this in two ways:

(1) There's a compile-time INVARIANT that needs to be maintained by
    developers in adding new system calls.  When building the kernel, it
    would be useful to have a compile-time assertion that causes a kernel
    compile to fail if an invalid system call is defined.  I.e., when
    init_sysent.c is generated, it should build in __CTASSERT's that all
    argument counts are consistent with the requirements of the hardware
    architecture being built for. 

(2) There's a run-time INVARIANT issue for loadable modules built by third
    parties who may not understand the limits on arguments on system calls
    for various architectures.  This can be handled by a check in the
    system call registration code, although since that's a non-critical
    performance path, I suggest testing the invariant even if INVARIANTS
    isn't compiled in.  In some ways, I'd rather handle this at
    compile-time for the module, but I think the infrastructure for
    hooking up system calls at compile-time for modules will make that
    more difficult as compared to statically compiled system calls.

Note that the discussion so far has not addressed the compile-time issue: 
which is a much better time to perform the tests -- it's something we can
test when the kernel is compiled, so why not?.  It also hasn't addressed
non-i386 systems, such as amd64, which have similar or identical concerns. 

With all due respect to the submitter, I think bugtraq was not the forum
to post this issue to, as that forum is typically preferred for
exploitable vulnerabilities.  A follow-up post to clarify that the initial
post described a possible avenue for programmer error when extending the
kernel, rather than an immediately exploitable vulnerability, might reduce
confusion. 

Robert N M Watson             FreeBSD Core Team, TrustedBSD Projects
robert at fledge.watson.org      Principal Research Scientist, McAfee Research