atf-c for kernel function unit testing

[Alan Somers]

On Mon, Mar 3, 2014 at 10:32 AM, Julio Merino <jmmv at freebsd.org> wrote:
> On Mon, Mar 3, 2014 at 11:52 AM, R. Tyler Croy <tyler at monkeypox.org> wrote:
>>
>> I'm wondering if anybody's explored using atf-c for adding unit tests to kernel
>> level functions or driver code.
>
> We do this extensively in NetBSD: we have a mechanism (rump) to build
> unmodified kernel-level code as user-space binaries and we later use
> atf to unit-test the code in controlled tiny environments. (You'd
> think about this as super-lightweight virtualization of the kernel.)
>
> Achieving this for any non-trivial kernel-level code is hard however,
> but I believe that this approach has huge benefits. For example: we
> are able to unit-test all file system and networking code in
> user-space without any chances of crashing the system; there is also
> the benefit of being able to control the test environment fully
> without relying on any state of the running kernel.
>
> For more details see:
>
> http://en.wikipedia.org/wiki/Anykernel
> http://wiki.netbsd.org/rumpkernel/
>
> If the code you are interested in testing can easily be built without
> ties to the kernel, you can relatively easily do the above. (In fact,
> we used to do this in NetBSD for some tiny areas of the code before we
> got rump.)
>
> At some point I also pondered the idea to write an interface and a
> kernel module to allow writing unit tests within the kernel. This
> might be interesting for some special cases (like testing live
> hardware maybe) but it's a very fragile setup that should not be
> pursued if other safer solutions suffice.

Rump is definitely the ideal method.  Lacking that, FreeBSD's options
are to build kernel code in userspace, or to run unit tests in the
kernel.  Doing the latter is not currently possible with ATF.  There
are some places where it's currently done ad-hoc, such as in xnb(4),
but it's not pretty.  Compiling kernel code in userspace isn't always
possible.  When I wrote xnb(4), almost all of its unit tests depended
on mbuf(9), and I found that there was no way to compile mbuf.h in
userspace.  ZFS is a rare exception.  Almost all of the ZFS code is
compiled in userspace as part of libzpool.  That's how ztest works.

My suggestion is to write new drivers to be compilable in userspace,
rely on functional tests for existing drivers, and find somebody
willing to port rump from NetBSD.

-Alan