(Security Regression Testsuites)Request for comments

[Robert Watson]

On Tue, 29 May 2007, zhouyi zhou wrote:

>  Where I am still confused:
>
> 1) Which area and direction should I focus. The security subsystem in 
> FreeBSD is large, which area deserves a testsuite in higher priority.

Off-hand, my feeling is I'd like us to consider three areas of testing:

- Correctness testing, in which we test general and edge cases in access
   control to make sure both the infrastructure and policies operate correctly.
   For example, using test policies to validate that the MAC Framework is
   correctly implemented, and using tests on real policies to determine that
   the real policies implement the desired access control.  Likewise, for
   audit, making sure that the right records are generated with the right
   tokens for the right events, that the selection model works, etc.

- Stability testing, in which we test general and edge cases to make sure the
   system is stable, especially under load, etc.

- ABI/API life cycle testing, in which we confirm that key data structures in
   the ABI remain stable over time, and that the interpretation remains
   consistent.  For example, the persisting data structures in on-disk labels,
   audit record formats, etc.

Obviously, doing all this in one summer is well out of scope, but I think some 
useful in-roads can be made in testing key areas, such as making sure that 
file system protections with MLS and Biba are correct, tests for audit, and so 
on.  You may want to look at Pawel's and my existing file system test tools 
(src/tools/regression in 7-CURRENT) to see some areas and approaches to 
testing.

> 2) The general structure of the testsuite: Will it be a userland application 
> package like stress2, or include a kernel module cooperation (like 
> security/mac_test) 3) How to write a testsuite that will prevent the furthor 
> violation of security instead of test the cases which are already corrected. 
> PF、IPFW and IPSec have already corrected their confliction with Mandatory 
> Access Control, I think the testcases for the already corrected problems 
> will not discover the newly generated problems, for example: test case for 
> the PF's synproxy state rule only verify PF have correctly add a correct tag 
> for Mandatory access control in function pf_send_tcp, how we discover a 
> problem which may create in the future by means of create a mbuf without a 
> correct tag for Mandatory access control in a new function?

I would suggest starting with a small set of test projects to evaluate 
approaches.  For example:

(1) Consider adding a new test policy couple with userland tools to make sure
     that access control checks occur as required for each system call.

(2) Add a set of user space tests that confirm that MLS is properly
     implemented for each system call.

(3) Add a set of user space tests that confirm that, for each system call, the
     right audit records are generated with the right tokens.

(4) Add a set of user space tests to confirm that audit record preselection is
     properly implemented.

These are a bit more bounded in scope, and should start to bring out common 
aspects to testing across security functions (i.e., "foreach system call").

Robert N M Watson
Computer Laboratory
University of Cambridge