FAIL: kernel fault injection

[Zachary Loafman]

Arch -

I'd like to contribute the kernel fault injection system that Isilon
uses. Before contributing it, I'd like to get approval for the APIs
involved.

Testing errors is hard. Let's say you have:

int foo(void) {
  [...]
  error = bar();
  if (error) {
    /* do stuff */
  }
}

.. but some_func() can't reliably be made to fail. How do you test it?
We added error injection macros that look like this:

int foo(void) {
  [...]
  error = bar();
  KFAIL_POINT_CODE(FP_KERN, bar_fails_foo, error = RETURN_VALUE);
  if (error) {
    /* do stuff */
  }
}

The KFAIL_POINT_CODE macro adds a sysctl MIB that allows
you to inject errors into the above code. For example:

# sysctl fail_point.kern.bar_fails_foo=".1%return(5)"

This says, ".1% of the time, evaluate the fail point code with 5 as
the RETURN_VALUE". If this were a standard errno, you could read the
above setting as "1/1000th of the time, pretend bar() returned EIO".

We also have a few wrappers around KFAIL_POINT_CODE that essentially
wrap common uses. For example, the above use can be shorthanded to:
  KFAIL_POINT_ERROR(FP_KERN, bar_fails_foo, error)

Currently, the sysctl parser accepts the following variants:
  return(x) - triggers the code with RETURN_VALUE set to x
  sleep(t) - sleep t milliseconds,
  panic/break - panic or break into the debugger
  print - print that the fail point was hit

In addition to the commands, we have a syntax to express the
when to evaluate those commands:
  p%<command> - evaluate command p% of the time (example above)
  5*<command> - evaluate command 5 times, then disable the expression

And you can compound with expr1->expr2, so, e.g.:
  5%return(5)->1%return(22):
    5% of the time, return 5, 1% of the remaining time, return 22
  5*return(0)->10*return(5)->1%return(19)
    return 0 for 5 times, then 5 for 10 times, and after those,
    return 19 1% of the time.
  1%5*return(22):
    1/100th of the time, return 22, but only do it 5 times total.

I've also attached an ascii rendering of a (rough draft) man page that
goes into more detail.

Comments?

...Zach
-------------- next part --------------
FAIL(9)                FreeBSD Kernel Developer's Manual               FAIL(9)

NAME
     KFAIL_POINT_CODE, KFAIL_POINT_RETURN, KFAIL_POINT_RETURN_VOID,
     KFAIL_POINT_ERROR, KFAIL_POINT_GOTO, fail_point, FP_KERN -- fail points

SYNOPSIS
     #include <sys/fail.h>

     KFAIL_POINT_CODE(parent, name, code);

     KFAIL_POINT_RETURN(parent, name);

     KFAIL_POINT_RETURN_VOID(parent, name);

     KFAIL_POINT_ERROR(parent, name, error_var);

     KFAIL_POINT_GOTO(parent, name, error_var, label);

DESCRIPTION
     Fail points are used to add code points where errors may be injected in a
     user controlled fashion. Fail points provide a convenient wrapper around
     user provided error injection code, providing a sysctl(9) MIB, and a
     parser for that MIB that describes how the error injection code should
     fire.

     The base fail point macro is KFAIL_POINT_CODE() where parent is a sysctl
     tree (frequently FP_KERN for kernel fail points, but various subsystems
     may wish to provide their own fail point trees), and name is the name of
     the MIB in that tree, and code is the error injection code. The code
     argument does not require braces, but it is considered good style to use
     braces for any multi-line code arguments. Inside the code argument, the
     evaluation of RETURN_VALUE is derived from the return() value set in the
     sysctl MIB. See SYSCTL SETTINGS below.

     The remaining KFAIL_POINT_*() macros are wrappers around common error
     injection paths:

     KFAIL_POINT_RETURN(parent, name)
       is the equivalent of KFAIL_POINT_CODE(..., return RETURN_VALUE)

     KFAIL_POINT_RETURN_VOID(parent, name)
       is the equivalent of KFAIL_POINT_CODE(..., return)

     KFAIL_POINT_ERROR(parent, name, error_var)
       is the equivalent of KFAIL_POINT_CODE(..., error_var = RETURN_VALUE)

     KFAIL_POINT_GOTO(parent, name, error_var, label)
       is the equivalent of KFAIL_POINT_CODE(...,
         { error_var = RETURN_VALUE; goto label;})

SYSCTL VARIABLES
     The KFAIL_POINT_*() macros add sysctl MIBs where specified. Many base
     kernel MIBs can be found in the fail_point.kern tree (referenced in code
     by FP_KERN ).

     The sysctl setting recognizes the following grammar:

       <fail_point> ::
           <term> ( "->" <term> )*

       <term> ::
           ( (<float> "%") | (<integer> "*" ) )*
           <type>
           [ "(" <integer> ")" ]

       <float> ::
           <integer> [ "." <integer> ] |
           "." <integer>

       <type> ::
           "off" | "return" | "sleep" | "panic" | "break" | "print"

     The <type> argument specifies which action to take:

     off     Take no action (does not trigger fail point code)

     return  Trigger fail point code with specified argument

     panic   Panic

     break   Break into the debugger.

     print   Print that the fail point executed

     The <float>% and <integer>* modifiers prior to <type> control when <type>
     is executed. The <float>% form (e.g. "1.2%") can be used to specify a
     probability that <type> will execute. The <integer>* form (e.g. "5*") can
     be used to specify the number of times <type> should be executed before
     this <term> is disabled. Only the last probability and the last count are
     used if multiple are specified, i.e. "1.2%2%" is the same as "2%". When
     both a probability and a count are specified, the probability is evalu-
     ated before the count, i.e. "2%5*" means "2% of the time, but only exe-
     cute it 5 times total".

     The operator -> can be used to express cascading terms. If you specify
     <term1>-><term2>, it means that if <term1> doesn't 'execute', <term2> is
     evaluated. For the purpose of this operator, the return() and print()
     operators are the only types that cascade. A return() term only cascades
     if the code executes, and a print() term only cascades when passed a non-
     zero argument.

EXAMPLES
     sysctl fail_point.kern.foobar="2.1%return(5)"
             21/1000ths of the time, execute code with RETURN_VALUE set to 5.

     sysctl fail_point.kern.foobar="2%return(5)->5%return(22)"
             2/100th of the time, execute code with RETURN_VALUE set to 5. If
             that doesn't happen, 5% of the time execute code with
             RETURN_VALUE set to 22.

     sysctl fail_point.kern.foobar="5*return(5)->0.1%return(22)"
             For 5 times, return 5. After that, 1/1000ths of the time, return
             22.

     sysctl fail_point.kern.foobar="0.1%5*return(5)"
             Return 5 for 1 in 1000 executions, but only execute 5 times
             total.

CAVEATS
     It's easy to shoot yourself in the foot by setting fail points too
     aggressively or setting too many in combination. For example, forcing
     malloc() to fail consistently is potentially harmful to uptime.

     The sleep() sysctl setting may not be appropriate in all situations. Cur-
     rently, fail_point_eval() does not verify whether the context is appro-
     priate for calling msleep().

FreeBSD 8.0                      May 10, 2009                      FreeBSD 8.0