Draft script for identifying [and someday documenting] tunables

Garrett Cooper gcooper at FreeBSD.org
Fri Feb 5 02:34:53 UTC 2010


Hi Tom, and other CURRENT folks,
    One of the items that needs to be fixed up on the documentation
page from what I've seen is that the kernel sysctl and tunables
information needs to be properly updated because it's either the
description is missing, out-of-date, or the sysctl OID or tunable has
been removed, etc.
    I came up with this [relatively] simple script for parsing
tunables -- took a crack at trying to get sysctl support in as well,
but hit a brick wall because the method for walking over the source
tree is fairly basic. What should be done to catch everything in an
automated manner is that there need to be some preprocessor defines
plugged into the tunable and sysctl macros so that it prints out the
proper description for the properties under inspection.
    Anyhow, if someone can propose a way to do that quickly and
cleanly, I'll write up a more standardized proposal with basic code
that can get plugged into the headers, so it punts out this info for
the script, which will produce an automated mapping of what tunables
are used, and the complete sysctl MIB.
Thanks!
-Garrett

#!/usr/bin/env python
"""

Traverse a source tree to produce a list of FreeBSD sysctls and tunables
for documentation purposes.

Copyright (c) 1992-2010 Cisco Systems, Inc. All rights reserved.

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions
are met:
1. Redistributions of source code must retain the above copyright
   notice, this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright
   notice, this list of conditions and the following disclaimer in the
   documentation and/or other materials provided with the distribution.

THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
ARE DISCLAIMED.  IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
SUCH DAMAGE.

"""

# NOTE: this is done in python because it's more flexible than sed, awk, etc.
#
# XXX (gcooper):
# 1. FreeBSD kernel tunables should be defined in an easy-to-understand
# self-documenting method as they aren't today (you can grab the information
# from the sysctl description, but it's messy parsing the description if it's
# not in the same file or the immediate area around the tunable definition);
# hence the description parsing is disabled.
# 2. This data should be piped via subprocess through cpp(1) to look at the
# tunable data, probably.
#

import optparse
import os
import re

PARSE_SYSCTLS, PARSE_TUNABLES = 1, 2
SYSCTL_RE = re.compile('^\s*SYSCTL_[A-Z_]+\s*\("(.+)", .+, "(.+)"\)')
TUNABLE_RE = re.compile('^\s*TUNABLE_[A-Z]+\s*\("([a-z\.]+)", .+\)')

class KernelProperty(object):
    """ A generic class for Sysctl and Tunable objects """

    def __init__ (self, name, **kwargs):
        """ Tunable objects constructor.

        name - tunable name; required.
        description - tunable description; optional.
        file - file where the info was spotted; optional.
        line - line where the tunable was defined; optional.

        XXX (gcooper): description is hardwired to None -- make this a
        requirement when self-documenting tunable info is completed...
        """
        if not name:
            raise ValueError('name must be defined to a non-zero length string')

        for i in [ 'description', 'file', 'line' ]:
            setattr(self, i, kwargs.get(i))

        #if not description:
        #    raise ValueError('description must be defined to a non-zero '
        #                     'length string')
        self.name = name

    def __str__ (self):
        """ `Prints' out the KernelProperty object in stringized form.

        The format is as follows:

        1. name
        2. file:line name
        3. name = "description"
        4. file:line name = "description"

        The format printed out all depends on the data provided by
        parse_kernel_objects(...).

        """

        # Provide the file / line info.
        # pylint: disable-msg=E1101
        if self.file and self.line:
            # pylint: disable-msg=E1101
            prefix = '%s:%d: ' % (self.file, self.line)
        else:
            prefix = ''

        # Omit the description.
        # pylint: disable-msg=E1101
        if self.description:
            # pylint: disable-msg=E1101
            suffix = ' = "%s"' % (self.description,)
        else:
            suffix = ''

        # NOTE (gcooper): if either prefix or suffix is empty, there will be
        # nasty heading / trailing whitespace. Hence, I used + concatenation
        # instead of ' '.join([..]), because this is considerably cheaper to
        # do than '%s%s%s' (and far less braindead)...
        return prefix + self.name + suffix

class Sysctl(KernelProperty):
    """ A class corresponding to a kernel sysctl OID property.
    """

class Tunable(KernelProperty):
    """ A class corresponding to a kernel tunable object.
    """

def main():
    """ Main function of course. """

    parser = optparse.OptionParser()

    parser.add_option('-d', '--srcdir', dest='srcdir', default='/usr/src/sys',
                      help=('source directory to look for sysctls and '
                            'tunables in'))

    #
    # XXX (gcooper): this is more difficult to do as the descriptions
    # themselves span multiple lines. Needless to say, it's a Orlando Bloom-ing
    # parsing mess...
    #
    #parser.add_option('-s', '--sysctl', dest='parse_sysctls', default=False,
    #                  action='store_true',
    #                  help='look for sysctls OIDs')

    parser.add_option('-t', '--tunable', dest='parse_tunables', default=False,
                      action='store_true',
                      help='look for tunables')

    parser.add_option('-v', '--verbose', dest='verbose', default=False,
                      action='store_true',
                      help='include file/line results for tunables.')

    opts, __ = parser.parse_args()

    kknobs = 0

    # XXX (garrcoop): sysctl parsing's a pain -- so it's disabled...
    # See above comment.
    kknobs = PARSE_TUNABLES
    #
    #if opts.parse_sysctls:
    #    kknobs |= PARSE_SYSCTLS
    #if opts.parse_tunables:
    #    kknobs |= PARSE_TUNABLES
    #
    #if not kknobs:
    #    kknobs = PARSE_SYSCTLS


    sysctls, tunables = comb_tree_for_kernel_objects(opts.srcdir,
                                                     kernel_knobs=kknobs,
                                                     verbose_info=opts.verbose,)

    if sysctls:
        print '\n'.join([ 'Sysctls:' ] + map(str, sysctls))

    if tunables:
        print '\n'.join([ 'Tunables:' ] + map(str, tunables))

def comb_tree_for_kernel_objects(srcdir, kernel_knobs, verbose_info=False):
    """ Comb a tree looking for kernel tunables in files under srcdir.

    - Returns a list of Tunable objects found in srcdir.
    - The file and line number the tunable was found on will be saved when
      verbose_info is set to True. See the Tunable class for more
      details.
    """

    sysctls = []
    tunables = []

    # Doing this makes relpath considerably simpler and cleaner than doing it
    # inside the loop.
    os.chdir(srcdir)

    for root, __, files in os.walk(srcdir):

        for _file in files:

            _file = os.path.relpath(os.path.join(root, _file))
            _sysctls, _tunables = parse_kernel_objects(_file, kernel_knobs,
                                                       verbose_info)
            sysctls.extend(_sysctls)
            tunables.extend(_tunables)

    return sysctls, tunables

def parse_kernel_objects(_file, kernel_knobs, verbose_info=False):
    """ Comb a tree looking for kernel tunables.

    - Returns a tuple of lists containing Sysctl and Tunable objects
      found in _file.
    - The file and line number the tunable was found on will be saved when
      verbose_info is set to True. See the KernelProperty class for more
      details.
    """

    kwargs = { }
    sysctls = []
    tunables = []

    parse_sysctls = kernel_knobs & PARSE_SYSCTLS
    parse_tunables = kernel_knobs & PARSE_TUNABLES

    if not (parse_sysctls or parse_tunables):
        # Needs to be an inclusive or of PARSE_SYSCTLS, PARSE_TUNABLES, or a
        # combination of the two knobs.
        raise ValueError('invalid value for kernel_knobs: %s'
                         % str(kernel_knobs))

    if verbose_info:
        kwargs['file'] = _file

    fd = open(_file, 'r')
    try:
        lines = fd.readlines()
    finally:
        fd.close()

    kwargs['line'] = 0

    for line in lines:

        kwargs['line'] += 1

        if parse_tunables:
            match = TUNABLE_RE.match(line)
        else:
            match = None

        if match:
            tunables.append(Tunable(match.group(1), **kwargs))
        elif parse_sysctls:
            match = SYSCTL_RE.match(line)
            if match:
                sysctls.append(Sysctl(match.group(1),
                                      description=match.group(2), **kwargs))

    return sysctls, tunables

if __name__ == '__main__':
    main()
-------------- next part --------------
A non-text attachment was scrubbed...
Name: find_freebsd_kernel_properties.py
Type: application/octet-stream
Size: 8663 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-current/attachments/20100205/ee857ada/find_freebsd_kernel_properties.obj


More information about the freebsd-current mailing list