svn commit: r227824 - in projects/diffused_head: sbin/ipfw
sbin/ipfw/diffuse_collector sbin/ipfw/diffuse_exporter
share/man/man4
Lawrence Stewart
lstewart at FreeBSD.org
Tue Nov 22 17:25:23 UTC 2011
Author: lstewart
Date: Tue Nov 22 17:25:22 2011
New Revision: 227824
URL: http://svn.freebsd.org/changeset/base/227824
Log:
- Manual pages for the DIFFUSE kernel module, exporter and collector
- An update to the ipfw.8 manual page to describe DIFFUSE's IPFW grammar
extensions.
Sponsored by: FreeBSD Foundation
Reviewed by: bz
Added:
projects/diffused_head/sbin/ipfw/diffuse_collector/diffuse_collector.8 (contents, props changed)
projects/diffused_head/sbin/ipfw/diffuse_exporter/diffuse_exporter.8 (contents, props changed)
projects/diffused_head/share/man/man4/diffuse.4 (contents, props changed)
Modified:
projects/diffused_head/sbin/ipfw/diffuse_collector/Makefile
projects/diffused_head/sbin/ipfw/diffuse_exporter/Makefile
projects/diffused_head/sbin/ipfw/ipfw.8
projects/diffused_head/share/man/man4/Makefile
Modified: projects/diffused_head/sbin/ipfw/diffuse_collector/Makefile
==============================================================================
--- projects/diffused_head/sbin/ipfw/diffuse_collector/Makefile Tue Nov 22 16:44:50 2011 (r227823)
+++ projects/diffused_head/sbin/ipfw/diffuse_collector/Makefile Tue Nov 22 17:25:22 2011 (r227824)
@@ -5,6 +5,6 @@ PROG= diffuse_collector
SRCS= diffuse_collector.c diffuse_proto.c
DPADD= ${LIBUTIL}
LDADD= -lutil
-NO_MAN=
+MAN= diffuse_collector.8
.include <bsd.prog.mk>
Added: projects/diffused_head/sbin/ipfw/diffuse_collector/diffuse_collector.8
==============================================================================
--- /dev/null 00:00:00 1970 (empty, because file is newly added)
+++ projects/diffused_head/sbin/ipfw/diffuse_collector/diffuse_collector.8 Tue Nov 22 17:25:22 2011 (r227824)
@@ -0,0 +1,165 @@
+.\"
+.\" Copyright (c) 2010
+.\" Swinburne University of Technology, Melbourne, Australia.
+.\" Copyright (c) 2011 The FreeBSD Foundation
+.\" All rights reserved.
+.\"
+.\" This software was developed at the Centre for Advanced Internet
+.\" Architectures, Swinburne University of Technology, by Sebastian Zander, made
+.\" possible in part by a gift from The Cisco University Research Program Fund, a
+.\" corporate advised fund of Silicon Valley Community Foundation.
+.\"
+.\" Portions of this documentation were written at the Centre for Advanced
+.\" Internet Architectures, Swinburne University of Technology, Melbourne,
+.\" Australia by Lawrence Stewart under sponsorship from the FreeBSD Foundation.
+.\"
+.\" Redistribution and use in source and binary forms, with or without
+.\" modification, are permitted provided that the following conditions
+.\" are met:
+.\" 1. Redistributions of source code must retain the above copyright
+.\" notice, this list of conditions and the following disclaimer.
+.\" 2. Redistributions in binary form must reproduce the above copyright
+.\" notice, this list of conditions and the following disclaimer in the
+.\" documentation and/or other materials provided with the distribution.
+.\"
+.\" THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+.\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+.\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+.\" ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE FOR
+.\" ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+.\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+.\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+.\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+.\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+.\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+.\" SUCH DAMAGE.
+.\"
+.\" $FreeBSD$
+.\"
+.Dd November 23, 2011
+.Dt diffuse_collector 8
+.Os
+.Sh NAME
+.Nm diffuse_collector
+.Nd receive flow information from one or more flow information sources and
+manage local firewall and/or traffic shaper actions.
+.Sh SYNOPSIS
+.Nm
+.Op Fl hnv
+.Op Fl c Ar config-file
+.Op Fl s Ar sctp-port
+.Op Fl t Ar tcp-port
+.Op Fl u Ar udp-port
+.Sh DESCRIPTION
+The
+.Nm
+utility receives flow information directly from classifier nodes via UDP, or
+indirectly from
+.Xr diffuse_exporter
+instances via UDP, TCP or SCTP.
+.Nm
+uses the flow information received to manage local firewall and/or traffic
+shaper actions.
+.Pp
+.Nm
+was tested with the
+.Xr ipfw 4
+firewall, but provisions exist in the code to simplify adding support for
+different firewalls in future.
+.Pp
+The following options are available:
+.Bl -tag -width "Ar actions-file" -offset indent
+.It Fl c Ar config-file
+Path to the plain text INI-style configuration file.
+.It Fl h
+Show help.
+.It Fl n
+Do not generate firewall commands.
+Useful for testing.
+.It Fl s Ar sctp-port
+SCTP port to listen on.
+.It Fl t Ar tcp-port
+TCP port to listen on.
+.It Fl u Ar udp-port
+UDP port to listen on.
+.It Fl v
+Increase verbosity.
+.El
+.Sh CONFIGURATION FILE
+The
+.Nm
+configuration file uses an INI style layout with key-value pair configuration
+items.
+There are currently three sections:
+.Qq general ,
+.Qq firewall
+and
+.Qq classactions .
+The example configuration file mentioned in the
+.Sx FILES
+section documents the available configuration options and their syntax.
+.Sh FILES
+.Bl -tag -width "conf" -compact -offset 0
+.It Pa /usr/share/examples/diffuse/collector.conf
+An example collector configuration file using
+.Xr ipfw 4
+for the underlying firewall.
+.El
+.Sh EXAMPLES
+Listen for flow information on UDP and TCP port 5000 and parse the
+/etc/collector.conf configuration file to obtain the rest of the required
+configuration:
+.Bd -literal -offset indent
+.Nm Ns
+ -c /etc/collector.conf -u 5000 -t 5000
+.Ed
+.Sh SEE ALSO
+.Xr diffuse 4 ,
+.Xr dummynet 4 ,
+.Xr ipfw 4 ,
+.Xr diffuse_exporter 8 ,
+.Xr ipfw 8
+.Sh ACKNOWLEDGEMENTS
+Development and testing of this software were made possible in part by grants
+from the FreeBSD Foundation and The Cisco University Research Program Fund, a
+corporate advised fund of Silicon Valley Community Foundation.
+.Sh HISTORY
+The
+.Nm
+utility is part of the DIFFUSE architecture and first appeared in
+.Fx 10.0 .
+.Pp
+DIFFUSE
+.Ns ( Em DI Ns stributed
+.Em F Ns irewall
+and
+.Em F Ns low-shaper
+.Em U Ns sing
+.Em S Ns tatistical
+.Em E Ns vidence )
+was first released in 2010 by Sebastian Zander whilst working on the DIFFUSE
+research project at Swinburne University of Technology's Centre for Advanced
+Internet Architectures, Melbourne, Australia, which was made possible in part by
+a gift from The Cisco University Research Program Fund, a corporate advised fund
+of Silicon Valley Community Foundation.
+More details are available at:
+.Pp
+http://caia.swin.edu.au/urp/diffuse/
+.Sh AUTHORS
+.An -nosplit
+The
+.Nm
+was written by
+.An Sebastian Zander Aq szander at swin.edu.au
+and later extended by
+.An Lawrence Stewart Aq lstewart at FreeBSD.org .
+.Pp
+This manual page was written by
+.An Sebastian Zander Aq szander at swin.edu.au
+and
+.An Lawrence Stewart Aq lstewart at FreeBSD.org .
+.Sh BUGS
+.Bl -dash
+.It
+IPv6 is currently unsupported.
+.El
Modified: projects/diffused_head/sbin/ipfw/diffuse_exporter/Makefile
==============================================================================
--- projects/diffused_head/sbin/ipfw/diffuse_exporter/Makefile Tue Nov 22 16:44:50 2011 (r227823)
+++ projects/diffused_head/sbin/ipfw/diffuse_exporter/Makefile Tue Nov 22 17:25:22 2011 (r227824)
@@ -5,6 +5,6 @@ PROG= diffuse_exporter
SRCS= diffuse_exporter.c diffuse_proto.c
DPADD= ${LIBUTIL}
LDADD= -lutil
-NO_MAN=
+MAN= diffuse_exporter.8
.include <bsd.prog.mk>
Added: projects/diffused_head/sbin/ipfw/diffuse_exporter/diffuse_exporter.8
==============================================================================
--- /dev/null 00:00:00 1970 (empty, because file is newly added)
+++ projects/diffused_head/sbin/ipfw/diffuse_exporter/diffuse_exporter.8 Tue Nov 22 17:25:22 2011 (r227824)
@@ -0,0 +1,146 @@
+.\"
+.\" Copyright (c) 2010
+.\" Swinburne University of Technology, Melbourne, Australia.
+.\" Copyright (c) 2011 The FreeBSD Foundation
+.\" All rights reserved.
+.\"
+.\" This software was developed at the Centre for Advanced Internet
+.\" Architectures, Swinburne University of Technology, by Sebastian Zander, made
+.\" possible in part by a gift from The Cisco University Research Program Fund, a
+.\" corporate advised fund of Silicon Valley Community Foundation.
+.\"
+.\" Portions of this documentation were written at the Centre for Advanced
+.\" Internet Architectures, Swinburne University of Technology, Melbourne,
+.\" Australia by Lawrence Stewart under sponsorship from the FreeBSD Foundation.
+.\"
+.\" Redistribution and use in source and binary forms, with or without
+.\" modification, are permitted provided that the following conditions
+.\" are met:
+.\" 1. Redistributions of source code must retain the above copyright
+.\" notice, this list of conditions and the following disclaimer.
+.\" 2. Redistributions in binary form must reproduce the above copyright
+.\" notice, this list of conditions and the following disclaimer in the
+.\" documentation and/or other materials provided with the distribution.
+.\"
+.\" THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+.\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+.\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+.\" ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE FOR
+.\" ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+.\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+.\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+.\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+.\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+.\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+.\" SUCH DAMAGE.
+.\"
+.\" $FreeBSD$
+.\"
+.Dd November 23, 2011
+.Dt diffuse_exporter 8
+.Os
+.Sh NAME
+.Nm diffuse_exporter
+.Nd forward flow information from a classifier node to one or more action nodes.
+.Sh SYNOPSIS
+.Nm
+.Op Fl hv
+.Op Fl a Ar prot://host:port , Ns ...
+.Op Fl c Ar host:port
+.Sh DESCRIPTION
+The
+.Nm
+utility acts as an intermediary between a DIFFUSE classifier node and one or
+more action nodes.
+It listens for UDP messages containing flow information sent by the in-kernel
+.Xr diffuse 4
+classifier and forwards the information to remote action nodes using UDP, TCP or
+SCTP.
+.Pp
+The following options are available:
+.Bl -tag -width "Ar prot://host:port,..." -offset indent
+.It Fl a Ar prot://host:port , Ns ...
+Comma separated (no spaces) list of action nodes to forward flow information to.
+.Ar proto
+must be either "udp", "tcp", or "sctp",
+.Ar host
+is the action node's IP address or fully qualified host name, and
+.Ar port
+is the port the action node is listening on.
+If no
+.Ar port
+is specified, 3191 is used as a default.
+.It Fl c Ar host:port
+The classifier node to listen for flow information from.
+.Ar host
+is the classifier node's IP address or fully qualified host name, and
+.Ar port
+is the port
+.Nm
+will listen on for messages from the classifier node.
+If no
+.Ar port
+is specified, 3191 is used as a default.
+.It Fl h
+Show help.
+.It Fl v
+Increase verbosity.
+.El
+.Sh EXAMPLES
+Listen for flow information from a local classifier on the default port and
+forward it to two different action nodes using TCP and SCTP respectively:
+.Bd -literal -offset indent
+.Nm Ns
+ -c localhost \\\
+ -a tcp://anode1.example.com:5000,sctp://anode2.example.com:5000
+.Ed
+.Sh SEE ALSO
+.Xr diffuse 4 ,
+.Xr dummynet 4 ,
+.Xr ipfw 4 ,
+.Xr ipfw 8 ,
+.Xr diffuse_collector 8
+.Sh ACKNOWLEDGEMENTS
+Development and testing of this software were made possible in part by grants
+from the FreeBSD Foundation and The Cisco University Research Program Fund, a
+corporate advised fund of Silicon Valley Community Foundation.
+.Sh HISTORY
+The
+.Nm
+utility is part of the DIFFUSE architecture and first appeared in
+.Fx 10.0 .
+.Pp
+DIFFUSE
+.Ns ( Em DI Ns stributed
+.Em F Ns irewall
+and
+.Em F Ns low-shaper
+.Em U Ns sing
+.Em S Ns tatistical
+.Em E Ns vidence )
+was first released in 2010 by Sebastian Zander whilst working on the DIFFUSE
+research project at Swinburne University of Technology's Centre for Advanced
+Internet Architectures, Melbourne, Australia, which was made possible in part by
+a gift from The Cisco University Research Program Fund, a corporate advised fund
+of Silicon Valley Community Foundation.
+More details are available at:
+.Pp
+http://caia.swin.edu.au/urp/diffuse/
+.Sh AUTHORS
+.An -nosplit
+The
+.Nm
+was written by
+.An Sebastian Zander Aq szander at swin.edu.au
+and later extended by
+.An Lawrence Stewart Aq lstewart at FreeBSD.org .
+.Pp
+This manual page was written by
+.An Sebastian Zander Aq szander at swin.edu.au
+and
+.An Lawrence Stewart Aq lstewart at FreeBSD.org .
+.Sh BUGS
+.Bl -dash
+.It
+IPv6 is currently unsupported.
+.El
Modified: projects/diffused_head/sbin/ipfw/ipfw.8
==============================================================================
--- projects/diffused_head/sbin/ipfw/ipfw.8 Tue Nov 22 16:44:50 2011 (r227823)
+++ projects/diffused_head/sbin/ipfw/ipfw.8 Tue Nov 22 17:25:22 2011 (r227824)
@@ -1,7 +1,7 @@
.\"
.\" $FreeBSD$
.\"
-.Dd August 20, 2011
+.Dd November 23, 2011
.Dt IPFW 8
.Os
.Sh NAME
@@ -91,6 +91,21 @@ in-kernel NAT.
.Oc
.Oc
.Ar pathname
+.Pp
+.Ss DIFFUSE
+.Nm
+.Brq Cm feature | mlclass | export
+.Ar name
+.Cm config
+.Ar config-options
+.Nm
+.Brq Cm feature | mlclass | export
+.Brq Cm delete | show
+.Ar name
+.Nm
+.Cm flowtable
+.Brq Cm show | zero | flush
+.Op expired
.Sh DESCRIPTION
The
.Nm
@@ -98,8 +113,10 @@ utility is the user interface for contro
.Xr ipfw 4
firewall, the
.Xr dummynet 4
-traffic shaper/packet scheduler, and the
-in-kernel NAT services.
+traffic shaper/packet scheduler, in-kernel NAT services, and
+.Xr diffuse 4 ,
+an extension that provides machine-learning based traffic classification and
+distributed firewalling / traffic shaping.
.Pp
A firewall configuration, or
.Em ruleset ,
@@ -336,6 +353,17 @@ commands are used to configure the traff
See the
.Sx TRAFFIC SHAPER (DUMMYNET) CONFIGURATION
Section below for details.
+.Ss DIFFUSE CONFIGURATION
+The
+.Nm
+.Cm feature , mlclass
+and
+.Cm export
+commands are used to configure
+.Xr diffuse 4 .
+See the
+.Sx DIFFUSE CONFIGURATION
+Section below for details.
.Pp
If the world and the kernel get out of sync the
.Nm
@@ -722,6 +750,14 @@ Divert packets that match this rule to t
socket bound to port
.Ar port .
The search terminates.
+.It Cm export Ar name
+Export flow information for flows classified by
+.Xr diffuse 4 .
+The information is exported according to the configuration of the export
+.Ar name
+(see
+.Sx DIFFUSE CONFIGURATION Ns ).
+The search continues with the next rule.
.It Cm fwd | forward Ar ipaddr | tablearg Ns Op , Ns Ar port
Change the next-hop on matching packets to
.Ar ipaddr ,
@@ -769,6 +805,12 @@ To enable
.Cm fwd
a custom kernel needs to be compiled with the option
.Cd "options IPFIREWALL_FORWARD" .
+.It Cm mlclass Ar name
+Classify flows using machine learning classifier
+.Ar name ,
+which was previously configured with a config command (see
+.Sx DIFFUSE CONFIGURATION Ns ).
+The search continues with the next rule.
.It Cm nat Ar nat_nr
Pass packet to a
nat instance
@@ -1249,6 +1291,19 @@ action followed by the comment.
.It Cm bridged
Alias for
.Cm layer2 .
+.It Cm class-tags Ar tag , Ns Ar ...
+This option can only be specified for rules with a
+.Cm mlclass
+action.
+If specified, packets will be tagged with the specified tag(s) according to
+their class(es).
+For example, a packet that classified as the first class will be tagged with the
+first tag.
+(See
+.Sx DIFFUSE CONFIGURATION Ns ).
+.Pp
+Tagged packets can then be matched in later rules using the tagged option
+described below.
.It Cm diverted
Matches only packets generated by a divert socket.
.It Cm diverted-loopback
@@ -1289,6 +1344,48 @@ IPSec authentication headers
.Pq Cm ah ,
and IPsec encapsulated security payload headers
.Pq Cm esp .
+.It Cm features Ar feature-name , Ns Ar ... Op unidirectional
+Enables the computation of one or more previously configured features
+named
+.Ar feature-name , Ns Ar ...
+(see
+.Sx DIFFUSE CONFIGURATION Ns ).
+Features will only be computed for flows
+whose packets match all rule options prior to
+.Cm features .
+Flows are bidirectional by default, unless
+.Ar unidirectional
+is specified here.
+.Pp
+If not explicitly specified, the features option is automatically generated for
+every rule that has one or more feature matches, match-if-class options, or a
+mlclass action.
+.It Qq Ar feature-stat-name Ns Bro < Ns | Ns <= Ns | Ns = Ns | Ns >= Ns | Ns > Brc Ns value
+Matches a packet if the feature
+.Ar feature-stat-name
+is smaller, smaller equal, equal, larger equal or larger than the specified
+value.
+A
+.Ar feature-stat-name
+is expressed as:
+.Pp
+.Op Bro Ar fwd Ns | Ns Ar bck Brc Ns . Ns
+.Ar stat-name . Ns Ar feature-name
+.Pp
+fwd and bck indicate the direction (bidirectional flows),
+.Ar stat-name
+defines the name of the statistics provided by the feature module, and
+.Ar feature-name
+is the name of the configured feature (see
+.Sx DIFFUSE CONFIGURATION Ns ).
+.Pp
+For bidirectional flows, omitting "fwd" or "bck" implies the forward direction
+of flows.
+For unidirectional flows or features computed in both directions of a
+bidirectional flow, neither "fwd" or "bck" should be specified.
+.Pp
+The whole option must be specified in double quotes if necessary
+to prevent the shell from interpreting the ">" etc.
.It Cm fib Ar fibnum
Matches a packet that has been tagged to use
the given FIB (routing table) number.
@@ -1537,6 +1634,14 @@ Values can be entered as decimal or hexa
and they are always printed as hexadecimal (unless the
.Cm -N
option is used, in which case symbolic resolution will be attempted).
+.It Cm match-if-class Ar classifier Ns : Ns Ar class , Ns Ar ...
+Matches packets that were classified by a previously configured classifier
+.Ar classifier
+as one of the classes listed.
+Each
+.Ar class
+can be a class name or a # followed by the class number (see
+.Sx DIFFUSE CONFIGURATION Ns ).
.It Cm proto Ar protocol
Matches packets with the corresponding IP protocol.
.It Cm recv | xmit | via Brq Ar ifX | Ar if Ns Cm * | Ar ipno | Ar any
@@ -2393,6 +2498,510 @@ interface is not available after process
so those packets are dropped in the output path.
Care should be taken to ensure that link-local packets are not passed to
.Nm dummynet .
+.Sh DIFFUSE CONFIGURATION
+With the
+.Xr diffuse 4
+extension,
+.Nm
+can be used to classify flows using machine learning and to export the
+information to remote hosts, which can perform actions on these classified
+flows, e.g. block or rate limit them.
+.Pp
+.Nm DIFFUSE
+operates by first using the firewall to select packets.
+Selected packets are grouped into bidirectional or unidirectional flows, for
+which characteristics (features) are computed.
+Based on the features, flows can then be classified using machine learning.
+Classified flows can be exported to remote hosts to trigger remote actions.
+.Pp
+Packets are classified into flows similar to
+.Nm Ns 's
+generation of dynamic rules (if keep-state is used).
+Bidirectional flows are like dynamic rule entries.
+However,
+.Nm DIFFUSE
+flows can be unidirectional too (if required).
+The timeout of flows also works the same way as for dynamic rules.
+The timeout values for different protocols are configurable (sysctl variables).
+.Pp
+.Nm DIFFUSE
+adds three new objects:
+.Bl -hang -width "classifier"
+.It Em feature
+A
+.Em feature
+is a configured instance of a feature module (implemented as a kernel module).
+Features are not single statistics but a set of statistics related to some
+characteristic of a flow, e.g. packet length.
+Features can be computed independently for each packet but they can also be
+computed over a series of packets, called a flow.
+.It Em classifier
+A
+.Em classifier
+is a configured instance of a machine learning classifier algorithm (implemented
+as a kernel module).
+A classifier takes a set of feature statistics as input and based on these
+assigns a class to the packet or flow.
+.It Em export
+An
+.Em export
+is a configured instance of the export function.
+The only way to currently export classified flow information from the kernel is
+via UDP.
+An
+.Em export
+sends the information directly or via a
+.Xr diffuse_exporter 8
+instance to one or more action node(s), which then based on the class of flows
+can perform various actions, such as blocking or shaping.
+.El
+.Ss FEATURES
+Before features can be used they must be configured using the config
+command:
+.Pp
+.Nm
+.Cm feature
+.Ar feature-name
+.Cm config module
+.Ar module-name
+.Op Ar module-options
+.Pp
+The
+.Ar feature-name
+is the name of the new feature instance.
+It cannot be the name of an existing feature instance.
+.Ar module-name
+is the name of a feature module.
+See
+.Xr diffuse 4
+for the list of existing modules.
+.Pp
+Since the "plen" and "iat" modules are used in the examples below we briefly
+explain their statistics here.
+The "plen" module computes minimum, mean, maximum, and variance of packet
+length, and the "iat" module computes minimum, mean, maximum, and variance
+of packet inter-arrival times.
+.Pp
+For each existing feature module
+.Nm diffuse
+will create a default feature instance with default configuration that
+has the same name as the module.
+If the default configuration is deemed adequate the default feature can be use
+as is without any further configuration.
+.Pp
+However, in many cases the default configuration will not be appropriate.
+The following command creates a new feature instance of the "plen" module:
+.Pp
+.Dl "ipfw feature myplen config module plen window 25"
+.Pp
+The
+.Ar module-options
+are module specific, and the -h parameter can be used to view available options
+e.g. the following command will show the plen module options:
+.Pp
+.Dl "ipfw feature myplen config module plen -h"
+.Pp
+To view the properties of configured features use:
+.Pp
+.Nm
+.Cm feature show
+.Brq Ar feature-name | Cm all
+.Pp
+This shows the details for the configured feature
+.Ar feature-name
+or for all configured features if "all" is specified.
+.Pp
+Configured features can be deleted using:
+.Pp
+.Nm
+.Cm feature delete
+.Ar feature-name
+.Ss CLASSIFIERS
+Before classifiers can be used they must be configured using the config
+command:
+.Pp
+.Nm
+.Cm mlclass
+.Ar classifier-name
+.Cm config algorithm
+.Ar algorithm-name
+.Op use-feature-stats Ar feature Ns , Ns ...
+.Op class-names Ar name Ns , Ns ...
+.Op confirm Ar number
+.Op Brq Cm sample Ar number | Cm rnd-sample Ar prob
+.Op Ar algorithm-options
+.Pp
+The
+.Ar classifier-name
+is the name of the new classifier instance.
+It cannot be the name of an existing classifier instance.
+.Ar algorithm-name
+is the name of a classifier module, which implements the actual classification
+algorithm.
+See
+.Xr diffuse 4
+for a list of classifiers.
+The "nbayes" algorithm used in the following examples classifies packets or
+flows based on the Bayes formula.
+.Pp
+By default the names of the feature statistics used are read from the model
+file.
+However, if the model file does not contain valid feature statistic names or one
+wants to use different statistics, the parameter
+.Cm use-feature-stats
+allows the definition of features used by the classifier as part of the
+classifier configuration.
+Feature statistics are specified as for feature matches (see rule options).
+.Pp
+By default the names of the classes are read from the model file too.
+However, the parameter
+.Cm class-names
+allows to override the names specified in the model file as part of the
+classifier configuration.
+.Pp
+The parameter
+.Cm confirm
+controls how many times a class has to be confirmed before a
+.Cm match-if-class
+will match.
+For example, if
+.Cm confirm
+is set to 2 a
+.Cm match-if-class
+will only match if at least 3 consecutive packets were classified as the same
+class.
+By default
+.Cm confirm
+is zero; as soon as all features are computed each packet can be matched with a
+.Cm match-if-class .
+.Pp
+By default a classifier classifies every packet for which all needed features
+have been computed.
+Sampling can be used to only classify a subset of packets to reduce CPU load.
+The parameter
+.Cm sample
+can be used to only execute the classifier every
+.Ar number
+packets.
+The parameter
+.Cm rnd-sample
+can be used to randomly sample packets for which the classifier is executed.
+The previous packet's class (if any) is assigned to non-sampled packets.
+.Pp
+To facilitate a quick classification of new flows the first packet
+where all features have been computed is always classified, regardless of the
+sampling parameters.
+Note: specify only
+.Cm sample
+or
+.Cm rnd-sample
+but not both!
+.Pp
+Since a classifier configuration totally depends on the classifier model, no
+default classifier instances are generated.
+.Pp
+The following command creates a new classifier instance for the "nbayes"
+algorithm using a fictitious classifier model called example_nbayes.diffuse.
+.Bd -literal -offset indent
+ ipfw mlclass myclass config algorithm nbayes model \\\
+ example_nbayes.diffuse
+.Ed
+.Pp
+The
+.Ar module-options
+depend on the algorithm.
+To view the available options the -h parameter can be used, for example:
+.Pp
+.Dl "ipfw mlclass myclass config algorithm nbayes -h"
+.Pp
+will show all the options the "nbayes" algorithm provides.
+.Pp
+To view the properties of configured classifiers use:
+.Pp
+.Nm
+.Cm mlclass show
+.Brq Ar class-name | Cm all
+.Pp
+This shows the details for the configured classifier
+.Ar class-name
+or for all configured classifiers if "all" is specified.
+.Pp
+Configured classifiers can be deleted using:
+.Pp
+.Nm
+.Cm mlclass delete
+.Ar class-name
+.Ss EXPORTS
+Before exports can be used they must be configured using the config command:
+.Pp
+.Nm
+.Cm export
+.Ar export-name
+.Cm config target udp:// Ns Ar host Ns
+.Op : Ns Ar port
+.Op action Ar action-name
+.Op action-params Ar action-params
+.Op min-batch Ar min-batch-number
+.Op max-batch Ar max-batch-number
+.Op max-delay Ar max-delay-number
+.Op confirm Ar confirm-number
+.Op unidirectional
+.Pp
+The
+.Ar export-name
+is the name of the new export instance.
+It cannot be the name of an existing export instance.
+.Ar action-name
+and
+.Ar action-params
+are the action name and parameters that are sent for matching flows.
+Note that the receiver may overrule these with locally specified actions.
+.Pp
+Note that the arguments of
+.Ar action-name
+and
+.Ar action-params
+are currently opaque values for DIFFUSE i.e. no checking is performed.
+Also note that the length of the arguments is currently limited to
+.Em seven
+characters and longer arguments are truncated.
+.Pp
+The argument of
+.Cm target
+specifies the receiver.
+The protocol must be UDP,
+.Ar host
+is the fully qualified host name, and
+.Ar port
+is the port number of the export target.
+.Pp
+.Cm min-batch-number
+is the minimum number of flows exported in one batch.
+Similarly,
+.Ar max-batch-number
+is the maximum number of flows exported in one batch (must be equal or larger
+than
+.Ar min-batch-number ).
+These parameters allow controlling the export batch size.
+Note that increasing
+.Ar min-batch-number
+also increases the delay for delivering flow information.
+.Pp
+.Ar max-delay-number
+specifies a maximum delay between the generation of export information and the
+actual export.
+Note that if
+.Ar max-delay-number
+is set (value > 0) the minimum batch size is still enforced, but the maximum
+batch size can now be exceeded (if more records are over the maximum delay than
+the maximum batch size).
+.Pp
+.Ar confirm-number
+is used to specify how many times a flow has to be consecutively classified as the
+.Em same
+class before it is exported.
+For example, if
+.Ar confirm-number
+is set to 2, information is only exported if the class was confirmed twice
+(meaning three consecutive classifications resulting in the same class).
+.Pp
+By default the receiver will treat flows as bidirectional, i.e. apply actions
+to both directions of a flow.
+The
+.Cm unidirectional
+option specifies that the receiver should treat flows as unidirectional.
+However, based on local configuration the receiver may still decide to treat
+flows differently than indicated by the classifier.
+.Pp
+Note that all UDP packets send by an export rule bypass the firewall.
+.Pp
+The following command creates a new export instance that exports data to
+localhost using the default port (port 3191).
+The minimum batch size is set to 2 and the maximum batch size is set to 10. Note
+that packets send by an export rule bypass the firewall.
+.Pp
+.Bd -literal -offset indent
+ ipfw export myexp config target udp://localhost min-batch 1 \\\
+ max-batch 5
+.Ed
+.Pp
+To view the properties of configured exports use:
+.Pp
+.Nm
+.Cm export show
+.Brq Ar export-name | Cm all
+.Pp
+This shows the details for the configured export
+.Ar export-name
+or for all configured exports if "all" is specified.
+.Pp
+Configured exports can be deleted using:
+.Pp
+.Nm
+.Cm export delete
+.Ar export-name
+.Pp
+.Ss FLOWTABLE
+.Nm DIFFUSE
+has a flow table to keep state of any flows for which features are computed, and
+which are classified using a classifier.
+The current list of flows in the table can be viewed using the following
+command:
+.Pp
+.Nm
+.Cm flowtable show
+.Op expired
+.Pp
+By default only active flows are shown.
+If expired is specified, expired flows are also shown. Note, that any expired
+flows are eventually deleted.
+.Pp
+The command outputs the list of features followed by one line for each flow in
+the table.
+Each line contains the following fields:
+.Bl -bullet
+.It
+Rule number of the rule that generated the flow
+.It
+Bucket number in the hash table
+.It
+Number of packets
+.It
+Number of bytes
+.It
+Expire time
+.It
+Protocol, source IP and port (UDP, TCP), destination IP and port (UDP, TCP)
+.It
+List of all feature statistics with their current values
+.It
+List of classes (if the flow has been classified)
+.El
+.Pp
+The packet and byte counters can be reset using:
+.Pp
+.Nm
+.Cm flowtable zero
+.Pp
+All entries can be removed from the flow table using:
+.Pp
+.Nm
+.Cm flowtable flush
+.Pp
+The flow table has a fixed number of buckets, but it can be re-sized.
+First, change the sysctl variable net.inet.ip.diffuse.ft_buckets to the desired
+value, and then flush the flow table.
+The sysctl variable net.inet.ip.diffuse.ft_curr_buckets shows the actual number
+of buckets.
+.Ss FLOWS AND FEATURE COMPUTATION
+The first rule with a features option a packet hits triggers the flow lookup and
+flow state generation.
+Later rules for the same packet then use a pointer to the flow entry.
+This means whether a flow is bidirectional or unidirectional is determined by
+the rule creating the flow, and later rules cannot change this.
+.Pp
+The use of bidirectional flows is recommended (the default).
+It is possible to use unidirectional flows with the "unidirectional" option of
+"features".
+Do not mix both types unless you know what you are doing.
+.Pp
+The "features" option is implicitly generated for rules with feature matches,
+"match-if-class" options or a classifier action, even if not explicitly
+specified.
+Implicit "features" options list all the features used by the feature matches or
+the classifier and will create bidirectional flows.
+.Pp
+Features of a flow are updated by every rule a packet hits that has a "features"
+option.
+Each feature is of course only updated once for each packet.
+The maximum number of features per flow is currently limited to
+.Em eight .
+Rules that attempt to update features past this limit do not match, as will
+subsequent rules that attempt to use the non-existing features.
+.Pp
+Note that since the first rule (with terminating action) that matches a packet
+terminates the search, rules that update features but do not see all packets
+effectively compute the features over a subset of packets.
+This allows to create hierarchical features, such as packet length statistics
+can be computed only for packets smaller/larger than a threshold.
+.Pp
+However, in other cases all features must be computed over all packets.
+An initial rule with non-terminating action does this:
+.Bd -literal -offset indent
+ ipfw add 1 count ip from any to any features plen,iat
+ ...
+.Ed
+.Pp
+Now the "plen" and "iat" features will always be computed over all
+IP packets regardless of the following rules.
+.Pp
+.Ss EXAMPLES
+The first example configures a new feature instance and uses it to match only
+flows with large packets in both directions:
+.Pp
+.Bd -literal -offset indent
+ ipfw feature myplen config module plen window 25
+ ipfw add allow ip from any to any "fwd.max.myplen>1000" \\\
+ "bck.max.myplen>1000"
+ ipfw add deny ip from any to any
+.Ed
+.Pp
+The following rules create a classifier instance and use it to classify flows.
+Then count actions are used to count the matching packets.
+(The example classifier model only works for IPv4.)
+.Bd -literal -offset indent
+ ipfw mlclass myclass config algorithm nbayes model \\\
+ et_vs_other_plenonly.nbayes.diffuse
+ ipfw add count ipv4 from any to any match-if-class myclass:#0
+ ipfw add count ipv4 from any to any match-if-class myclass:#1
+.Ed
+.Pp
+The following rules do the same but use the "tag" action and "tagged" option:
+.Bd -literal -offset indent
+ ipfw mlclass myclass config algorithm nbayes model \\\
+ et_vs_other_plenonly.nbayes.diffuse
+ ipfw add mlclass myclass ipv4 from any to any class-tags 10,20
+ ipfw add count ipv4 from any to any tagged 10
+ ipfw add count ipv4 from any to any tagged 20
+.Ed
+.Pp
+The following rules create a classifier instance and an export instance.
+Here we explicitly specify the feature statistics and class names.
+Instead of class numbers class names are used in the "match-if-class" here.
+Flows of class "et" are then exported.
+.Bd -literal -offset indent
+ ipfw mlclass myclass config algorithm nbayes model \\\
+ et_vs_other_plenonly.nbayes.diffuse \\\
+ use-feature-stats fwd.min.myplen,fwd.mean.myplen,\\\
+ fwd.max.myplen,fwd.stdev.myplen,bck.min.myplen,\\\
+ bck.mean.myplen,bck.max.myplen,bck.stdev.myplen \\\
+ class-names other,et
+ ipfw myexp config target udp://localhost min-batch 5
+ ipfw add export myexp ipv4 from any to any match-if-class myclass:et
+.Ed
+.Ss SYSCTLS
+.Nm DIFFUSE
*** DIFF OUTPUT TRUNCATED AT 1000 LINES ***
More information about the svn-src-projects
mailing list