[PATCH] Netdump for review and testing -- preliminary version

[Attilio Rao]

In the last weeks I worked for porting the netdump infrastructure to
FreeBSD-CURRENT on the behalf of Sandvine Incorporated.
Netdump is a framework that aims for handling kernel coredumps over
the TCP/IP suite in order to dump to a separate machine than the
running one. That may be used on an interesting number of cases
involving disk-less workstations, disk driver debugging or embedded
devices.

GENERAL FRAMEWORK ARCHITECTURE

Netdump is composed, right now, by an userland "server" and a kernel
"client". The former is run on the target machine (where the dump will
phisically happen) and it is responsible for receiving  the packets
containing coredumps frame and for correctly writing them on-disk.
The latter is part of the kernel installed on the source machine
(where the dump is initiated) and is responsible for building
correctly UDP packets containing the coredump frames, pushing through
the network interface and routing them appropriately.

While the server may appear as, pretty much, a simple userland deamon
dealing with UDP packets, the client imposes interesting problems as
long as its activity is linked to handling kernel core dumping. More
precisely, as long as the client is part of the dumping mechanism and
the kernel may be in general panic conditions, netdump must ensure
"robustness" of operations. That is partially achieved by reworking
(and someway replicating) locally some UDP, ARP and IP operations,
hardsetting some values (like the default gateway, the destination and
the client address) and reducing further interactions with the kernel
just to the network interface acitivities.
More specifically, it implements a very basic UDP / IPv4 / ARP stack
separate from the standard stack (since that may not be in a
consistent state).
It can dump to a server on the same LAN or via a router (correctly
specifying the connection gateway).
In order to receive packet on critical conditions, netdump polls the
interface. Every network driver can implement hooks to be used by
netdump independently by DEVICE_POLLING option, even if it is
probabilly a good idea to share some code among them. The reference
set of hooks is contained into "struct netdump_methods".
And if_lem/if_em driver modifies may be set as reference for netdump
hooks implementation.

In order to work into an "up and running" system (meant as with all
the devices in place) the netdump handler hooks as a pre-sync handler
(differently from other dumping routines). It however suffers some
problems typical of other dumping mechanism. For example, on DDB
entering unlocked version of polling handler is used, in order to
reduce the risk of deadlocks during inspections*. That reflects, among
the netdump methods, the existence of 2 versions of polling hooks,
where the "unlocked" is meant as reducing locking as much as possible.

PATCH AND FURTHER WORK

The patch is not totally complete and it is not intended to be
committed in SVN yet. What I'm looking for now is more testing and
review (in particular in terms of architecture) coverage by community.
The server should be in realtively "committable" state, though, but I
encourage its stress-testing. A manpage is provided along that should
be very easy to understand how to use it.

Things that can be further improved, as it is now, in the client, are:
- Deciding if hardcoding of the kernel parameter is done properly. I
personally don't like the sysctl usage and I would prefer an userland
small utility used to testing and maybe add some tunables for enabling
netdump early in the boot. You may have several opinions on this
though.
- VIMAGE and IPv6 support.
- More drivers support. Right now only if_em (and if_lem) are
converted to use netdump and can be used as a draft for other device
drivers. if_ixgb should came along in the final, committing, version
too. In general I think that all drivers supporting device polling
could easilly support also netdump
- Ideally dumpsys() in FreeBSD is too much disk-activity oriented. It
should be made, instead, more neutral and more flexible to cope better
with different interfaces. It is a quite a bit of work, however, and
beyond the scope of netdump introduction (even if it could be
beneficial for it)

Netdump has been developed on a FreeBSD project branch located here:
svn://svn.freebsd.org/base/projects/sv/

which could also forsee further informations about every single
change. However, for your convenience, also a patch has been made
public which is located here (against FreeBSD-CURRENT at 213246):
http://www.freebsd.org/~attilio/Sandvine/STABLE_8/netdump/netdump_alpha_0.diff

In order to enable the client it is enough to add at your kernel config:
options        NETDUMP_CLIENT

NETDUMP_CLIENT_DEBUG can be specified too in order to have further
debuggin traces but I'd encourage to use them just for developing
Netdump itself.

TYPICAL USAGE

This is a set of the available sysctls for netdump:
# sysctl -d net.dump
net.dump: netdump
net.dump.enable: enable network dump
net.dump.retries: times to retransmit lost packets
net.dump.polls: times to poll NIC per retry
net.dump.nic: NIC to dump on
net.dump.gateway: dump default gateway
net.dump.client: dump client

And when compiled with the NETDUMP_CLIENT_DEBUG option:
net.dump.serve# sysctl -d debug.netdump
debug.netdump: Netdump debugging
debug.netdump.crash: force crashingr: dump server

A tipycal setup for netdump can be the following:

Run the server on the receiving interface:

# ifconfig em2
em2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
       options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
       ether 00:30:48:28:b2:7a
       inet 10.135.14.118 netmask 0xfffffffc broadcast 10.135.14.119
       media: Ethernet autoselect (1000baseT <full-duplex>)
       status: active
# netdumpsrv -a 10.135.14.118
Listening on IP 10.135.14.118
Default: dumping on /var/crash/

While on the client:
# netstat -nr
Routing tables

Internet:
Destination        Gateway            Flags    Refs      Use  Netif Expire
10.128.0.0/10      10.135.12.113      UGS        18      149    em2
10.135.12.112/30   link#3             U           0        0    em2
127.0.0.1          link#5             UH          0      118    lo0
...
# ifconfig em2
em2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
       options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
       ether 00:30:48:28:9f:b2
       inet6 fe80::230:48ff:fe28:9fb2%em2 prefixlen 64 scopeid 0x3
       inet 10.135.12.114 netmask 0xfffffffc broadcast 10.135.12.115
       nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
       media: Ethernet autoselect (1000baseT <full-duplex>)
       status: active
# sysctl net.dump.server=10.135.14.118
net.dump.server: 0.0.0.0 -> 10.135.14.118
# sysctl net.dump.client=10.135.12.114
net.dump.client: 0.0.0.0 -> 10.135.12.114
# sysctl net.dump.gateway=10.135.12.113
net.dump.gateway: 0.0.0.0 -> 10.135.12.113
# sysctl net.dump.nic=em2
net.dump.nic: none -> em2
# sysctl net.dump.enable=1
net.dump.enable: 0 -> 1

and at next dumping operation on client, it must netdump.

Thanks,
Attilio

[* The dumping infrastructure in FreeBSD has several weaknesses that
needs to be discussed and treacted separately]


-- 
Peace can only be achieved by understanding - A. Einstein