Running the network stack without Giant -- what to try and when
Robert Watson
rwatson at FreeBSD.org
Sat Jul 17 22:37:50 PDT 2004
As many of you have seen from status reports, e-mails, bug reports, etc,
the FreeBSD Project has been working for some time on getting the network
stack to run in parallel on multiple CPUs. We're now at a point where a
substantial amount of functionality appears to run pretty successfully
without the Giant lock, and we're ready for more people to start running
it that way so we can find and fix problems. Let me start by enumerating
a few caveats:
- While we've been doing pretty heavy testing in MPSAFE configurations,
the nature of multiprocessor development and adapting code for MP safety
means that it's unlikely this will "just work" for all takers. However,
it may well work for many.
- We've focussed primarily on getting mainstream network configurations
to run without Giant: this means that less mainstream subsystems (parts
of IPv6, some netgraph nodes, IPX, etc) are currently unsafe without the
Giant lock turned on. Some less mainstream network devices are also not
currently able to operate without Giant. There is active work in all of
these area to remedy this issue.
- You may run into hard to diagnose problems. We'd like to try to
diagnose them anyway, but if you start to experience new problems,
you'll want to go read the Handbook chapter on preparing kernel bug
reports and diagnosing problems. You'll also want to be prepared to run
the system with INVARIANTS and WITNESS turned on.
- Not all workloads will experience a performance benefit -- some, for
various reasons, will get worse. However, several interesting
performance loads get measurably better. If you don't see an
improvement, or you see things get worse, please don't be surprised --
you may want to look at some of the suggestions I make below on ways to
make the results more predictable. Generally, you shouldn't see
substantial performance degradation, if any, but it can't be ruled out,
especially due to scheduling issues, etc.
- We can and will destroy your data. We don't mean to, because we like
your data, and we try not to, but this is, after all, operating system
development.
With all that in mind, now is in fact a good time to start experimenting
with things, as these changes appear to be relatively stable in our
initial testing. Note that there is some current instability in the CVS
HEAD, and so I'd ask for some caution in reporting problems as being
caused by debug.mpsafenet -- it may or may not be our fault :-). I've
disabled PREEMPTION locally for thread centric testing, but haven't needed
to for other testing.
Here's some technical information on how to get started:
(1) Determine if all of the stack components you will operate with are
MPsafe. For common configurations, answering the following questions
will help you decide this:
- Are you using IPv6, IPX, ATM, or KAME IPSEC? If you answered
yes to any of these questions, it is not yet safe for you to run
without Giant.
- Are your using Netgraph? If yes, it may be that you are not yet
able to run without Giant. It is worth giving it a try, but you
may experience panics, etc, especially in MP configurations.
- Are you using SLIP or kernel PPP (not to be confused with user
ppp, which is what most FreeBSD users use with modems).
- Are you using any physical network interfaces other than the
following: bge, dc, em, ep, fxp, rl, sis, xl, wi.
The following may well work: en, gx, pcn, sf. However, they
have not been marked MPSAFE by the driver maintainer.
NOTE: Do you maintain a network interface driver? Is it not on
this list? Shame on you! Or maybe shame on me for not listing
it, even though it should work. Drop me a private e-mail with
and questions or comments. Please update the busdma driver
status web page with your driver's status.
(2) If you are comfortable that you are using an MPSAFE-supported
configuration, then you can use the following tunable in loader.conf
to disable the Giant lock over the network stack on your system:
debug.mpsafenet="1"
Note that this is a boot-time only flag; you can inspect the setting
with a sysctl, but it cannot currently be changed at runtime.
Do a dmesg and confirm that all your probed network interfaces are
marked as MPSAFE or not GIANT LOCKED (or whatever we call it now). If
you have a network interface that is still GIANT LOCKED, it may not be
able to function correctly with debug.mpsafenet=1. However, if you're
not actively using it, it probably won't cause a problem. For
example, firewire network interfaces can't currently be used with
debug.mpsafenet=1. However, if idle, they shouldn't cause any
problems. We're currently working to improve compatibility with
device drivers that aren't mpsafe, and hope to have a prototype soon.
Some notes:
On SMP-centric performance measurements, such as local UNIX domain socket
use by MySQL on MP systems, I've observed 30%-40% performance improvements
by disabling Giant (some details below). My recommended configuration for
testing out the impact of disabling Giant on MP systems is:
- Set "options ADAPTIVE_MUTEXES" -- this seems to help a lot with
contention and load.
- Disable HTT. In my workloads, which tended to pound the kernel,
this hurt quite a bit. Obviously, the effectiveness of HTT
depends on the instruction mix, so this may not be for you.
- Pick one of ULE and 4BSD, and then try the other. I found 4BSD
helped a lot for MySQL, but I've seen other benchmarks with quite
different results.
- For stability purposes with MySQL, I currently have to disable
PREEMPTION, as the MySQL benchmarks I use are pretty
thread-centric and trigger preemption-related bugs with the kernel
threading bits.
- If you want to measure performance, make sure to disable
INVARIANTS, INVARIANTS_SUPPORT, WITNESS, etc.
Some notes on bug reporting:
- Make sure to identify that you are running with debug.mpsafenet.
If the problem is reproduceable, make sure to indicate if it goes
away or persists when you disable debug.mpsafenet.
- If you appear to be experiencing a hang/deadlock, please try
running with WITNESS. I'd actually like to see most people
running with WITNESS for a bit to shake out lock order issues, as
I've introduced a lot of orders. If experiencing lock order
reversals, please include the full console warning including
stack trace.
- INVARIANTS also considered good. Even if you aren't running
with WITNESS, do run with INVARIANTS.
- If you experience a hang, see if you can get into DDB -- if you
are having problems getting in using a console break, try a serial
console. When debugging, at minimum DDB 'ps' output, along with
traces of interesting processes. Typically interesting will be
processes that appear to be involved in the hang, etc.
Obviously, this requires some intuition about what causes the
hang and I can't offer hard and fast rules here.
- Experimenting with debug.mpsafenet=1 and UP is also interesting,
not just SMP. With PREEMPTION turned on, it may result in lower
latency and/or lower throughput. Or not. Regardless, it's
interesting -- you don't have to have SMP to give it a spin.
FYI, while results can and will vary, I was pleased to observe moving from
a UP->MP speedup of 1.07 on a dual-processor box to a speedup of 1.42 with
the supersmack benchmark using 11 workers and 1000 select transactions
with MySQL. For reference, that was with the 4BSD scheduler and adaptive
mutexes. For loopback netperf with TCP and UDP, I observed no change in
performance (well, 1% better for UDP RR, but basically no change). Note
that the MySQL benchmark here is basically a UNIX domain socket IPC test,
and so real world databases will give pretty different results since they
won't be pure IPC. The results appear to be very sensitive to the choice
of scheduler, and for a variety of reasons I've preferred 4BSD during
recent testing (not least, better results in terms of throughput).
There are a lot of people who have been working on this for quite some
time -- I can't thank them all here, but I will point at the netperf web
page as a place to look for ongoing patches, change logs, and some
credits:
http://www.watson.org/~robert/freebsd/netperf/
I try to keep it up to date about once a week or so as I drop new patch
sets.
Robert N M Watson FreeBSD Core Team, TrustedBSD Projects
robert at fledge.watson.org Principal Research Scientist, McAfee Research
More information about the freebsd-current
mailing list