Network-related device drivers and MPSAFEty: status, and plans for 7.0

[Robert Watson]

Just in case you missed it, a significant activity in the FreeBSD source 
tree for the past five years has been SMPng, the movement from a single 
"Giant" lock around the kernel to alternative synchronization models that 
include atomic operations, fine-grained locking, per-CPU data structures, 
etc.  A significant part of this work was to remove the Giant lock from 
the network stack, where it has now been largely removed.  The remaining 
sections consist of:

- Legacy network protocol code which needs to be removed or fixed.  This
   includes KAME IPSEC (not FAST_IPSEC), some older ATM portions, and some
   tty-related parts (SLIP, kernel PPP).  For most of these portions, which
   I'll send e-mail about in a week or two to cover the state of, we now
   either have owners or a plan.  There are some notable exceptions.

- A number of network device drivers, which run under a compatibility mode
   that requires acquiring Giant in the inbound path, and possibly
   deferring work to a Giant-holding context in the outbound path.

This e-mail is about the second category -- device drivers which still run 
with or require Giant.  These in turn can be broken out into two 
categories:

- Device drivers that appear to have been modified to run MPSAFE, and are
   possibly running partly Giant-free (i.e., perhaps the outbound path but
   not interrupt handler, or vice versa).

- Device drivers that haven't been touched and rely on Giant entirely.

The plan for FreeBSD 7.0 (i.e., HEAD) is that all network device drivers 
will be required to *operate without Giant*.  This is necessary because 
all network protocol parts will be running without Giant, and the 
compatibility shims that allow older device drivers to run with Giant will 
be removed.  Obviously, the details are subject to discussion, but I 
thought it would be helpful to do a sweep of the drivers and identify ones 
that either require some further work, or are definitely on the chopping 
board.  Nothing will be removed until within a few months of the RELENG_7 
branch, so we have quite a bit of time (about a year) to get this cleaned 
up before device drivers start disappearing.  This will not be my last 
e-mail on the topic, you will get intermittent updates on how things are 
going.  Specifically, however:

- All network device drivers should be running INTR_MPSAFE (exceptions for
   device driver frameworks that don't support it).

- No network device drivers should be running IFF_NEEDSGIANT (possibly
   excepting device driver frameworks that don't support it).

First, a list of device drivers in the "partly or mostly done" category. 
Some require a lot of testing and work; others require a bit of code 
deletion or changing a flag and a bit of testing.  If you are the 
maintainer/owner of one of these drivers, or play one on TV, I would love 
to see this work done by the end of August.  And ideally, I'd love to see 
it MFC'd to RELENG_6.  In general, this is low-risk work.

if_an		Aironet Communications 4500/4800 wireless network adapter
 		driver.  Mutexes present and exercised; not marked
 		IFF_NEEDSGIANT, but not marked INTR_MPSAFE.  Maybe we can
 		just throw the switch?

if_cp		Cronyx Tau-PCI WAN adapters -- runs MPSAFE, but still has
 		frobts to run un-MPSAFE.  That should be easy to fix.

if_ct		Cronyx-Tau adapter -- runs MPSAFE, but still has frobs to
 		run un-MPSAFE.  That should be easy to fix.

if_cx		Driver for synchronous/asynchronous Cronyx Sigma WAN
 		adapters -- runs MPSAFE, but still has frobs to run
 		un-MPSAFE.  That should be easy to fix :-).

if_dc		Tulip driver -- mutexes present, not used unless IS_MPSAFE
 		set to non-0, so un-exercised.

if_el		Ethernet driver for 3Com Etherlink 3C501 devices.
 		Mutxes present and exercised, not marked MPSAFE.

if_en		Device driver for Midway-based ATM interfaces.  Mutexes
 		present and exercised; not marked IFF_NEEDSGIANT, but not
 		marked INTR_MPSAFE.  Maybe we can just throw the switch?

if_fatm		Device driver for Fore PCA200E ATM interfaces.  Mutexes
 		present and exercised; not marked IFF_NEEDSGIANT, but not
 		marked INTR_MPSAFE.  Maybe we can just throw the switch?

if_fxp		Intel EtherExpress PRO/100 Ethernet device driver.
 		Mutexes present and exercised; not marked IFF_NEEDSGIANT,
 		but not marked INTR_MPSAFE (see if_fxpvar.h#define).
 		Also, Giant passed into busdma for call-back lock.  Maybe
 		we can just throw the switch?

if_hatm		Device driver for Fore/Marconi HE155 and HE622 ATM
 		interfaces.  Appears MPSAFE and ready to go, only it looks
 		like it uses a non-default kenv value to key MPSAFEty.
 		Maybe we can just throw the switch?

if_hme		Sun Microelectronics STP2002-STQ Ethernet interfaces
 		device driver.  Generally fine, but Giant referenced in
 		budma callback.  Can this be fixed easily?

if_my		Myson Technology Ethernet PCI driver.  Mutexes present and
 		exercised, not marked IFF_NEEDSGIANT, but not not marked
 		INTR_MPSAFE.  Maybe we can just throw the switch?

if_nve		NVIDIA nForce MCP Networking Adapter device driver.
 		Mutexes present and exercised; not marked IFF_NEEDSGIANT,
 		but not marked INTR_MPSAFE.  Also, busdma callbacks
 		reference Giant.  Maybe we can just throw the switch?

if_patm		Device driver for IDT77252 based ATM interfaces (ProSum
 		and IDT).  Mutexes present and exercised, not marked
 		IFF_NEEDSGIANT, but not marked INTR_MPSAFE.  Maybe we can
 		just throw the switch?

if_pcn		AMD Am79c972 fast ethernet PCI NIC driver. Mutexes
 		present and exercised, not marked MPSAFE.

if_pdq		DEC PDQ FDDI Controller.  Mutexes present and exercised.
 		Not marked IFF_NEEDSGIANT, but also not marked
 		INTR_MPSAFE.  A property of EISA MPSAFEty?

if_sf		Adaptec AIC-6915 "Starfire" PCI fast ethernet driver.
 		Mutexes present and exercised, not marked MPSAFE.

if_ste		Sundance Technologies ST201 Fast Ethernet device driver.
 		Mutexes present and exercised, not marked MPSAFE.

if_ti		Alteon Networks Tigon PCI gigabit ethernet driver.
 		Mutexes present and exercised, not marked MPSAFE.

if_tl		Texas Instruments ThunderLAN Ethernet device driver.
 		Mutexes present and exercised, not marked MPSAFE.

if_wb		Winbond fast ethernet PCI NIC driver.  Mutexes present
 		and exercised, not marked MPSAFE.

if_owi		Lucent Hermes, Intersil PRISM and Spectrum24 IEEE 802.11
 		driver (old).  Mutexes present and exercised, not marked
 		MPSAFE.

if_aue, if_axe, if_cdce, if_cue, if_kue, if_rue, if_udav, if_ural
 		Countless USB device drivers -- locking present,
 		exercised, but all running under Giant, presumably due to
 		the USB framework.  Someone has a big TODO item here.

if_wl		T1 speed ISA/radio lan card.  Mutexes present and
 		exercised, not marked IFF_NEEDSGIANT, but not marked
 		INTR_MPSAFE.  Easy to fix?


The second category was device drivers that basically haven't been touched 
at all with respect to synchronization since they were introduced into the 
tree, and in most cases, these drivers were introduced into the tree 
before SMPng began.  These drivers are often aging ISA device drivers, and 
are often quite simple in structure so could be made MPSAFE quite easily, 
at least if you had the hardware on hand to test.  There's probably quite 
a bit of this hardware floating around, so my expectation is that at least 
a third of these drivers will be made MPSAFE before I start having to 
start setting deadlines.  If you have, or better yet, regularly use one of 
these drivers, please take a look at doing the work!

Here they are:

if_ar		synchronous Digi/Arnet device driver -- has mutexes, but
 		not used really.

if_arl		Aironet Arlan 655 wireless network adapter driver -- no
 		locks at all.

if_awi		AMD PCnetMobile IEEE 802.11 PCMCIA wireless network
 		driver -- no locks at all.

if_cm		SMC Arcnet Ethernet device driver -- no locks at all.

if_cnw		Netwave AirSurfer wireless network driver -- no locks at
 		all.

if_cs		Crystal Semiconductor CS8900 and CS8920 NICs -- no locks
 		at all.

if_de		DEC 21040 PCI Ethernet Controller -- no locks at all.

if_ed		NE-2000 and WD-80x3 Ethernet driver -- no locks at all.

if_ex		Ethernet device driver for the Intel EtherExpress Pro/10
 		and Pro/10+ -- macros defined, but no locking at all.

if_fe		Fujitsu MB86960A/MB86965A based Ethernet adapters -- no
 		locks at all.

if_fwe		Ethernet emulation driver for FireWire -- no locks at all.

if_fwip		IP over FireWire driver -- no locks at all.

if_gem		ERI/GEM/GMAC Ethernet device driver -- no locks at all.

if_harp		Pseudo physical interface driver for HARP -- no locks at
 		all.

if_ie		8-bit and 16-bit ISA ethernet cards based on the Intel
 		i82586 chip -- no locks at all.

if_ic		I2C bus IP driver -- no locks at all.

if_lge		Level 1 LXT1001 NetCellerator PCI Gigabit Ethernet adapter
 		driver -- no locks at all.

if_lnc		AMD Lance/PCnet Ethernet device driver -- no locks at all.

if_mn		Unlocked netgraph node?

if_oltr		Olicom Token Ring device driver -- no locks at all.

if_plip		Printer port Internet Protocol driver -- no locks at all.

if_ray		Raytheon Raylink/Webgear Aviator PCCard driver -- no locks
 		at all.

if_sbsh		Granch SBNI16 SHDSL modem device driver -- no locks at
 		all.

if_sbni		Granch SBNI12 leased line modem driver -- no locks at all.

if_snc		National Semiconductor DP8393X SONIC Ethernet adapter
 		driver -- no locks at all.

if_sr		Synchronous RISCom/N2 / WANic 400/405 device driver -- no
 		locks at all.

if_tx		SMC 83c17x Fast Ethernet device driver -- no locks at all.

if_txp		3Com 3XP Typhoon/Sidewinder (3CR990) Ethernet interface --
 		has mutexes, but they're not used.

if_vx		3Com EtherLink III / Fast EtherLink III (3c59x) Ethernet
 		driver -- no locks at all.

if_xe		Xircom PCMCIA Ethernet device driver -- no locks at all.

This list might have a few mis-categorized, I might have missed a few 
device drivers, etc.  If you are using a device driver and it's not marked 
MPSAFE, and it's also not on this list, you should still worry, and 
ideally do something about it.

We have about a year -- that's lots of time, if people start now. :-)

Robert N M Watson