newbus integration of MOD_QUIESCE (was Re: cvs commit: src/sbin/kldunload kldunload.8 kldunload.c )

[M. Warner Losh]

In message: <Pine.NEB.3.96L.1040713183350.88111S-100000 at fledge.watson.org>
            Robert Watson <rwatson at freebsd.org> writes:
: That said, I'm not sure I see the kldunload state changes as incompatible
: with the notion of having most kernel modules become newbussified,

I don't either.  I just wanted to point out
	1) there's no committed newbus hooks
	2) there are plans for similar things in newbus
	3) no design effort was made to accomidate newbus (which I
	   shouldn't have raised)
and we got off in a bunch of different directions...  Let's see if we
can get it back on track.

: I suggest creating a thread named "Newbus the world" on arch@, in which
: the major classes of kernel modules and extension interfaces are
: documented, along with ideas about how they could usefully turn into
: Newbus-enabled pieces.  Poul-Henning's list is a decent starting point,
: but probably not a complete list.  Remember to tell people why switching
: to Newbus would be an improvement, especially for modules and module
: systems that don't clearly map onto synthetic or real devices, such as
: Netgraph nodes, MAC policy modules, schedulers, network protocols, file
: systems, etc.

I think that's likely overkill at this point.  I think you are reading
way more into this than I'd wanted.  And you've gotten my objection
backward, which is taking this discussion away from fruitful waters
which I hope to make progress on the problem at hand.

I don't object that everything isn't newbused.  I object that newbus
interaction wasn't considered enough before this was implemented.
Leaving aside a he said/she said recounting, I'd like to address my
frustion at hand.  I'm trying to answer the question 'Now that this is
in the tree, what should newbus do when one of its modules gets a
MOD_QUIESCE?'

The only documentation I have on this is:

	Add a MOD_QUIESCE event for modules.  This should return error
	(EBUSY) of the module is in use.

[[ assuming the 'of' is a typo for 'if' ]]

and

     On unload it is first called with what set to MOD_QUIESCE .  If the
     unload was not forced, a non-zero return will prevent the unload from
     happening.

     If the unload continues what is set to MOD_UNLOAD.  If the module returns
     non-zero to this, the unload will not happen.

     The difference between MOD_QUIESCE and MOD_UNLOAD is that the module
     should fail MOD_QUIESCE if it is currently in use, whereas MOD_UNLOAD
     should only fail if it is impossible to unload the module, for instance
     because there are memory references to the module which cannot be
     revoked.

I've not seen other documentation that explains the event in more
detail.  Nor do I see comments in the code explaining it any better.
>From reading the code and the docs, MOD_QUIESCE is just a 'are you
idle?' question.

The nasty case I've come up with is what happens when the module is
idle (not busy), but becomes busy (not idle) after the MOD_QUIESCE
call?  Right now newbus modules that receive a MOD_UNLOAD call attempt
to detach all instances of devices contained in that module.  If I
have a way to poll the driver to see if it is busy (which is
relatively easy to implement), then if it becomes busy after the
MOD_QUIESCE call, I get a MOD_UNLOAD which would force instances to
detach.

Case 1:

	driver becomes busy
	MOD_QUIESCE comes in, newbus returns EBUSY

Case 2:

	MOD_QUIESCE comes in, newbus returns 0
	driver becomes busy
	MOD_UNLOAD comes in, newbus detaches the device

In case 1 we have the correct results.  In case 2 we do not unless we
know newbus/the driver remembers it has been quiesced and fails the
operation that causes it to become busy instead of allowing it to
succeed.  If this is true, this implies there's more to the
MOD_QUIESCE call than just asking the question 'Are you idle?'  If
not, then how do we get out of the race here?

Part of the problem here is that some drivers will take a detach
request as a command, and others as a request.  Most of them will
detach unless they really really really can't unwide the state.  Some
will only detach if nothing is open.  Network drivers generally just
detach since they are never open (but may be configured, etc).  So
this is presently a bit of a rats nest of conflicting meanings that
makes it hard to do some things people have suggested:
	1) load a driver and then unload it if nothing attaches
	2) unload drivers (modules) that have no attached drivers but
	   leave the attached ones alone.
MOD_QUIESCE will help these cases a lot, but I'd like to get all the
different meanings in the tree in alignment, and to do that I think I
need a better definition.

So ignore the grumpiness that I had earlier in the day and focus on
this message.

Warner