newbus integration of MOD_QUIESCE

[M. Warner Losh]

In message: <36244.1089791458 at critter.freebsd.dk>
            "Poul-Henning Kamp" <phk at phk.freebsd.dk> writes:
: In message <20040713.174512.11239675.imp at bsdimp.com>, "M. Warner Losh" writes:
: >I've not seen other documentation that explains the event in more
: >detail.  Nor do I see comments in the code explaining it any better.
: >>From reading the code and the docs, MOD_QUIESCE is just a 'are you
: >idle?' question.
: 
: More like "If you are idle, prepare to be unloaded, otherwise yell".

OK.  So there is a race here.  The modules have to cope with the lag
between the MOD_QUIESCE call and the MOD_UNLOAD that will come later.
Can the modules assume that if they get a MOD_QUIESCE that they WILL
get a MOD_UNLOAD later if they return 0?

: >The nasty case I've come up with is what happens when the module is
: >idle (not busy), but becomes busy (not idle) after the MOD_QUIESCE
: >call?
: 
: That is for the module to handle: if it returns zero to MOD_QUIESCE
: it knows it will be unloaded and shouldn't accept any new business.

That's a fairly high standard to live up to.  It would mean setting a
flag and not accepting any new business.  I thought in irc discussions
that this was a 'am I busy or not' call, but it is clear that this has
side effects.  There's no way to ask the module if it is idle or not,
so this is just a small increment over what we have now.

: Finegrained guidance for what a module should do in MOD_QUISCE:
: 
:     Network driver:
: 
: 	If any instance is IF_UP, return EBUSY.
: 	Otherwise detach all instances and do not attach to
: 	any new instances offered.

This sounds a lot like what we call detach in newbus.  Yet this is
different than what we'd talked about in terms of the
'notify/shutdown' split that we've been designing.  This makes it
harder to do that.  I don't like this guidance.  It doesn't map to
anything in newbus that we have or we're planning.

So this would also act as a gate to all further probe routines
returning 'good' as well?  That's a lot of side effects.

This makes it hard to map into other plans we have for newbus, and
hard for us to provide a hook into newbus drivers for this module
message.

Of course whta we have today is very racy (race inserting a CardBus
card and unloading the module), so things aren't made that much worse
by thse change, but it doesn't help much either.

:     Device driver:
: 
: 	If any instance is opened, return EBUSY.
: 	Otherwise destroy_dev all instances and do not attach to
: 	any new instances offered.

Same as the above.  Likely with a lot of wrinkles for bus drivers.  If
pccard were to receive this, does it consider itself busy because it
has children drivers?  Or should it pass the request down to them?
Since the children drivers aren't necessarily modules, we can't just
rely on them.

I think it should behave the same way as if you'd called MOD_QUIESCE
on the child device.  Eg Give we have an ep attached to pccard.
kldunload if_ep should behave the same way as kldunload pccard.  If
if_ep is up, then it can veto the quiesce and the unload won't
happen.

In addition, these sorts of recommendations fundamentally change the
way that drivers unload these days.  Today, kldunload of a typical
network driver will detach it, even if it is up or busy.  It is an
extraordinary driver that does anything differently.

Do you plan on fixing all the drivers/modules in the tree that doesn't
do this now?  Having inconsistant unload behavior is one of the
problems we have in the tree, and it seems like this would make it
worse.

Also, in preparing this message, I noticed a number of drivers that
return EINVAL instead of EOPNOSUPP for MOD_ events they don't
understand.  There are several that also return 0 for events that they
don't understand.  These should be corrected in the tree.  Do you have
plans to do this?

:     GEOM module:
: 
: 	If any provider has non-zero access counts, return EBUSY.
: 	Otherwise orphan all providers, destroy or wither all
: 	geoms + consumers and do not taste/configure new instances.
: 
: MOD_UNLOAD on the other hand means:
: 
:     All kinds of module:
: 
: 	Clean all instances even if they are in use.
: 
: 	Return error only if something cannot be cleaned up, for
: 	instance memory references into the module we cannot reclaim.
: 
: 	The fact that an in-use instance is cleaned makes other
: 	code in the kernel with insufficient error-checks fail is
: 	_not_ a reason to return error, the other code should be
: 	fixed instead.
: 
: And as I said earlier:  if you give -f to kldunload the MOD_QUISCE
: error return is ignored and MOD_UNLOAD issued anyway.

And none of this helps us to unload idle drivers, but I guess that
wasn't your goal.

Finally, I'm not trying to be negative here.  There are just a lot of
issues that haven't been considered yet.  I'm a little grumpy that I
was specifically told that you didn't want to hear about newbus
issues, but I'm trying to be constructive in my comments here none the
less.  I am looking to implement something in newbus with these
messages, so I'm trying to understand it at a high level of detail.

Warner