I'm counting my threads, one, two, three, four, five... [1]

[Poul-Henning Kamp]

I've completed my sweep of the kernel and I belive that we now have a
reliable count of the number of threads currently inside any particular
cdev and consequently, with a bit of adding up, also for each cdevsw.

There are two soft spots marked with XXX (devfs_rule and vm_mmap)
and one big hack, snp(4).

The next step is to add a new method to the cdevsw which purges threads
from the driver.  The best name I can come up with is d_evict().  The
drivers should make this function wakeup(9) on anything a thread could
be sleeping on for the relevant cdev and the tread on waking up should
examine whatever bits it can, realize the hw/driver/whatever is gone
and exit the driver with ENXIO error return.

If this cdevsw method is present, destroy_dev() will use it and will
sleep until all threads have cleared out.

So for a typical driver things will look like this:

foo_evict(struct cdev *dev)
{
	struct foo_sc *sc;

	/* XXX: NB: no locks, only wakeups */
	sc = dev->si_drv1;
	wakeup(sc->bing);
	wakeup(sc->bongle);
	wakeup(sc->bang);
}

foo_detach(dev)
{
	...

	sc->flags |= GONE;
	destroy_dev(sc->cdev);
	/* XXX: we now know there are no threads involved with this cdev */
	...
}

destroy_dev() will work more or less this way:


	...
	dev_lock();
	csw = cdev->si_devsw;
	cdev->si_devsw = NULL
	while (csw->d_evict != NULL && cdev->si_threacount > 0) {
		(csw->d_evict)(cdev);
		msleep(&dev_lock, ..., hz/10);
	}
	dev_unlock();
	...

I will also add a convenience function called destroy_cdevsw() which
will call destroy_dev() on all cdevs using the cdevsw in question.

I belive this gives us the handle we need to unload drivers and remove
hardware without panicing in the lower layers of the kernel.  The
higher layers may still have a thing or two to learn in this respect.

Poul-Henning

[1]  It used to be possible to legally download the D:A:D tune which
inspired the subject, but I can't find the link anymore, sorry :-(

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.