Geom / destroy_dev() deadlock

[Steven Haber]

Hey FreeBSD devs,

I hope this is the right forum for this. I haven't used the FreeBSD
mailing lists before. I'm a relatively new hire at EMC Isilon. We are
using FreeBSD 7.0 as the basis for our scale-out NAS product line. We
are frequently hitting the deadlock described here:

http://lists.freebsd.org/pipermail/freebsd-geom/2010-February/003888.htm
l

The race is that destroy_dev() sleeps indefinitely waiting for thread
references to drop from a dev. In the case of geom, we hold the topology
lock the whole time we're in the dev layer. In devfs_open() and
devfs_close(), we take a ref on the dev before calling into to geom.

The proposed solution of destroy_dev_sched() makes sense to me. I am
trying to understand the necessity of Alexander Motin's additional
changes. destroy_dev_tq() holds the devmtx during the destroy and devfs
uses this lock to take refs before calling into geom. To me it seems we
should be able to protect the cdev with just the devmtx, provided
consumers of d_close(), d_open(), etc. ref and rel the dev
appropriately.

Steven Haber
Software Engineer, EMC Isilon
(206) 753-1526