How do I synchronize with the return from a bio_done handler?

From: Olivier Delande <olivier.delande_at_gmail.com>
Date: Mon, 11 Oct 2021 09:51:22 UTC
(Apologies if freebsd-geom is not the most appropriate list to which
to post this. This is not strictly about the GEOM framework, but
rather about some of its ingredients: struct bio and asynchronous
completion callbacks. That is why I chose this list over more generic
ones like freebsd-drivers. Please let me know if I should move this
elsewhere.)

I am writing a kernel module that performs some I/O on a block device.
To this end, I create struct bio instances with g_new_bio() which I
then submit to the block device's d_strategy method. In my bio_done
handler, I handle the result of the operation, then free the struct
bio with g_destroy_bio(). This strategy seems to be working.

I wonder how to properly shut down my module. I can easily arrange to
stop submitting new bios, but how can I then wait until no one is
running my bio_done handler any more? Keeping track of the number of
in-flight bios (e.g. by incrementing a counter before posting a bio,
and decrementing it right before returning from the bio_done handler)
is not enough, because there is a race window between the final
decrement bringing the counter to 0 and the actual return(s) from the
handler. I need to synchronize with something happening after the
handler has returned. Unfortunately, since freeing the bio inside the
handler as I do seems acceptable (other drivers, like
cam/ctl/ctl_backend_block.c, do so as well), it is likely that the bio
is never touched after the handler has returned. Is there any other
point where something synchronizes with the bio_done handlers?
(Perhaps releasing the reference to the block device and its switch
table with dev_relthread() does wait for the handlers to finish
running, but this is just a wild guess.)

Best regards,
Olivier