Spurious witness warning when destroying spin mtx

[Ryan Stone]

Today I saw a spurious witness warning for "acquiring duplicate lock of
same type".  The root cause is that when running mtx_destroy on a spinlock
that is held by the current thread, mtx_destroy calls spinlock_exit()
before calling WITNESS_UNLOCK, which opens up a window in which the CPU can
be interrupted and attempt to acquire another spinlock of the same type as
the one being destroyed.  This patch should fix it:

diff --git a/sys/kern/kern_mutex.c b/sys/kern/kern_mutex.c
index 2f13863..96f43f8 100644
--- a/sys/kern/kern_mutex.c
+++ b/sys/kern/kern_mutex.c
@@ -918,16 +918,16 @@ _mtx_destroy(volatile uintptr_t *c)
        else {
                MPASS((m->mtx_lock & (MTX_RECURSED|MTX_CONTESTED)) == 0);

+               lock_profile_release_lock(&m->lock_object);
+               /* Tell witness this isn't locked to make it happy. */
+               WITNESS_UNLOCK(&m->lock_object, LOP_EXCLUSIVE, __FILE__,
+                   __LINE__);
+
                /* Perform the non-mtx related part of mtx_unlock_spin(). */
                if (LOCK_CLASS(&m->lock_object) == &lock_class_mtx_spin)
                        spinlock_exit();
                else
                        curthread->td_locks--;
-
-               lock_profile_release_lock(&m->lock_object);
-               /* Tell witness this isn't locked to make it happy. */
-               WITNESS_UNLOCK(&m->lock_object, LOP_EXCLUSIVE, __FILE__,
-                   __LINE__);
        }

        m->mtx_lock = MTX_DESTROYED