fix for per-mount i/o counting in ffs

[Bruce Evans]

On Wed, 18 May 2016, Bruce Evans wrote:

> ...
> How does any use of ump->um_devvp work?
>
> I tried revoke(2) on the devvp of a mounted file system.  This worked
> to give v_type = VBAD and v_rdev = NULL, but didn't crash.  ffs_unmount()
> checked for the bad vnode, unlike most places, and failed to clear
> si_mountpt.
>
> Normal use doesn't have revokes, but if the vnode is reclaimed instead
> of just becoming bad, then worse things probably happen.  I think vnode
> ...

I still haven't generated a crash, but revoke certainly does one bad thing:
it breaks detection of busy devices so that the same device can be mounted
more than once.  GEOM was supposed to allow multiple mounts for ro mounts,
but this gave garbage pointers and is turned off.  To turn it back on, use:

     % mount -o ro /dev/ad4s4a /i		# my normal mount
     % mount -o ro /dev/ad4s4a /i		# fails with EBUSY
     % revoke /dev/dev/ad4s4a
     % ls /i					# seems to work
     % mount -o ro /dev/ad4s4a /i		# doesn't fail; clobbers ptrs
     % ls /i					# seems to work
     % umount /i					# seems to work, but clobbers
     % ls /i					# top of stack still there
     % umount /i					# seems to work

Crashes can probably be arranged by writing to the device after it is
revoked.  The device is supposed to be exclusive access or at least ro,
but revoke breaks that.  Or just put 2 independent valid file systems on
the same device in advance or by writing, so as to clobber the pointers
better.

The exclusive access can also be broken using separate devfs instances:

     % mount -o ro /dev/ad4s4a /i
     % mkdir /tmp/dev
     % mount -t devfs devfs /tmp/dev		# normal sort of use for jails?
     % mount -o rw /tmp/dev/ad4s4a /i		# doesn't fail; can even be rw

Perhaps this doesn't clobber pointers near bufobj as badly as the turned off
code, but it certainly clobbers si_mountpt.  Each new mount sets si_mountpt
in the shared cdev struct.  The first unmount sets this to NULL so I think
it never points to garbage.  It just points to the wrong mount struct or
is turned off.

The case of multiple devfs instances has a chance of working since devvp
is separate so assigments to devvp->v_bufobj don't clobber previous
mouts.

I now remember that this prevented me finding a fix for the i/o
counting.  Multiple mounts were supposed to work, but obviously a
single pointer in the cdev cannot work for multiple mounts.  I think it
was removed (breaking the i/o counting) because it was too hard to fix
it to work even for a single mount (since allowing multiple mounts gives
pointer clobbering problems).

Bruce