Snapshots that won't delete [was: Re: ZFS on root booting...]

[Ben Morrow]

Quoth Adam McDougall <mcdouga9 at egr.msu.edu>:
> 
> Were you running a newer kernel with an older format zpool?  I heard
> ixsystems had customers doing that and ran into corruption when they
> tried to modify the zpool in some way (expand? I don't remember).
> http://www.bsdnow.tv/episodes/2014_07_09-zfs_war_stories

Oh! Might this be what's causing a problem I've been meaning to ask
about?

My desktop at home is running (a patched, but not anywhere to do with
ZFS) 10-STABLE from a while ago, with a zpool that was created under
8.2-R and is still at version 15. I have been deliberately not upgrading
it, because I saw no reason to and it seemed safer to leave things as
they were.

Recently, though, my dump script has started having occasional problems
with snapshots that won't delete. Pending further investigation I have
been renaming them to allow the recursive delete to succeed, and (so
far) rebooting has always made it possible to get rid of them. 

/home/mauzo% sudo zfs destroy zroot/DATA/R at broken-dump-20140906
cannot destroy snapshot zroot/DATA/R at broken-dump-20140906: dataset is busy
/home/mauzo% mount | grep @
/home/mauzo% zfs list -o name,origin | grep broken-dump
/home/mauzo% zfs list -o name,userrefs zroot/DATA/R at broken-dump-20140906
NAME                               USERREFS
zroot/DATA/R at broken-dump-20140906         0
/home/mauzo% zfs holds zroot/DATA/R at broken-dump-20140906
NAME                               TAG  TIMESTAMP
/home/mauzo%

Presumably this is indicative of some sort of serious problem with
either the pool or the filesystem, and the only permanent solution is to
rebuild the whole thing? ZFS doesn't seem to have a fsck-equivalent for
the ZFS layer. I've run a scrub, and it found no problems.

I was about to upgrade the machine to the latest 10-STABLE, but I can
put that off if anyone thinks this is worth investigating. Once I reboot
I'm fairly sure the current stuck snapshots will fix themselves, and of
course I can't be sure when this will happen again.

[Also: it would be really nice to have an alias for 'zfs destroy' that
will only destroy snapshots. When running something like 'zfs destroy -r
zroot/DATA at whatever' I find I have to check at least three times that I
have definitely put that @ in the right place before pressing Return.
*Especially* when I'm doing it because I'm running a dump, and it's just
fallen over because it can't delete a snapshot.]

Ben