slow "zfs destroy snapshot" with predictable time pattern

[Royce Williams]

On Sun, Feb 27, 2011 at 9:30 AM, Royce Williams
<royce.williams at gmail.com> wrote:
> I thought that 'zfs snapshot destroy' should be fast (on the order of
> a few seconds), but mine are taking a predictably long time on a
> pretty modest filesystem (details below).
>
> I discovered this when a typo caused many more snapshots than I
> intended (every minute!); I had about 12,000 of them before I noticed.
>  Destroying the first snapshot took about 39 wallclock seconds on an
> otherwise idle system.  A few more destroys took almost exactly the
> same amount of time.
>
> I know little about ZFS under the hood, but I wanted to investigate a
> little bit. I scripted a loop of 'time zfs destroy snapshot' and let
> it run overnight.  Each destroy was consistently taking 37-40 seconds,
> but then after hundreds of deletions in that time range, I saw a
> jagged spike, followed by a consistent drop that has stayed in the
> 23-25s range:
>
> [hours of 38-39s destroys snipped]
>
> real    0m38.205s
> real    0m38.455s
> real    0m38.580s
> real    0m37.414s
> real    0m35.330s <-- small drop here
> real    0m35.347s
> real    0m35.380s
> real    0m35.355s
> real    0m35.255s
> real    0m35.514s
> real    0m35.422s
> real    0m35.464s
> real    0m46.121s   <-- small spike here
> real    0m44.630s
> real    0m46.021s
> real    1m19.443s   <-- big spike here
> real    0m40.896s
> real    0m22.848s  <-- drop into the 20s range
> real    0m29.039s
> real    0m29.831s
> real    0m26.348s
> real    0m22.623s
> real    0m29.314s
> real    0m29.589s
> real    0m26.573s
> real    0m22.773s
>
> [hours of of 23-25s destroys snipped]
>
> I know very little about ZFS under the hood, but this model might fit the facts:
>
> * Normally, 'zfs destroy snapshot' is fast (on the order of a few seconds);
>
> * 'zfs destroy snapshot' has to briefly analyze all snapshots prior to
> destruction;
>
> * A particular 'problem' snapshot can slow that full analysis by a
> consistent amount of time;
>
> * Destroying that 'problem' snapshot drops the analysis time by that amount.
>
>
> If my model is correct, I'm going to see one or more spikes, followed
> by corresponding drops, until the destroys return to a reasonable
> rate.
>
> This guy had a problem that might also fit that model -- that
> particular snapshots can be very slow, and removing them removes the
> time delay. That thread notes that it was due to a low-memory
> condition, and OpenSolaris bug 6542681 was filed for it.  I do not
> think that my problem is because of low memory.
>
>    http://www.mail-archive.com/zfs-discuss@opensolaris.org/msg07647.html
>
>
> I have stopped the destroys in case the remaining 'problem' snapshot is useful.
>
> The system is 8.1-SECURITY, amd64, 4GB RAM, no sysctl or loader
> tweaks, ZFS v3, zpool v14, single 58GB ZFS pool.
>
> # zfs list
> NAME              USED  AVAIL  REFER  MOUNTPOINT
> atoz-backup      15.4G  58.0G  25.5K  /atoz-backup
> atoz-backup/usr  15.3G  58.0G  14.8G  /atoz-backup/usr
>
> # df -ki | egrep 'atoz|Filesystem'
> Filesystem                               1024-blocks     Used
> Avail Capacity iused     ifree %iused  Mounted on
> atoz-backup                                 60789979       25
> 60789953     0%       6 121579907    0%   /atoz-backup
> atoz-backup/usr                             76281655 15491701
> 60789953    20%  714124 121579907    1%   /atoz-backup/usr

I left this message open-ended.  I also left out the hardware info.
What I meant to get at was:

* Is ~35 seconds roughly normal for a "zfs destroy snapshot" on a
filesystem with 723K files, single disk, under no other I/O or CPU
load, with an Intel ICH9 SATA300 talking to a Seagate ST3160812AS
(SATA 3)?  Googling seems to indicate that it's not, even if I had
12,000 snapshots.

* If this speed is not normal, does this smell like a bug?  If so, I
can help recreate or test, maybe on 8.2 or 9-CURRENT.

* If this speed is normal, what search terms would have yielded that
fact? (slow OR speed OR problem OR performance) plus "zfs destroy
snapshot" didn't yield much that applied.

* Is my model valid?  I reasoned that if a snapshot is a separate set
of references to all files in the filesystem, and the filesystem
hasn't changed much over time, then destroying each snapshot should
take a relatively constant amount of time. Even if my time expectation
was wrong (that it would be quite a bit shorter, even for 723,000
files), I would not expect the destroy time to drop by 50% in the
middle of the destroy series ... so I think that I'm missing
something.

* If my model isn't valid, what is a better model?  For example, I had
gotten down to just over 10,000 snapshots when I stopped the series of
destroys; could there be a snapshot count threshold, under which each
destroy now takes less time, in stair-step fashion?

In other words: I'm trying to understand what I'm seeing. :-)

Also, sorry for the bad subject line - "zfs destroy snapshot" is
obviously the correct command; I've updated the subject line.

Royce