slow "zfs destroy snapshot" with predictable time pattern
royce.williams at gmail.com
Mon Feb 28 00:05:11 UTC 2011
On Sun, Feb 27, 2011 at 9:30 AM, Royce Williams
<royce.williams at gmail.com> wrote:
> I thought that 'zfs snapshot destroy' should be fast (on the order of
> a few seconds), but mine are taking a predictably long time on a
> pretty modest filesystem (details below).
> I discovered this when a typo caused many more snapshots than I
> intended (every minute!); I had about 12,000 of them before I noticed.
> Destroying the first snapshot took about 39 wallclock seconds on an
> otherwise idle system. A few more destroys took almost exactly the
> same amount of time.
> I know little about ZFS under the hood, but I wanted to investigate a
> little bit. I scripted a loop of 'time zfs destroy snapshot' and let
> it run overnight. Each destroy was consistently taking 37-40 seconds,
> but then after hundreds of deletions in that time range, I saw a
> jagged spike, followed by a consistent drop that has stayed in the
> 23-25s range:
> [hours of 38-39s destroys snipped]
> real 0m38.205s
> real 0m38.455s
> real 0m38.580s
> real 0m37.414s
> real 0m35.330s <-- small drop here
> real 0m35.347s
> real 0m35.380s
> real 0m35.355s
> real 0m35.255s
> real 0m35.514s
> real 0m35.422s
> real 0m35.464s
> real 0m46.121s <-- small spike here
> real 0m44.630s
> real 0m46.021s
> real 1m19.443s <-- big spike here
> real 0m40.896s
> real 0m22.848s <-- drop into the 20s range
> real 0m29.039s
> real 0m29.831s
> real 0m26.348s
> real 0m22.623s
> real 0m29.314s
> real 0m29.589s
> real 0m26.573s
> real 0m22.773s
> [hours of of 23-25s destroys snipped]
> I know very little about ZFS under the hood, but this model might fit the facts:
> * Normally, 'zfs destroy snapshot' is fast (on the order of a few seconds);
> * 'zfs destroy snapshot' has to briefly analyze all snapshots prior to
> * A particular 'problem' snapshot can slow that full analysis by a
> consistent amount of time;
> * Destroying that 'problem' snapshot drops the analysis time by that amount.
> If my model is correct, I'm going to see one or more spikes, followed
> by corresponding drops, until the destroys return to a reasonable
> This guy had a problem that might also fit that model -- that
> particular snapshots can be very slow, and removing them removes the
> time delay. That thread notes that it was due to a low-memory
> condition, and OpenSolaris bug 6542681 was filed for it. I do not
> think that my problem is because of low memory.
> I have stopped the destroys in case the remaining 'problem' snapshot is useful.
> The system is 8.1-SECURITY, amd64, 4GB RAM, no sysctl or loader
> tweaks, ZFS v3, zpool v14, single 58GB ZFS pool.
> # zfs list
> NAME USED AVAIL REFER MOUNTPOINT
> atoz-backup 15.4G 58.0G 25.5K /atoz-backup
> atoz-backup/usr 15.3G 58.0G 14.8G /atoz-backup/usr
> # df -ki | egrep 'atoz|Filesystem'
> Filesystem 1024-blocks Used
> Avail Capacity iused ifree %iused Mounted on
> atoz-backup 60789979 25
> 60789953 0% 6 121579907 0% /atoz-backup
> atoz-backup/usr 76281655 15491701
> 60789953 20% 714124 121579907 1% /atoz-backup/usr
I left this message open-ended. I also left out the hardware info.
What I meant to get at was:
* Is ~35 seconds roughly normal for a "zfs destroy snapshot" on a
filesystem with 723K files, single disk, under no other I/O or CPU
load, with an Intel ICH9 SATA300 talking to a Seagate ST3160812AS
(SATA 3)? Googling seems to indicate that it's not, even if I had
* If this speed is not normal, does this smell like a bug? If so, I
can help recreate or test, maybe on 8.2 or 9-CURRENT.
* If this speed is normal, what search terms would have yielded that
fact? (slow OR speed OR problem OR performance) plus "zfs destroy
snapshot" didn't yield much that applied.
* Is my model valid? I reasoned that if a snapshot is a separate set
of references to all files in the filesystem, and the filesystem
hasn't changed much over time, then destroying each snapshot should
take a relatively constant amount of time. Even if my time expectation
was wrong (that it would be quite a bit shorter, even for 723,000
files), I would not expect the destroy time to drop by 50% in the
middle of the destroy series ... so I think that I'm missing
* If my model isn't valid, what is a better model? For example, I had
gotten down to just over 10,000 snapshots when I stopped the series of
destroys; could there be a snapshot count threshold, under which each
destroy now takes less time, in stair-step fashion?
In other words: I'm trying to understand what I'm seeing. :-)
Also, sorry for the bad subject line - "zfs destroy snapshot" is
obviously the correct command; I've updated the subject line.
More information about the freebsd-fs