ZFS directory with a large number of files

[Peter Jeremy]

On 2011-Aug-02 08:39:03 +0100, "seanrees at gmail.com" <seanrees at gmail.com> wrote:
>On my FreeBSD 8.2-S machine (built circa 12th June), I created a
>directory and populated it over the course of 3 weeks with about 2
>million individual files. As you might imagine, a 'ls' of this
>directory took quite some time.
>
>The files were conveniently named with a timestamp in the filename
>(still images from a security camera, once per second) so I've since
>moved them all to timestamped directories (yyyy/MM/dd/hh/mm). What I
>found though was the original directory the images were in is still
>very slow to ls -- and it only has 1 file in it, another directory.

I've also seen this behaviour on Solaris 10 after cleaning out a
directory with a large number of files (though not as pathological as
your case).  I tried creating and deleting entries in an unsuccessful
effort to trigger directory compaction.  I wound up moving the
remaining contents into a new directory, deleting the original one
and renaming the new directory.

It would appear te be a garbage collection bug in ZFS.

On 2011-Aug-02 13:10:27 +0300, Daniel Kalchev <daniel at digsys.bg> wrote:
>On 02.08.11 12:46, Daniel O'Connor wrote:
>> I am pretty sure UFS does not have this problem. i.e. once you 
>> delete/move the files out of the directory its performance would be 
>> good again. 
>
>UFS would be the classic example of poor performance if you do this.

Traditional UFS (including Solaris) behave badly in this scenario but
4.4BSD derivatives will release unused space at the end of a directory
and have smarts to more efficiently skip unused entries at the start
of a directory.

-- 
Peter Jeremy
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 196 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20110803/0dd25d6e/attachment.pgp