ZFS directory with a large number of files

[seanrees at gmail.com]

Hi there,

I Googled around and checked the PRs and wasn't successful in finding
any reports of what I'm seeing. I'm hoping someone here can help me
debug what's going on.

On my FreeBSD 8.2-S machine (built circa 12th June), I created a
directory and populated it over the course of 3 weeks with about 2
million individual files. As you might imagine, a 'ls' of this
directory took quite some time.

The files were conveniently named with a timestamp in the filename
(still images from a security camera, once per second) so I've since
moved them all to timestamped directories (yyyy/MM/dd/hh/mm). What I
found though was the original directory the images were in is still
very slow to ls -- and it only has 1 file in it, another directory.

To clarify:
% ls second
[lots of time and many many files enumerated]
% # rename files using rename script
% ls second
[wait ages]
2011 dead
% mkdir second2 && mv second/2011 second2
% ls second2
[fast!]
2011
% ls second
[still very slow]
dead
% time ls second
dead/
gls -F --color  0.00s user 1.56s system 0% cpu 3:09.61 total

(timings are similar for /bin/ls)

This data is stored on a striped ZFS pool (version 15, though the
kernel reports version 28 is available but zpool upgrade seems to
disagree), 2T in size. I've run zpool scrub with no effect. ZFS is
busily driving the disks away; my iostat monitoring has all three
drives in the zpool running at 40-60% busy for the duration of the ls
(it was quiet before).

I've attached truss to the ls process. It spends a lot of time here:
fstatfs(0x5,0x7fffffffe0d0,0x800ad5548,0x7fffffffdfd8,0x0,0x0) = 0 (0x0)

I'm thinking there's some old ZFS metadata that it's looking into, but
I'm not sure how to best dig into this to understand what's going on
under the hood.

Can anyone perhaps point me the right direction on this?

Thanks,

Sean