ZFS directory with a large number of files

[Jeremy Chadwick]

On Tue, Aug 02, 2011 at 10:16:35AM +0100, seanrees at gmail.com wrote:
> On Tue, Aug 2, 2011 at 10:08 AM, Jeremy Chadwick
> <freebsd at jdc.parodius.com> wrote:
> > On Tue, Aug 02, 2011 at 08:39:03AM +0100, seanrees at gmail.com wrote:
> >> On my FreeBSD 8.2-S machine (built circa 12th June), I created a
> >> directory and populated it over the course of 3 weeks with about 2
> >> million individual files.
> >
> > I'll keep this real simple:
> >
> > Why did you do this?
> >
> > I hope this was a stress test of some kind. ?If not:
> 
> Not really, but it turned into one.
> 
> The camera I was using had the ability (rather handily) to upload a
> still image once per second via FTP to a server of my choosing. It
> didn't have the ability to organize them for me in a neat directory
> hierarchy. So on holidays I went for 3 weeks and came back to ~2M
> images in the same directory.

I equate this to the following conversation with a doctor: "I shoved 2
million cotton balls into my ear, and now my hearing is sub-par".  "Why
did you do this?"   In this situation, the correct reply to the
physician is: "because I wasn't thinking, doc".

I suppose the reason I'm being so brash is that people doing this always
brings into question the thought process that went into the decision to
do said thing (both on your part, as well as the engineers of your
camera who thought such a feature would be a wise choice, especially
with an interval of 1 picture per second[1]).

When I was being taught the ropes of system administration at Oregon
State, the team of crotchety UNIX admins there made it quite clear that
there were things you just Did Not Do(tm) to computer systems.  Shoving
thousands of files into a single directory with no hierarchy was one of
them.

Again I will point this out: it doesn't matter what filesystem is used,
they all will suffer (just to different degrees) in this situation.
This is just one of those things where I have to say: Please Don't Do
This(tm).  At least this is an educational experience for you.

> > This is the 2nd or 3rd mail in recent months from people saying "I
> > decided to do something utterly stupid with my filesystem[1] and now I'm
> > asking why performance sucks".
> >
> > Why can people not create proper directory tree layouts to avoid this
> > problem regardless of what filesystem is used? ?I just don't get it.
> 
> I'm not sure it's utterly stupid; I didn't expect legendarily fast
> performance from 'ls' or anything else that enumerated the contents of
> the directory when all the files were there.

You shouldn't expect any kind of performance, at all, on any filesystem,
when/if you do this.  The effects can linger for quite some time.  I
can't come up with a good analogy at the moment.

> Now that the files are neatly organized, I expected fstatfs() on the
> directory to become fast again. It isn't. I'd like to understand why
> (or maybe learn a new trick or two about inspecting ZFS...)

Ronald's recommendation should address the problem, or at least diminish
it in the least.

[1]: I would also strongly advocate contacting your camera manufacturer
and asking them to add some extremely simple/basic code for adding a
directory hierarchy when the images are put on an FTP server.  The
amount of code is extremely nominal.

-- 
| Jeremy Chadwick                                jdc at parodius.com |
| Parodius Networking                       http://www.parodius.com/ |
| UNIX Systems Administrator                   Mountain View, CA, US |
| Making life hard for others since 1977.               PGP 4BD6C0CB |