FreeBSD/ZFS on 9.3-RELEASE chews up memory with "wide" directories when calling readdir, etc; causes trap 12 panics

[Alan Somers]

On Tue, Apr 7, 2015 at 12:37 AM, Garrett Cooper <yaneurabeya at gmail.com> wrote:
> Hi,
>         Long story short, I had a lot of mail spooled up in /var/spool. When I did ls /var/spool, ZFS chewed up almost all 12GB of my memory in <10 mins (because there were enough files there) and the system eventually panicked because [I assume that a memory allocation failed and] a trap 12 panic was caught. I don’t have the exact details, but it should be relatively easy to repro (YMMV if you have a boatload of RAM):
>
> repro_end=10000000000
> for i in $(seq 1 $repro_end); do mktemp tmp.XXXXXXXXXXXX; done
> ls
>
>         This might be ameliorated via r281026, but this change is only available in CURRENT (so far), and I haven’t tested it.
>         Are there any comments about this scalability issue with FreeBSD/ZFS?
> Thanks,


I spent the last ~ 24 hours creating 58,567,635 empty files in one
directory.  I can ls it without crashing on a machine with 32 GB RAM.

# /usr/bin/time -l ls /tmp/tmp | wc
     1061.21 real       225.54 user        36.61 sys
   9720268  maximum resident set size
        28  average shared memory size
         8  average unshared data size
       128  average unshared stack size
   2425013  page reclaims
         0  page faults
         0  swaps
    108036  block input operations
         0  block output operations
         0  messages sent
         0  messages received
         0  signals received
    108004  voluntary context switches
      2428  involuntary context switches
 58567635 58567635 644243985

-Alan