FreeBSD/ZFS on 9.3-RELEASE chews up memory with "wide" directories when calling readdir, etc; causes trap 12 panics

[Garrett Cooper]

On Apr 8, 2015, at 9:15, Alan Somers <asomers at freebsd.org> wrote:

> On Tue, Apr 7, 2015 at 12:37 AM, Garrett Cooper <yaneurabeya at gmail.com> wrote:
>> Hi,
>>        Long story short, I had a lot of mail spooled up in /var/spool. When I did ls /var/spool, ZFS chewed up almost all 12GB of my memory in <10 mins (because there were enough files there) and the system eventually panicked because [I assume that a memory allocation failed and] a trap 12 panic was caught. I don’t have the exact details, but it should be relatively easy to repro (YMMV if you have a boatload of RAM):
>> 
>> repro_end=10000000000
>> for i in $(seq 1 $repro_end); do mktemp tmp.XXXXXXXXXXXX; done
>> ls
>> 
>>        This might be ameliorated via r281026, but this change is only available in CURRENT (so far), and I haven’t tested it.
>>        Are there any comments about this scalability issue with FreeBSD/ZFS?
>> Thanks,
> 
> I spent the last ~ 24 hours creating 58,567,635 empty files in one
> directory.  I can ls it without crashing on a machine with 32 GB RAM.
> 
> # /usr/bin/time -l ls /tmp/tmp | wc
>     1061.21 real       225.54 user        36.61 sys
>   9720268  maximum resident set size
>        28  average shared memory size
>         8  average unshared data size
>       128  average unshared stack size
>   2425013  page reclaims
>         0  page faults
>         0  swaps
>    108036  block input operations
>         0  block output operations
>         0  messages sent
>         0  messages received
>         0  signals received
>    108004  voluntary context switches
>      2428  involuntary context switches
> 58567635 58567635 644243985

Strange. I wish I could gather more details about how many files were located there. Distance and time that they were created might manner; in my case there were _many_ spooled up emails that hadn’t been sent because I didn’t take the time to fix my SMTP settings via comcast/gmail.

RAM amount might matter too. 12GB vs 32GB is a bit of a difference.

Thanks!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 496 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20150408/93ae97ce/attachment.sig>