8.0-R-p2 ZFS: unixbench causing kmem exhaustion panic

[Ivan Voras]

2010/1/13 Doug Poland <doug at polands.org>:
>
> On Wed, January 13, 2010 12:35, Ivan Voras wrote:
>>>>
>>>> Try adding vfs.zfs.arc_max=512M to /boot/loader.conf.
>>>>
>>> Would you suggest tweaking the vm.kmem_size tunables in addition to
>>> arc_max?
>>
>> No, unless they auto-tune to something lesser than approximately
>> arc_max*3.
>>
>> I try to set arc_max to be a third (or a quarter) the kmem_size, and
>> tune kmem_size ad_hoc to suit the machine and its purpose.
>>
>> The reason for this is that "arc_max" is just a guideline, not a hard
>> limit... the ZFS ARC usage can and will spike to much larger values,
>> usually in the most inopportune moment.
>>
> This is the state of the machine when it panicked this time:
>
> panic: kmem_malloc(131072): kmem_map too small: 1296957440 total
> allocated
> cpuid = 1
>
> /boot/loader.conf: vfs.zfs.arc_max=512M
> vfs.numvnodes: 660
> vfs.zfs.arc_max: 536870912
> vfs.zfs.arc_meta_limit: 134217728
> vfs.zfs.arc_meta_used: 7006136
> vfs.zfs.arc_min: 67108864
> vfs.zfs.zil_disable: 0
> vm.kmem_size: 1327202304
> vm.kmem_size_max: 329853485875

(from the size of arc_max I assume you did remember to reboot after
changing loader.conf and before testing again but just checking - did
you?)

Can you monitor and record kstat.zfs.misc.arcstats.size sysctl while
the test is running (and crashing)?

This looks curious - your kmem_max is ~~ 1.2 GB, arc_max is 0.5 GB and
you are still having panics. Is there anything unusual about your
system? Like unusually slow CPU, unusually fast or slow drives?

I don't have any ideas smarter than reducing arc_max by half then try
again and continue reducing it until it works. It would be very
helpful if you could monitor the kstat.zfs.misc.arcstats.size sysctl
while you are doing the tests to document what is happening to the
system. If it by any chance stays the same you should probably monitor
"vmstat -m".


> Using a handy little script I found posted in several places, I was
> monitoring memory:
>
> TEXT     15373968       14.66   MiB
> DATA   1536957440       1465.76 MiB
> TOTAL  1552331408       1480.42 MiB
>
> Where TEXT = a sum of kldstat memory values
> and   DATA = a sum of vmstat -m values
>
> Is there a next step to try, or is this chasing a wild goose?
>
>
> --
> Regards,
> Doug
>
>