8.0-R-p2 ZFS: unixbench causing kmem exhaustion panic
ivoras at freebsd.org
Wed Jan 13 19:58:16 UTC 2010
2010/1/13 Doug Poland <doug at polands.org>:
> On Wed, January 13, 2010 12:35, Ivan Voras wrote:
>>>> Try adding vfs.zfs.arc_max=512M to /boot/loader.conf.
>>> Would you suggest tweaking the vm.kmem_size tunables in addition to
>> No, unless they auto-tune to something lesser than approximately
>> I try to set arc_max to be a third (or a quarter) the kmem_size, and
>> tune kmem_size ad_hoc to suit the machine and its purpose.
>> The reason for this is that "arc_max" is just a guideline, not a hard
>> limit... the ZFS ARC usage can and will spike to much larger values,
>> usually in the most inopportune moment.
> This is the state of the machine when it panicked this time:
> panic: kmem_malloc(131072): kmem_map too small: 1296957440 total
> cpuid = 1
> /boot/loader.conf: vfs.zfs.arc_max=512M
> vfs.numvnodes: 660
> vfs.zfs.arc_max: 536870912
> vfs.zfs.arc_meta_limit: 134217728
> vfs.zfs.arc_meta_used: 7006136
> vfs.zfs.arc_min: 67108864
> vfs.zfs.zil_disable: 0
> vm.kmem_size: 1327202304
> vm.kmem_size_max: 329853485875
(from the size of arc_max I assume you did remember to reboot after
changing loader.conf and before testing again but just checking - did
Can you monitor and record kstat.zfs.misc.arcstats.size sysctl while
the test is running (and crashing)?
This looks curious - your kmem_max is ~~ 1.2 GB, arc_max is 0.5 GB and
you are still having panics. Is there anything unusual about your
system? Like unusually slow CPU, unusually fast or slow drives?
I don't have any ideas smarter than reducing arc_max by half then try
again and continue reducing it until it works. It would be very
helpful if you could monitor the kstat.zfs.misc.arcstats.size sysctl
while you are doing the tests to document what is happening to the
system. If it by any chance stays the same you should probably monitor
> Using a handy little script I found posted in several places, I was
> monitoring memory:
> TEXT 15373968 14.66 MiB
> DATA 1536957440 1465.76 MiB
> TOTAL 1552331408 1480.42 MiB
> Where TEXT = a sum of kldstat memory values
> and DATA = a sum of vmstat -m values
> Is there a next step to try, or is this chasing a wild goose?
More information about the freebsd-questions