8.0-R-p2 ZFS: unixbench causing kmem exhaustion panic

[Ivan Voras]

2010/1/14 Doug Poland <doug at polands.org>:
>
> On Thu, January 14, 2010 08:50, Ivan Voras wrote:
>> 2010/1/14 Doug Poland <doug at polands.org>:
>>>>>
>>>>> kstat.zfs.misc.arcstats.size
>>>>>
>>>>> seemed to fluctuate between about 164,000,00 and 180,000,000 bytes
>>>>> during this last run
>>>>
>>>> Is that with or without panicking?
>>>>
>>> with a panic
>>>
>>>
>>>> If the system did panic then it looks like the problem is a memory
>>>> leak somewhere else in the kernel, which you could confirm by
>>>> monitoring vmstat -z.
>>>>
>>> I'll give that a try.  Am I looking for specific items in vmstat
>>> -z?   arc*, zil*, zfs*, zio*?  Please advise.
>>
>> You should look for whatever is allocating all your memory between 180
>> MB (which is your ARC size) and 1.2 GB (which is your kmem size).
>>
>
> OK, another run, this time back to vfs.zfs.arc_max=512M in
> /boot/loader.conf, and a panic:
>
> panic: kmem malloc(131072): kmem map too small: 1294258176 total
> allocated
>
> I admit I do not fully understand what metrics are important to proper
> analysis of this issue.  In this case, I was watching the following
> within 1 second of the panic:
>
> sysctl kstat.zfs.misc.arcstats.size: 41739944
> sysctl vfs.numvnodes: 678
> sysctl vfs.zfs.arc_max: 536870912
> sysctl vfs.zfs.arc_meta_limit: 134217728
> sysctl vfs.zfs.arc_meta_used: 7228584
> sysctl vfs.zfs.arc_min: 67108864
> sysctl vfs.zfs.cache_flush_disable: 0
> sysctl vfs.zfs.debug: 0
> sysctl vfs.zfs.mdcomp_disable: 0
> sysctl vfs.zfs.prefetch_disable: 1
> sysctl vfs.zfs.recover: 0
> sysctl vfs.zfs.scrub_limit: 10
> sysctl vfs.zfs.super_owner: 0
> sysctl vfs.zfs.txg.synctime: 5
> sysctl vfs.zfs.txg.timeout: 30
> sysctl vfs.zfs.vdev.aggregation_limit: 131072
> sysctl vfs.zfs.vdev.cache.bshift: 16
> sysctl vfs.zfs.vdev.cache.max: 16384
> sysctl vfs.zfs.vdev.cache.size: 10485760
> sysctl vfs.zfs.vdev.max_pending: 35
> sysctl vfs.zfs.vdev.min_pending: 4
> sysctl vfs.zfs.vdev.ramp_rate: 2
> sysctl vfs.zfs.vdev.time_shift: 6
> sysctl vfs.zfs.version.acl: 1
> sysctl vfs.zfs.version.dmu_backup_header: 2
> sysctl vfs.zfs.version.dmu_backup_stream: 1
> sysctl vfs.zfs.version.spa: 13
> sysctl vfs.zfs.version.vdev_boot: 1
> sysctl vfs.zfs.version.zpl: 3
> sysctl vfs.zfs.zfetch.array_rd_sz: 1048576
> sysctl vfs.zfs.zfetch.block_cap: 256
> sysctl vfs.zfs.zfetch.max_streams: 8
> sysctl vfs.zfs.zfetch.min_sec_reap: 2
> sysctl vfs.zfs.zil_disable: 0
> sysctl vm.kmem_size: 1327202304
> sysctl vm.kmem_size_max: 329853485875
> sysctl vm.kmem_size_min: 0
> sysctl vm.kmem_size_scale: 3
>
>
> vmstat -z | egrep -i 'zfs|zil|arc|zio|files'
> ITEM                     SIZE     LIMIT      USED      FREE  REQUESTS
> Files:                     80,        0,      116,      199,   850713
> zio_cache:                720,        0,    53562,       98, 86386955
> arc_buf_hdr_t:            208,        0,     1193,       31,    11990
> arc_buf_t:                 72,        0,     1180,      120,    11990
> zil_lwb_cache:            200,        0,    11580,     2594,    62407
> zfs_znode_cache:          376,        0,      605,       55,      654
>
> vmstat -m |grep solaris|sed 's/K//'|awk '{print "vm.solaris:", $3*1024}'
>
>
>  solaris: 1285068800
>
>
> The value I see as the culprit is vmstat -m | grep solaris.  This
> value fluctuates wildly during the run and is always near kmem_size at
> the time of the panic.
>
> Again, I'm not sure what to look for here, and you are patiently
> helping me along in this process.  If you have any tips or can point
> me to docs on how to easily monitor these values, I will endeavor to
> do so.

The only really important ones should be kstat.zfs.misc.arcstats.size
(which you very rarely print) and vm.kmem_size. The "solaris" entry
above should be near  kstat.zfs.misc.arcstats.size in all cases.

But I don't have any more ideas here. Try taking this post (also
include kstst.zfs.misc.arcstats.size) to the freebsd-fs@ mailing list.