vm.kmem_size_max and vm.kmem_size capped at 329853485875 (~307GB)

[Gezeala M. Bacuño II]

On Mon, Aug 20, 2012 at 8:22 AM, Alan Cox <alc at rice.edu> wrote:
> On 08/18/2012 19:57, Gezeala M. Bacuño II wrote:
>>
>> On Sat, Aug 18, 2012 at 12:14 PM, Alan Cox<alc at rice.edu>  wrote:
>>>
>>> On 08/17/2012 17:08, Gezeala M. Bacuño II wrote:
>>>>
>>>> On Fri, Aug 17, 2012 at 1:58 PM, Alan Cox<alc at rice.edu>   wrote:
>>>>>
>>>>> vm.kmem_size controls the maximum size of the kernel's heap, i.e., the
>>>>> region where the kernel's slab and malloc()-like memory allocators
>>>>> obtain
>>>>> their memory.  While this heap may occupy the largest portion of the
>>>>> kernel's virtual address space, it cannot occupy the entirety of the
>>>>> address
>>>>> space.  There are other things that must be given space within the
>>>>> kernel's
>>>>> address space, for example, the file system buffer map.
>>>>>
>>>>> ZFS does not, however, use the regular file system buffer cache. The
>>>>> ARC
>>>>> takes its place, and the ARC abuses the kernel's heap like nothing
>>>>> else.
>>>>> So, if you are running a machine that only makes trivial use of a
>>>>> non-ZFS
>>>>> file system, like you boot from UFS, but store all of your data in ZFS,
>>>>> then
>>>>> you can dramatically reduce the size of the buffer map via boot loader
>>>>> tuneables and proportionately increase vm.kmem_size.
>>>>>
>>>>> Any further increases in the kernel virtual address space size will,
>>>>> however, require code changes.  Small changes, but changes nonetheless.
>>>>>
>>>>> Alan
>>>>>
<<snip>>
>>> Your objective should be to reduce the value of "sysctl vfs.maxbufspace".
>>> You can do this by setting the loader.conf tuneable "kern.maxbcache" to
>>> the
>>> desired value.
>>>
>>> What does your machine currently report for "sysctl vfs.maxbufspace"?
>>>
>> Here you go:
>> vfs.maxbufspace: 54967025664
>> kern.maxbcache: 0
>
>
> Try setting kern.maxbcache to two billion and adding 50 billion to the
> setting of vm.kmem_size{,_max}.
>
>

Thank you. We'll try this and post back results.

>> Other (probably) relevant values:
>> vfs.hirunningspace: 16777216
>> vfs.lorunningspace: 11206656
>> vfs.bufdefragcnt: 0
>> vfs.buffreekvacnt: 2
>> vfs.bufreusecnt: 320149
>> vfs.hibufspace: 54966370304
>> vfs.lobufspace: 54966304768
>> vfs.maxmallocbufspace: 2748318515
>> vfs.bufmallocspace: 0
>> vfs.bufspace: 10490478592
>> vfs.runningbufspace: 0
>>
>> Let me know if you need other tuneables or sysctl values. Thanks a lot
>> for looking into this.
>>
>