vm.kmem_size_max and vm.kmem_size capped at 329853485875 (~307GB)

[Marie Bacuno II]

On Aug 16, 2012, at 18:47, Garrett Cooper <yanegomi at gmail.com> wrote:

> On Thu, Aug 16, 2012 at 6:44 PM, Garrett Cooper <yanegomi at gmail.com> wrote:
>> On Thu, Aug 16, 2012 at 5:46 PM, Gezeala M. Bacuño II <gezeala at gmail.com> wrote:
>>> Hello fellow listers,
>>> 
>>> On a server with 512GB RAM it appears that vm.kmem_size_max is not
>>> being auto-tuned to use >329853485875 (~307GB).
>>> 
>>> On this machine vm.kmem_size is equal to vm.kmem_size_max
>>> 
>>> # from sysctl
>>> vm.kmem_size_max: 329853485875
>>> vm.kmem_size: 329853485875
>>> 
>>> On a machine with 1GB of RAM, I have successfully set vm.kmem_size_max
>>> to 330GB and vm.kmem_size automatically adjusts to 1GB even if I
>>> manually set it in /boot/loader.conf.
>>> 
>>> But on the machine with 512GB of RAM it just resets. For the machine
>>> to boot, we need to go to the loader prompt and issue:
>>> 
>>> OK set vm.kmem_size_max="300G"
>>> OK boot
>>> 
>>> On all PCBSD (8,9) or FreeBSD (8.1,8.2,9) machines we have,
>>> vm.kmem_size_max is always set to 329853485875.
>>> 
>>> How can I increase vm.kmem_size_max to use at least 500GB? And how is
>>> 329853485875 determined (formula)? I need to increase vm.kmem_size_max
>>> and vm.kmem_size so I can set vfs.zfs.arc_max (ZFS ARC) to use say
>>> 490GB.
>>> 
>>> I'm browsing thru the source code at
>>> http://fxr.watson.org/fxr/search?v=FREEBSD9&string=vm.kmem_size_max
>>> and I'm still trying to make sense of how vm.kmem_size_max is
>>> computed.
>>> 
>>> I have posted the same topic on forums.freebsd.org but I'm not getting
>>> any recommendations.
>>> 
>>> Please see the link for additional details:
>>> http://forums.freebsd.org/showthread.php?t=33977
>> 
>> Have you tried defining VM_KMEM_SIZE_MAX to your target value?
>> 
>> Its architecture specific BTW... see
>> sys/<architecture>/include/vmparam.h -- look for `VM_KMEM_SIZE_MAX`.
> 
> Also, it's a tunable, not a sysctl... so you need to set the value in
> /boot/loader.conf .
> -Garrett

Thanks for the quick reply.

Yes, had it set on /boot/loader.conf and by trial and error on the loader prompt.

We were able to bump it to 400G successfully. Tried 500G and 450G and the machine just spews out garbage in the screen.

The latest output from "zfs-stats -a" with vm.kmem_size_max="400G" is in the forum: http://forums.freebsd.org/showthread.php?t=33977

About the code, I am looking into amd64 arch. Still checking the values of the variables.. Can't just retrieve them using getconf. If you can point me to a doxygen like documentation appreciate it a lot.

Where does the constant value 329853485875 came from?

Thanks!