misc/129396: [kern] cannot set vm.kmem_size_max beyond 3GB

[Weldon Godfrey]

>Number:         129396
>Category:       misc
>Synopsis:       [kern] cannot set vm.kmem_size_max beyond 3GB
>Confidential:   no
>Severity:       serious
>Priority:       low
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Wed Dec 03 18:20:01 UTC 2008
>Closed-Date:
>Last-Modified:
>Originator:     Weldon Godfrey
>Release:        FreeBSD 7.1-PRERELEASE amd64
>Organization:
ENA
>Environment:
FreeBSD store1.mail.ena.net 7.1-PRERELEASE FreeBSD 7.1-PRERELEASE #2: Thu Nov 20 10:41:36 CST 2008     root at store1.mail.ena.net:/usr/obj/usr/src/sys/GENERIC  amd64
>Description:
I noticed during my testing, while trying to copy two 2TB files at the same time (I am using ZFS), when the files get to a certain size, the system panics.  It didn't dump core or log a reason in syslog, however, one time it managed not to reboot (just hang after panic) and I was able to capture that it was complaining that kmem was too small.

I thought that was odd, since I have set the max to 15GB (the system has 16GB of RAM), so I checked sysctl and vm.kmem_max was only at 3GB.  This is in my loader.conf:
vm.kmem_size_max="16106127360"
vm.kmem_size="1073741824"
checking right after boot, the vm.kmem_size was set, it was at 1073741824

here is sysctl right after reboot with the above loader.conf:
store1# sysctl -a | grep kmem
vm.kmem_size_scale: 3
vm.kmem_size_max: 3221225472
vm.kmem_size_min: 0
vm.kmem_size: 1073741824
store1# sysctl -a | grep kern.maxvn
kern.maxvnodes: 100000
store1#



As a work-a-round, I tried to set these options in the kernel config and recompile the kernel, it won't compile, this is the error:

cc -c -O2 -frename-registers -pipe -fno-strict-aliasing  -std=c99 -g -Wall -Wredundant-decls -Wnested-externs -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Winline -Wcast-qual  -Wundef -Wno-pointer-sign -fformat-extensions -nostdinc  -I. -I/usr/src/sys -I/usr/src/sys/contrib/altq -D_KERNEL -DHAVE_KERNEL_OPTION_HEADERS -include opt_global.h -fno-common -finline-limit=8000 --param inline-unit-growth=100 --param large-function-growth=1000  -mcmodel=kernel
-mno-red-zone-mfpmath=387 -mno-sse -mno-sse2 -mno-mmx -mno-3dnow -msoft-float -fno-asynchronous-unwind-tables -ffreestanding -Werror /usr/src/sys/kern/kern_malloc.c
cc1: warnings being treated as errors
/usr/src/sys/kern/kern_malloc.c: In function 'kmeminit':
/usr/src/sys/kern/kern_malloc.c:598: warning: large integer implicitly truncated to unsigned type
*** Error code 1


>How-To-Repeat:
try to set vm.kmem_size_max or OPTIONS VM_KMEM_SIZE_MAX beyond 3221225472
>Fix:
I am guessing these tunables need a larger cast?

u_int vm_kmem_size;
SYSCTL_UINT(_vm, OID_AUTO, kmem_size, CTLFLAG_RD, &vm_kmem_size, 0,
    "Size of kernel memory");
 
u_int vm_kmem_size_min;
SYSCTL_UINT(_vm, OID_AUTO, kmem_size_min, CTLFLAG_RD, &vm_kmem_size_min, 0,  
    "Minimum size of kernel memory");
 
u_int vm_kmem_size_max;
SYSCTL_UINT(_vm, OID_AUTO, kmem_size_max, CTLFLAG_RD, &vm_kmem_size_max, 0,
    "Maximum size of kernel memory");
   
u_int vm_kmem_size_scale;
SYSCTL_UINT(_vm, OID_AUTO, kmem_size_scale, CTLFLAG_RD, &vm_kmem_size_scale, 0,
    "Scale factor for kernel memory size");

>Release-Note:
>Audit-Trail:
>Unformatted: