svn commit: r243631 - in head/sys: kern sys

[Alan Cox]

On 01/11/2013 05:38, Jayachandran C. wrote:
> On Tue, Jan 8, 2013 at 3:12 AM, Andre Oppermann <andre at freebsd.org> wrote:
>> On 07.01.2013 20:32, Alan Cox wrote:
>>> On 01/07/2013 12:47, Oleksandr Tymoshenko wrote:
>>>> On 12/27/2012 6:46 PM, Oleksandr Tymoshenko wrote:
>>>>> On 12/18/2012 1:59 AM, Alan Cox wrote:
>>>>>> On 12/17/2012 23:40, Oleksandr Tymoshenko wrote:
>>>>>>> On 2012-12-08, at 1:21 PM, Alan Cox <alc at rice.edu> wrote:
>>>>>>>
>>>>>>>> On 12/08/2012 14:32, Andre Oppermann wrote:
>>>>>>> .. skipped ..
>>>>>>>
>>>>>>>>> The trouble seems to come from NSFBUFS which is (512 + maxusers *
>>>>>>>>> 16)
>>>>>>>>> resulting in a kernel map of (512 + 400 * 16) * PAGE_SIZE =
>>>>>>>>> 27MB.  This
>>>>>>>>> seem to be pushing it with the smaller ARM kmap layout.
>>>>>>>>>
>>>>>>>>> Does it boot and run when you set the tunable kern.ipc.nsfbufs=3500?
>>>>>>>>>
>>>>>>>>> ARM does have a direct map mode as well which doesn't require the
>>>>>>>>> allocation
>>>>>>>>> of sfbufs.  I'm not sure which other problems that approach has.
>>>>>>>>>
>>>>>>>> Only a few (3?) platforms use it.  It reduces the size of the user
>>>>>>>> address space, and translation between physical addresses and
>>>>>>>> direct map
>>>>>>>> addresses is not computationally trivial as it is on other
>>>>>>>> architectures, e.g., amd64, ia64.  However, it does try to use large
>>>>>>>> page mappings.
>>>>>>>>
>>>>>>>>
>>>>>>>>> Hopefully alc@ (added to cc) can answer that and also why the
>>>>>>>>> kmap of
>>>>>>>>> 27MB
>>>>>>>>> manages to wrench the ARM kernel.
>>>>>>>>>
>>>>>>>> Arm does not define caps on either the buffer map size (param.h)
>>>>>>>> or the
>>>>>>>> kmem map size (vmparam.h).  It would probably make sense to copy
>>>>>>>> these
>>>>>>>> definitions from i386.
>>>>>>> Adding caps didn't help. I did some digging and found out that
>>>>>>> although address range
>>>>>>> 0xc0000000 .. 0xffffffff is indeed valid for ARM in general actual
>>>>>>> KVA space varies for
>>>>>>> each specific hardware platform. This "real" KVA is defined by
>>>>>>> <virtual_avail, virtual_end>
>>>>>>> pair and ifI use them instead of <VM_MIN_KERNEL_ADDRESS,
>>>>>>> VM_MAX_KERNEL_ADDRESS>
>>>>>>> in init_param2 function my pandaboard successfully boots. Since
>>>>>>> former pair is used for defining
>>>>>>> kernel_map boundaries I believe it should be used for auto tuning
>>>>>>> as well.
>>>>>>
>>>>>> That makes sense.  However, "virtual_avail" isn't the start of the
>>>>>> kernel address space.  The kernel map always starts at
>>>>>> VM_MIN_KERNEL_ADDRESS.  (See kmem_init().)  "virtual_avail" represents
>>>>>> the next unallocated virtual address in the kernel address space at an
>>>>>> early point in initialization.  "virtual_avail" and "virtual_end"
>>>>>> aren't
>>>>>> used after that, or outside the VM system.  Please use
>>>>>> vm_map_min(kernel_map) and vm_map_max(kernel_map) instead.
>>>>>
>>>>> I checked: kernel_map is not available (NULL) at this point.  So we
>>>>> can't use it to
>>>>> determine real KVA size. Closest thing we can get is
>>>>> virtual_avail/virtual_end pair.
>>>>>
>>>>> Andre, could you approve attached patch for commit or suggest better
>>>>> solution?
>>>>
>>>> Any update on this one? Can I proceed with commit?
>>>>
>>> Sorry, I've been away from my e-mail since the 30th, and I'm now in the
>>> process of getting caught up.  Give me a day or so to look at this.
> I see an issue with commit on MIPS XLP platform as well.
>
> With 16 GB physical memory, the ncallout is calculated to be 538881
> (since it is based on maxfiles - which is now based on the physical
> memory). Due to this, the callwheel allocation per cpu is 16MB
> (callwheelsize is 1MB). And on a 32 CPU machine, the total allocation
> for callouts comes to 32*16MB = 512MB.
>
> I have worked around this issue for now by increasing VM_KMEM_SIZE_MAX
> (which is 200MB now) - but I think a better fix is needed for this.
>

MIPS should use a definition for VM_KMEM_SIZE_MAX that scales with the
kernel address space size, like amd64, i386, and sparc64, and not a
fixed number.  I think that the following should work for both 32- and
64-bit processors:

Index: mips/include/vmparam.h
===================================================================
--- mips/include/vmparam.h      (revision 245229)
+++ mips/include/vmparam.h      (working copy)
@@ -130,10 +130,11 @@
 #endif
 
 /*
- * Ceiling on amount of kmem_map kva space.
+ * Ceiling on the amount of kmem_map KVA space: 40% of the entire KVA
space.
  */
 #ifndef VM_KMEM_SIZE_MAX
-#define        VM_KMEM_SIZE_MAX        (200 * 1024 * 1024)
+#define        VM_KMEM_SIZE_MAX        ((VM_MAX_KERNEL_ADDRESS - \
+    VM_MIN_KERNEL_ADDRESS + 1) * 2 / 5)
 #endif
 
 /* initial pagein size of beginning of executable file */