svn commit: r243631 - in head/sys: kern sys

[Jayachandran C.]

On Tue, Jan 8, 2013 at 3:12 AM, Andre Oppermann <andre at freebsd.org> wrote:
> On 07.01.2013 20:32, Alan Cox wrote:
>>
>> On 01/07/2013 12:47, Oleksandr Tymoshenko wrote:
>>>
>>> On 12/27/2012 6:46 PM, Oleksandr Tymoshenko wrote:
>>>>
>>>> On 12/18/2012 1:59 AM, Alan Cox wrote:
>>>>>
>>>>> On 12/17/2012 23:40, Oleksandr Tymoshenko wrote:
>>>>>>
>>>>>> On 2012-12-08, at 1:21 PM, Alan Cox <alc at rice.edu> wrote:
>>>>>>
>>>>>>> On 12/08/2012 14:32, Andre Oppermann wrote:
>>>>>>
>>>>>> .. skipped ..
>>>>>>
>>>>>>>> The trouble seems to come from NSFBUFS which is (512 + maxusers *
>>>>>>>> 16)
>>>>>>>> resulting in a kernel map of (512 + 400 * 16) * PAGE_SIZE =
>>>>>>>> 27MB.  This
>>>>>>>> seem to be pushing it with the smaller ARM kmap layout.
>>>>>>>>
>>>>>>>> Does it boot and run when you set the tunable kern.ipc.nsfbufs=3500?
>>>>>>>>
>>>>>>>> ARM does have a direct map mode as well which doesn't require the
>>>>>>>> allocation
>>>>>>>> of sfbufs.  I'm not sure which other problems that approach has.
>>>>>>>>
>>>>>>> Only a few (3?) platforms use it.  It reduces the size of the user
>>>>>>> address space, and translation between physical addresses and
>>>>>>> direct map
>>>>>>> addresses is not computationally trivial as it is on other
>>>>>>> architectures, e.g., amd64, ia64.  However, it does try to use large
>>>>>>> page mappings.
>>>>>>>
>>>>>>>
>>>>>>>> Hopefully alc@ (added to cc) can answer that and also why the
>>>>>>>> kmap of
>>>>>>>> 27MB
>>>>>>>> manages to wrench the ARM kernel.
>>>>>>>>
>>>>>>> Arm does not define caps on either the buffer map size (param.h)
>>>>>>> or the
>>>>>>> kmem map size (vmparam.h).  It would probably make sense to copy
>>>>>>> these
>>>>>>> definitions from i386.
>>>>>>
>>>>>> Adding caps didn't help. I did some digging and found out that
>>>>>> although address range
>>>>>> 0xc0000000 .. 0xffffffff is indeed valid for ARM in general actual
>>>>>> KVA space varies for
>>>>>> each specific hardware platform. This "real" KVA is defined by
>>>>>> <virtual_avail, virtual_end>
>>>>>> pair and ifI use them instead of <VM_MIN_KERNEL_ADDRESS,
>>>>>> VM_MAX_KERNEL_ADDRESS>
>>>>>> in init_param2 function my pandaboard successfully boots. Since
>>>>>> former pair is used for defining
>>>>>> kernel_map boundaries I believe it should be used for auto tuning
>>>>>> as well.
>>>>>
>>>>>
>>>>> That makes sense.  However, "virtual_avail" isn't the start of the
>>>>> kernel address space.  The kernel map always starts at
>>>>> VM_MIN_KERNEL_ADDRESS.  (See kmem_init().)  "virtual_avail" represents
>>>>> the next unallocated virtual address in the kernel address space at an
>>>>> early point in initialization.  "virtual_avail" and "virtual_end"
>>>>> aren't
>>>>> used after that, or outside the VM system.  Please use
>>>>> vm_map_min(kernel_map) and vm_map_max(kernel_map) instead.
>>>>
>>>>
>>>> I checked: kernel_map is not available (NULL) at this point.  So we
>>>> can't use it to
>>>> determine real KVA size. Closest thing we can get is
>>>> virtual_avail/virtual_end pair.
>>>>
>>>> Andre, could you approve attached patch for commit or suggest better
>>>> solution?
>>>
>>>
>>> Any update on this one? Can I proceed with commit?
>>>
>>
>> Sorry, I've been away from my e-mail since the 30th, and I'm now in the
>> process of getting caught up.  Give me a day or so to look at this.

I see an issue with commit on MIPS XLP platform as well.

With 16 GB physical memory, the ncallout is calculated to be 538881
(since it is based on maxfiles - which is now based on the physical
memory). Due to this, the callwheel allocation per cpu is 16MB
(callwheelsize is 1MB). And on a 32 CPU machine, the total allocation
for callouts comes to 32*16MB = 512MB.

I have worked around this issue for now by increasing VM_KMEM_SIZE_MAX
(which is 200MB now) - but I think a better fix is needed for this.

JC.