kva size on amd64

[Konstantin Belousov]

On Fri, Feb 01, 2013 at 11:47:23AM +0200, Andriy Gapon wrote:
> on 31/01/2013 20:30 Alan Cox said the following:
> > Try developing a different allocation strategy for the kmem_map. 
> > First-fit is clearly not working well for the ZFS ARC, because of
> > fragmentation.  For example, instead of further enlarging the kmem_map,
> > try splitting it into multiple submaps of the same total size,
> > kmem_map1, kmem_map2, etc.  Then, utilize these akin to the "old" and
> > "new" spaces of a copying garbage collector or storage segments in a
> > log-structured file system.  However, actual copying from an "old" space
> > to a "new" space may not be necessary.  By the time that the "new" space
> > from which you are currently allocating fills up or becomes sufficiently
> > fragmented that you can't satisfy an allocation, you've likely created
> > enough contiguous space in an "old" space.
> > 
> > I'll hypothesize that just a couple kmem_map submaps that are .625 of
> > physical memory size would suffice.  The bottom line is that the total
> > virtual address space should be less than 2x physical memory.
> > 
> > In fact, maybe the system starts off with just a single kmem_map, and
> > you only create additional kmem_maps on demand.  As someone who doesn't
> > use ZFS that would actually save me physical memory that is currently
> > being wasted on unnecessary preallocated page table pages for my
> > kmem_map.  This begins to sound like option (1) that you propose above.
> > 
> > This might also help to keep physical memory fragmentation in check.
> 
> Alan,
> 
> very interesting suggestions, thank you!
> 
> Of course, this is quite a bit more work than just jacking up some limit :-)
> So, it could be a while before any code materializes.
> 
> Actually, I have been obsessed quite for some time with an idea of confining ZFS
> to its own submap.  But ZFS does its allocations through malloc(9) and uma(9)
> (depending on configuration). It seemed like a bit of work to provide support
> for per-zone or per-tag submaps in uma and malloc.
> What is your opinion of this approach?
Definitely not being Alan.

I think that the rework of the ZFS memory management should remove the
use of uma or kmem_alloc() at all. From what I heard in part from you,
there is no reason to keep the filesystem caches mapped full time.

I hope to commit shortly a facilities that would allow ZFS to easily
manage copying for i/o from the unmapped set of pages. The checksumming
you mentioned would require some more work, but this does not look
unsurmountable. Having ZFS use raw vm_page_t for caching would also
remove the pressure on KVA.

> 
> P.S.
> BTW, do I understand correctly that the reservation of kernel page tables
> happens through vm_map_insert -> pmap_growkernel ?

Yes. E.g. kmem_suballoc->vm_map_find->vm_map_insert->pmap_growkernel.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 834 bytes
Desc: not available
URL: <http://lists.freebsd.org/pipermail/freebsd-arch/attachments/20130201/843068d9/attachment.sig>