bhyve guest's memory representation & live migration using COW

[Mark Johnston]

On Mon, Mar 04, 2019 at 10:52:44AM +0200, Elena Mihailescu wrote:
> I'm writing this e-mail to keep you in the loop with the live
> migration progress and to ask for advice regarding the process of
> cleaning the dirty bit. At Rod Grimes's suggestion, I added the
> virtualization list at CC.
> 
> I had some issues with the vmm_dirty bit (a bit from oflags) and
> initially I thought that there some issues related to the fact that in
> some cases oflags is directly updated (oflags = <smth>) and with the
> fact that the dirty bit is set otherwise that using vm_page_dirty(),
> but it seems that the issues I had were from other sources and now,
> after cleaning the code, and refactored a bit, it seems to work fine.
> Some of the code snippets that I thought that could affect the
> migration are at [1].
> 
> However, if I dump the guest memory after the live migration (the
> guest is not yet started after migration) and compare it with the
> source guest's ctx->baseaddr, there are some differences and I don't
> know from where they may come from.
> 
> One of the things I must implement is related to the dirty bit
> mechanism. For now, I do not clear the dirty bit after I migrate a
> page.

Note that there is code that only calls vm_page_dirty(m) if
m->dirty != VM_PAGE_BITS_ALL.  One example is in vm_page_advise().
So if the page is dirty from the kernel's point of view and you clear
the VMM dirty bit, subsequent modifications may not be propagated to the
vm_page state.

> This is not a wrong implementation because in every round, I
> migrate all the pages that have been modified ever (since the guest
> was started), but it is not an optimal implementation. To clear the
> dirty bit, I've tried different mechanism such as:
> - pmap_remove_write - kernel panic: pde entry not found for a wired mapped page
> - vm_object_page_clean
> - vm_map_sync -> kernel panic
> - pmap_protect - remove write permission, copy page and add write
> permission -> kernel panic; pmap_demote_pde: page table page for a
> wired mapping is missing
>
> Any idea about a way of forcing a page to be cleared (so the physical
> dirty bit to be cleared)?

I'm not sure exactly what you mean, but pmap_clear_modify(m) will clear
the modification bit for mappings of m.  It seems like you want
something roughly like:

	vm_page_xbusy(m);
	if (pmap_is_modified(m))
		m->dirty = m->vmm_dirty = VM_PAGE_BITS_ALL;
	pmap_clear_modify(m);
	vm_page_xunbusy(m);

Or do we need to restrict the operation to a specific mapping of m?