[GSOC] bhyve instruction caching

[Mihai Carabas]

On Sun, Jun 15, 2014 at 8:58 PM, Mihai Carabas <mihai.carabas at gmail.com> wrote:
> On Sun, Jun 8, 2014 at 12:38 AM, Mihai Carabas <mihai.carabas at gmail.com> wrote:
>>>
>>> These days I've started a discussion with Neel about some
>>> microbenchmarking mechanisms. I will come with some more details next
>>> week.
>> I've built a microbenchmarking kernel module which is accessing the
>> lapic->id for 1000000 and than I calculate the average of an access
>> (each access needs to be emulated by the hypervisor).
>>
>> I've also implemented the instuction caching mechanism. At each emulation:
>> - I check to see if I have that particular instruction cached
>> - if not I will cache it in a particula structure named "struct vie_cached" [1]
>> - if it's cached I just use that instruction
>>
>> Right now I am working on write-protecting the pages where the
>> instruction reside. I will come with some more details/results when I
>> finished this part too (there are some SMP issues I'm still debating
>> with Neel).
>
> I added the write protection for the pages where instruction resides
> (I mark them with PROT_READ|PROT_EXECUTE using vm_map_protect). When a
> page fault is raised on that page I delete the instruction from the
> cache. I've tested the instruction caching feature and it works ok (no
> bugs reveal). The results on the microbenchmarking are looking OK, but
> they aren't conclusive because I'm not write protecting the pages of
> the pagetable (which will eat more time). Right now for 1000000
> accesses to the lapic-id we have an average of 6200 ticks per VM_EXIT
> for an instruction emulation. Without instruction caching the average
> is at about 10000ticks. Like I've said before, this results aren't
> very conclusive until I introduce all protections needed. I will come
> in the next weeks with more results when I have the caching feature
> fully implemented.
>
> I've also introduce a new systl to enable/disable instruction caching
> "hw.vmm.instruction_cache".
In the last two weeks I developed the mechanism of write-protecting
the VM pagetable: every page of the pagetable that was pointing to the
memory page where the instruction resides was made read-only. I had
some trouble with the locking mechanism when having a virtual machine
with more than 1 vCPU (one vCPU was write protecting and immediatly
another VCPU was remove the protection resulting in a corruption of
the write-protection and cache logic). Neel helped me solve this issue
with some best effort checks and are working OK.

Also I migrated to sx_* locks due to the fact the rm_* locks didn' let
me sleep in them (at some point when raising the protection in the VM
subsystem a sleep may be encountered).

At the end of the week I will come with some new results with the
microbenchmarking.

Also I'm looking for new real-world benchmarks to see where we can
benefict a lot from the instruction caching feature.

Thanks,
Mihai