[GSOC] bhyve instruction caching

[Mihai Carabas]

>
>>
>> Also I'm looking for new real-world benchmarks to see where we can
>> benefict a lot from the instruction caching feature.
> I've started doing to kind of benchmarks:
> * CPU intensive (ex: a bash script that increments in a loop a variable)
> * benchmarks that creates lot of short processes (not CPU intensive
> but short running I/O intensive): make buildworld -j2
>
> I will come with some concrete results in the next week. Meanwhile I
> will add some new statistics like "Cache hits" and "Cache evictions"
> to see how my cache behavious in each of the benchmarks.
I have added the statistics about cache hits and evictions using the
vmm_stat_* infrastructure. I added three type of variables:
- VM_INST_CACHE_HITS - the number of cache hits
- VM_INST_CACHE_INSERTIONS - the number of cache insertions
- VM_INST_CACHE_EVICTIONS[4] - an array with cache evictions
corresponding to each level of a page table (basically the pagetable
which caused the eviction)

1) CPU intensive (simple bash script) with 2 VCPUs
"""
#!/usr/local/bin/bash
a=0
while true
do
a=$((a+1))
done
"""
After 11 minutes of running here are the results:

VCPU0:
number of instruction cache hits         699519
number of instruction cache evictions[0] 7139
number of instruction cache evictions[1] 0
number of instruction cache evictions[2] 0
number of instruction cache evictions[3] 0
number of instruction cache insertions   10395

VCPU1:
number of instruction cache hits         840485
number of instruction cache evictions[0] 8926
number of instruction cache evictions[1] 0
number of instruction cache evictions[2] 0
number of instruction cache evictions[3] 0
number of instruction cache insertions   5743

As you can see the number of evictions is very low for this kind of process

2)  "make buildworld -j2" with 2 VCPUs:

vcpu:0
number of instruction cache hits                19204630
number of instruction cache evictions[0]        8563694
number of instruction cache evictions[1]        1131
number of instruction cache evictions[2]        0
number of instruction cache evictions[3]        0
number of instruction cache insertions          8688733

vcpu:1
number of instruction cache hits                12930500
number of instruction cache evictions[0]        9173381
number of instruction cache evictions[1]        1457
number of instruction cache evictions[2]        0
number of instruction cache evictions[3]        0
number of instruction cache insertions          9051295

As you can see in this case, the cache hit is much lower due to the
high-rate eviction caused mainly by the PML4 entry. This is caused by
the make infrastructure which is creating/deleting a lot of small
processes.

This is the analysis about the cache hit metric. The code writing for
this project is at the end and I will come at the end of the week with
some high-level measurements, comparing the running times with the
non-cache implementation.

Thanks,
Mihai