Virtual performance

[Vlad GALU]

On 2/17/07, Ivan Voras <ivoras at fer.hr> wrote:
> I haven't been using virtual machines for production much, but this is
> likely to change in the near future. After running some benchmarks, it
> looks like there's something very bad with performance under VMWare.
>
> I've tried two things: the big "VMWare Infrastructure" product, version
> 3.0.1 and the small, free (as in beer) VMWare Infrastructure, but the
> findings are the same. As an illustration, consider this header from "top":
>
> last pid: 29892;  load averages:  1.20,  1.16,  1.09    up 0+00:45:22
> 01:03:37
> 38 processes:  3 running, 35 sleeping
> CPU states:  3.8% user,  0.0% nice, 96.2% system,  0.0% interrupt,  0.0%
> idle
> Mem: 29M Active, 251M Inact, 117M Wired, 2080K Cache, 112M Buf, 3363M Free
> Swap: 5120M Total, 5120M Free
>
> Note the "system" time - while this statistic might look like a getpid()
> loop, it is, in fact, a sample from the middle of a `make buildkernel`
> on 6.2-release. In fact, on a non-virtual machine, a getpid() loop comes
> out rougly 60% in sys time + 40% in user time. I don't know why would
> compilation, a memory and CPU intensive process, require so much sys
> time, and the same is noticeable when running practically any program.
>
> All this time is not spent in a kernel thread - top showing system
> threads shows them all as quiescent.
>
> Running unixbench, I see that performance is mainly lost on benchmarks
> that do context switching, while numeric performance stays approximately
> the same. Most performance loss is on "context1" and "pipe" benchmarks.
>
> Here's a typical result of running unixbench for pipe and context1
> benchmark on the above machine:
>
>                      INDEX VALUES
> TEST                                        BASELINE     RESULT      INDEX
> Pipe Throughput                              12440.0    48528.7       39.0
> Pipe-based Context Switching                  4000.0     9593.8       24.0
>
> These should (the INDEX field), for this machine, be somewhere between
> 350 - 500. Other tests that frequently make kernel calls (like: execl,
> file system, syscall overhead) are also very much affected.
>
> Some things I've tried (without luck):
>
> - several different machines and CPUs (Intel, AMD), all on i386 arch
> - tried both RELENG_6 and HEAD (WITNESS, INVARIANTS disabled) **
> - disabling and enabling SMP **
> - disabling and enabling PREEMPTION, ADAPTIVE_MUTEXES and ADAPTIVE_GIANT
> - disabling and enabling kern.sched.ipiwakeup.enabled
> - decreasing kern.hz
> - changing timecounter to TSC
>
> [**] : There's a special case here: I've just finished compiling a UP
> kernel for HEAD, and when running unixbench, the "Pipe throughput"
> benchmark of unixbench gets an order of magnitude better (to ~~480). I'm
> too tired now to dig further on it.
>
> The peculiar thing is that such a slowdown is not present in Linux and
> Windows - there it it's something like 10% slowdown at most (compared to
> the same machine in non-virtual mode), while doing the same comparison
> on FreeBSD it comes on the order of 4x - 5x slower.
>
> I think this finding should be easily reproducable and verifyable -
> anyone with a Windows or Linux machine can install the free VMWare
> Server and load up FreeBSD as guest.
>
> I don't know whose fault this is, VMWares or FreeBSD's, but
> virtualization is popular, and since FreeBSD is very much lagging behind
> for server-side virtualization (Xen, VMWare, etc. - jails and vimage
> don't count since they're FreeBSD-specific), it would be a shame if it
> also got marked as bad at client-side.
>
> Any ideas on what to try next to improve the performance?

   At work, we have several build guests running FreeBSD. Overall it
seems to be a timer problem when running in VMware. Running
vmware-guestd helped in almost all circumstances. The worst cases we
saw were in processes who were sleep()-ing.


-- 
If it's there, and you can see it, it's real.
If it's not there, and you can see it, it's virtual.
If it's there, and you can't see it, it's transparent.
If it's not there, and you can't see it, you erased it.