become worse now - Re: one virtualbox vm disrupts all vms and entire network

Steve Tuts yiz5hwi at gmail.com
Fri Jul 13 20:34:18 UTC 2012


On Mon, Jul 9, 2012 at 9:11 AM, Steve Tuts <yiz5hwi at gmail.com> wrote:

>
>
> On Tue, Jun 12, 2012 at 6:24 PM, Gary Palmer <gpalmer at freebsd.org> wrote:
>
>> On Thu, Jun 07, 2012 at 03:56:22PM -0400, Steve Tuts wrote:
>> > On Thu, Jun 7, 2012 at 3:54 AM, Steve Tuts <yiz5hwi at gmail.com> wrote:
>> >
>> > >
>> > >
>> > > On Thu, Jun 7, 2012 at 2:58 AM, Bernhard Fr?hlich <decke at bluelife.at
>> >wrote:
>> > >
>> > >> On Do.,   7. Jun. 2012 01:07:52 CEST, Kevin Oberman <
>> kob6558 at gmail.com>
>> > >> wrote:
>> > >>
>> > >> > On Wed, Jun 6, 2012 at 3:46 PM, Steve Tuts <yiz5hwi at gmail.com>
>> wrote:
>> > >> > > On Wed, Jun 6, 2012 at 3:50 AM, Bernhard Froehlich
>> > >> > > <decke at freebsd.org>wrote:
>> > >> > >
>> > >> > > > On 05.06.2012 20:16, Bernhard Froehlich wrote:
>> > >> > > >
>> > >> > > > > On 05.06.2012 19:05, Steve Tuts wrote:
>> > >> > > > >
>> > >> > > > > > On Mon, Jun 4, 2012 at 4:11 PM, Rusty Nejdl
>> > >> > > > > > <rnejdl at ringofsaturn.com> wrote:
>> > >> > > > > >
>> > >> > > > > >  On 2012-06-02 12:16, Steve Tuts wrote:
>> > >> > > > > > >
>> > >> > > > > > >  Hi, we have a Dell poweredge server with a dozen
>> interfaces.
>> > >> > > > > > >  It hosts
>> > >> > > > > > > > a
>> > >> > > > > > > > few guests of web app and email servers with
>> > >> > > > > > > > VirtualBox-4.0.14.  The host
>> > >> > > > > > > > and all guests are FreeBSD 9.0 64bit.  Each guest is
>> bridged
>> > >> > > > > > > > to a distinct
>> > >> > > > > > > > interface.  The host and all guests are set to 10.0.0.0
>> > >> > > > > > > > network NAT'ed to
>> > >> > > > > > > > a
>> > >> > > > > > > > cicso router.
>> > >> > > > > > > >
>> > >> > > > > > > > This runs well for a couple months, until we added a
>> new
>> > >> > > > > > > > guest recently.
>> > >> > > > > > > > Every few hours, none of the guests can be connected.
>>  We
>> > >> > > > > > > > can only connect
>> > >> > > > > > > > to the host from outside the router.  We can also go
>> to the
>> > >> > > > > > > > console of the
>> > >> > > > > > > > guests (except the new guest), but from there we can't
>> ping
>> > >> > > > > > > > the gateway 10.0.0.1 any more.  The new guest just
>> froze.
>> > >> > > > > > > >
>> > >> > > > > > > > Furthermore, on the host we can see a vboxheadless
>> process
>> > >> > > > > > > > for each guest,
>> > >> > > > > > > > including the new guest.  But we can not kill it, not
>> even
>> > >> > > > > > > > with "kill -9".
>> > >> > > > > > > > We looked around the web and someone suggested we
>> should use
>> > >> > > > > > > > "kill -SIGCONT" first since the "ps" output has the
>> "T" flag
>> > >> > > > > > > > for that vboxheadless process for that new guest, but
>> that
>> > >> > > > > > > > doesn't help.  We also
>> > >> > > > > > > > tried all the VBoxManager commands to poweroff/reset
>> etc
>> > >> > > > > > > > that new guest,
>> > >> > > > > > > > but they all failed complaining that vm is in Aborted
>> state.
>> > >> > > > > > > >  We also tried
>> > >> > > > > > > > VBoxManager commands to disconnect the network cable
>> for
>> > >> > > > > > > > that new guest,
>> > >> > > > > > > > it
>> > >> > > > > > > > didn't complain, but there was no effect.
>> > >> > > > > > > >
>> > >> > > > > > > > For a couple times, on the host we disabled the
>> interface
>> > >> > > > > > > > bridging that new
>> > >> > > > > > > > guest, then that vboxheadless process for that new
>> guest
>> > >> > > > > > > > disappeared (we
>> > >> > > > > > > > attempted to kill it before that).  And immediately all
>> > >> > > > > > > > other vms regained
>> > >> > > > > > > > connection back to normal.
>> > >> > > > > > > >
>> > >> > > > > > > > But there is one time even the above didn't help - the
>> > >> > > > > > > > vboxheadless process
>> > >> > > > > > > > for that new guest stubbonly remains, and we had to
>> reboot
>> > >> > > > > > > > the host.
>> > >> > > > > > > >
>> > >> > > > > > > > This is already a production server, so we can't
>> upgrade
>> > >> > > > > > > > virtualbox to the
>> > >> > > > > > > > latest version until we obtain a test server.
>> > >> > > > > > > >
>> > >> > > > > > > > Would you advise:
>> > >> > > > > > > >
>> > >> > > > > > > > 1. is there any other way to kill that new guest
>> instead of
>> > >> > > > > > > > rebooting? 2. what might cause the problem?
>> > >> > > > > > > > 3. what setting and test I can do to analyze this
>> problem?
>> > >> > > > > > > > ______________________________****_________________
>> > >> > > > > > > >
>> > >> > > > > > > >
>> > >> > > > > > > I haven't seen any comments on this and don't want you to
>> > >> > > > > > > think you are being ignored but I haven't seen this but
>> also,
>> > >> > > > > > > the 4.0 branch was buggier
>> > >> > > > > > > for me than the 4.1 releases so yeah, upgrading is
>> probably
>> > >> > > > > > > what you are looking at.
>> > >> > > > > > >
>> > >> > > > > > > Rusty Nejdl
>> > >> > > > > > > ______________________________****_________________
>> > >> > > > > > >
>> > >> > > > > > >
>> > >> > > > > > >  sorry, just realize my reply yesterday didn't go to the
>> list,
>> > >> > > > > > > so am
>> > >> > > > > > re-sending with some updates.
>> > >> > > > > >
>> > >> > > > > > Yes, we upgraded all ports and fortunately everything went
>> back
>> > >> > > > > > and especially all vms has run peacefully for two days
>> now.  So
>> > >> > > > > > upgrading to the latest virtualbox 4.1.16 solved that
>> problem.
>> > >> > > > > >
>> > >> > > > > > But now we got a new problem with this new version of
>> > >> virtualbox:
>> > >> > > > > > whenever
>> > >> > > > > > we try to vnc to any vm, that vm will go to Aborted state
>> > >> > > > > > immediately. Actually, merely telnet from within the host
>> to the
>> > >> > > > > > vnc port of that vm will immediately Abort that vm.  This
>> > >> > > > > > prevents us from adding new vms. Also, when starting vm
>> with vnc
>> > >> > > > > > port, we got this message:
>> > >> > > > > >
>> > >> > > > > > rfbListenOnTCP6Port: error in bind IPv6 socket: Address
>> already
>> > >> > > > > > in use
>> > >> > > > > >
>> > >> > > > > > , which we found someone else provided a patch at
>> > >> > > > > >
>> > >>
>> http://permalink.gmane.org/**gmane.os.freebsd.devel.**emulation/10237<
>> > >> http://permalink.gmane.org/gmane.os.freebsd.devel.emulation/10237>
>> > >> > > > > >
>> > >> > > > > > So looks like when there are multiple vms on a ipv6 system
>> (we
>> > >> > > > > > have 64bit FreeBSD 9.0) will get this problem.
>> > >> > > > > >
>> > >> > > > >
>> > >> > > > > Glad to hear that 4.1.16 helps for the networking problem.
>> The VNC
>> > >> > > > > problem is also a known one but the mentioned patch does not
>> work
>> > >> > > > > at least for a few people. It seems the bug is somewhere in
>> > >> > > > > libvncserver so downgrading net/libvncserver to an earlier
>> version
>> > >> > > > > (and rebuilding virtualbox) should help until we come up
>> with a
>> > >> > > > > proper fix.
>> > >> > > > >
>> > >> > > >
>> > >> > > > You are right about the "Address already in use" problem and
>> the
>> > >> > > > patch for it so I will commit the fix in a few moments.
>> > >> > > >
>> > >> > > > I have also tried to reproduce the VNC crash but I couldn't.
>> > >> Probably
>> > >> > > > because
>> > >> > > > my system is IPv6 enabled. flo@ has seen the same crash and
>> has no
>> > >> > > > IPv6 in his kernel which lead him to find this commit in
>> > >> > > > libvncserver:
>> > >> > > >
>> > >> > > >
>> > >> > > > commit 66282f58000c8863e104666c30cb67**b1d5cbdee3
>> > >> > > > Author: Kyle J. McKay <mackyle at gmail.com>
>> > >> > > > Date:   Fri May 18 00:30:11 2012 -0700
>> > >> > > >     libvncserver/sockets.c: do not segfault when
>> > >> > > > listenSock/listen6Sock == -1
>> > >> > > >
>> > >> > > > http://libvncserver.git.**
>> > >> sourceforge.net/git/gitweb.**cgi?p=libvncserver/
>> > >> > > > **libvncserver;a=commit;h=**66282f5<
>> > >>
>> http://libvncserver.git.sourceforge.net/git/gitweb.cgi?p=libvncserver/libvncserver;a=commit;h=66282f5
>> > >> >
>> > >> > > >
>> > >> > > >
>> > >> > > > It looks promising so please test this patch if you can
>> reproduce
>> > >> the
>> > >> > > > crash.
>> > >> > > >
>> > >> > > >
>> > >> > > > --
>> > >> > > > Bernhard Froehlich
>> > >> > > > http://www.bluelife.at/
>> > >> > > >
>> > >> > >
>> > >> > > Sorry, I tried to try this patch, but couldn't figure out how to
>> do
>> > >> > > that. I use ports to compile everything, and can see the file is
>> at
>> > >> > >
>> > >>
>> /usr/ports/net/libvncserver/work/LibVNCServer-0.9.9/libvncserver/sockets.c
>> > >> > > .  However, if I edit this file and do make clean, this patch is
>> wiped
>> > >> > > out before I can do "make" out of it.  How to apply this patch
>> in the
>> > >> > > ports?
>> > >> >
>> > >> > To apply patches to ports:
>> > >> > # make clean
>> > >> > # make patch
>> > >> > <Apply patch>
>> > >> > # make
>> > >> > # make deinstall
>> > >> > # make reinstall
>> > >> >
>> > >> > Note that the final two steps assume a version of the port is
>> already
>> > >> > installed. If not: 'make install'
>> > >> > I you use portmaster, after applying the patch: 'portmaster -C
>> > >> > net/libvncserver' --
>> > >>
>> > >> flo has already committed the patch to net/libvncserver so I guess it
>> > >> fixes the problem. Please update your portstree and verify that it
>> works
>> > >> fine.
>> > >>
>> > >
>> > > I confirmed after upgrading all ports and noticing libvncserver
>> upgraded
>> > > to 0.99_1 and reboot, then I can vnc to the vms now.  Also, starting
>> vms
>> > > with vnc doesn't have that error now, instead it issues the following
>> info,
>> > > so all problem are solved.
>> > >
>> > > 07/06/2012 03:49:14 Listening for VNC connections on TCP port 5903
>> > > 07/06/2012 03:49:14 Listening for VNC connections on TCP6 port 5903
>> > >
>> > > Thanks everyone for your great help!
>> > >
>> >
>> > Unfortunately, seems that the original problem of one vm disrupts all
>> vms
>> > and entire network appears to remain, albeit to less scope.  After
>> running
>> > on virtualbox-ose-4.1.16_1 and libvncserver-0.9.9_1 for 12 hours, all
>> vms
>> > lost connection again.  Also, phpvirtualbox stopped responding, and
>> > attempts to restart vboxwebsrv hanged.  And trying to kill (-9) the
>> > vboxwebsrv process won't work.  The following was the output of "ps
>> > aux|grep -i box" at that time:
>> >
>> > root 3322  78.7 16.9 4482936 4248180  ??  Is    3:42AM   126:00.53
>> > /usr/local/bin/VBoxHeadless --startvm vm1
>> > root 3377   0.2  4.3 1286200 1078728  ??  Is    3:42AM    15:39.40
>> > /usr/local/bin/VBoxHeadless --startvm vm2
>> > root 3388   0.1  4.3 1297592 1084676  ??  Is    3:42AM    15:06.97
>> > /usr/local/bin/VBoxHeadless --startvm vm7 -n -m 5907 -o
>> jtlgjkrfyh9tpgjklfds
>> > root 2453   0.0  0.0 141684   7156  ??  Ts    3:38AM     4:14.09
>> > /usr/local/bin/vboxwebsrv
>> > root 2478   0.0  0.0  45288   2528  ??  S     3:38AM     1:29.99
>> > /usr/local/lib/virtualbox/VBoxXPCOMIPCD
>> > root 2494   0.0  0.0 121848   5380  ??  S     3:38AM     3:13.96
>> > /usr/local/lib/virtualbox/VBoxSVC --auto-shutdown
>> > root 3333   0.0  4.3 1294712 1079608  ??  Is    3:42AM    19:35.09
>> > /usr/local/bin/VBoxHeadless --startvm vm3
>> > root 3355   0.0  4.3 1290424 1079332  ??  Is    3:42AM    16:43.05
>> > /usr/local/bin/VBoxHeadless --startvm vm5
>> > root 3366   0.0  8.5 2351436 2140076  ??  Is    3:42AM    17:32.35
>> > /usr/local/bin/VBoxHeadless --startvm vm6
>> > root 3598   0.0  4.3 1294520 1078664  ??  Ds    3:50AM    15:01.04
>> > /usr/local/bin/VBoxHeadless --startvm vm4 -n -m 5904 -o
>> > u679y0uojlkdfsgkjtfds
>> >
>> > You can see the vboxwebsrv process has the "T" flag there, and the
>> > vboxheadless process for vm4 has "D" flag there.  Both of such
>> processes I
>> > can never kill them, not even with "kill -9".  So on the host I disabled
>> > the interface bridged to vm4 and restarted network, and fortunately both
>> > the vm4 and the vboxwebsrv processed disappeared.  And at that point all
>> > other vms regained network.
>> >
>> > There may be one hope that the "troublemaker" may be limited to one of
>> the
>> > vms that started with vnc, although there was no vnc connection at that
>> > time, and the other vm with vnc was fine.  And this is just a hopeful
>> guess.
>> >
>> > Also I found no log or error message related to virtualbox in any log
>> > file.  The VBoxSVC.log only had some information when started but never
>> > since.
>>
>> If this is still a problem then
>>
>> ps alxww | grep -i box
>>
>> may be more helpful as it will show the wait channel of processes stuck
>> in the kernel.
>>
>> Gary
>>
>
> We avoided this problem by running all vms without vnc.  But forgot this
> problem and left one vm on with vnc, together with the other few running
> vms yesterday, and hit this problem again on virtualbox 4.1.16.  Only the
> old trick of turning off the host interface corresponding to the vm with
> vnc and then restarting host network got us out of the problem.
>
> We then upgraded virtualbox to 4.1.18, turning off all vms, wait until "ps
> aux|grep -i box" reported nothing, then started all vms.  And let no vm
> with vnc running.
>
> Still the problem hit us again.  Here is the output of " ps alxww | grep
> -i box" as you suggested:
>
> 1011    42725    1    0    20    0    1289796    1081064    IPRT S
> Is    ??    30:53.24  VBoxHeadless --startvm vm5
>
> after "kill -9 42725", the line changed to
>
> 1011    42725    1    0    20    0    1289796    1081064    keglim
> Ts    ??    30:53.24  VBoxHeadless --startvm vm5
>
> after "kill -9" for another vm, the line changed to something like
>
> 1011    42754    1    0    20    0    1289796    1081064    -    Ts
> ??    30:53.24  VBoxHeadless --startvm vm7
>
> and controlvm command don't work, and these command stuck there
> themselves.  The following are their outputs:
>
> 0    89572    79180    0    21    0    44708    1644    select     I+
> v6    0:00.01    VBoxManage controlvm projects_outside acpipowerbutton
> 0    89605    89586    0    21    0    44708    2196    select     I+
> v7    0:00.01    VBoxManage controlvm projects_outside poweroff
>
> We now rebooted the host, and left no vm with vnc running.
>

The problem has become more rampant now.  After rebooting and running
virtualbox-ose-4.1.18, and no vm was started with console, the around 10
vms, bridged to each of its own dedicated interface, get no network
connection a couple times a day. Most times it would recover itself after
about 10 minutes, sometimes we have to restart host network which
immediately restore all connections.


More information about the freebsd-emulation mailing list