Fwd: bhyve clock jitter

Andrea Brancatelli abrancatelli at schema31.it
Thu Apr 14 06:03:35 UTC 2016


Hi ...

I have a strange issue about the clock of some FreeBSD Bhyve guests, maybe someone of you can help me to diagnose it and work on a solution.

In detail, sometimes (without an apparent regularity) one of four bhyve guest's clock goes exactly 7 minutes and 10 seconds in the future.
 
This clock stays unaligned for about 20 minutes, then, without any human intervention, the clock realign to proper condition.

My configuration is:

envrm02 (bhyve HOST, 10.2-RELEASE-p14)
-> og-qa (bhyve guest - 10.2-RELEASE-p14)
-> og-ci (bhyve guest - 10.2-RELEASE-p14)
-> og-test (bhyve guest - 10.2-RELEASE-p14)
-> og-prod-rm (bhyve guest - 10.2-RELEASE-p14)

All the og-* machines just have a running copy of tomcat.

I built a simple test to catch the error with python/fabric... the test is:

fab -R ENV,OG,OG-PROD -- date | awk -F ': ' '/2016/{print $2, $1}'

which simply run the command "date" on all hosts.

So, the result is:

@Mac-mini ~/p/f/> ./check_clock.sh
Thu Apr  7 17:38:00 CEST 2016 [envrm02] out
Thu Apr  7 17:38:00 CEST 2016 [og-qa] out
Thu Apr  7 17:38:00 CEST 2016 [og-ci] out
Thu Apr  7 17:38:00 CEST 2016 [og-test] out
Thu Apr  7 17:38:00 CEST 2016 [og-prod-rm] out

@Mac-mini ~/p/f/> ./check_clock.sh
Thu Apr  7 18:38:00 CEST 2016 [envrm02] out
Thu Apr  7 18:45:10 CEST 2016 [og-qa] out
Thu Apr  7 18:38:00 CEST 2016 [og-ci] out
Thu Apr  7 18:38:00 CEST 2016 [og-test] out
Thu Apr  7 18:38:00 CEST 2016 [og-prod-rm] out

@Mac-mini ~/p/f/> ./check_clock.sh
Thu Apr  7 18:41:05 CEST 2016 [envrm02] out
Thu Apr  7 18:48:15 CEST 2016 [og-qa] out
Thu Apr  7 18:41:05 CEST 2016 [og-ci] out
Thu Apr  7 18:41:05 CEST 2016 [og-test] out
Thu Apr  7 18:41:05 CEST 2016 [og-prod-rm] out

@Mac-mini ~/p/f/> ./check_clock.sh
Thu Apr  7 18:46:51 CEST 2016 [envrm02] out
Thu Apr  7 18:46:52 CEST 2016 [og-qa] out
Thu Apr  7 18:46:52 CEST 2016 [og-ci] out
Thu Apr  7 18:46:53 CEST 2016 [og-test] out
Thu Apr  7 18:46:53 CEST 2016 [og-prod-rm] out

The configuration is the same for all hosts and is this:

@Mac-mini ~/p/f/> fab -R ENV,OG,OG-PROD -- cat /etc/rc.conf | grep ntpd
[envrm02] out: ntpd_enable="YES"
[og-qa] out: ntpd_enable="YES"
[og-qa] out: ntpd_sync_on_start="YES"
[og-ci] out: ntpd_enable="YES"
[og-ci] out: ntpd_sync_on_start="YES"
[og-test] out: ntpd_enable="YES"
[og-test] out: ntpd_sync_on_start="YES"
[og-prod-rm] out: ntpd_enable="YES"
[og-prod-rm] out: ntpd_sync_on_start="YES"

@Mac-mini ~/p/f/> fab -R ENV,OG,OG-PROD -- cat /etc/ntp.conf | grep -v 'out: #'
[envrm02] Executing task '<remainder>'
[envrm02] run: cat /etc/ntp.conf
[envrm02] out:
[envrm02] out: server 0.freebsd.pool.ntp.org iburst
[envrm02] out: server 1.freebsd.pool.ntp.org iburst
[envrm02] out: server 2.freebsd.pool.ntp.org iburst
[envrm02] out:
[envrm02] out: restrict default limited kod nomodify notrap nopeer noquery
[envrm02] out: restrict -6 default limited kod nomodify notrap nopeer noquery
[envrm02] out: restrict 127.0.0.1
[envrm02] out: restrict -6 ::1
[envrm02] out: restrict 127.127.1.0

[og-qa] Executing task '<remainder>'
[og-qa] run: cat /etc/ntp.conf
[og-qa] out:
[og-qa] out: server 0.freebsd.pool.ntp.org iburst
[og-qa] out: server 1.freebsd.pool.ntp.org iburst
[og-qa] out: server 2.freebsd.pool.ntp.org iburst
[og-qa] out:
[og-qa] out: restrict default limited kod nomodify notrap nopeer noquery
[og-qa] out: restrict -6 default limited kod nomodify notrap nopeer noquery
[og-qa] out: restrict 127.0.0.1
[og-qa] out: restrict -6 ::1
[og-qa] out: restrict 127.127.1.0

[og-ci] Executing task '<remainder>'
[og-ci] run: cat /etc/ntp.conf
[og-ci] out:
[og-ci] out: server 0.freebsd.pool.ntp.org iburst
[og-ci] out: server 1.freebsd.pool.ntp.org iburst
[og-ci] out: server 2.freebsd.pool.ntp.org iburst
[og-ci] out:
[og-ci] out: restrict default limited kod nomodify notrap nopeer noquery
[og-ci] out: restrict -6 default limited kod nomodify notrap nopeer noquery
[og-ci] out: restrict 127.0.0.1
[og-ci] out: restrict -6 ::1
[og-ci] out: restrict 127.127.1.0

[og-test] Executing task '<remainder>'
[og-test] run: cat /etc/ntp.conf
[og-test] out:
[og-test] out: server 0.freebsd.pool.ntp.org iburst
[og-test] out: server 1.freebsd.pool.ntp.org iburst
[og-test] out: server 2.freebsd.pool.ntp.org iburst
[og-test] out:
[og-test] out: restrict default limited kod nomodify notrap nopeer noquery
[og-test] out: restrict -6 default limited kod nomodify notrap nopeer noquery
[og-test] out: restrict 127.0.0.1
[og-test] out: restrict -6 ::1
[og-test] out: restrict 127.127.1.0


Some observation: 
* The clock of envrm02 (the HOST) always seems to be correct.
* The issue happens on a (apparently) random guest (not always the same one).
* The issue happens always on one host at a time.


The bhyve command line is, roughly, this one:

/usr/sbin/bhyveload -m 2048 -d /san_storage/VMfs/$machine/$machine.img $machine
/usr/sbin/bhyve -c 2 -m 2048 -A -H -P -s 0,hostbridge -s 2,ahci-hd,/san_storage/VMfs/$machine/$machine.img -s 4,virtio-net,$tap -s 31,lpc -l com1,stdio $machine

Thanks in advance.

 

-- 
Andrea Brancatelli
Schema31 S.p.a.
Responsabile IT

ROMA - BO - FI - PA 
ITALY
Tel: +39.06.98.358.472
Cell: +39.331.2488468
Fax: +39.055.71.880.466
Società del Gruppo SC31 ITALIA


More information about the freebsd-virtualization mailing list