PINE64 - 12.0-CURRENT r324563 - ntpd can't keep time

Tue Nov 14 11:29:10 UTC 2017

[Top post of an FYI and other short notes.]

FYI: My Pine64+ 2GB context uses an external USB3
powered hub with a USB3 SSD on one of the Pine64+ 2GB
USB2 ports in order to hold the UFS root file system.
(I can boot with just the sdcard but rarely do.)

(I've only used ZFS in a couple of contexts, all with
more than 2 GiBytes of RAM.)

The -r320599 'works' point means that a (slow) bisect
may be possible in your context to narrow the range of
what changes in head might be at issue in your context.
What is the smallest -r?????? figure above -r320599
that you have seen the problem with? Could you test
some approximate mid-point between the two? (A
difficulty with this can be avoiding other known
problems in the history.)

[End top post]

On 2017-Nov-14, at 1:39 AM, Mark Millard <markmi at dsl-only.net> wrote:

> On 2017-Nov-13, at 11:06 PM, Henri Hennebert <hlh at restart.be> wrote:
> 
>>> . . .
>> 
>> I believe that the clock of the Pine64+ is going too fast and that the 2 servers where polled and so show this offset/jitter. In an other occurrence of this problem, if I wait long enough, all servers display huge offset.
>> In a old version of Freebsd12, when dev.cpu.0.freq was accessible, the problem appear when I force the frequency to 1200.
> 
> May be the time is just being damaged on occasion
> or read-incorrectly sometimes? (Once seen as wrong,
> the adjustment process will gradually change time
> towards what it calculates as a supposedly better
> time.)
> 
> Good to know that eventually  all the servers show
> a large jitter(?) at the same time. (Jitter reflects
> more history and so may be a better reference.)
> 
>> In the same network, other computers (amd64) get no problem with this ntpd configuration.
> 
> Good to know. It helps localize the issue.
> 
>>> It looks to me like you need to avoid those 2 severs,
>>> substituting some others or some such. If some
>>> communication channel(s) involved are corrupting data
>>> then simply switching servers might not avoid the
>>> issue?
>>> I finally got a hold of the rpi3 and updated it to
>>> -r325700 from back a -r320123. it has been up 9 hr
>>> 40 min and date is still showing the correct time,
>>> no evidence of huge offsets or huge jitter.
>>> The Pine64+ 2GB is also at -r325700 now and has been
>>> up for 2 days. It also is working fine. Again no
>>> evidence of huge offsets or huge jitter.
>> 
>> Did you run heavy jobs running on this system? My Pine64+ is managing my connection to internet (mpd5), named, dhcp, my mail (sendmail + cyrus-imap) and dspam with postgres as db. The problem crops up when for example I run a port upgrade.
> 
> I did a complete, from-scratch -j4 buildworld buildkernel
> on the Pine64+ 2GB, keeping it busy for a little under
> 18 hr + 10 min as I remember. top showed that it used some
> swapspace during this. I included:
> 
> WITH_CLANG=
> WITH_CLANG_IS_CC=
> WITH_CLANG_FULL=
> WITH_CLANG_EXTRAS=
> WITH_LLD_BOOTSTRAP=
> WITH_LLD=
> WITH_LLD_IS_LD=
> WITH_LLDB=
> WITH_BOOT=
> 
> But it did claim to reuse the system clang instead
> of building a bootstrap clang compiler. Yet the
> WITH_LLD_BOOTSTRAP= did lead to a bunch of
> bootstrap activity anyway, so two builds of lld
> related materials. (The purpose of the build
> was to see if it would complete or not, extra work
> made for a better test.)
> 
> It was on an earlier version than -r325700 
> that I last rebuilt my ports but rebuilding the
> ports has never caused a time problem either.
> The ports builds were via poudriere.
> 
> But I do not use the Pine64+ 2GB for any of those
> other things that you list as using yours for. I
> do ssh over ethernet into the Pine64+ to work on
> it but it is not a service provider at all, just
> a machine for me to do build experiments with
> (native and cross builds) and other forms of problem
> investigations. nfs is present but little/rarely
> used, and only temporarily in use at that.
> 
> As a step in potential isolation of what contributes
> vs. what does not, could you temporarily revert to
> not using/running the Pine64+ 2GB for the other
> activities and try the ports builds and/or buildworld
> buildkernel? Does it still have the problem in
> something like my simpler context?
> 
> If time no long jumps, then you could add back an
> activity and try again until the additions lead to
> the time problem. That might give a clue what could
> be interfering with time.
> 
> Also: I do not know if you have access to a 2nd
> Pine64 to check for a device-specific problem by
> switching devices temporarily. Such might also
> allow keeping your services active while checking
> the simpler context.
> 
> If Pine64s had a general time problem, there would
> be a general problem for everyone else. I think the
> stage of investigation is still: try to isolate a
> smaller context sufficient to have the problem but
> for which an even smaller context is observed to
> not have the problem. Try to narrow the range of
> differences between observed-working and
> observed-failing to gain a clue about what
> contributes to the problem vs. what does not.
> 
> 
> 
> FYI: Sitting idle the Pine64+ 2GB here shows
> in top:
> 
> sshd: <?>@pts/0 (sshd)
> sshd: <?> [priv] (sshd)
> /usr/sbin/sshd
> /usr/sbin/ntpd -g -c /etc/ntp.conf -p /var/run/ntpd.pid -f /var/db/ntpd.drift
> sendmail: accepting connections (sendmail)
> sendmail: Queue runner at 00:30:00 for /var/spool/clientmqueue (sendmail)
> /sbin/devd
> top -CawPosize
> login [pam] (<login>)
> su (<su>)
> top -CawPosize
> /usr/sbin/mountd -r
> su (sh)
> -sh (sh)
> -sh (<sh>)
> nfsd: master (nfsd)
> dhclient: awg0 (dhclient)
> /usr/sbin/cron -s (<cron>)
> dhclient: awg0 [priv] (dhclient)
> /usr/sbin/syslogd -s
> /usr/sbin/rpcbind
> nfsd: server (nfsd)
> 
> (I do not normally use sendmail for anything. It
> just sits there while the Pine64+ 2GB is up. nfs
> is normally idle but is used on occasion.)
> 
> FYI: The installed ports list (far more is built
> by poudriere):
> 
> # pkg info
> atf-0.21                       C, C++ and shell libraries to write ATF-compliant test programs
> binutils-2.28,1                GNU binary tools
> bonnie-2.0.6_1                 Performance Test of Filesystem I/O
> bonnie++-1.97.3                Performance Test of Filesystem I/O
> ca_root_nss-3.32.1             Root certificate bundle from the Mozilla Project
> curl-7.56.1                    Command line tool and library for transferring data with URLs
> dialog4ports-0.1.6             Console Interface to configure ports
> dtrace-toolkit-1.0_1           Collection of useful scripts for DTrace
> dwarfdump-20161124             Tool to display DWARF debugging information in ELF files
> expat-2.2.1                    XML 1.0 parser written in C
> freebsd-release-manifests-20171003 FreeBSD release manifests
> gcc7-7.2.0_2                   GNU Compiler Collection 7
> gdb-8.0.1_1                    GNU GDB of newer version than comes with the system
> gettext-runtime-0.19.8.1_1     GNU gettext runtime libraries and programs
> git-lite-2.14.3                Distributed source code management tool (lite package)
> gmp-6.1.2                      Free library for arbitrary precision arithmetic
> indexinfo-0.3                  Utility to regenerate the GNU info page index
> iorate-3.05                    General purpose storage I/O benchmarking tool
> iozone-3.457                   Performance Test of Sequential File I/O
> kyua-0.13_4,3                  Testing framework for infrastructure software
> libedit-3.1.20170329_2,1       Command line editor library
> libidn2-2.0.4                  Implementation of IDNA2008 internationalized domain names
> libnghttp2-1.27.0              HTTP/2.0 C Library
> libunistring-0.9.7             Unicode string library
> lua52-5.2.4                    Small, compilable scripting language providing easy access to C code
> lutok-0.4_6                    Lightweight C++ API for Lua
> mpc-1.0.3                      Library of complex numbers with arbitrarily high precision
> mpfr-3.1.6                     Library for multiple-precision floating-point computations
> patch-2.7.5                    GNU patch utility
> pcre-8.40_1                    Perl Compatible Regular Expressions library
> perl5-5.24.3                   Practical Extraction and Report Language
> pkg-1.10.1                     Package manager
> portlint-2.17.13               Verifier for FreeBSD port directory
> portmaster-3.17.10             Manage your ports without external databases or languages
> poudriere-devel-3.1.99.20171028 Port build and test system
> randomio-1.4                   Multithreaded disk i/o microbenchmark
> readline-7.0.3_1               Library for editing command lines as they are typed
> rsync-3.1.2_7                  Network file distribution/synchronization utility
> sqlite3-3.20.1_2               SQL database engine in a C library
> stress-1.0.4                   Tool to impose load on and stress test Unix-like systems
> sudo-1.8.21p2                  Allow others to run commands as root
> unzip-6.0_7                    List, test, and extract compressed files from a ZIP archive
> wget-1.19.2                    Retrieve files from the Net via HTTP(S) and FTP
> zip-3.0_1                      Create/update ZIP files compatible with PKZIP
> 
> So it is enough to keep the Pine64+ 2GB busy
> for a a notable time when rebuilt. In particular,
> gcc7-7.2.0_2 (and what it requires) takes a fair
> time to build.
> 
> The poudriere build of the ports is currently the
> primary thing the ports are used for: build tests
> to catch build failures, should they occur.

===
Mark Millard
markmi at dsl-only.net