NTP 4.2.8 loses sync, system time far out

[jggimi]

Thank you for the detailed explanation.

(I will guess that you will not see significant performance gain by having virtual multiprocessor guests. Even thought MP performance improvements have been made and are continuing, the kernel uses Big Lock serialization.)

[MatthiasKoch]

Rebooted the machine in question once more with the option "Check and upgrade VMware Tools before each power on" (I am positive that this had been checked all the time anyway). This time the panel is displaying

VMware Tools: Running, version:2147483647 (Guest Managed)
DNS name: <machine's FQDN>
IP Addresses: <machine's IP>

correctly - VMware Tools are running now, for whatever reason (no log entrys whatsoever, though). Anyway the behaviour is unchanged - ntpd starts, corrects the time, and then breaks (running and claiming to correct the system time by ever growing amounts of seconds while not doing anything).

I now have stopped all services on the defector manually, putting it in the same state as the reference machine as shown per ps ax and lo - it's finally working. For the first time I see ntpd populating and updating /var/db/ntp.drift, alongside the message

Nov 18 13:11:40 n2 ntpd[52657]: adjusting clock frequency by -0.294265 to 22.404ppm

This has been the last ntpd message in /var/log/daemon in three hours, and the clock is still perfectly in sync.

I will keep it like this over the weekend and check on Monday. Then I'll re-activate the services one by one and attempt to reproduce the error.

Stopped services were:

• Nagios 4.2.2 (built from the latest downloaded version)
• Apache (apache-httpd-2.4.23 from packages)
• php70_fpm (php-7.0.8p0 from packages)
• NPCD (pnp4nagios-0.6.25p3 from packages)

Hope I'll be able to narrow the problem further down.

[MatthiasKoch]

OK - looks like I've finally found something.

First off - the clock has been running over the weekend, drifting no more than 0.00x seconds.

I have gradually re-enabled the stopped services one by one and have found that it is NPCD (a pnp4nagios daemon) that causes the problem. It is reproducable. Whether it's running alone or in conjunction with others makes no difference. At the moment NPCD starts, the system time is slowed down until it reaches a point (normally within minutes) where ntpd cannot fix it any longer. As soon as it stops, the time is slowly corrected until it eventually reaches it's normal drift window.

EDIT: Nagios, too, is making the clock drift. Apache and php70_fpm are not affecting it.

[jggimi]

A quick Google search found me this:

https://support.nagios.com/forum/vie...hp?f=7&t=36275

Within this thread is a comment (from user "jolson") that I believe describes pnp4nagios as a clock-adjusting application. If so, then perhaps you can change the application operating mode to eliminate the problem?

http://docs.pnp4nagios.org/pnp-0.6/modes

The OpenBSD port of pnp4nagios has a Maintainer you can contact for OpenBSD-specific application support, though it is not clear to me if this is an OpenBSD-specific problem. The Maintainer's Email address can be found with:

$ pkg_info pnp4nagios

[MatthiasKoch]

Originally Posted by jggimi

Within this thread is a comment (from user "jolson") that I believe describes pnp4nagios as a clock-adjusting application.

I agree that jolson seems to believe it is. There is however no hint in neither the description nor the corresponding conf-files that gives me a reason to believe that pnp4nagios sets the system clock. On top of that, it wouldn't make any sense. After all this isn't Linux, where systemd reconfigures your core daemons.

I admit I am unfamiliar with pnp4nagios, but I've used (and am currently using) Nagios installations on Linux machines both on physical machines and VMware virtual machines). And Nagios itself, too, is slowing down the system clock (although not as drastic as npcd), and I've never seen anything only close to that on Linux.

I'll drop jolson a message and ask for clarification.

[jggimi]

Test your pnp4nagios system on real hardware.

• If the problem recurs, then you have have an application problem or an OpenBSD problem.
• If the problem does not recur, then you have a virtual machine problem, either with VMWare, the OpenBSD kernel implementation of VMWare Tools, or with the virtualization technology of the underlying hardware.