chrony fails to start, nm-dispatcher hook causes high CPU load

[Kevin Otte]

Chrony is failing to start correctly, and the chronyc command launched by the NetworkManager dispatcher hook is going to 100% CPU.

ProblemType: Bug
DistroRelease: Ubuntu 18.04
Package: chrony 3.2-4ubuntu4.1
ProcVersionSignature: Ubuntu 4.15.0-24.26-generic 4.15.18
Uname: Linux 4.15.0-24-generic x86_64
ApportVersion: 2.20.9-0ubuntu7.2
Architecture: amd64
CurrentDesktop: XFCE
Date: Tue Jul 3 19:50:55 2018
InstallationDate: Installed on 2018-04-12 (82 days ago)
InstallationMedia: Xubuntu 18.04 LTS "Bionic Beaver" - Beta amd64 (20180404)
SourcePackage: chrony
UpgradeStatus: No upgrade log present (probably fresh install)

[Kevin Otte (nivex)]

Jul 3 19:44:07 icarus systemd[1]: chrony.service: Start operation timed out. Terminating.
Jul 3 19:44:07 icarus systemd[1]: chrony.service: Failed with result 'timeout'.
Jul 3 19:44:07 icarus systemd[1]: Failed to start chrony, an NTP client/server.

[Joshua Powers (powersj)]

Thank you for taking the time to file a bug report.

There is not enough information to begin working on this or narrowing down where the issue may be. Are you running a custom configuration for chrony? What other changes were made to the system? Any other data you can pass on to help triage the issue?

Since there is not enough information in your report to begin triage or to
differentiate between a local configuration problem and a bug in Ubuntu, I
am marking this bug as "Incomplete". We would be grateful if you would:
provide a more complete description of the problem, explain why you
believe this is a bug in Ubuntu rather than a problem specific to your
system, and then change the bug status back to "New".

For local configuration issues, you can find assistance here:
http://www.ubuntu.com/support/community

[Kevin Otte (nivex)]

The information I provided is taken from a VM upon logging in after initial boot. I provided my chrony.conf as attachment precisely to head off the question about custom config.

The logs were not clear on why chrony is not starting cleanly. Regardless, attempts to call chronyc by the NM dispatcher are hanging an entire CPU. Neither of these are particularly good.

Interestingly, this configuration was working fine for awhile and only recently started displaying these issues.

[Joshua Powers (powersj)]

Ah sorry I missed the chrony attachment!

The only differences appear to be:

> refclock PHC /dev/ptp_kvm poll 3 dpoll -2 refid KVM
> server atlantis-int.home.nivex.net offline

Did your system have /dev/ptp_kvm already? My test bionic VM does not have that device.

$ ethtool -T ens3
...
PTP Hardware Clock: none
...

I'll look more Monday.

[Joshua Powers (powersj)]

Ah! I found a system that has it I'll try to reproduce there on Monday.

On the host:
/dev/ptp0

and

$ ethtool -T eno1
...
PTP Hardware Clock: 0
...

[Kevin Otte (nivex)]

ptp_kvm is being loaded in /etc/modules on my machine

[Joshua Powers (powersj)]

Noticed another chrony issue, bug 1779621, come in about failing to start. If you wouldn't mind reading over that and seeing if you have similar issues that might be good.

I am going to wait for a chrony expert to get back from vacation and have him review both of these.

[Andreas Hasenack (ahasenack)]

Do you have any apparmor messages in the dmesg output of that machine where chrony is failing to start?

After the first reboot, if you issue a sudo systemctl restart chrony, does it work? Or same problem?

You said it's a desktop, how is the network brought up there? Is it only available after the user logs in?

You also said at some point it was a VM, is that just for testing and debugging this problem, and you have a "real" desktop elsewhere, or is your real use case that VM?

[Kevin Otte (nivex)]

icarus is my virtual desktop to test all manner of things. It is so named for if I fly it too close to the sun I just rebuild it. As such, I did just that, and was unable to reproduce this issue.

[Andreas Hasenack (ahasenack)]

A key difference between a vm and a "real" desktop could be networking, specially if wifi is involved. For example, in some cases the network might only be available after the user logs in. Do you think this could have been the case here?

[Kevin Otte (nivex)]

chrony will still start even if the network isn't up. It will just not report any reachability to peers (or possibly any peers at all). For some reason chrony was straight up not running. Not sure why that was the case.

My guess is the high CPU from the chronyc command invoked by the nm-dispatcher script was because systemd had reserved the control socket. The client ended up spinning talking to a socket going nowhere.

The system was also in a weird state where the lightdm login was not appearing until after a login on one of the virtual consoles, so it probably had other issues.