[MIR] Chrony in 18.04

[Christian Ehrhardt ]

--- MIR ---

1. Availability: The package is Ubuntu universe and builds for the architectures it is designed to work on.

2. Rationale:
 2.1 NTP in general is needed quite a lot, but we want to exchange ntpd
     which is the current implementation in main with chrony for 18.04.
 2.2 Security: chrony was considered easier to be maintained easier in
     terms of security and provide a more modern ntp experience as well.
 2.3 Efficiency: Furthermore several cloud people seem to be interested to
     change to chrony in the guests for its lower memoy/cpu footprint
     (efficiency I guess).
2.4 related to this MIR 6 years ago this is the same but for Fedora.
    See: https://fedoraproject.org/wiki/Features/ChronyDefaultNTP
    IIRC some limitations that were present have been eliminated since, so
    it is even better than it was back then.
2.5 In general one has to realize that in a systemd-timesync world
    ntp/chrony are mostly for the "serving" portion of an ntp service, and
    not so much about the client (unless you the better accuracy vs
    timesyncd is needed).

3. Security: In fact the request came in by security Team, so I guess I call this section done

3. Quality assurance
 3.1 configuration ease - works after installation
 3.2 no high prio debconf
 3.3 usability (no major issues in Debian nor Ubuntu)
     asked Paul in regard to the ntp charm in comment #5
 3.4 long-term >=high bugs (none in Debian nor Ubuntu)
 3.5 Debian/Ubuntu bugs look reasonable maintained
 3.6 does not deal with hard to support exotic hardware (other than ntpd
     btw). If used this can be done through universe package GPSD (no
     dependency)
 3.7 Test suite runs on build (some skipped if not env applicable)
 3.8 debian/watch exists
 3.9 not depending on obsoleted packages

4.1 It does not face graphical UI
4.2 It is unfortunately not internationalized as far as I could see in the source

5. Dependencies - there is one not in main libtomcrypt
   We don't want it in main either, instead we want to fix bug 1744328 and then use libnss which is in main already.

6. Not found major Policy or FSH violations that would have to be fixed.

7. Maintenance
  7.1 Upstream - is maintained well (and better than ntpd it seems
       according to some discussisons)
  7.2 Ubuntu - Owning Team would be Ubuntu (in exchange to drop ntp)

8. Background information:
  Fulfills the same role as ntp, yet according to the security Team would
  be preferred for them.

--- Affected Packages ---

I'll add all those as bug tasks.
Once the MIR has passed the state of uncertainty (e.g. would it be blocked by one of the dependent bug being not doable at all) then please work on these into 18.04. Here a list what is affected in the listed packages:

Maas - needs to change dependencies and maybe template
cloud-init - needs to support writing ntp config to chrony instead of ntpd
ceph-base - change recommends from ntpd to chrony (it only intends to get good time and doesn't care via which dameon that is, so that should be ok to be change)
ntp charm - switch to chrony for >=18.04
chrony - MIR itself (discussion here and eventually seeding)

--- Depending on further Bugs ---
In my initial evaluation I uncovered (and filed) a set of bugs that I consider requirement to make it fully ready:
Reminder - tracking state here might be out of sync, I'll only change them to Done once complete and not care about interim status changes.

DONE - bug 1744662 - add chrony apparmor profile
DONE - bug 1744328 - make src:libnss libfreebl3 usable by other programs
COMMITTED - bug 1744664 - use Ubuntu time servers
COMMITTED - bug 1744072 - d/control: use to nss instead of tomcrypt
Some more cleanups in Chrony are optional but useful.

Other Related Bugs

 * https://bugs.launchpad.net/cloud-init/+bug/1731619 (cloud-init)

[Christian Ehrhardt  (paelzer)]

Current TODOs to get the MIR started:
1. complete the template
2. check dependencies and file MIRs as needed
3. Add bug tasks for all other affected packages

[Christian Ehrhardt  (paelzer)]

TOOD: add docs like serverguide to move to chrony

[Christian Ehrhardt  (paelzer)]

TODO: add tasks for charms by paul geer

[Christian Ehrhardt  (paelzer)]

Discussion about usability of libnss forked into bug 1744328

[Christian Ehrhardt  (paelzer)]

Hi Paul,
I subscribed you as I wanted to clarify something.
Back in [1], you mentioned it was important to you to get ntpdate (single shot cli) and ntpd (daemon) to work together nicely for the ntp charm.

Now if the ntp charm would be modified to use chrony from 18.04 onward, would that break it completely as chrony has no direct ntpdate counterpart that I'd know of?

[1]: https://bugs.launchpad.net/ubuntu/+source/ntp/+bug/1706818

[Christian Ehrhardt  (paelzer)]

bug 1718227 covers the need for hook integration from ifup to systemd, this is a soft prereq to consider it fully complete for 18.04

[Christian Ehrhardt  (paelzer)]

While some things are up in the air we should step this forward as good as we can, so @ubuntu-mir Team pleas ack and set it so the next status (if ok) so that the security Team can do an official check and ack as well.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Simon Déziel (sdeziel)]

RE: ntpdate equivalent, upstream recommends "chrony -q" with or without a config file.

https://chrony.tuxfamily.org/faq.html#_does_code_chronyd_code_have_an_ntpdate_mode

[Ken Dreyer (Red Hat) (kdreyer-redhat)]

Ceph tracker to switch from ntpd to chronyd: http://tracker.ceph.com/issues/22751

[Christian Ehrhardt  (paelzer)]

Thanks Simin and Ken, both great to know about!

[Christian Ehrhardt  (paelzer)]

We will also have to rewrite parts of the server guide
- Chrony usage in general
- Maybe how to convert a config from ntp to chrony

[Christian Ehrhardt  (paelzer)]

FYI: Debian accepted all our apparmor changes already plus a few cleanups - synced that new version into Bionic

[Seth Arnold (seth-arnold)]

I reviewed chrony version 3.2-1build1 as checked into bionic. This isn't a
full security audit but rather a quick gauge of maintainability.

- There are ten CVEs in our database; the fixes mostly aren't enumerated
  in our database, but many of the descriptions sound like they were
  handed out 'conservatively' -- errors in administration-level command
  channel or a malicious peer server operator in a position to interpose
  traffic from another peer server.

  I like the paranoia.

- chrony is a new, simpler, smaller, safer, ntp daemon. It's suitable for
  client and server use, and supports some hardware drivers, NIC
  timestamping, but perhaps not as many features as our old NTPD.

- Build-Depends: debhelper, bison, libedit-dev, libtomcrypt-dev,
  libcap-dev, pps-tools, libseccomp-dev, pkg-config, asciidoctor

- libtomcrypt dependency is being worked on; apparently nss is an option
  once we expose an "internal only" library.

- Does daemonize, nicely
- pre/post inst/rm scripts have autogenerated sections. Also:
- postinst script creates _chrony user and group, chowns /var/log/chrony
  and /var/lib/chrony
- postinst cleans up after previous version "key" file (authentication has
  been simplified in newer versions) in a complicated set of comparisons
- postrm removes /var/lib/chrony/, /etc/chrony/, _chrony user and group
- Initscript uses start-stop-daemon to start chrony
- systemd unit file is simple
- No dbus services
- No setuid files
- chronyc and chronyd executables in PATH
- No sudo fragments
- No udev rules
- test suite run at build; not comprehensive, but nice to have
- clean build logs

- sendmail is spawned to send mail via popen(). All variables are under
  control of configuration file. No error handling in case the admin sets
  the "mail to" variable to something silly long or dangerous, but this is
  very low risk.

- Memory management looked careful
- file io looked careful
- logging looked careful
- TZ environment variable used to gather information on leap seconds,
  looked careful
- Privileged operations looked careful
- I did not inspect cryptography
- Privileged portions of the code, privsep-style, looked careful; I did
  not inspect privsep for safety
- Extensive networking, looked careful
- No temporary file handling
- No WebKit
- No JavaScript
- No PolicyKit
- Clean cppcheck

Errors are checked religiously, coding style is unique and awkward but not
a real impediment to maintenance. Obviously ntp is an involved protocol
and probably further flaws will be found -- and we will rely upon
upstream's help for all but the simplest of issues. It looks
professionally programmed.

The only issue I found has no security relevance but may be slightly
surprising:

- reference() uses snprintf() to build a string to call sendmail; the
  username may not fit in the allocated space, and the code gets no
  warning about this.

  Any shell metacharacters in this setting would interfere with proper
  operation of the program.

I'd like to see this addressed for reliability reasons but it's not a
pressing issue.

Security team ACK for promoting chrony to main.

Thanks

[Christian Ehrhardt  (paelzer)]

FYI - It seems a bit dead here, but most work atm is going into dependent bug 1744328

[Christian Ehrhardt  (paelzer)]

ifup/down hooks are undirected (just chronyc offline/online).
There are networkmanager dispatchers which are smarter.

All of this is to allow to handle lossy/changing connections which is far more a laptop or similar (=>NetworkManager) than a server.

We might consider moving on without a solution.
Cyphermox mentioned he will look to provide a solution to hook into events again at some point (Part of the netplan transition) that is based on netlink events I think.

If all but the hooks are complete we can still move on IMHO.

[David Britton (dpb)]

re: ifup/down hooks --

In the end, it's the same situation with either ntpd or chrony. let's just add it to the tasks to do after promotion in general for 18.04. I wouldn't conflate the MIR with this point at all.

[Paul Gear (paulgear)]

On 22/01/18 17:27, ChristianEhrhardt wrote:
> Hi Paul,
> I subscribed you as I wanted to clarify something.
> Back in [1], you mentioned it was important to you to get ntpdate (single shot cli) and ntpd (daemon) to work together nicely for the ntp charm.
>
> Now if the ntp charm would be modified to use chrony from 18.04 onward,
> would that break it completely as chrony has no direct ntpdate
> counterpart that I'd know of?
>
> [1]: https://bugs.launchpad.net/ubuntu/+source/ntp/+bug/1706818
>
Hi Christian,

My current plan is to write a new reactive charm for chrony rather than
trying to retrofit the ntp charm to support chrony.  I would expect that
the functionality which relies on ntpdate will drop out.  I don't have a
timeframe for this, however.

Regards,
Paul

[Christian Ehrhardt  (paelzer)]

Builds complete against new nss, also all other bugs we wanted are grouped.
New chrony uploaded to bionic - once passed we can do the seeding.

[Launchpad Janitor (janitor)]

This bug was fixed in the package chrony - 3.2-2ubuntu2

---------------
chrony (3.2-2ubuntu2) bionic; urgency=medium

  * d/control: use to nss instead of tomcrypt (in main) (LP: #1744072)
  * d/chrony.conf: use ubuntu ntp pool and server (LP: #1744664)
  * d/chrony.default, d/chrony.service: support /etc/default/chrony
    DAEMON_OPTS in systemd environment (LP: #1746081)
  * d/chrony.service: properly start after networking (LP: #1746458)
  * d/usr.sbin.chronyd: allow to create /run/chrony on demand (LP: #1746444)

 -- Christian Ehrhardt <email address hidden> Fri, 19 Jan 2018 09:45:38 +0100

[Christian Ehrhardt  (paelzer)]

Ok, this now has all prereqs resolved.
It is ready for the actual MIR + seed change.
Setting the state back to new (we reused the bugno, but it is for the MIR actually)

So todo's now are:
@MIR Team ack and set fix committed
@Cpaelzer - Propose a seed change.

[Christian Ehrhardt  (paelzer)]

MP's for the seed changes are up:
platform: https://code.launchpad.net/~paelzer/ubuntu-seeds/18.04-ntp-to-chrony-platform/+merge/337257
ubuntu: https://code.launchpad.net/~paelzer/ubuntu-seeds/18.04-ntp-to-chrony-ubuntu/+merge/337256

Waiting now for:
- the general MIR team ack and setting to fix committed on this bug.
- a review ack on the two MPs above

[Paul Gear (paulgear)]

@paelzer: As I looked at chrony's config and the options which would be needed in a new chrony charm, I found that most of them were common with ntp, so I'm going to start work on a branch of the ntp charm which supports switching between ntp and chrony. I'll link the branch here when I have some progress to report.

[Nish Aravamudan (nacc)]

I reviewed chrony; it's fine to MIR (and has security team approval). MIR ACKed.

[Christian Ehrhardt  (paelzer)]

Thanks Nish that you took a look.
I merged the two open and already approved seed changing branches.

Will also do the ceph upload soon.
All those will make chrony show up in component mismatches to then be added.
(and hopefully ntp will show up soon after for demotion to universe)

[Steve Langasek (vorlon)]

Override component to main
chrony 3.2-2ubuntu3 in bionic amd64: universe/admin/extra/100% -> main
chrony 3.2-2ubuntu3 in bionic arm64: universe/admin/extra/100% -> main
chrony 3.2-2ubuntu3 in bionic armhf: universe/admin/extra/100% -> main
chrony 3.2-2ubuntu3 in bionic i386: universe/admin/extra/100% -> main
chrony 3.2-2ubuntu3 in bionic ppc64el: universe/admin/extra/100% -> main
chrony 3.2-2ubuntu3 in bionic s390x: universe/admin/extra/100% -> main
6 publications overridden.

[Christian Ehrhardt  (paelzer)]

The ceph change to modify the dependencies (reorder recommends) are in proposed and hopefully soon to migrate.

Of the rather time critical bits (to demote ntp in time before FF) what is left is the change in MAAS. Since the sprint there was no reply by MAAS yet, so pinging on IRC in addition to this bug update.

Those two depend on ntp in d/control:
- maas-region-api
- maas-rack-controller

You could likely even keep most of the tests as-is, but the custom ntp config (src/provisioningserver/ntp/config.py?) would need to be changed I assume.

[Launchpad Janitor (janitor)]

This bug was fixed in the package ceph - 12.2.2-0ubuntu2

---------------
ceph (12.2.2-0ubuntu2) bionic; urgency=medium

  * d/control: Re-order Recommends to prefer chrony over time-daemon
    (chrony/openntp) and ntp for Ubuntu (LP: #1744072).

 -- Christian Ehrhardt <email address hidden> Fri, 16 Feb 2018 09:19:21 +0100

[Christian Ehrhardt  (paelzer)]

FYI Proposed documentation update: https://code.launchpad.net/~paelzer/serverguide/serverguide-chrony-18.04/+merge/338892

[Paul Gear (paulgear)]

I've added initial support for chrony to the ntp charm: https://code.launchpad.net/~paulgear/ntp-charm/+git/ntp-charm/+merge/340780

Very lightly tested at present - nagios check known to be non-working, other features should work. It's available as cs:~paulgear/ntp if anyone would like to test: https://jujucharms.com/u/paulgear/ntp/

[Matthias Klose (doko)]

clout-init and maas are already in main. why are these still open?

[Christian Ehrhardt  (paelzer)]

Hi Mathias - that was for both to support configuring chrony for ntp services.
Both are done AFAIK, setting fix released.