[SRU] ubuntu-advantage-tools (27.2.2 -> 27.3) Xenial, Bionic, Focal, Hirsute

[Grant Orndorff]

[Impact]
This release sports both bug-fixes and new features and we would like to
make sure all of our supported customers have access to these
improvements. The notable ones are:

* more robust error handling when determining the cloud we're on LP: #1940131 LP: #1938207 LP: #1944676
* disallows fips on focal aws/azure LP: #1939449 LP: #1939932
* adds/changes to ua-related recurring jobs:
  - change in frequency to existing job: updates the apt and motd esm update messaging: every 6 hours
  - new job: updates the contract details and status: every 12 hours
  - new job: ONLY ON GCP (implemented as separate timer that is only activated on GCP LTS when not attached): checks for license changes and auto-attaches if a pro license was added: every 5 minutes
* adds support for ros/ros-updates entitlements with --beta flag

With this change, the ua-message timer is renamed to ua-timer, as it has a more generic functionality: it triggers sub jobs which need to be executed periodically. One of those is exactly the job which updates the messaging - which has its interval reduced to 6h, as well as the timer itself. There is also a job to update the client status every 12h, and a third one to collect metrics, which is disabled for this release.

See the changelog entry below for a full list of changes and bugs.

[Test Case]
The following development and SRU process was followed:
https://wiki.ubuntu.com/UbuntuAdvantageToolsUpdates

The ubuntu-advantage-tools team will be in charge of attaching the artifacts and
console output of the appropriate run to the bug. ubuntu-advantage-tools team
members will not mark ‘verification-done’ until this has happened.

<TODO After released to -proposed: attach integration test artifacts>

In addition to the automated integration testing, here are manual test instructions to verify that the packaging changes are functional. All of these tests use the ua-client/staging PPA, which has the 27.3 release build in it.

Manual Test 1:
Here we verify that the cloud-id command changes in postinst work correctly such that the installation succeeds even if the cloud-id command fails.
```
lxc launch ubuntu-daily:impish dev-i
lxc exec dev-i -- /bin/sh -c "printf 'exit 1' > /usr/bin/cloud-id"
lxc exec dev-i -- chmod +x /usr/bin/cloud-id
lxc exec dev-i -- cloud-id
# CHECK: successfully faking cloud-init error. command should've exited 1
lxc exec dev-i -- add-apt-repository -yu ppa:ua-client/staging
lxc exec dev-i -- apt install -y ubuntu-advantage-tools
# CHECK: installation should've succeeded despite failing cloud-init
lxc delete dev-i --force
```

Manual Test 2:
Here we verify that the new license check timer only runs on LTS instances that are identified as running on GCP.
```
# should not be enabled when not gcp, on LTS (focal)
lxc launch ubuntu-daily:focal dev-f
lxc exec dev-f -- add-apt-repository -yu ppa:ua-client/staging
lxc exec dev-f -- apt install -y ubuntu-advantage-tools
lxc exec dev-f -- systemctl list-timers --all
lxc exec dev-f -- systemctl status ua-license-check.timer
# CHECK: should not be running
lxc exec dev-f -- reboot
lxc exec dev-f -- systemctl list-timers --all
lxc exec dev-f -- systemctl status ua-license-check.timer
# CHECK: still should not be running
lxc delete dev-f --force

# fake gcp by overwriting cloud-id, still should not be enabled because not on LTS (impish)
lxc launch ubuntu-daily:impish dev-i
lxc exec dev-i -- /bin/sh -c "printf 'echo gce' > /usr/bin/cloud-id"
lxc exec dev-i -- chmod +x /usr/bin/cloud-id
lxc exec dev-i -- cloud-id
# CHECK: successfully faking gcp. output should be "gce" (with an "e")
lxc exec dev-i -- add-apt-repository -yu ppa:ua-client/staging
lxc exec dev-i -- apt install -y ubuntu-advantage-tools
lxc exec dev-i -- systemctl list-timers --all
lxc exec dev-i -- systemctl status ua-license-check.timer
# CHECK: should not be running
lxc delete dev-i --force

# fake gcp by overwriting cloud-id, on LTS (focal), should be enabled
lxc launch ubuntu-daily:focal dev-f
lxc exec dev-f -- /bin/sh -c "printf 'echo gce' > /usr/bin/cloud-id"
lxc exec dev-f -- chmod +x /usr/bin/cloud-id
lxc exec dev-f -- cloud-id
# CHECK: successfully faking gcp. output should be "gce" (with an "e")
lxc exec dev-f -- add-apt-repository -yu ppa:ua-client/staging
lxc exec dev-f -- apt install -y ubuntu-advantage-tools
lxc exec dev-f -- systemctl list-timers --all
lxc exec dev-f -- systemctl status ua-license-check.timer
# CHECK: should be enabled
lxc delete dev-f --force
```

Manual Test 3:
Here we verify that the old ua-messaging.{timer,service} artifacts are properly removed on upgrade.
```
lxc launch ubuntu-daily:impish dev-i
lxc exec dev-i -- /bin/sh -c "ls -1 /etc/systemd/system/timers.target.wants /usr/lib/systemd/system /usr/lib/systemd/system /var/lib/systemd/deb-systemd-helper-enabled/timers.target.wants /var/lib/systemd/deb-systemd-helper-enabled/ | grep ua-messaging"
# CHECK: verify several ua-messaging artifacts
lxc exec dev-i -- add-apt-repository -yu ppa:ua-client/staging
lxc exec dev-i -- apt install -y ubuntu-advantage-tools
lxc exec dev-i -- /bin/sh -c "ls -1 /etc/systemd/system/timers.target.wants /usr/lib/systemd/system /usr/lib/systemd/system /var/lib/systemd/deb-systemd-helper-enabled/timers.target.wants /var/lib/systemd/deb-systemd-helper-enabled/ | grep ua-messaging"
# CHECK: verify ua-messaging artifacts are not left behind
lxc delete dev-i --force
```

Manual Test 4:
Here we verify that if the user had disabled the old ua-messaging timer, then we will carry that preference forward to the new ua-timer timer.
```
lxc launch ubuntu-daily:impish dev-i
lxc exec dev-i -- systemctl list-timers --all
lxc exec dev-i -- systemctl status ua-messaging.timer
# CHECK: verify ua-messaging.timer is enabled
lxc exec dev-i -- systemctl stop ua-messaging.timer
lxc exec dev-i -- systemctl disable ua-messaging.timer
lxc exec dev-i -- systemctl list-timers --all
lxc exec dev-i -- systemctl status ua-messaging.timer
# CHECK: verify ua-messaging.timer is disabled
lxc exec dev-i -- add-apt-repository -yu ppa:ua-client/staging
lxc exec dev-i -- apt install -y ubuntu-advantage-tools
lxc exec dev-i -- systemctl list-timers --all
lxc exec dev-i -- systemctl status ua-timer.timer
# CHECK: verify ua-timer.timer is disabled
lxc delete dev-i --force
```

Manual Test 5:
Here we verify that the new log files are appropriately created on install, rotated by logrotate, and deleted on purge.
```
lxc launch ubuntu-daily:impish dev-i
lxc exec dev-i -- add-apt-repository -yu ppa:ua-client/staging
lxc exec dev-i -- apt install -y ubuntu-advantage-tools
lxc exec dev-i -- /bin/sh -c 'ls -l /var/log/ubuntu-advantage*'
# CHECK: verify that three ua log files were created, are owned by root, and have 600 permissions
lxc exec dev-i -- /bin/sh -c "printf testcontent > /var/log/ubuntu-advantage.log"
lxc exec dev-i -- /bin/sh -c "printf testcontent > /var/log/ubuntu-advantage-timer.log"
lxc exec dev-i -- /bin/sh -c "printf testcontent > /var/log/ubuntu-advantage-license-check.log"
lxc exec dev-i -- logrotate --force /etc/logrotate.d/ubuntu-advantage-tools
lxc exec dev-i -- /bin/sh -c 'ls -l /var/log/ubuntu-advantage*'
# CHECK: verify all 3 logs were rotated
lxc exec dev-i -- /bin/sh -c "printf testcontent > /var/log/ubuntu-advantage.log"
lxc exec dev-i -- /bin/sh -c "printf testcontent > /var/log/ubuntu-advantage-timer.log"
lxc exec dev-i -- /bin/sh -c "printf testcontent > /var/log/ubuntu-advantage-license-check.log"
lxc exec dev-i -- /bin/sh -c 'ls /var/log/ubuntu-advantage*'
# CHECK: verify all ua log files exist including rotated versions
lxc exec dev-i -- apt purge -y ubuntu-advantage-tools
lxc exec dev-i -- /bin/sh -c "ls /var/log/"
# CHECK: verify that all ua log files are removed
lxc delete dev-i --force
```

[Regression Potential]
In order to mitigate the regression potential, the results of the
aforementioned integration tests are attached to this bug.

We moved the trigger of the apt and motd messaging updates from a dedicated systemd timer to a shared timer that conditionally calls the messaging updates in our python code. This adds complexity. If we made a mistake, then either the job won't get called frequently enough or will get called too frequently. If the former, then some esm updates related messaging will be out of date in apt and motd. If the latter, then cpu cycles will be wasted in needlessly updating messages.

We touched postinst to handle cloud-id failures more robustly. Touching postinst is always scary because it is the most likely way for us to break upgrades. In theory this change made upgrades less likely to fail, but if we made a mistake, it could cause new unexpected failures.

We added more recurring jobs in the service of new features. This increases complexity and potential for mistakes. In particular, we have strived to avoid excessive logging from these jobs. If we made a mistake in our logging, we could inadvertently fill up disks with useless logs. Additional recurring jobs will also use more cpu over time than previous versions. This is at least partially addressed below.

We instrumented a high frequency timer to only run on GCP, but if we made a mistake, this could be accidentally activated on non-GCP machines, which would be a waste. (See below for additional high frequency timer discussion).

We check if the ua-messaging timer was disabled prior to this update, and if so we also disable the new ua-timer timer in systemd. Failing to do so would keep enabled a service that the user had explicitly disabled in the past, resulting in a unwanted behavior. Our migration of this user configuration only covers the case where the user ran `systemctl disable` to disable the old timer. If they disabled the timer in a different way, then their configuration will not be carried forward. Furthermore, this is a somewhat complicated postinst addition, and carries all the normal risks of editing postinst.

[Discussion]
Our timer on GCP runs every 5 minutes. This is necessary to support timely upgrades of gcp instances from standard ubuntu to ubuntu pro. We need to poll the metadata endpoint frequently to catch the license change in a timely manner. We exit as early as possible if there is nothing to be done for any given timer trigger. From our testing, this has minimal overall system performance impact. <TODO @chad.smith insert details and link to spreadsheet>

[Changelog]

  * d/tools.postinst:
    - consider cloud to be "none" on any cloud-id error
    - purge old ua-messaging.timer/service files
  * systemd:
    - remove ua-messaging.timer/service
    - add new ua-timer.timer that runs every 2 hours
    - add new ua-license_check.timer that runs every 5 minutes only if
      activated by ua-license-check.path
  * New upstream release 27.3
    - ros:
      + add beta support to enable ros and ros-updates
      + add support for "required services" so that esm-infra and esm-apps
        get auto-enabled when enabling ros or ros-updates
      + add support for "dependent services" so that user gets prompted to
        disable ros/ros-updates if they disable esm-infra/esm-apps
    - fips:
      + allow fips on GCP bionic now that optimized kernel is ready
      + disallow enabling fips on focal on clouds until cloud-optimized focal
        fips-certified kernel is ready (LP: #1939449, LP: #1939932)
      + print warning about generic fips kernel if cloud-id fails
    - cloud:
      + rely only on cloud-id to determine cloud type (LP: #1940131)
      + catch errors when determining cloud type (LP: #1938207) (GH: #1541)
    - azure:
      + bump IMDS API version to support Azure published images
    - cli:
      + collect-logs command that creates a tar file with debug-relevant logs
        and status info (GH: #463)
      + clean locks on exceptions more thoroughly to avoid false "Operation in
        progress" status messages
      + retain past service state after detach
      + shows better error message when a port value in a proxy is invalid
    - non-unicode locale support:
      + remove unicode-only characters from help file
      + don't print unicode-only characters in ua fix if non-utf8 locale
        (GH: #1463)
    - ua-timer.timer:
      + introduce a single systemd timer to handle ua recurring jobs
      + timer runs every 2 hours to support most frequent timer job
      + recurring job intervals are configurable in uaclient.conf
      + individual jobs are disabled if their interval is set to 0
      - status job:
        + update ua status every 12 hours
      - messaging job:
        + update APT/MOTD ESM messaging every 6 hours
      - metering job:
        + disabled until infrastructure is ready
        + for attached machines only, periodically update contract server with
          status information for proper contract metering
    - ua-license-check.timer:
      + only runs on LTS GCP instances that are not attached
      + runs every 5 minutes to check if gcp instance has license required to
        auto-attach
    - logs:
      + fixes duplicate logging (GH: #553)
    - tests and support:
      + remove groovy integration tests
      + various improvements to integration tests

[Steve Langasek (vorlon)]

One of the packaging changes in the SRU which I don't see an explicit bug for (so I'm attaching the question here) is:

+++ ubuntu-advantage-tools-27.3~21.10.1/debian/ubuntu-advantage-tools.postinst 2021-09-21 13:02:06.000000000 +0000
[...]
+configure_log_file() {
+ log_path=$1
+ if [ ! -f $log_path ]; then
+ touch $log_path
+ fi
+ chmod 0600 $log_path
+ chown root:root $log_path
+}
[...]
- if [ ! -f /var/log/ubuntu-advantage.log ]; then
- touch /var/log/ubuntu-advantage.log
- fi
- chmod 0600 /var/log/ubuntu-advantage.log
- chown root:root /var/log/ubuntu-advantage.log
+ configure_log_file /var/log/ubuntu-advantage.log
+ configure_log_file /var/log/ubuntu-advantage-timer.log
+ configure_log_file /var/log/ubuntu-advantage-license-check.log
+
[...]

It is unusual for maintainer scripts to pre-create log files. If these files are removed, do the tools recreate them with the correct permissions? (If so: is this code needed at all?)

[Steve Langasek (vorlon)]

+enable_periodic_license_check() {
+ cloud_id=$(cloud-id 2>/dev/null) || cloud_id=""
+ if echo "$cloud_id" | grep -q "^gce"; then
+ if check_is_lts "${UBUNTU_CODENAME}"; then
+ if [ ! -f $MACHINE_TOKEN_FILE ]; then
+ touch $LICENSE_CHECK_MARKER_FILE
+ fi
+ fi
+ fi
+}

What is the expected behavior here if the user upgrades to this version of the package on an LTS, and then upgrades to a non-LTS release?

What is the expected behavior if the user is running on a non-LTS release, and then upgrades to a non-LTS release?

(Current behavior with this implementation: on upgrade to an LTS release, the timer will be enabled, because a new version of the package is configured and this code runs again. On upgrade to a non-LTS release, the timer will not be disabled.)

[Grant Orndorff (orndorffgrant)]

Thank you Steve for taking a look.

You are right on all counts.

It appears the code to pre-create log files is unnecessary. If you consider this a blocker then we can remove it in this release. Otherwise we will remove it in 27.4

We purposefully kept the purpose of the enable_periodic_license_check function to _enabling_ when run on GCP LTS non-attached instances. You are right that the timer will not be disabled during postinst when upgrading to a non-LTS; however, the timer will notice that it is non-LTS and disable itself next time it runs.

The code that does this is here:
https://github.com/canonical/ubuntu-advantage-client/blob/release-27/uaclient/jobs/license_check.py#L36

And our integration test for the timer disabling itself on non-LTS is here:
https://github.com/canonical/ubuntu-advantage-client/blob/release-27/features/license_check.feature#L47

[Steve Langasek (vorlon)]

Thanks, your responses to the previous two comments are satisfactory and I don't think changes are needed for this SRU for those points.

+ # We should only perform this check on UA version that have the
+ # ua-messaging.timer: 27.0 until 27.2. This will also guarantee
+ # that on 27.3 and forward, we will not run this logic.
+ if dpkg --compare-versions "$PREVIOUS_PKG_VER" ge-nl "27.0~"; then
+ if dpkg --compare-versions "$PREVIOUS_PKG_VER" lt-nl "27.3~"; then
+ if [ -x "/usr/bin/deb-systemd-helper" ]; then
+ if [ -d /run/systemd/system ]; then
+ if ! deb-systemd-helper --quiet was-enabled $OLD_MESSAGING_TIMER; then

This could be written without all the nesting as:
    if dpkg --compare-versions "$PREVIOUS_PKG_VER" lt "27.0~" \
       || dpkg --compare-versions "$PREVIOUS_PKG_VER" ge "27.3~"; then
        return
    fi
    if [ -x "/usr/bin/deb-systemd-helper" ] && [ -d /run/systemd/system ] \
       && ! deb-systemd-helper --quiet was-enabled $OLD_MESSAGING_TIMER; then

Under what circumstances would deb-systemd-helper not be present? init-system-helpers is part of the minimal task from trusty onwards; it is Essential: yes from bionic onwards; and ubuntu-advantage-tools itself could reasonably declare a dependency on it.

What is the expected behavior if we don't have a running systemd (such as in a chroot, and /run/systemd/system is not present)? Why are you short-circuiting in this case, rather than calling deb-systemd-helper and letting it handle these cases?

Also, for some reason you are calling deb-systemd-helper *enable*, not disable?

I think this should be reduced to:

    if ! deb-systemd-helper --quiet was-enabled $OLD_MESSAGING_TIMER; then
        deb-systemd-helper disable $UA_TIMER_NAME > /dev/null 2>&1 || true
        systemctl stop $UA_TIMER_NAME > /dev/null 2>&1 || true
    fi

the other checks are either superfluous, or actively harmful.

[Grant Orndorff (orndorffgrant)]

Thank you Steve, you raise several good points.

1. The excessive nesting.
I agree that it is annoying to read several layers of nested ifs. As an individual issue, we don't think this is a blocker; however if we decide to rewrite this function for this SRU for the below reasons, we can use the early return like you suggest. If not, then we will simplify this for next release.

2. The check for deb-systemd-helper existing.
There's a longer story as to why we thought this check was a good idea at the time, but the end result is that we were mistaken and it is superfluous. We don't think this check hurts anything but it doesn't help either. We can remove it in our next release unless you think it blocks this one.

3. The check for systemd existing.
This is present because we were copying a pattern that deb-systemd-helper uses in its autosnippets. Whenever d-s-h appends a snippet that uses systemctl, it makes this check first. For example, a snippet that is added to our postinst by d-s-h:

# Automatically added by dh_systemd_start
if [ -d /run/systemd/system ]; then
        systemctl --system daemon-reload >/dev/null || true
        deb-systemd-invoke start ua-timer.timer >/dev/null || true
fi
# End automatically added section

However, it turns out our use of systemctl in this case is also unnecessary. We don't need to run systemctl stop in this case because ua-timer.timer will never have been started when this chunk of code runs. Again, this shouldn't hurt anything. We can remove it in our next release unless you think it blocks this one.

4. Running deb-systemd-helper enable is confusing.
The use of enable was intentional. We attempted to explain in the comment above that line. The reason is that we want to take advantage of d-s-h's autosnippet for enabling/updating config for ua-timer.timer which appends the following to the bottom of postinst:

# Automatically added by dh_systemd_enable
# This will only remove masks created by d-s-h on package removal.
deb-systemd-helper unmask ua-timer.timer >/dev/null || true

# was-enabled defaults to true, so new installations run enable.
if deb-systemd-helper --quiet was-enabled ua-timer.timer; then
        # Enables the unit on first installation, creates new
        # symlinks on upgrades if the unit file has changed.
        deb-systemd-helper enable ua-timer.timer >/dev/null || true
else
        # Update the statefile to add new symlinks (if any), which need to be
        # cleaned up on purge. Also remove old symlinks.
        deb-systemd-helper update-state ua-timer.timer >/dev/null || true
fi
# End automatically added section

The chunk of code that we are talking about happens before this d-s-h autosnippet, and the intent is to prevent this d-s-h snippet from running "enable".
By running "enable" and then "disable" we update the d-s-h state such that "was-enabled" will fail and so that snippet will not run "enable".

We acknowledge that this solution is confusing, to put it charitably. We've evaluated a number of alternatives:

4.1. Straightforward simplification
deb-systemd-helper enable $UA_TIMER_NAME > /dev/null 2>&1 || true
deb-systemd-helper disable $UA_TIMER_NAME > /dev/null 2>&1 || t...

[Chad Smith (chad.smith)]

Thanks for the review here Steve.

Grant has already started working on your prior review comments 1 & 2 to reduce some of the complexity and excessive nesting for next release to aid in maintainability of this section.

For the comment about "the other checks are either superfluous, or actively harmful."

I'm not certain how this section will be harmful here, if we are in chroots, we should be skipping this section in general because our initial `if [ -d /run/systemd/system ]` check. If this code happens to run in a chroot w/ run/systemd/system present we also avoid non-zero exit in the failure case if systemctl or deb-systremd-helper error.

I presumed this was also why debhelper snippets is automatically adding the similar check prior to invoking `systemctl ...`.

Are you suggesting we don't use systemctl or deb-systemd-helper tools directly and instead just attempt to unlink systemd timer/service link files to ensure we continue to disable ua-timer.timer if ua-messaging.timer is not represented as enabled?

[Steve Langasek (vorlon)]

On Tue, Oct 19, 2021 at 07:48:03PM -0000, Chad Smith wrote:
> For the comment about "the other checks are either superfluous, or
> actively harmful."

> I'm not certain how this section will be harmful here, if we are in
> chroots, we should be skipping this section in general because our
> initial `if [ -d /run/systemd/system ]` check. If this code happens to
> run in a chroot w/ run/systemd/system present we also avoid non-zero
> exit in the failure case if systemctl or deb-systremd-helper error.

The point is that, if the system is *currently* mounted as a chroot, and
this postinst runs, it will not to disable the new service in accordance
with the admin's choice regarding the old service. If furthermore in the
course of normal operation this is a booted system and not a chroot, this
means that on next boot (and subsequently), the new service will be started
when the intent was that it not be.

This is an unlikely scenario in the grand scheme of things, but is not
impossible. If someone has disabled this service; and the instance ends up
in a broken state, so the user mounts the root drive from another instance,
chroots into it, and runs package updates as part of fixing it; then you
could have exactly this situation.

So not applying the "disable" policy when running the postinst in a chroot
is a bug. We should avoid introducing bugs in SRUs when possible, however
corner case.

> I presumed this was also why debhelper snippets is automatically adding
> the similar check prior to invoking `systemctl ...`.

debhelper doesn't wrap calls to 'deb-systemd-helper enable' with checks
for whether the system currently has a running systemd. The
'deb-systemd-helper enable' and 'deb-systemd-helper disable' calls are
explicitly intended to update the state of the unit even if systemd is not
currently running.

> Are you suggesting we don't use systemctl or deb-systemd-helper tools
> directly and instead just attempt to unlink systemd timer/service link
> files to ensure we continue to disable ua-timer.timer if ua-
> messaging.timer is not represented as enabled?

That would be an option, but seems unnecessary vs just calling
'deb-systemd-helper disable $UA_TIMER_NAME > /dev/null 2>&1 || true' here.

On Fri, Oct 15, 2021 at 05:59:56PM -0000, Grant Orndorff wrote:

> 4. Running deb-systemd-helper enable is confusing.
> The use of enable was intentional. We attempted to explain in the comment
> above that line. The reason is that we want to take advantage of d-s-h's
> autosnippet for enabling/updating config for ua-timer.timer which appends
> the following to the bottom of postinst:

> # Automatically added by dh_systemd_enable
> # This will only remove masks created by d-s-h on package removal.
> deb-systemd-helper unmask ua-timer.timer >/dev/null || true

> # was-enabled defaults to true, so new installations run enable.
> if deb-systemd-helper --quiet was-enabled ua-timer.timer; then
> # Enables the unit on first installation, creates new
> # symlinks on upgrades if the unit file has changed.
> deb-systemd-helper enable ua-timer.timer >/dev/null || true
> else
> # Update the statefile to add new symlinks (if any),...

[Chad Smith (chad.smith)]

Thank you for the additional context on the potential failure paths here. Agreed we shouldn't knowingly introduce a bug no matter how corner case the condition may be.

>> In what case do you see incorrect behavior by calling (the more obviously
named) 'deb-systemd-helper disable'?

I actually don't think there was a specific case that was Just an attempt to be explicit about the stop and disable calls from systemctl (stop was not necessary anyway).

In fact, Grant has already proposed changes to drop the systemctl disable call anyway and adopt the `deb-systemd-helper disable` instead
https://github.com/canonical/ubuntu-advantage-client/pull/1842/files#diff-7ca7735b78964990b8c31fc79692c988e6904a1058d4f752765cd59b54a882aeR338-R352

If this minor diff is reasonable to reduce the conditional nesting and avoid a corner case issue with invoking systemctl directly from within a chrooted disk, then I'll talk w/ Grant and others tomorrow to see if we can get an upload queued with that change.

[Grant Orndorff (orndorffgrant)]

Hi Steve,

> Ok, now I see - but that means deb-systemd-helper's auxiliary state will be
> permanently out of sync with the systemd state, I think?

We are purposefully manipulating d-s-h's state here, but it never becomes out of sync with the systemd state.

> Also, this
> dh_systemd_enable snippet calls deb-systemd-helper enable if 'was-enabled'
> was true, but I don't see that you want to call enable again, as this is
> either a no-op, or it will enable the service which we explicitly are aiming
> to disable.

Exactly! :) We need `was-enabled` to fail or else the timer will get enabled. This is why we are calling "enable" then "disable" up above. I will explain via example below.

> In what case do you see incorrect behavior by calling (the more obviously
> named) 'deb-systemd-helper disable'?

If we only call "disable", the total relevant script will look like this

# our call
deb-systemd-helper disable ua-timer.timer >/dev/null || true
# d-s-h snippet
deb-systemd-helper unmask ua-timer.timer >/dev/null || true
if deb-systemd-helper --quiet was-enabled ua-timer.timer; then
        deb-systemd-helper enable ua-timer.timer >/dev/null || true
else
        deb-systemd-helper update-state ua-timer.timer >/dev/null || true
fi

In this case, "was-enabled" will _succeed_, and "deb-systemd-helper enable" in the d-s-h snippet will run. This is due to "was-enabled" having an explicit special case for the first install: If "deb-systemd-helper enable" has never been run before, then "was-enabled" will succeed. They note this in a comment in their autosnippet.
So as a result, if we _only_ run "deb-systemd-helper disable", then the timer will end up getting enabled.

Now, suppose we adopt the simplified version of what we're doing here (https://github.com/canonical/ubuntu-advantage-client/pull/1842/commits/f8b1a7222eeeea21f7b416a2bdd3d08e688cbef7).

Then the total relevant script will be this:

# our calls
deb-systemd-helper enable ua-timer.timer >/dev/null || true
deb-systemd-helper disable ua-timer.timer >/dev/null || true
# d-s-h snippet
deb-systemd-helper unmask ua-timer.timer >/dev/null || true
if deb-systemd-helper --quiet was-enabled ua-timer.timer; then
        deb-systemd-helper enable ua-timer.timer >/dev/null || true
else
        deb-systemd-helper update-state ua-timer.timer >/dev/null || true
fi

In this case, "was-enabled" will _fail_. So the d-s-h snippet will not run the "deb-systemd-helper enable" line. "was-enabled" fails here because d-s-h was used to enable the timer at least once in the past, but the timer is currently disabled.
So as a result, if we run "enable" and then "disable", then the timer will end up disabled.

We have tested all of the above. The timer is correctly disabled if we call "enable" then "disable". The timer is not correctly disabled if we only call "disable".

If you agree, then we will apply this commit https://github.com/canonical/ubuntu-advantage-client/pull/1842/commits/f8b1a7222eeeea21f7b416a2bdd3d08e688cbef7 to this release to at least make the code here simpler.

Edit: And of course, this change would fix the systemd concerns by no longer calling systemctl directly.

[Grant Orndorff (orndorffgrant)]

Steve and Chad: I just finished a manual test of:

1. Disabling ua-messaging.timer
2. In a chroot, upgrade ubuntu-advantage-tools
3. Check if ua-timer.timer was disabled

With the current upload this is unsuccessful, as Steve has predicted.

With our proposed change (https://github.com/canonical/ubuntu-advantage-client/commit/2d7da9e5ab3d1ed482c6c8f1f8be3bba7e403b40), then it is successful.

So I think we should go ahead and re-upload with this patch.

[Chad Smith (chad.smith)]

+1 on this Grant. Let's queue another upload today if we can

[Steve Langasek (vorlon)]

Hello Grant, or anyone else affected,

Accepted ubuntu-advantage-tools into impish-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubuntu-advantage-tools/27.3~21.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-impish to verification-done-impish. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-impish. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Steve Langasek (vorlon)]

Hello Grant, or anyone else affected,

Accepted ubuntu-advantage-tools into hirsute-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubuntu-advantage-tools/27.3~21.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-hirsute to verification-done-hirsute. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-hirsute. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Steve Langasek (vorlon)]

Hello Grant, or anyone else affected,

Accepted ubuntu-advantage-tools into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubuntu-advantage-tools/27.3~20.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Steve Langasek (vorlon)]

Hello Grant, or anyone else affected,

Accepted ubuntu-advantage-tools into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubuntu-advantage-tools/27.3~18.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Steve Langasek (vorlon)]

Hello Grant, or anyone else affected,

Accepted ubuntu-advantage-tools into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubuntu-advantage-tools/27.3~16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Lucas Albuquerque Medeiros de Moura (lamoura)]

I have run all of the manual tests with the proposed package and they are all working as expected

[Grant Orndorff (orndorffgrant)]

We have run the full ubuntu-advantage-tools integration test suite against the version in -proposed. The results are attached. All tests passed.

You can verify the correct version was used by checking the output of the first test in each file, which prints the version number.

Next we will run all the manual tests described on this bug and upload logs. Then we will mark verification done.

[Grant Orndorff (orndorffgrant)]

Attached are logs for all the manual tests for all series. All are working as expected.

Marking verification done now.

[Launchpad Janitor (janitor)]

This bug was fixed in the package ubuntu-advantage-tools - 27.3~21.10.1

---------------
ubuntu-advantage-tools (27.3~21.10.1) impish; urgency=medium

  * d/tools.postinst:
    - consider cloud to be "none" on any cloud-id error
    - purge old ua-messaging.timer/service files
    - keep ua-timer.timer disabled if ua-messaging.timer was disabled by
      the user
    - properly configure both ubuntu-advantage-timer and
      ubuntu-advantage-licence-check logs
  * d/tools.postrm:
    - remove ubuntu-advantage-timer and ubuntu-advantage-license-check logs
      during purge
  * systemd:
    - remove ua-messaging.timer/service
    - add new ua-timer.timer that runs every 6 hours
    - add new ua-license_check.timer that runs every 5 minutes only if
      activated by ua-license-check.path
  * New upstream release 27.3 (LP: #1942929)
    - ros:
      + add beta support to enable ros and ros-updates
      + add support for "required services" so that esm-infra and esm-apps
        get auto-enabled when enabling ros or ros-updates
      + add support for "dependent services" so that user gets prompted to
        disable ros/ros-updates if they disable esm-infra/esm-apps
    - fips:
      + allow fips on GCP bionic now that optimized kernel is ready
      + disallow enabling fips on focal on clouds until cloud-optimized focal
        fips-certified kernel is ready (LP: #1939449, LP: #1939932)
      + print warning about generic fips kernel if cloud-id fails
    - cloud:
      + rely only on cloud-id to determine cloud type (LP: #1940131)
      + catch errors when determining cloud type
        (LP: #1938207, LP: #1944676) (GH: #1541)
    - azure:
      + bump IMDS API version to support Azure published images
    - cli:
      + collect-logs command that creates a tar file with debug-relevant logs
        and status info (GH: #463)
      + clean locks on exceptions more thoroughly to avoid false "Operation in
        progress" status messages
      + retain past service state after detach
      + shows better error message when a port value in a proxy is invalid
    - non-unicode locale support:
      + remove unicode-only characters from help file
      + don't print unicode-only characters in ua fix if non-utf8 locale
        (GH: #1463)
    - logrotate:
      + add logrotate functionality for ubuntu-advantage-timer.log.
      + Fix root:root logrotate permissions.
    - ua-timer.timer:
      + introduce a single systemd timer to handle ua recurring jobs
      + timer runs every 2 hours to support most frequent timer job
      + recurring job intervals are configurable in uaclient.conf
      + individual jobs are disabled if their interval is set to 0
      - status job:
        + update ua status every 12 hours
      - messaging job:
        + update APT/MOTD ESM messaging every 6 hours
      - metering job:
        + disabled until infrastructure is ready
        + for attached machines only, periodically update contract server with
          status information for proper contract metering
    - ua-license-check.timer:
      + only runs on LTS GCP instances that are not attached
      + runs every 5 minutes to check if gcp instance has license r...

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for ubuntu-advantage-tools has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package ubuntu-advantage-tools - 27.3~21.04.1

---------------
ubuntu-advantage-tools (27.3~21.04.1) hirsute; urgency=medium

  * Backport new upstream release: (LP: #1942929) to hirsute

ubuntu-advantage-tools (27.3~21.10.1) impish; urgency=medium

  * d/tools.postinst:
    - consider cloud to be "none" on any cloud-id error
    - purge old ua-messaging.timer/service files
    - keep ua-timer.timer disabled if ua-messaging.timer was disabled by
      the user
    - properly configure both ubuntu-advantage-timer and
      ubuntu-advantage-licence-check logs
  * d/tools.postrm:
    - remove ubuntu-advantage-timer and ubuntu-advantage-license-check logs
      during purge
  * systemd:
    - remove ua-messaging.timer/service
    - add new ua-timer.timer that runs every 6 hours
    - add new ua-license_check.timer that runs every 5 minutes only if
      activated by ua-license-check.path
  * New upstream release 27.3 (LP: #1942929)
    - ros:
      + add beta support to enable ros and ros-updates
      + add support for "required services" so that esm-infra and esm-apps
        get auto-enabled when enabling ros or ros-updates
      + add support for "dependent services" so that user gets prompted to
        disable ros/ros-updates if they disable esm-infra/esm-apps
    - fips:
      + allow fips on GCP bionic now that optimized kernel is ready
      + disallow enabling fips on focal on clouds until cloud-optimized focal
        fips-certified kernel is ready (LP: #1939449, LP: #1939932)
      + print warning about generic fips kernel if cloud-id fails
    - cloud:
      + rely only on cloud-id to determine cloud type (LP: #1940131)
      + catch errors when determining cloud type
        (LP: #1938207, LP: #1944676) (GH: #1541)
    - azure:
      + bump IMDS API version to support Azure published images
    - cli:
      + collect-logs command that creates a tar file with debug-relevant logs
        and status info (GH: #463)
      + clean locks on exceptions more thoroughly to avoid false "Operation in
        progress" status messages
      + retain past service state after detach
      + shows better error message when a port value in a proxy is invalid
    - non-unicode locale support:
      + remove unicode-only characters from help file
      + don't print unicode-only characters in ua fix if non-utf8 locale
        (GH: #1463)
    - logrotate:
      + add logrotate functionality for ubuntu-advantage-timer.log.
      + Fix root:root logrotate permissions.
    - ua-timer.timer:
      + introduce a single systemd timer to handle ua recurring jobs
      + timer runs every 2 hours to support most frequent timer job
      + recurring job intervals are configurable in uaclient.conf
      + individual jobs are disabled if their interval is set to 0
      - status job:
        + update ua status every 12 hours
      - messaging job:
        + update APT/MOTD ESM messaging every 6 hours
      - metering job:
        + disabled until infrastructure is ready
        + for attached machines only, periodically update contract server with
          status information for proper contract metering
    - ua-license-check.timer:
   ...

[Launchpad Janitor (janitor)]

This bug was fixed in the package ubuntu-advantage-tools - 27.3~20.04.1

---------------
ubuntu-advantage-tools (27.3~20.04.1) focal; urgency=medium

  * Backport new upstream release: (LP: #1942929) to focal

ubuntu-advantage-tools (27.3~21.10.1) impish; urgency=medium

  * d/tools.postinst:
    - consider cloud to be "none" on any cloud-id error
    - purge old ua-messaging.timer/service files
    - keep ua-timer.timer disabled if ua-messaging.timer was disabled by
      the user
    - properly configure both ubuntu-advantage-timer and
      ubuntu-advantage-licence-check logs
  * d/tools.postrm:
    - remove ubuntu-advantage-timer and ubuntu-advantage-license-check logs
      during purge
  * systemd:
    - remove ua-messaging.timer/service
    - add new ua-timer.timer that runs every 6 hours
    - add new ua-license_check.timer that runs every 5 minutes only if
      activated by ua-license-check.path
  * New upstream release 27.3 (LP: #1942929)
    - ros:
      + add beta support to enable ros and ros-updates
      + add support for "required services" so that esm-infra and esm-apps
        get auto-enabled when enabling ros or ros-updates
      + add support for "dependent services" so that user gets prompted to
        disable ros/ros-updates if they disable esm-infra/esm-apps
    - fips:
      + allow fips on GCP bionic now that optimized kernel is ready
      + disallow enabling fips on focal on clouds until cloud-optimized focal
        fips-certified kernel is ready (LP: #1939449, LP: #1939932)
      + print warning about generic fips kernel if cloud-id fails
    - cloud:
      + rely only on cloud-id to determine cloud type (LP: #1940131)
      + catch errors when determining cloud type
        (LP: #1938207, LP: #1944676) (GH: #1541)
    - azure:
      + bump IMDS API version to support Azure published images
    - cli:
      + collect-logs command that creates a tar file with debug-relevant logs
        and status info (GH: #463)
      + clean locks on exceptions more thoroughly to avoid false "Operation in
        progress" status messages
      + retain past service state after detach
      + shows better error message when a port value in a proxy is invalid
    - non-unicode locale support:
      + remove unicode-only characters from help file
      + don't print unicode-only characters in ua fix if non-utf8 locale
        (GH: #1463)
    - logrotate:
      + add logrotate functionality for ubuntu-advantage-timer.log.
      + Fix root:root logrotate permissions.
    - ua-timer.timer:
      + introduce a single systemd timer to handle ua recurring jobs
      + timer runs every 2 hours to support most frequent timer job
      + recurring job intervals are configurable in uaclient.conf
      + individual jobs are disabled if their interval is set to 0
      - status job:
        + update ua status every 12 hours
      - messaging job:
        + update APT/MOTD ESM messaging every 6 hours
      - metering job:
        + disabled until infrastructure is ready
        + for attached machines only, periodically update contract server with
          status information for proper contract metering
    - ua-license-check.timer:
      +...

[Launchpad Janitor (janitor)]

This bug was fixed in the package ubuntu-advantage-tools - 27.3~18.04.1

---------------
ubuntu-advantage-tools (27.3~18.04.1) bionic; urgency=medium

  * Backport new upstream release: (LP: #1942929) to bionic

ubuntu-advantage-tools (27.3~21.10.1) impish; urgency=medium

  * d/tools.postinst:
    - consider cloud to be "none" on any cloud-id error
    - purge old ua-messaging.timer/service files
    - keep ua-timer.timer disabled if ua-messaging.timer was disabled by
      the user
    - properly configure both ubuntu-advantage-timer and
      ubuntu-advantage-licence-check logs
  * d/tools.postrm:
    - remove ubuntu-advantage-timer and ubuntu-advantage-license-check logs
      during purge
  * systemd:
    - remove ua-messaging.timer/service
    - add new ua-timer.timer that runs every 6 hours
    - add new ua-license_check.timer that runs every 5 minutes only if
      activated by ua-license-check.path
  * New upstream release 27.3 (LP: #1942929)
    - ros:
      + add beta support to enable ros and ros-updates
      + add support for "required services" so that esm-infra and esm-apps
        get auto-enabled when enabling ros or ros-updates
      + add support for "dependent services" so that user gets prompted to
        disable ros/ros-updates if they disable esm-infra/esm-apps
    - fips:
      + allow fips on GCP bionic now that optimized kernel is ready
      + disallow enabling fips on focal on clouds until cloud-optimized focal
        fips-certified kernel is ready (LP: #1939449, LP: #1939932)
      + print warning about generic fips kernel if cloud-id fails
    - cloud:
      + rely only on cloud-id to determine cloud type (LP: #1940131)
      + catch errors when determining cloud type
        (LP: #1938207, LP: #1944676) (GH: #1541)
    - azure:
      + bump IMDS API version to support Azure published images
    - cli:
      + collect-logs command that creates a tar file with debug-relevant logs
        and status info (GH: #463)
      + clean locks on exceptions more thoroughly to avoid false "Operation in
        progress" status messages
      + retain past service state after detach
      + shows better error message when a port value in a proxy is invalid
    - non-unicode locale support:
      + remove unicode-only characters from help file
      + don't print unicode-only characters in ua fix if non-utf8 locale
        (GH: #1463)
    - logrotate:
      + add logrotate functionality for ubuntu-advantage-timer.log.
      + Fix root:root logrotate permissions.
    - ua-timer.timer:
      + introduce a single systemd timer to handle ua recurring jobs
      + timer runs every 2 hours to support most frequent timer job
      + recurring job intervals are configurable in uaclient.conf
      + individual jobs are disabled if their interval is set to 0
      - status job:
        + update ua status every 12 hours
      - messaging job:
        + update APT/MOTD ESM messaging every 6 hours
      - metering job:
        + disabled until infrastructure is ready
        + for attached machines only, periodically update contract server with
          status information for proper contract metering
    - ua-license-check.timer:
     ...

[Launchpad Janitor (janitor)]

This bug was fixed in the package ubuntu-advantage-tools - 27.3~16.04.1

---------------
ubuntu-advantage-tools (27.3~16.04.1) xenial; urgency=medium

  * Backport new upstream release: (LP: #1942929) to xenial

ubuntu-advantage-tools (27.3~21.10.1) impish; urgency=medium

  * d/tools.postinst:
    - consider cloud to be "none" on any cloud-id error
    - purge old ua-messaging.timer/service files
    - keep ua-timer.timer disabled if ua-messaging.timer was disabled by
      the user
    - properly configure both ubuntu-advantage-timer and
      ubuntu-advantage-licence-check logs
  * d/tools.postrm:
    - remove ubuntu-advantage-timer and ubuntu-advantage-license-check logs
      during purge
  * systemd:
    - remove ua-messaging.timer/service
    - add new ua-timer.timer that runs every 6 hours
    - add new ua-license_check.timer that runs every 5 minutes only if
      activated by ua-license-check.path
  * New upstream release 27.3 (LP: #1942929)
    - ros:
      + add beta support to enable ros and ros-updates
      + add support for "required services" so that esm-infra and esm-apps
        get auto-enabled when enabling ros or ros-updates
      + add support for "dependent services" so that user gets prompted to
        disable ros/ros-updates if they disable esm-infra/esm-apps
    - fips:
      + allow fips on GCP bionic now that optimized kernel is ready
      + disallow enabling fips on focal on clouds until cloud-optimized focal
        fips-certified kernel is ready (LP: #1939449, LP: #1939932)
      + print warning about generic fips kernel if cloud-id fails
    - cloud:
      + rely only on cloud-id to determine cloud type (LP: #1940131)
      + catch errors when determining cloud type
        (LP: #1938207, LP: #1944676) (GH: #1541)
    - azure:
      + bump IMDS API version to support Azure published images
    - cli:
      + collect-logs command that creates a tar file with debug-relevant logs
        and status info (GH: #463)
      + clean locks on exceptions more thoroughly to avoid false "Operation in
        progress" status messages
      + retain past service state after detach
      + shows better error message when a port value in a proxy is invalid
    - non-unicode locale support:
      + remove unicode-only characters from help file
      + don't print unicode-only characters in ua fix if non-utf8 locale
        (GH: #1463)
    - logrotate:
      + add logrotate functionality for ubuntu-advantage-timer.log.
      + Fix root:root logrotate permissions.
    - ua-timer.timer:
      + introduce a single systemd timer to handle ua recurring jobs
      + timer runs every 2 hours to support most frequent timer job
      + recurring job intervals are configurable in uaclient.conf
      + individual jobs are disabled if their interval is set to 0
      - status job:
        + update ua status every 12 hours
      - messaging job:
        + update APT/MOTD ESM messaging every 6 hours
      - metering job:
        + disabled until infrastructure is ready
        + for attached machines only, periodically update contract server with
          status information for proper contract metering
    - ua-license-check.timer:
     ...

[Launchpad Janitor (janitor)]

This bug was fixed in the package ubuntu-advantage-tools - 27.3~21.10.1

---------------
ubuntu-advantage-tools (27.3~21.10.1) impish; urgency=medium

  * d/tools.postinst:
    - consider cloud to be "none" on any cloud-id error
    - purge old ua-messaging.timer/service files
    - keep ua-timer.timer disabled if ua-messaging.timer was disabled by
      the user
    - properly configure both ubuntu-advantage-timer and
      ubuntu-advantage-licence-check logs
  * d/tools.postrm:
    - remove ubuntu-advantage-timer and ubuntu-advantage-license-check logs
      during purge
  * systemd:
    - remove ua-messaging.timer/service
    - add new ua-timer.timer that runs every 6 hours
    - add new ua-license_check.timer that runs every 5 minutes only if
      activated by ua-license-check.path
  * New upstream release 27.3 (LP: #1942929)
    - ros:
      + add beta support to enable ros and ros-updates
      + add support for "required services" so that esm-infra and esm-apps
        get auto-enabled when enabling ros or ros-updates
      + add support for "dependent services" so that user gets prompted to
        disable ros/ros-updates if they disable esm-infra/esm-apps
    - fips:
      + allow fips on GCP bionic now that optimized kernel is ready
      + disallow enabling fips on focal on clouds until cloud-optimized focal
        fips-certified kernel is ready (LP: #1939449, LP: #1939932)
      + print warning about generic fips kernel if cloud-id fails
    - cloud:
      + rely only on cloud-id to determine cloud type (LP: #1940131)
      + catch errors when determining cloud type
        (LP: #1938207, LP: #1944676) (GH: #1541)
    - azure:
      + bump IMDS API version to support Azure published images
    - cli:
      + collect-logs command that creates a tar file with debug-relevant logs
        and status info (GH: #463)
      + clean locks on exceptions more thoroughly to avoid false "Operation in
        progress" status messages
      + retain past service state after detach
      + shows better error message when a port value in a proxy is invalid
    - non-unicode locale support:
      + remove unicode-only characters from help file
      + don't print unicode-only characters in ua fix if non-utf8 locale
        (GH: #1463)
    - logrotate:
      + add logrotate functionality for ubuntu-advantage-timer.log.
      + Fix root:root logrotate permissions.
    - ua-timer.timer:
      + introduce a single systemd timer to handle ua recurring jobs
      + timer runs every 2 hours to support most frequent timer job
      + recurring job intervals are configurable in uaclient.conf
      + individual jobs are disabled if their interval is set to 0
      - status job:
        + update ua status every 12 hours
      - messaging job:
        + update APT/MOTD ESM messaging every 6 hours
      - metering job:
        + disabled until infrastructure is ready
        + for attached machines only, periodically update contract server with
          status information for proper contract metering
    - ua-license-check.timer:
      + only runs on LTS GCP instances that are not attached
      + runs every 5 minutes to check if gcp instance has license r...