[SRU] ubuntu-advantage-tools (27.2.2 -> 27.3) Xenial, Bionic, Focal, Hirsute
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
ubuntu-advantage-tools (Ubuntu) |
Fix Released
|
Undecided
|
Unassigned | ||
Xenial |
Fix Released
|
Undecided
|
Unassigned | ||
Bionic |
Fix Released
|
Undecided
|
Unassigned | ||
Focal |
Fix Released
|
Undecided
|
Unassigned | ||
Hirsute |
Fix Released
|
Undecided
|
Unassigned | ||
Impish |
Fix Released
|
Undecided
|
Unassigned |
Bug Description
[Impact]
This release sports both bug-fixes and new features and we would like to
make sure all of our supported customers have access to these
improvements. The notable ones are:
* more robust error handling when determining the cloud we're on LP: #1940131 LP: #1938207 LP: #1944676
* disallows fips on focal aws/azure LP: #1939449 LP: #1939932
* adds/changes to ua-related recurring jobs:
- change in frequency to existing job: updates the apt and motd esm update messaging: every 6 hours
- new job: updates the contract details and status: every 12 hours
- new job: ONLY ON GCP (implemented as separate timer that is only activated on GCP LTS when not attached): checks for license changes and auto-attaches if a pro license was added: every 5 minutes
* adds support for ros/ros-updates entitlements with --beta flag
With this change, the ua-message timer is renamed to ua-timer, as it has a more generic functionality: it triggers sub jobs which need to be executed periodically. One of those is exactly the job which updates the messaging - which has its interval reduced to 6h, as well as the timer itself. There is also a job to update the client status every 12h, and a third one to collect metrics, which is disabled for this release.
See the changelog entry below for a full list of changes and bugs.
[Test Case]
The following development and SRU process was followed:
https:/
The ubuntu-
console output of the appropriate run to the bug. ubuntu-
members will not mark ‘verification-done’ until this has happened.
<TODO After released to -proposed: attach integration test artifacts>
In addition to the automated integration testing, here are manual test instructions to verify that the packaging changes are functional. All of these tests use the ua-client/staging PPA, which has the 27.3 release build in it.
Manual Test 1:
Here we verify that the cloud-id command changes in postinst work correctly such that the installation succeeds even if the cloud-id command fails.
```
lxc launch ubuntu-daily:impish dev-i
lxc exec dev-i -- /bin/sh -c "printf 'exit 1' > /usr/bin/cloud-id"
lxc exec dev-i -- chmod +x /usr/bin/cloud-id
lxc exec dev-i -- cloud-id
# CHECK: successfully faking cloud-init error. command should've exited 1
lxc exec dev-i -- add-apt-repository -yu ppa:ua-
lxc exec dev-i -- apt install -y ubuntu-
# CHECK: installation should've succeeded despite failing cloud-init
lxc delete dev-i --force
```
Manual Test 2:
Here we verify that the new license check timer only runs on LTS instances that are identified as running on GCP.
```
# should not be enabled when not gcp, on LTS (focal)
lxc launch ubuntu-daily:focal dev-f
lxc exec dev-f -- add-apt-repository -yu ppa:ua-
lxc exec dev-f -- apt install -y ubuntu-
lxc exec dev-f -- systemctl list-timers --all
lxc exec dev-f -- systemctl status ua-license-
# CHECK: should not be running
lxc exec dev-f -- reboot
lxc exec dev-f -- systemctl list-timers --all
lxc exec dev-f -- systemctl status ua-license-
# CHECK: still should not be running
lxc delete dev-f --force
# fake gcp by overwriting cloud-id, still should not be enabled because not on LTS (impish)
lxc launch ubuntu-daily:impish dev-i
lxc exec dev-i -- /bin/sh -c "printf 'echo gce' > /usr/bin/cloud-id"
lxc exec dev-i -- chmod +x /usr/bin/cloud-id
lxc exec dev-i -- cloud-id
# CHECK: successfully faking gcp. output should be "gce" (with an "e")
lxc exec dev-i -- add-apt-repository -yu ppa:ua-
lxc exec dev-i -- apt install -y ubuntu-
lxc exec dev-i -- systemctl list-timers --all
lxc exec dev-i -- systemctl status ua-license-
# CHECK: should not be running
lxc delete dev-i --force
# fake gcp by overwriting cloud-id, on LTS (focal), should be enabled
lxc launch ubuntu-daily:focal dev-f
lxc exec dev-f -- /bin/sh -c "printf 'echo gce' > /usr/bin/cloud-id"
lxc exec dev-f -- chmod +x /usr/bin/cloud-id
lxc exec dev-f -- cloud-id
# CHECK: successfully faking gcp. output should be "gce" (with an "e")
lxc exec dev-f -- add-apt-repository -yu ppa:ua-
lxc exec dev-f -- apt install -y ubuntu-
lxc exec dev-f -- systemctl list-timers --all
lxc exec dev-f -- systemctl status ua-license-
# CHECK: should be enabled
lxc delete dev-f --force
```
Manual Test 3:
Here we verify that the old ua-messaging.
```
lxc launch ubuntu-daily:impish dev-i
lxc exec dev-i -- /bin/sh -c "ls -1 /etc/systemd/
# CHECK: verify several ua-messaging artifacts
lxc exec dev-i -- add-apt-repository -yu ppa:ua-
lxc exec dev-i -- apt install -y ubuntu-
lxc exec dev-i -- /bin/sh -c "ls -1 /etc/systemd/
# CHECK: verify ua-messaging artifacts are not left behind
lxc delete dev-i --force
```
Manual Test 4:
Here we verify that if the user had disabled the old ua-messaging timer, then we will carry that preference forward to the new ua-timer timer.
```
lxc launch ubuntu-daily:impish dev-i
lxc exec dev-i -- systemctl list-timers --all
lxc exec dev-i -- systemctl status ua-messaging.timer
# CHECK: verify ua-messaging.timer is enabled
lxc exec dev-i -- systemctl stop ua-messaging.timer
lxc exec dev-i -- systemctl disable ua-messaging.timer
lxc exec dev-i -- systemctl list-timers --all
lxc exec dev-i -- systemctl status ua-messaging.timer
# CHECK: verify ua-messaging.timer is disabled
lxc exec dev-i -- add-apt-repository -yu ppa:ua-
lxc exec dev-i -- apt install -y ubuntu-
lxc exec dev-i -- systemctl list-timers --all
lxc exec dev-i -- systemctl status ua-timer.timer
# CHECK: verify ua-timer.timer is disabled
lxc delete dev-i --force
```
Manual Test 5:
Here we verify that the new log files are appropriately created on install, rotated by logrotate, and deleted on purge.
```
lxc launch ubuntu-daily:impish dev-i
lxc exec dev-i -- add-apt-repository -yu ppa:ua-
lxc exec dev-i -- apt install -y ubuntu-
lxc exec dev-i -- /bin/sh -c 'ls -l /var/log/
# CHECK: verify that three ua log files were created, are owned by root, and have 600 permissions
lxc exec dev-i -- /bin/sh -c "printf testcontent > /var/log/
lxc exec dev-i -- /bin/sh -c "printf testcontent > /var/log/
lxc exec dev-i -- /bin/sh -c "printf testcontent > /var/log/
lxc exec dev-i -- logrotate --force /etc/logrotate.
lxc exec dev-i -- /bin/sh -c 'ls -l /var/log/
# CHECK: verify all 3 logs were rotated
lxc exec dev-i -- /bin/sh -c "printf testcontent > /var/log/
lxc exec dev-i -- /bin/sh -c "printf testcontent > /var/log/
lxc exec dev-i -- /bin/sh -c "printf testcontent > /var/log/
lxc exec dev-i -- /bin/sh -c 'ls /var/log/
# CHECK: verify all ua log files exist including rotated versions
lxc exec dev-i -- apt purge -y ubuntu-
lxc exec dev-i -- /bin/sh -c "ls /var/log/"
# CHECK: verify that all ua log files are removed
lxc delete dev-i --force
```
[Regression Potential]
In order to mitigate the regression potential, the results of the
aforementioned integration tests are attached to this bug.
We moved the trigger of the apt and motd messaging updates from a dedicated systemd timer to a shared timer that conditionally calls the messaging updates in our python code. This adds complexity. If we made a mistake, then either the job won't get called frequently enough or will get called too frequently. If the former, then some esm updates related messaging will be out of date in apt and motd. If the latter, then cpu cycles will be wasted in needlessly updating messages.
We touched postinst to handle cloud-id failures more robustly. Touching postinst is always scary because it is the most likely way for us to break upgrades. In theory this change made upgrades less likely to fail, but if we made a mistake, it could cause new unexpected failures.
We added more recurring jobs in the service of new features. This increases complexity and potential for mistakes. In particular, we have strived to avoid excessive logging from these jobs. If we made a mistake in our logging, we could inadvertently fill up disks with useless logs. Additional recurring jobs will also use more cpu over time than previous versions. This is at least partially addressed below.
We instrumented a high frequency timer to only run on GCP, but if we made a mistake, this could be accidentally activated on non-GCP machines, which would be a waste. (See below for additional high frequency timer discussion).
We check if the ua-messaging timer was disabled prior to this update, and if so we also disable the new ua-timer timer in systemd. Failing to do so would keep enabled a service that the user had explicitly disabled in the past, resulting in a unwanted behavior. Our migration of this user configuration only covers the case where the user ran `systemctl disable` to disable the old timer. If they disabled the timer in a different way, then their configuration will not be carried forward. Furthermore, this is a somewhat complicated postinst addition, and carries all the normal risks of editing postinst.
[Discussion]
Our timer on GCP runs every 5 minutes. This is necessary to support timely upgrades of gcp instances from standard ubuntu to ubuntu pro. We need to poll the metadata endpoint frequently to catch the license change in a timely manner. We exit as early as possible if there is nothing to be done for any given timer trigger. From our testing, this has minimal overall system performance impact. <TODO @chad.smith insert details and link to spreadsheet>
[Changelog]
* d/tools.postinst:
- consider cloud to be "none" on any cloud-id error
- purge old ua-messaging.
* systemd:
- remove ua-messaging.
- add new ua-timer.timer that runs every 2 hours
- add new ua-license_
activated by ua-license-
* New upstream release 27.3
- ros:
+ add beta support to enable ros and ros-updates
+ add support for "required services" so that esm-infra and esm-apps
get auto-enabled when enabling ros or ros-updates
+ add support for "dependent services" so that user gets prompted to
disable ros/ros-updates if they disable esm-infra/esm-apps
- fips:
+ allow fips on GCP bionic now that optimized kernel is ready
+ disallow enabling fips on focal on clouds until cloud-optimized focal
+ print warning about generic fips kernel if cloud-id fails
- cloud:
+ rely only on cloud-id to determine cloud type (LP: #1940131)
+ catch errors when determining cloud type (LP: #1938207) (GH: #1541)
- azure:
+ bump IMDS API version to support Azure published images
- cli:
+ collect-logs command that creates a tar file with debug-relevant logs
and status info (GH: #463)
+ clean locks on exceptions more thoroughly to avoid false "Operation in
progress" status messages
+ retain past service state after detach
+ shows better error message when a port value in a proxy is invalid
- non-unicode locale support:
+ remove unicode-only characters from help file
+ don't print unicode-only characters in ua fix if non-utf8 locale
(GH: #1463)
- ua-timer.timer:
+ introduce a single systemd timer to handle ua recurring jobs
+ timer runs every 2 hours to support most frequent timer job
+ recurring job intervals are configurable in uaclient.conf
+ individual jobs are disabled if their interval is set to 0
- status job:
+ update ua status every 12 hours
- messaging job:
+ update APT/MOTD ESM messaging every 6 hours
- metering job:
+ disabled until infrastructure is ready
+ for attached machines only, periodically update contract server with
status information for proper contract metering
- ua-license-
+ only runs on LTS GCP instances that are not attached
+ runs every 5 minutes to check if gcp instance has license required to
auto-attach
- logs:
+ fixes duplicate logging (GH: #553)
- tests and support:
+ remove groovy integration tests
+ various improvements to integration tests
Related branches
- Athos Ribeiro (community): Approve
- Canonical Server Core Reviewers: Pending requested
-
Diff: 14912 lines (+6589/-2545)133 files modified.gitignore (+42/-0)
.pre-commit-config.yaml (+4/-0)
Jenkinsfile (+8/-1)
README.md (+87/-6)
RELEASES.md (+239/-132)
debian/changelog (+74/-0)
debian/rules (+7/-3)
debian/ubuntu-advantage-tools.logrotate (+4/-1)
debian/ubuntu-advantage-tools.postinst (+77/-9)
debian/ubuntu-advantage-tools.postrm (+2/-0)
dev-requirements.txt (+3/-2)
features/_version.feature (+1/-2)
features/attach_invalidtoken.feature (+4/-4)
features/attach_validtoken.feature (+96/-204)
features/attached_commands.feature (+266/-76)
features/attached_enable.feature (+199/-6)
features/attached_status.feature (+29/-0)
features/aws-ids.yaml (+3/-3)
features/cloud.py (+111/-69)
features/enable_fips_cloud.feature (+606/-0)
features/enable_fips_vm.feature (+410/-0)
features/environment.py (+73/-86)
features/install_uninstall.feature (+46/-0)
features/license_check.feature (+120/-0)
features/proxy_config.feature (+7/-0)
features/staging_commands.feature (+2/-534)
features/steps/steps.py (+243/-46)
features/ubuntu_pro.feature (+27/-27)
features/ubuntu_upgrade.feature (+100/-13)
features/ubuntu_upgrade_unattached.feature (+91/-0)
features/unattached_commands.feature (+74/-24)
features/unattached_status.feature (+47/-15)
features/util.py (+90/-64)
help_data.yaml (+19/-1)
integration-requirements.txt (+1/-1)
lib/license_check.py (+28/-0)
lib/patch_status_json.py (+2/-2)
lib/reboot_cmds.py (+6/-7)
lib/timer.py (+121/-0)
lib/upgrade_lts_contract.py (+2/-2)
pyproject.toml (+4/-0)
setup.py (+1/-0)
sru/release-27.3/gcp_auto_attach_test.sh (+35/-0)
sru/release-27.3/ua-messaging-disabled.sh (+28/-0)
systemd/ua-license-check.path (+14/-0)
systemd/ua-license-check.service (+11/-0)
systemd/ua-license-check.timer (+12/-0)
systemd/ua-timer.service (+2/-3)
systemd/ua-timer.timer (+2/-3)
tools/refresh-aws-pro-ids (+15/-4)
tools/refresh-keyrings.sh (+4/-1)
tools/run-integration-tests.py (+191/-0)
tools/ua-test-credentials.example.yaml (+20/-0)
tox.ini (+13/-9)
uaclient-devel.conf (+5/-0)
uaclient.conf (+4/-0)
uaclient/apt.py (+5/-13)
uaclient/cli.py (+204/-75)
uaclient/clouds/__init__.py (+2/-7)
uaclient/clouds/aws.py (+3/-9)
uaclient/clouds/azure.py (+4/-11)
uaclient/clouds/gcp.py (+28/-10)
uaclient/clouds/identity.py (+24/-48)
uaclient/clouds/tests/test_aws.py (+1/-1)
uaclient/clouds/tests/test_azure.py (+3/-3)
uaclient/clouds/tests/test_gcp.py (+1/-1)
uaclient/clouds/tests/test_identity.py (+29/-74)
uaclient/config.py (+154/-62)
uaclient/conftest.py (+6/-10)
uaclient/contract.py (+68/-28)
uaclient/defaults.py (+4/-0)
uaclient/entitlements/__init__.py (+9/-13)
uaclient/entitlements/base.py (+221/-48)
uaclient/entitlements/cc.py (+3/-9)
uaclient/entitlements/cis.py (+3/-9)
uaclient/entitlements/esm.py (+8/-11)
uaclient/entitlements/fips.py (+59/-43)
uaclient/entitlements/livepatch.py (+17/-24)
uaclient/entitlements/repo.py (+28/-64)
uaclient/entitlements/ros.py (+28/-0)
uaclient/entitlements/tests/conftest.py (+23/-27)
uaclient/entitlements/tests/test_base.py (+150/-18)
uaclient/entitlements/tests/test_cc.py (+2/-4)
uaclient/entitlements/tests/test_cis.py (+2/-5)
uaclient/entitlements/tests/test_esm.py (+3/-5)
uaclient/entitlements/tests/test_fips.py (+82/-23)
uaclient/entitlements/tests/test_livepatch.py (+4/-13)
uaclient/entitlements/tests/test_repo.py (+2/-6)
uaclient/jobs/__init__.py (+14/-0)
uaclient/jobs/license_check.py (+62/-0)
uaclient/jobs/metering.py (+29/-0)
uaclient/jobs/tests/__init__.py (+0/-0)
uaclient/jobs/tests/test_gcp_auto_attach.py (+144/-0)
uaclient/jobs/update_messaging.py (+8/-24)
uaclient/jobs/update_state.py (+10/-0)
uaclient/pip.py (+0/-2)
uaclient/security.py (+63/-65)
uaclient/serviceclient.py (+78/-13)
uaclient/snap.py (+1/-2)
uaclient/status.py (+29/-15)
uaclient/tests/test_apt.py (+8/-9)
uaclient/tests/test_cli.py (+94/-11)
uaclient/tests/test_cli_attach.py (+6/-6)
uaclient/tests/test_cli_auto_attach.py (+10/-11)
uaclient/tests/test_cli_collect_logs.py (+135/-0)
uaclient/tests/test_cli_config_set.py (+32/-12)
uaclient/tests/test_cli_config_show.py (+8/-6)
uaclient/tests/test_cli_config_unset.py (+10/-8)
uaclient/tests/test_cli_detach.py (+7/-8)
uaclient/tests/test_cli_disable.py (+6/-7)
uaclient/tests/test_cli_enable.py (+15/-6)
uaclient/tests/test_cli_fix.py (+3/-4)
uaclient/tests/test_cli_refresh.py (+17/-19)
uaclient/tests/test_cli_status.py (+33/-13)
uaclient/tests/test_config.py (+75/-12)
uaclient/tests/test_contract.py (+58/-9)
uaclient/tests/test_gpg.py (+1/-0)
uaclient/tests/test_patch_status_json.py (+2/-2)
uaclient/tests/test_pip.py (+3/-3)
uaclient/tests/test_reboot_cmds.py (+6/-7)
uaclient/tests/test_security.py (+169/-110)
uaclient/tests/test_serviceclient.py (+110/-12)
uaclient/tests/test_snap.py (+0/-1)
uaclient/tests/test_status.py (+4/-3)
uaclient/tests/test_ua_timer.py (+138/-0)
uaclient/tests/test_update_messaging.py (+13/-14)
uaclient/tests/test_upgrade_lts_contract.py (+1/-0)
uaclient/tests/test_util.py (+72/-20)
uaclient/tests/test_version.py (+2/-3)
uaclient/types.py (+3/-0)
uaclient/util.py (+122/-69)
uaclient/version.py (+2/-2)
ubuntu-advantage.1 (+74/-3)
description: | updated |
description: | updated |
description: | updated |
description: | updated |
description: | updated |
description: | updated |
description: | updated |
description: | updated |
description: | updated |
description: | updated |
description: | updated |
description: | updated |
Changed in ubuntu-advantage-tools (Ubuntu Impish): | |
status: | New → Triaged |
status: | Triaged → In Progress |
Changed in ubuntu-advantage-tools (Ubuntu Xenial): | |
status: | New → In Progress |
Changed in ubuntu-advantage-tools (Ubuntu Bionic): | |
status: | New → In Progress |
Changed in ubuntu-advantage-tools (Ubuntu Focal): | |
status: | New → In Progress |
Changed in ubuntu-advantage-tools (Ubuntu Hirsute): | |
status: | New → In Progress |
Changed in ubuntu-advantage-tools (Ubuntu): | |
status: | Incomplete → Fix Committed |
One of the packaging changes in the SRU which I don't see an explicit bug for (so I'm attaching the question here) is:
+++ ubuntu- advantage- tools-27. 3~21.10. 1/debian/ ubuntu- advantage- tools.postinst 2021-09-21 13:02:06.000000000 +0000 log_file( ) { ubuntu- advantage. log ]; then ubuntu- advantage. log ubuntu- advantage. log ubuntu- advantage. log ubuntu- advantage. log ubuntu- advantage- timer.log ubuntu- advantage- license- check.log
[...]
+configure_
+ log_path=$1
+ if [ ! -f $log_path ]; then
+ touch $log_path
+ fi
+ chmod 0600 $log_path
+ chown root:root $log_path
+}
[...]
- if [ ! -f /var/log/
- touch /var/log/
- fi
- chmod 0600 /var/log/
- chown root:root /var/log/
+ configure_log_file /var/log/
+ configure_log_file /var/log/
+ configure_log_file /var/log/
+
[...]
It is unusual for maintainer scripts to pre-create log files. If these files are removed, do the tools recreate them with the correct permissions? (If so: is this code needed at all?)