[ Impact ]
On ubuntu-advantage-tools v32.2, currently in -proposed, we are hitting apparmor DENIED errors on the apt update hook which executes esm-cache.service.
This ONLY happens if the version with the apparmor profiles is installed on a Focal system which has been upgraded from Bionic, using do-release-upgrade.
It seems that despite covering /usr/bin/ in the profile on Focal for commands like uname or systemctl, we don't account for /bin/. However, when coming from a Bionic system, /bin/ is an actual folder instead of a symlink (as expected on a fresh Focal machine).
This happens because of the usr-merge[1] effort. On fresh focal systems, we have symlinks replacing top-level directories like /bin, /sbin, and others:
root@f-pristine:~# ls -la /{bin,lib,lib*,sbin}
lrwxrwxrwx 1 root root 7 May 24 21:40 /bin -> usr/bin
lrwxrwxrwx 1 root root 7 May 24 21:40 /lib -> usr/lib
lrwxrwxrwx 1 root root 7 May 24 21:40 /lib -> usr/lib
lrwxrwxrwx 1 root root 9 May 24 21:40 /lib32 -> usr/lib32
lrwxrwxrwx 1 root root 9 May 24 21:40 /lib64 -> usr/lib64
lrwxrwxrwx 1 root root 10 May 24 21:40 /libx32 -> usr/libx32
lrwxrwxrwx 1 root root 8 May 24 21:40 /sbin -> usr/sbin
In bionic, these are actual directories:
root@b:~# ls -lad /{bin,lib,lib*,sbin}
drwxr-xr-x 1 root root 2472 Jun 7 2023 /bin
drwxr-xr-x 1 root root 438 Jun 7 2023 /lib
drwxr-xr-x 1 root root 438 Jun 7 2023 /lib
drwxr-xr-x 1 root root 40 Jun 7 2023 /lib64
drwxr-xr-x 1 root root 3694 Jun 7 2023 /sbin
In a focal system that was upgraded from bionic, the usr-merge is not done, and this focal system will retain the bionic top-level directories.
Logs:
2024-05-24 03:09:16,344:WARNING:root:XXX apparmor DENIED begin
2024-05-24 03:09:16,344:WARNING:root:May 24 03:09:09 rtp kernel: [237304.232128] audit: type=1400 audit(1716530949.314:82839): apparmor="DENIED" operation="exec" class="file" namespace="root//lxd-upro-behave-bionic-system-under-test-0524-025458284620_<var-snap-lxd-common-lxd>" profile="ubuntu_pro_esm_cache" name="/bin/uname" pid=108713 comm="python3" requested_mask="x" denied_mask="x" fsuid=1000000 ouid=1000000
May 24 03:09:09 rtp kernel: [237304.261953] audit: type=1400 audit(1716530949.346:82840): apparmor="DENIED" operation="exec" class="file" namespace="root//lxd-upro-behave-bionic-system-under-test-0524-025458284620_<var-snap-lxd-common-lxd>" profile="ubuntu_pro_apt_news" name="/bin/uname" pid=108714 comm="python3" requested_mask="x" denied_mask="x" fsuid=1000000 ouid=1000000
May 24 03:09:09 rtp kernel: [237304.456301] audit: type=1400 audit(1716530949.538:82841): apparmor="DENIED" operation="exec" class="file" namespace="root//lxd-upro-behave-bionic-system-under-test-0524-025458284620_<var-snap-lxd-common-lxd>" profile="ubuntu_pro_esm_cache//cloud_id" name="/bin/uname" pid=108719 comm="cloud-id" requested_mask="x" denied_mask="x" fsuid=1000000 ouid=1000000
May 24 03:09:09 rtp kernel: [237304.514651] audit: type=1400 audit(1716530949.598:82842): apparmor="DENIED" operation="exec" class="file" namespace="root//lxd-upro-behave-bionic-system-under-test-0524-025458284620_<var-snap-lxd-common-lxd>" profile="ubuntu_pro_esm_cache//cloud_id" name="/bin/systemctl" pid=108721 comm="cloud-id" requested_mask="x" denied_mask="x" fsuid=1000000 ouid=1000000
May 24 03:09:11 rtp kernel: [237306.797550] audit: type=1400 audit(1716530951.878:82843): apparmor="DENIED" operation="exec" class="file" namespace="root//lxd-upro-behave-bionic-system-under-test-0524-025458284620_<var-snap-lxd-common-lxd>" profile="ubuntu_pro_esm_cache" name="/bin/uname" pid=109364 comm="python3" requested_mask="x" denied_mask="x" fsuid=1000000 ouid=1000000
May 24 03:09:11 rtp kernel: [237306.827422] audit: type=1400 audit(1716530951.910:82844): apparmor="DENIED" operation="exec" class="file" namespace="root//lxd-upro-behave-bionic-system-under-test-0524-025458284620_<var-snap-lxd-common-lxd>" profile="ubuntu_pro_apt_news" name="/bin/uname" pid=109365 comm="python3" requested_mask="x" denied_mask="x" fsuid=1000000 ouid=1000000
May 24 03:09:12 rtp kernel: [237307.022790] audit: type=1400 audit(1716530952.106:82845): apparmor="DENIED" operation="exec" class="file" namespace="root//lxd-upro-behave-bionic-system-under-test-0524-025458284620_<var-snap-lxd-common-lxd>" profile="ubuntu_pro_esm_cache//cloud_id" name="/bin/uname" pid=109370 comm="cloud-id" requested_mask="x" denied_mask="x" fsuid=1000000 ouid=1000000
May 24 03:09:12 rtp kernel: [237307.074546] audit: type=1400 audit(1716530952.158:82846): apparmor="DENIED" operation="exec" class="file" namespace="root//lxd-upro-behave-bionic-system-under-test-0524-025458284620_<var-snap-lxd-common-lxd>" profile="ubuntu_pro_esm_cache//cloud_id" name="/bin/systemctl" pid=109372 comm="cloud-id" requested_mask="x" denied_mask="x" fsuid=1000000 ouid=1000000
May 24 03:09:14 rtp kernel: [237309.142413] audit: type=1400 audit(1716530954.226:82847): apparmor="DENIED" operation="exec" class="file" namespace="root//lxd-upro-behave-bionic-system-under-test-0524-025458284620_<var-snap-lxd-common-lxd>" profile="ubuntu_pro_apt_news" name="/bin/uname" pid=109856 comm="python3" requested_mask="x" denied_mask="x" fsuid=1000000 ouid=1000000
2024-05-24 03:09:16,344:WARNING:root:XXX apparmor DENIED end
1. https://wiki.debian.org/UsrMerge
[ Test Plan ]
These were caught by the automated verification tests for v32.2 in -proposed. If all of the automated verification tests pass for the version with the fix (32.3), then that will be considered a verification for this bug as well.
The specific tests to be executed for this are:
1. The Bionic to Focal upgrade tests:
- features/ubuntu_upgrade.feature:50 Attached upgrade -- @1.2 ubuntu release
- features/ubuntu_upgrade.feature:51 Attached upgrade -- @1.3 ubuntu release
- features/ubuntu_upgrade_unattached.feature:62 Unattached upgrade -- @1.2 ubuntu release
2. The following Focal tests which verify the esm cache working:
- features/unattached_commands.feature:370 esm cache failures don't generate errors -- @1.2 ubuntu release
- all of features/security-status.feature
[ Where problems could occur ]
The fix edits the template for the ubuntu_pro_esm_cache apparmor profile. If mistakes were made, it may cause new apparmor denials or other related issues, ultimately meaning esm-cache.service wouldn't run properly, preventing esm update notifications from being displayed on unattached machines.
Given the nature of the change needed for this fix, it is very unlikely that we are breaking anything else: we are making the rules more permissive than they were before. However, if any typo is present, we may be breaking the esm-cache.service as mentioned before.
I spoke to Andreas and Renan about this bug just now. I'm told that 32.2 in proposed passes all verification tests against the exceptional test plan except for this bug. But we also need this bug fixed and don't want to have to wait for the additional time a full test rerun would take along with a further ageing period.
If we were to land exceptional SRU from proposed into updates right now, then perform a regular SRU for a 32.3 with this bug fixed, then the second SRU would only require normal SRU verification for the particular fix being applied (test plan to include both the B->F upgrade case and the fresh installation F case, but checking apparmor behaviour only).
Given that exceptional test plan verification against 32.2 is already complete and passed except for this bug, I think that not releasing 32.2 in proposed currently, replacing it with a 32.3, and then SRU-verifying only this bug is equivalent from a QA perspective and therefore acceptable and preferable in this case. We would then be able to release 32.3 to updates as normal.
Plan of action:
1. Update this bug "Where problems could occur" with the specifics that Renan mentioned earlier, and link to this comment to explain what is going on from both this bug and tracking bug 2060732.
2. Complete the Test Plan for this bug to the satisfaction of one SRU team member (Andreas?) treating it as the Test Plan like one would for a regular SRU of this fix only (exercising AppArmor sufficiently for both the B->F upgrade and F fresh install cases and a smoke test to make sure the client still works if not included implicitly in exercising AppArmor is sufficient IMHO; use of automation is fine as always).
3. Complete the verification report for 32.2 in bug 2060732 as normal, except without the final change of the tags since we do not intend to release it.
4. Accept 32.3 into proposed (for all series) to address this bug 2067319. The upload should use -v to include the previous upload as well.
5. The bugs already verified (eg. tracking bug 2060732 but also any others previously verified) can be re-marked as verified with only a link to this comment as explanation (no re-verification required).
6. SRU-verify this bug 2067319 like you would a regular SRU.
7. Release when ready. I also agree that we can skip the ageing period as the only testing expected to be done is by us.