snap "core" is not removable: snap is used by the model (and ubuntu-advantage-tools used to install it by default on all installs)

[Dimitri John Ledkov]

ubuntu-advantage-tools installs "core" snap even though canonical-livepatch snap exists for every base

it seems like instead of installing core22/stable on jammy systems, it installs latest/stable with core snap base, making core snap uninstallable.

I will try to reproduce this, but this is off.

Jammy+ systems, must not have core snap as unremovable anymore.

canonical-livepatch-client in latest/stable got updated to use base:core22, however all systems that tracked that channel in October 2023 are now polluted with irremovable core snap.

[Testcase]

snap install canonical-livepatch-client
snap refresh --channel core/stable canonical-livepatch-client (installs core snap as revision 246 is base:core)
snap refresh --channel latest/stable
snap remove core (impossible)
(refresh to core18/stable, core20/stable latest/stable to get rid of old revisions that use core)

$ grep -h '^base:' /snap/*/*/snap/snapcraft.yaml | sort -u
base: bare
base: core18
base: core20
base: core22

$ snap remove core
error: cannot remove "core": snap "core" is not removable: snap is used by the model

Potentially on my system previous revisions of the snap still hold back core snap being installed.

=== summary
snapd: once core snap is installed, it can never be removed

canonical-livepatch-client: does not provide base:bare build, nor does the default track point to most current base (today "base:core22"); nor do latest/stable/ubuntu-MM.YY branches exist that use matching base (i.e. latest/stable/ubuntu-22.04 publishes the core22/stable snap revisions)

ubuntu-advantage-tools: installs default track livepatch-client, which doesn't match a given LTS release

ubuntu-release-upgrader: doens't currently switch canonical-livepatch-client to use modern/matching base snap track.

===

ProblemType: Bug
DistroRelease: Ubuntu 22.04
Package: ubuntu-advantage-tools 29.4~22.04
ProcVersionSignature: Ubuntu 6.2.0-34.34~22.04.1-generic 6.2.16
Uname: Linux 6.2.0-34-generic x86_64
NonfreeKernelModules: zfs zunicode zavl icp zcommon znvpair
ApportVersion: 2.20.11-0ubuntu82.5
Architecture: amd64
CasperMD5CheckResult: pass
CurrentDesktop: ubuntu:GNOME
Date: Thu Oct 12 00:22:38 2023
InstallationDate: Installed on 2023-04-28 (166 days ago)
InstallationMedia: Ubuntu 22.04.2 LTS "Jammy Jellyfish" - Release amd64 (20230223)
SourcePackage: ubuntu-advantage-tools
UpgradeStatus: No upgrade log present (probably fresh install)
cloud-id.txt-error: Invalid command specified 'cloud-id'.
uaclient.conf:
 contract_url: https://contracts.canonical.com
 log_level: debug

[Dimitri John Ledkov (xnox)]

Just confirmed by launching daily jammy VM, performed `pro enable` with company token, and core snap got installed and now cannot be removed.

[Grant Orndorff (orndorffgrant)]

Thanks for reporting this xnox

We had de-prioritized the feature to install the per-release coreXX livepatch-client at the recommendation of the livepatch team. They are moving livepatch-client to a "bare" base which solves the problem entirely.

However, that isn't done yet, and I don't know the latest estimate on when that will get done. I'll ping them and loop them in here.

If this is urgent, we can implement setting the coreXX/stable channel per release in the mean time until the bare base version is ready.

Can you help us understand the criticality/urgency of the problem? If I understand correctly, a Pro jammy system will get latest/stable livepatch-client, which has the "core" base. The livepatch-client works, but because it is based on "core", then the user can't uninstall "core" without also uninstalling livepatch-client. That is unideal, and maybe I just haven't had enough coffee yet today, but is there something that makes that critically urgent to fix?

Same question in practical terms: Is this critical enough that we should pause the current release process and put this feature in v30 that will be released mid-November? The next scheduled release is v31 in Feb 2024.

[Grant Orndorff (orndorffgrant)]

Update from livepatch team: The "bare" base isn't scheduled for release in the near future, so if this is urgent, it is indeed something pro-client will need to fix using the coreXX/stable channels per release.

[Kian Parvin (kian-parvin)]

Fyi this has been the modus operandi for at least the last 2 years from what I can tell. Not to say that it's okay, but rather that could probably drop it from a critical. This Github Issue (sorry private repo https://github.com/canonical/livepatch-client/issues/181) had the original request for the multiple Snaps per core.

And as Grant mentioned, Livepatch team have investigated moving the Snap to the Bare base to alleviate this issue but there is some remaining technical work there and we haven't prioritised it as this was never an issue.

[Dimitri John Ledkov (xnox)]

There are two bugs here:

1) pro client must never pull in non-matching base, this is prohibited by the seeded snaps policy in Ubuntu Archive, which it is currently violating. It is absolutely critical that it must never install non-matching base, meaning I will request removal of livepatch feature from Ubuntu Archive for 24.04 release if this is not fixed for 24.04 release to install core24/stable channel

2) pro client must fix this for upgrades, and refresh all existing installs on all LTS releases to an appropraite coreXX/stable channel before "core" goes EOL in April 2026 (ticking time bomb). Or have base:bare published in latest/stable, or publish a supported base snap into latest/stable (i.e. core22 base promoted to latest/stable). Another alternative is to make core22 the default track, but that's still kicking the can down the road by 8 years, as it will not be suitable for 24.04 release

3) Even after we fix pro client to switch/referesh everyone off "core" base livepatch-client, this is still not enough. As currently, once "core" snap gets installed it can never ever ever be removed from a system. This is I believe a snapd bug. Meaning today, whilst jammy systems start out without "core" snap that is about to go EOL, they gain it, and can never remove it again.

This is absolutely critical to solve in some way. Which way it is solved, doesn't matter. But you cannot all reference each other's implementations, and deprioritise all of them. As at least one of them should be scheduled to be fixed soon. (as in the current situation, where base:bare is deprioritised, and installing/refreshing to the matching coreXX/stable track is also deprioritised).

You are currently exposing modern ubuntu systems to the risk of forcing to run and make available obsolete and vulnerable binaries on modern Ubuntu releases. Recent security exploits have been demonstrated to use unrelated binaries available on the host system, to gain advantage (see recent ssh-agent attack that use random available files on the host, with the right properties to dlopen them to stage remote code executation and prviledge escalation attacks).

[Dimitri John Ledkov (xnox)]

ubuntu-release-upgrader should know about unique track names for purpose of switching channels.

[Kian Parvin (kian-parvin)]

As discussed, the Livepatch-client snap from core22 will be moved to the latest/stable channel this week to promote a better base snap.

[Andrew Phelps (andrewphelpsj)]

Hello, the source of this bug has been identified, and a PR has been opened to address the issue: https://github.com/snapcore/snapd/pull/13509

[Ernest Lotter (ernestl)]

Fix will be release in snapd 2.61.2

[karl stevens (tas-mania-deactivatedaccount)]

On 24.04 Ubuntu essentially re-branded appimages as snaps and then blocked all appimages from running.

Is this supposed to protect the OS from it's own users?
All they have to do is boot from a thumb-drive, reformat Ubuntu and move on to more open system.

[Ernest Lotter (ernestl)]

Update: This fix introduced an compatibility issue with 2.61.2, and will instead become available in snapd 2.62