[SRU] update-manager crashes when ua security-status response is an error

[Gregory C. Psaltakis]

Impact
------

Update Manager fails to launch in the LTS releases if the 'ua security-status' command fails. It tries to parse its output to generate the list of ESM-Apps and ESM-Infra updates that would be available if the system were attached to Ubuntu Pro.

However, if ua finds some problem with unrelated stuff, such as Livepatch (as evidenced by the attached ubuntu-advantage.log), the command just raises an error, crashing Update Manager.

Although Update Manager could catch the error, it would result in not showing Ubuntu Pro updates, simply because the information is not there.

Better would be to ditch 'ua security-status' in favor of the Ubuntu Pro API. The updates() end point[1] will provide the information Update Manager wants even if the Ubuntu Advantage daemon has issues with other services such as Livepatch (as evidenced by #10), and it does not raise exceptions.

[1] https://canonical-ubuntu-pro-client.readthedocs-hosted.com/en/latest/references/api/#u-pro-packages-updates-v1

Test Plan
---------

Attach your machine to Ubuntu Pro and make sure Livepatch is enabled in the output of 'ua status'. Then, simulate a failing Livepatch with

--->
mv /snap/bin/canonical-livepatch /snap/bin/canonical-livepatch.bak
>/snap/bin/canonical-livepatch printf '%s\n' '#!/bin/sh' 'exit 1'
chmod +x /snap/bin/canonical-livepatch
<----

Launch update-manager, it should open and not crash.

Where problems could occur
--------------------------

In the Ubuntu Pro section of the updates, in form of wrong package data (e.g. package name, version number etc.) in the description pane when selecting a package. The section could not show at all, or the program could crash right away.

Original bug report
-------------------

Below is a run in my computer. Please send a mail on what to do

fuga2:~> update-manager -c -d
Unexpected error(s) occurred.
For more details, see the log: /var/log/ubuntu-advantage.log
To file a bug run: ubuntu-bug ubuntu-advantage-tools
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/UpdateManager/backend/InstallBackendAptdaemon.py", line 448, in _on_finished
    self._action_done(action,
  File "/usr/lib/python3/dist-packages/UpdateManager/backend/__init__.py", line 365, in _action_done
    self.window_main.start_available()
  File "/usr/lib/python3/dist-packages/UpdateManager/UpdateManager.py", line 241, in start_available
    self.refresh_cache()
  File "/usr/lib/python3/dist-packages/UpdateManager/UpdateManager.py", line 459, in refresh_cache
    self._get_ua_security_status()
  File "/usr/lib/python3/dist-packages/UpdateManager/UpdateManager.py", line 285, in _get_ua_security_status
    s = json.load(p.stdout)
  File "/usr/lib/python3.10/json/__init__.py", line 293, in load
    return loads(fp.read(),
  File "/usr/lib/python3.10/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.10/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.10/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
^CTraceback (most recent call last):
  File "/usr/bin/update-manager", line 125, in <module>
    Gtk.main()
  File "/usr/lib/python3/dist-packages/gi/overrides/Gtk.py", line 1687, in main
    with register_sigint_fallback(Gtk.main_quit):
  File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
    next(self.gen)
  File "/usr/lib/python3/dist-packages/gi/_ossighelper.py", line 237, in register_sigint_fallback
    signal.default_int_handler(signal.SIGINT, None)
KeyboardInterrupt

fuga2:~> ubuntu-bug ubuntu-advantage-tools
fuga2:~> update.go:85: cannot change mount namespace according to change mount (/var/lib/snapd/hostfs/usr/local/share/doc /usr/local/share/doc none bind,ro 0 0): cannot open directory "/usr/local/share": permission denied
update.go:85: cannot change mount namespace according to change mount (/var/lib/snapd/hostfs/usr/share/xubuntu-docs /usr/share/xubuntu-docs none bind,ro 0 0): cannot open directory "/var/lib": permission denied
Gtk-Message: 11:23:47.656: Not loading module "atk-bridge": The functionality is provided by GTK natively. Please try to not load it.

fuga2:~>

ProblemType: Bug
DistroRelease: Ubuntu 22.04
Package: ubuntu-advantage-tools 31.2~22.04
ProcVersionSignature: Ubuntu 5.15.0-100.110-generic 5.15.143
Uname: Linux 5.15.0-100-generic x86_64
ApportVersion: 2.20.11-0ubuntu82.5
Architecture: amd64
CasperMD5CheckResult: unknown
CurrentDesktop: ubuntu:GNOME
Date: Sun Mar 17 11:22:30 2024
PackageArchitecture: all
SourcePackage: ubuntu-advantage-tools
UpgradeStatus: Upgraded to jammy on 2024-03-09 (7 days ago)
apparmor_logs.txt: Mar 17 11:15:18 fuga2 kernel: audit: type=1400 audit(1710666917.933:11): apparmor="STATUS" operation="profile_load" profile="unconfined" name="ubuntu_pro_apt_news" pid=1057 comm="apparmor_parser"
cloud-id.txt-error: Invalid command specified 'cloud-id'.
livepatch-status.txt-error: Failed running command '/snap/bin/canonical-livepatch status' [exit(1)]. Message: error executing status: open /var/snap/canonical-livepatch/264/livepatchd.err: permission denied
uaclient.conf:
 contract_url: https://contracts.canonical.com
 log_level: debug

[Alberto Contreras (aciba)]

Thanks, Gregorym, for reporting this.

I do not see anything wrong in the attached log files regarding ubuntu-pro-client. The status looks good and there are no error traces.

Per your error traceback, it looks like update-manager is trying to json-decode the output of some, presumably pro, command, and failing. I have attached update-manager to the bug.

Checking into update-manager source code in jammy, I think the command is `pro security-status --format=json`, could you please attach the output of that command?

[Alberto Contreras (aciba)]

This looks like a duplicate of https://bugs.launchpad.net/ubuntu/+source/ubuntu-advantage-tools/+bug/2058158 .

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Giacomo Massa (jeq0)]

Hi, I've reported duplicate bug 2058158. I post the output of `pro security-status --format=json` below and attach as text file my ubuntu-avantage log.

Unexpected error(s) occurred.
For more details, see the log: /var/log/ubuntu-advantage.log
To file a bug run: ubuntu-bug ubuntu-advantage-tools

[Alberto Contreras (aciba)]

Thanks for the logs.

The output of `pro security-status --format=json` and the internal error look correct, so I'm going to mark the ubuntu-advantage-tools bit as invalid.

It feels like this is something that should be handled and fixed in update-manager. https://bugs.launchpad.net/ubuntu/+source/update-manager/+bug/2049785 sounds very similar to this.

[Nathan Teodosio (nteodosio)]

Giacomo or anyone else getting "Unexpected error(s) occurred." from 'ua security-status --format=json', can you please also attach the output of

  python3 -c 'import uaclient.api.u.pro.packages.updates.v1 as ua;print(ua.updates().updates)'

> The output of `pro security-status --format=json` and the internal error look correct, so I'm going to mark the ubuntu-advantage-tools bit as invalid.

I don't understand this bit. I wouldn't say that "Unexpected error(s) occurred." looks correct, quite on the contrary, so I'm reverting the task but please let me know if you disagree.

In any case, update-manager is also "to blame" insofar as it doesn't validate the output of that command and just directly tries to parse it as a JSON. For it, there are two possible ways forward:

1. If the output of the command is not a JSON, just skip all the Pro logic.
   Positive: Keep changes to a minimal for a backport to the LTS series.
   Negative: Buries the error and so we will not realize that the Pro logic was disabled, which is unintended behavior.
2. Use the API[1] as shown above.
   Positive: No exception is raised by the end point, and possibly it also gives the correct output even when the pro command would fail reported. This looks quite probable by the way, as from the log the command failed because of Livepatch, which update-manager has no interest in, but let's see if the affected users come back with that information.
   Negative: Changes are more encompassing as to see in [2].

[1] https://canonical-ubuntu-pro-client.readthedocs-hosted.com/en/latest/references/api/#u-pro-packages-updates-v1
[2] https://code.launchpad.net/~nteodosio/update-manager/+git/update-manager/+merge/459172

[Nathan Teodosio (nteodosio)]

The ubuntu-advantage-tools side had been discussed internally and "invalid" concluded as the best state because the best that could be done is to improve the error message, so I'm setting that back.

[Giacomo Massa (jeq0)]

Hi Nathan, here is the output from the python3 command as requested:

[UpdateInfo{'download_size': 3590030, 'origin': 'it.archive.ubuntu.com', 'package': 'librados2', 'provided_by': 'standard-updates', 'status': 'upgrade_available', 'version': '17.2.7-0ubuntu0.22.04.1'}, UpdateInfo{'download_size': 3547412, 'origin': 'it.archive.ubuntu.com', 'package': 'librbd1', 'provided_by': 'standard-updates', 'status': 'upgrade_available', 'version': '17.2.7-0ubuntu0.22.04.1'}, UpdateInfo{'download_size': 115463212, 'origin': 'brave-browser-apt-release.s3.brave.com', 'package': 'brave-browser', 'provided_by': 'standard-updates', 'status': 'upgrade_available', 'version': '1.64.109'}, UpdateInfo{'download_size': 2412, 'origin': 'it.archive.ubuntu.com', 'package': 'linux-image-oem-22.04c', 'provided_by': 'standard-security', 'status': 'upgrade_available', 'version': '6.1.0.1036.37'}, UpdateInfo{'download_size': 1674, 'origin': 'it.archive.ubuntu.com', 'package': 'linux-modules-ipu6-oem-22.04b', 'provided_by': 'standard-security', 'status': 'upgrade_available', 'version': '6.1.0.1036.37'}, UpdateInfo{'download_size': 2370, 'origin': 'it.archive.ubuntu.com', 'package': 'linux-modules-ipu6-oem-22.04c', 'provided_by': 'standard-security', 'status': 'upgrade_available', 'version': '6.1.0.1036.37'}, UpdateInfo{'download_size': 1654, 'origin': 'it.archive.ubuntu.com', 'package': 'linux-oem-22.04b', 'provided_by': 'standard-security', 'status': 'upgrade_available', 'version': '6.1.0.1036.37'}, UpdateInfo{'download_size': 1710, 'origin': 'it.archive.ubuntu.com', 'package': 'linux-oem-22.04c', 'provided_by': 'standard-security', 'status': 'upgrade_available', 'version': '6.1.0.1036.37'}, UpdateInfo{'download_size': 2284, 'origin': 'it.archive.ubuntu.com', 'package': 'linux-headers-oem-22.04c', 'provided_by': 'standard-security', 'status': 'upgrade_available', 'version': '6.1.0.1036.37'}, UpdateInfo{'download_size': 1674, 'origin': 'it.archive.ubuntu.com', 'package': 'linux-modules-ivsc-oem-22.04b', 'provided_by': 'standard-security', 'status': 'upgrade_available', 'version': '6.1.0.1036.37'}, UpdateInfo{'download_size': 2378, 'origin': 'it.archive.ubuntu.com', 'package': 'linux-modules-ivsc-oem-22.04c', 'provided_by': 'standard-security', 'status': 'upgrade_available', 'version': '6.1.0.1036.37'}]

[Nathan Teodosio (nteodosio)]

Reproduced in Bionic:

--->
% canonical-livepatch
snap-confine has elevated permissions and is not confined but should be. [...]
% systemctl status snapd-apparmor
[...]
Mär 22 10:58:04 snapd-apparmor[3198]: Failed to load policy-features from '/etc/apparmor.d/abi/kernel-5.4-outoftree-network': No such file or directory
<---

So I guess the whole story is: /etc/apparmor.d/abi/kernel-5.4-outoftree-network is missing, which is causing snapd-apparmor to fail, which is causing Livepatch to fail, which is causing 'ua security-status' to fail, which is causing update-manager to fail.

[Giacomo Massa (jeq0)]

I don't know if this helps but I've tested the patch https://code.launchpad.net/~nteodosio/update-manager/+git/update-manager/+ref/jammy-json-to-api for Jammy by nteodosio and it fixes the problem.

[Nathan Teodosio (nteodosio)]

Many thanks for your proactivity, Giacomo!, it is always very helpful to know beforehand that a proposed change really fixes the real end user case — that's all that matters after all.

[Timo Aaltonen (tjaalton)]

Hello Gregory, or anyone else affected,

Accepted update-manager into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/update-manager/1:22.04.20 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Timo Aaltonen (tjaalton)]

Hello Gregory, or anyone else affected,

Accepted update-manager into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/update-manager/1:20.04.10.21 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Timo Aaltonen (tjaalton)]

Hello Gregory, or anyone else affected,

Accepted update-manager into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/update-manager/1:18.04.12 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Giacomo Massa (jeq0)]

Hi Timo and everybody, just tested the proposed package for Jammy (22.04.4), version 1:22.04.20 and it fixes the bug for me same as it did applying the patch directly to UpdateManager.py before.

Preliminary:
- reverted UpdateManager.py to its original nonworking state
> bug is present

-updated the package to proposed version, then

Testing procedure:

mv /snap/bin/canonical-livepatch /snap/bin/canonical-livepatch.bak
>/snap/bin/canonical-livepatch printf '%s\n' '#!/bin/sh' 'exit 1'
chmod +x /snap/bin/canonical-livepatch
update-manager -c -d

Results:
software opens, runs and gives results without crashing.

[Nathan Teodosio (nteodosio)]

Thank you for verifying the SRU for Jammy, Giacomo! You probably know that, but: After having verified the fix by running the test case, you may want to move the original Livepatch file back with 'mv /snap/bin/canonical-livepatch.bak /snap/bin/canonical-livepatch', in case it were at least partially working.

[Nathan Teodosio (nteodosio)]

In a Ubuntu Pro attached Bionic virtual machine, I installed the 1:18.04.12 update-manager from proposed, sabotaged Livepatch with

--->
mv /snap/bin/canonical-livepatch /snap/bin/canonical-livepatch.bak
>/snap/bin/canonical-livepatch printf '%s\n' '#!/bin/sh' 'exit 1'
chmod +x /snap/bin/canonical-livepatch
<---

and launched update-manager, which opened and worked normally.

[Nathan Teodosio (nteodosio)]

Also confirmed the bug goes away in a Ubuntu Pro attached Focal virtual machine. Installed the 1:20.04.10.21 update-manager from proposed, sabotaged Livepatch with

--->
mv /snap/bin/canonical-livepatch /snap/bin/canonical-livepatch.bak
>/snap/bin/canonical-livepatch printf '%s\n' '#!/bin/sh' 'exit 1'
chmod +x /snap/bin/canonical-livepatch
<---

and launched update-manager, which opened and worked normally.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted update-manager (1:22.04.20) for jammy have finished running.
The following regressions have been reported in tests triggered by the package:

update-notifier/unknown (armhf)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/jammy/update_excuses.html#update-manager

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted update-manager (1:20.04.10.21) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

ubuntu-release-upgrader/1:20.04.41 (armhf)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#update-manager

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Andreas Hasenack (ahasenack)]

Focal still has an ubuntu-release-upgrader regression on armhf. I triggered that one again.

Jammy is ok.

Bionic is ok, but can't be released before focal.

The xenial upload is still in unapproved, probably because there is another xenial SRU going on
 in xenial-proposed: 1:16.04.21

[Launchpad Janitor (janitor)]

This bug was fixed in the package update-manager - 1:22.04.20

---------------
update-manager (1:22.04.20) jammy; urgency=medium

  * Replace Popen to 'ua security-status --format=json' by the Ubuntu Pro API.
    The former is not stable enough and would break the program once in a
    while when something unrelated to Update Manager (Livepatch, for instance)
    went awry. The updates() end point from the API, on the other hand,
    does not raise exceptions. LP: #2058133.

 -- Nathan Pratta Teodosio <email address hidden> Fri, 22 Mar 2024 08:24:04 +0100

[Andreas Hasenack (ahasenack)]

The verification of the Stable Release Update for update-manager has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Rainer Bruns (rainer-bruns)]

Hallo Andreas,

also so ganz schlau bin ich aus dem Ganzen nicht geworden. Denn meine
Aktualisierungsverwaltung hakt nach wie vor.

Wenn ich hingegen den von Dir übermittelten Befehl eingebe im Terminal,
aktualisiert das alles problemlos.

mfG

Rainer Bruns

Am 11.04.24 um 21:08 schrieb Andreas Hasenack:
> Focal still has an ubuntu-release-upgrader regression on armhf. I
> triggered that one again.
>
> Jammy is ok.
>
> Bionic is ok, but can't be released before focal.
>
> The xenial upload is still in unapproved, probably because there is another xenial SRU going on
> in xenial-proposed: 1:16.04.21
>

[Nathan Teodosio (nteodosio)]

 > Denn meine Aktualisierungsverwaltung hakt nach wie vor.

Derzeit ist das Update nur für Jammy veröffentlicht (was ist deine
Version?) und allerdings nur vor kurzer Zeit. Das heißt, dass es noch
nicht für jedermann verfügbar ist (c.f. phased updates[1]).

[1] https://wiki.ubuntuusers.de/Aktualisierungen/phased_update/

[Nathan Teodosio (nteodosio)]

SRU Team: The phasing in 22.04 was blocked due to LP:2004482. That is not a problem of update-manager though, but caused by a namespace conflict between ubuntu-advantage-tools and opcua-client-gui when installed via Pip[1].

I think therefore that the phasing should be continued.

[1] https://github.com/canonical/ubuntu-pro-client/issues/2434