pro-client execs `apt list --installed` instead of using python3-apt - leading to utf-8 errors

[sistemas_grupo-cps]

[ Impact ]

If for any reason the user's system has any non-utf8 character as part of the `apt list --installed` command, the Pro Client will fail to parse the return output and break.

Independent of why this happens, the Pro Client should not be relying on CLI output when there are Python bindings available to perform the same task - and python3-apt provides ways to list installed packages.

The fix to this bug is to rely on the programmatic integration with APT instead of calling the CLI.

[ Test Plan ]

Listing installed packages is part of the regular operation of the Pro Client in many scenarios covered by the integration / acceptance tests.

For this particular issue:
- Launch a container for the release to be tested
- Install a non-utf-8 locale and set it as the default. e.g.
    - sudo locale-gen fr_FR
    - sudo dpkg-reconfigure locales # select fr_FR
    - # log out and log in
- pro status
- see it fail
- install the package from -proposed
- see it does not fail anymore

[ Where problems could occur ]

Switching from the CLI call to python3-apt brings the risk of wrong implementation of the integration, which would lead to a different package list to be returned by python3-apt. The regression potential is mitigated by extensive testing - unit, integration and acceptance tests rely on this functionality.

[ Original description ]

["2024-04-10T08:28:17.952", "DEBUG", "ubuntupro.system", "load_file", 517, "Reading file: /etc/os-release", {}]
["2024-04-10T08:28:17.953", "DEBUG", "ubuntupro.system", "load_file", 517, "Reading file: /proc/version_signature", {}]
["2024-04-10T08:28:18.575", "ERROR", "ubuntupro.cli", "wrapper", 1768, "Unhandled exception, please file a bug", {"exc_info": "Traceback (most recent call
 last):\n File \"/usr/lib/python3/dist-packages/uaclient/cli/__init__.py\", line 1705, in wrapper\n return func(*args, **kwargs)\n File \"/usr/lib/py
thon3/dist-packages/uaclient/cli/__init__.py\", line 1838, in main\n return_value = args.action(args, cfg=cfg, extra_args=extra_args)\n File \"/usr/li
b/python3/dist-packages/uaclient/cli/__init__.py\", line 1475, in action_status\n status, ret = actions.status(\n File \"/usr/lib/python3/dist-package
s/uaclient/actions.py\", line 173, in status\n status = ua_status.status(cfg=cfg, show_all=show_all)\n File \"/usr/lib/python3/dist-packages/uaclient/
status.py\", line 418, in status\n response = _unattached_status(cfg)\n File \"/usr/lib/python3/dist-packages/uaclient/status.py\", line 283, in _unat
tached_status\n resources = get_available_resources(cfg)\n File \"/usr/lib/python3/dist-packages/uaclient/contract.py\", line 680, in get_available_re
sources\n resources = client.available_resources()\n File \"/usr/lib/python3/dist-packages/uaclient/contract.py\", line 114, in available_resources\n
   activity_info = self._get_activity_info()\n File \"/usr/lib/python3/dist-packages/uaclient/contract.py\", line 413, in _get_activity_info\n \"deskt
op\": system.is_desktop(),\n File \"/usr/lib/python3/dist-packages/uaclient/system.py\", line 397, in is_desktop\n for package in apt.get_installed_pa
ckages():\n File \"/usr/lib/python3/dist-packages/uaclient/apt.py\", line 727, in get_installed_packages\n out, _ = system.subp([\"apt\", \"list\", \"
--installed\"])\n File \"/usr/lib/python3/dist-packages/uaclient/system.py\", line 703, in subp\n out, err = _subp(\n File \"/usr/lib/python3/dist-pa
ckages/uaclient/system.py\", line 649, in _subp\n out_result = out.decode(\"utf-8\") if out else \"\"\nUnicodeDecodeError: 'utf-8' codec can't decode b
yte 0xe1 in position 109: invalid continuation byte"}]
["2024-04-10T08:30:25.404", "DEBUG", "ubuntupro.config", "parse_config", 625, "Using client configuration file at /etc/ubuntu-advantage/uaclient.conf", {}
]
["2024-04-10T08:30:25.408", "DEBUG", "ubuntupro.system", "load_file", 517, "Reading file: /etc/ubuntu-advantage/uaclient.conf", {}]
["2024-04-10T08:30:25.409", "DEBUG", "ubuntupro.files.files", "read", 60, "Tried to load /var/lib/ubuntu-advantage/user-config.json but file does not exis
t", {}]
["2024-04-10T08:30:25.421", "DEBUG", "ubuntupro.http", "configure_web_proxy", 146, "Setting no_proxy: 169.254.169.254,[fd00:ec2::254],metadata", {}]
["2024-04-10T08:30:25.422", "DEBUG", "ubuntupro.http", "configure_web_proxy", 154, "Setting global proxy dict", {}]
["2024-04-10T08:30:25.422", "DEBUG", "ubuntupro.cli", "main", 1825, "Executed with sys.argv: ['/usr/bin/pro', 'attach', '<REDACTED>']", {}]
["2024-04-10T08:30:25.422", "DEBUG", "ubuntupro.files.files", "read", 60, "Tried to load /var/lib/ubuntu-advantage/private/machine-token.json but file doe
s not exist", {}]
["2024-04-10T08:30:25.422", "DEBUG", "ubuntupro.lock", "__enter__", 77, "spin lock starting for pro attach", {}]
["2024-04-10T08:30:25.423", "DEBUG", "ubuntupro.system", "write_file", 556, "Writing file /run/ubuntu-advantage/notices/60-operation_in_progress atomicall
y via tempfile /run/ubuntu-advantage/notices/tmparyyoirn", {}]
["2024-04-10T08:30:25.423", "DEBUG", "ubuntupro.system", "write_file", 556, "Writing file /var/lib/ubuntu-advantage/lock atomically via tempfile /var/lib/
ubuntu-advantage/tmpn1yxl91o", {}]
["2024-04-10T08:30:25.424", "DEBUG", "ubuntupro.system", "write_file", 556, "Writing file /run/ubuntu-advantage/notices/60-operation_in_progress atomicall
y via tempfile /run/ubuntu-advantage/notices/tmpwqyq85v1", {}]
["2024-04-10T08:30:25.424", "DEBUG", "ubuntupro.files.files", "read", 60, "Tried to load /var/lib/ubuntu-advantage/private/machine-token.json but file doe
s not exist", {}]
["2024-04-10T08:30:25.424", "DEBUG", "ubuntupro.system", "load_file", 517, "Reading file: /etc/machine-id", {}]
["2024-04-10T08:30:25.424", "DEBUG", "ubuntupro.system", "load_file", 517, "Reading file: /etc/os-release", {}]
["2024-04-10T08:30:25.425", "DEBUG", "ubuntupro.system", "load_file", 517, "Reading file: /proc/version_signature", {}]
["2024-04-10T08:30:25.970", "DEBUG", "ubuntupro.system", "ensure_file_absent", 576, "Removed file: /run/ubuntu-advantage/notices/60-operation_in_progress"
, {}]
["2024-04-10T08:30:25.970", "DEBUG", "ubuntupro.system", "ensure_file_absent", 576, "Removed file: /var/lib/ubuntu-advantage/lock", {}]
["2024-04-10T08:30:25.971", "ERROR", "ubuntupro.cli", "wrapper", 1768, "Unhandled exception, please file a bug", {"exc_info": "Traceback (most recent call
 last):\n File \"/usr/lib/python3/dist-packages/uaclient/cli/__init__.py\", line 1705, in wrapper\n return func(*args, **kwargs)\n File \"/usr/lib/py
thon3/dist-packages/uaclient/cli/__init__.py\", line 1838, in main\n return_value = args.action(args, cfg=cfg, extra_args=extra_args)\n File \"/usr/li
b/python3/dist-packages/uaclient/cli/__init__.py\", line 257, in new_f\n return f(args, cfg=cfg, **kwargs)\n File \"/usr/lib/python3/dist-packages/uac
lient/cli/__init__.py\", line 202, in new_f\n return f(*args, **kwargs)\n File \"/usr/lib/python3/dist-packages/uaclient/cli/__init__.py\", line 186,
in new_f\n retval = f(*args, cfg=cfg, **kwargs)\n File \"/usr/lib/python3/dist-packages/uaclient/cli/__init__.py\", line 1305, in action_attach\n a
ctions.attach_with_token(cfg, token=token, allow_enable=allow_enable)\n File \"/usr/lib/python3/dist-packages/uaclient/actions.py\", line 65, in attach_w
ith_token\n new_machine_token = contract_client.add_contract_machine(\n File \"/usr/lib/python3/dist-packages/uaclient/util.py\", line 92, in decorato
r\n return f(*args, **kwargs)\n File \"/usr/lib/python3/dist-packages/uaclient/contract.py\", line 92, in add_contract_machine\n activity_info = se
lf._get_activity_info()\n File \"/usr/lib/python3/dist-packages/uaclient/contract.py\", line 413, in _get_activity_info\n \"desktop\": system.is_deskt
op(),\n File \"/usr/lib/python3/dist-packages/uaclient/system.py\", line 397, in is_desktop\n for package in apt.get_installed_packages():\n File \"/
usr/lib/python3/dist-packages/uaclient/apt.py\", line 727, in get_installed_packages\n out, _ = system.subp([\"apt\", \"list\", \"--installed\"])\n Fi
le \"/usr/lib/python3/dist-packages/uaclient/system.py\", line 703, in subp\n out, err = _subp(\n File \"/usr/lib/python3/dist-packages/uaclient/syste
m.py\", line 649, in _subp\n out_result = out.decode(\"utf-8\") if out else \"\"\nUnicodeDecodeError: 'utf-8' codec can't decode byte 0xe1 in position
109: invalid continuation byte"}]
["2024-04-10T08:38:29.763", "DEBUG", "ubuntupro.config", "parse_config", 625, "Using client configuration file at /etc/ubuntu-advantage/uaclient.conf", {}
]
["2024-04-10T08:38:29.767", "DEBUG", "ubuntupro.system", "load_file", 517, "Reading file: /etc/ubuntu-advantage/uaclient.conf", {}]
["2024-04-10T08:38:29.768", "DEBUG", "ubuntupro.files.files", "read", 60, "Tried to load /var/lib/ubuntu-advantage/user-config.json but file does not exis
t", {}]
["2024-04-10T08:38:29.780", "DEBUG", "ubuntupro.http", "configure_web_proxy", 146, "Setting no_proxy: 169.254.169.254,[fd00:ec2::254],metadata", {}]
["2024-04-10T08:38:29.780", "DEBUG", "ubuntupro.http", "configure_web_proxy", 154, "Setting global proxy dict", {}]
["2024-04-10T08:38:29.780", "DEBUG", "ubuntupro.cli", "main", 1825, "Executed with sys.argv: ['/usr/bin/pro', 'attach', '<REDACTED>']", {}]
["2024-04-10T08:38:29.781", "DEBUG", "ubuntupro.files.files", "read", 60, "Tried to load /var/lib/ubuntu-advantage/private/machine-token.json but file doe
s not exist", {}]
["2024-04-10T08:38:29.781", "DEBUG", "ubuntupro.lock", "__enter__", 77, "spin lock starting for pro attach", {}]
["2024-04-10T08:38:29.781", "DEBUG", "ubuntupro.system", "write_file", 556, "Writing file /run/ubuntu-advantage/notices/60-operation_in_progress atomicall
y via tempfile /run/ubuntu-advantage/notices/tmplx6ehdag", {}]
["2024-04-10T08:38:29.782", "DEBUG", "ubuntupro.system", "write_file", 556, "Writing file /var/lib/ubuntu-advantage/lock atomically via tempfile /var/lib/
ubuntu-advantage/tmp6ml_ltfj", {}]
["2024-04-10T08:38:29.782", "DEBUG", "ubuntupro.system", "write_file", 556, "Writing file /run/ubuntu-advantage/notices/60-operation_in_progress atomicall
y via tempfile /run/ubuntu-advantage/notices/tmpvxf_b1v0", {}]
["2024-04-10T08:38:29.783", "DEBUG", "ubuntupro.files.files", "read", 60, "Tried to load /var/lib/ubuntu-advantage/private/machine-token.json but file doe
s not exist", {}]
["2024-04-10T08:38:29.783", "DEBUG", "ubuntupro.system", "load_file", 517, "Reading file: /etc/machine-id", {}]
["2024-04-10T08:38:29.783", "DEBUG", "ubuntupro.system", "load_file", 517, "Reading file: /etc/os-release", {}]
["2024-04-10T08:38:29.784", "DEBUG", "ubuntupro.system", "load_file", 517, "Reading file: /proc/version_signature", {}]
["2024-04-10T08:38:30.366", "DEBUG", "ubuntupro.system", "ensure_file_absent", 576, "Removed file: /run/ubuntu-advantage/notices/60-operation_in_progress"
, {}]
["2024-04-10T08:38:30.367", "DEBUG", "ubuntupro.system", "ensure_file_absent", 576, "Removed file: /var/lib/ubuntu-advantage/lock", {}]
["2024-04-10T08:38:30.367", "ERROR", "ubuntupro.cli", "wrapper", 1768, "Unhandled exception, please file a bug", {"exc_info": "Traceback (most recent call
 last):\n File \"/usr/lib/python3/dist-packages/uaclient/cli/__init__.py\", line 1705, in wrapper\n return func(*args, **kwargs)\n File \"/usr/lib/py
thon3/dist-packages/uaclient/cli/__init__.py\", line 1838, in main\n return_value = args.action(args, cfg=cfg, extra_args=extra_args)\n File \"/usr/li
b/python3/dist-packages/uaclient/cli/__init__.py\", line 257, in new_f\n return f(args, cfg=cfg, **kwargs)\n File \"/usr/lib/python3/dist-packages/uac
lient/cli/__init__.py\", line 202, in new_f\n return f(*args, **kwargs)\n File \"/usr/lib/python3/dist-packages/uaclient/cli/__init__.py\", line 186,
in new_f\n retval = f(*args, cfg=cfg, **kwargs)\n File \"/usr/lib/python3/dist-packages/uaclient/cli/__init__.py\", line 1305, in action_attach\n a
ctions.attach_with_token(cfg, token=token, allow_enable=allow_enable)\n File \"/usr/lib/python3/dist-packages/uaclient/actions.py\", line 65, in attach_w
ith_token\n new_machine_token = contract_client.add_contract_machine(\n File \"/usr/lib/python3/dist-packages/uaclient/util.py\", line 92, in decorato
r\n return f(*args, **kwargs)\n File \"/usr/lib/python3/dist-packages/uaclient/contract.py\", line 92, in add_contract_machine\n activity_info = se
lf._get_activity_info()\n File \"/usr/lib/python3/dist-packages/uaclient/contract.py\", line 413, in _get_activity_info\n \"desktop\": system.is_deskt
op(),\n File \"/usr/lib/python3/dist-packages/uaclient/system.py\", line 397, in is_desktop\n for package in apt.get_installed_packages():\n File \"/
usr/lib/python3/dist-packages/uaclient/apt.py\", line 727, in get_installed_packages\n out, _ = system.subp([\"apt\", \"list\", \"--installed\"])\n Fi
le \"/usr/lib/python3/dist-packages/uaclient/system.py\", line 703, in subp\n out, err = _subp(\n File \"/usr/lib/python3/dist-packages/uaclient/syste
m.py\", line 649, in _subp\n out_result = out.decode(\"utf-8\") if out else \"\"\nUnicodeDecodeError: 'utf-8' codec can't decode byte 0xe1 in position
109: invalid continuation byte"}]

ProblemType: Bug
DistroRelease: Ubuntu 20.04
Package: ubuntu-advantage-tools 31.2~20.04
ProcVersionSignature: Ubuntu 5.15.0-101.111~20.04.1-generic 5.15.143
Uname: Linux 5.15.0-101-generic x86_64
NonfreeKernelModules: falcon_lsm_serviceable falcon_nf_netcontain falcon_kal falcon_lsm_pinned_16407 falcon_lsm_pinned_16206
ApportVersion: 2.20.11-0ubuntu27.27
Architecture: amd64
CasperMD5CheckResult: skip
Date: Wed Apr 10 08:28:11 2024
PackageArchitecture: all
ProcEnviron:
 TERM=xterm
 PATH=(custom, no user)
 LANG=es_ES
 SHELL=/bin/bash
SourcePackage: ubuntu-advantage-tools
UpgradeStatus: No upgrade log present (probably fresh install)
apparmor_logs.txt:

cloud-id.txt-error: Invalid command specified 'cloud-id'.
livepatch-status.txt-error: Invalid command specified '/snap/bin/canonical-livepatch status'.
ua-status.json-error: Failed running command 'pro status --format json' [exit(1)]. Message:
uaclient.conf:
 contract_url: https://contracts.canonical.com
 log_level: debug

[Grant Orndorff (orndorffgrant)]

Hi sistemascps!

Thank you for the bug report and sorry for the delay in responding. It looks like a non-UTF-8 character is present in the output of `apt list --installed`.

The fix will be to replace the usage of `apt list --installed` with using the python3-apt library to get the list of installed packages.

[Launchpad Janitor (janitor)]

This bug was fixed in the package ubuntu-advantage-tools - 33

---------------
ubuntu-advantage-tools (33) oracular; urgency=medium

  * d/apparmor: adjust the esm_cache apparmor profile to allow reading of dpkg
    data directory (LP: #2067810) (GH: #3137)
  * New upstream release 33 (LP: #2069237)
    - apt: use Python bindings instead of apt CLI to query for installed
      packages (LP: #2060769) (LP: #2068744)
    - beta: drop support for beta services
    - contracts: add support for contracts which target a specific series
    - fips: change enable functionality to ensure all packages with a FIPS
      candidate are upgraded to the FIPS version (GH: #2667)
    - fix:
      + add the current_status field to the plan api return object
      + change recommended attach method to magic attach (GH: #3040)
    - livepatch: prefer the term 'coverage' instead of 'support' in messaging
      (GH: #3063)
    - realtime:
      + auto-select the raspi variant when appropriate
      + inform the user when auto-selecting a variant

 -- Renan Rodrigo <email address hidden> Thu, 13 Jun 2024 00:19:54 -0300

[Robie Basak (racb)]

Hello sistemas_grupo-cps, or anyone else affected,

Accepted ubuntu-advantage-tools into noble-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubuntu-advantage-tools/33~24.04 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-noble to verification-done-noble. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-noble. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Robie Basak (racb)]

Hello sistemas_grupo-cps, or anyone else affected,

Accepted ubuntu-advantage-tools into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubuntu-advantage-tools/33~22.04 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Robie Basak (racb)]

Hello sistemas_grupo-cps, or anyone else affected,

Accepted ubuntu-advantage-tools into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubuntu-advantage-tools/33~20.04 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Robie Basak (racb)]

Hello sistemas_grupo-cps, or anyone else affected,

Accepted ubuntu-advantage-tools into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubuntu-advantage-tools/33~18.04 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Robie Basak (racb)]

Hello sistemas_grupo-cps, or anyone else affected,

Accepted ubuntu-advantage-tools into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubuntu-advantage-tools/33~16.04 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.