UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte

[Seth Arnold]

[Impact]
This bug affects anyone trying to report a bug about ubuntu-advantage-tools using apport if any of the status/log files in their system contains non-utf8 characters. Although this is potentially a rare situation, there is no reason to fail because of that.

[Test Case]
<TODO>

[Regression Potential]
In order to mitigate the regression potential, the results of the
aforementioned tests are attached to this bug.

As the change here is making an except block broader than it was before, we could potentially be ignoring other bugs not related to this that would eventually happen in the apport call. However, we want to ignore any file which is not collected for any reason no matter what, so this is not considered a major problem.

[Discussion]
The load_file function expects utf-8 because some strings must be redacted before posted to Launchpad. Trying to get those files without decoding them and checking for the strings could potentially lead to leaks, which would be way worse than not getting the file at all.

If bug information is missing because we ignore some file(s), we can ask the reporter for more information in the bug itself.

[Original Description]

Hello, ubuntu-advantage-tools version 27.11.2~18.04.1 has some unfortunate python problems in the apport package hooks that complicates filing bugs against ubuntu-advantage-tools on 18.04 LTS:

$ sudo ubuntu-bug ubuntu-advantage-tools

*** Collecting problem information

The collected information can be sent to the developers to improve the
application. This might take a few minutes.
..............ERROR: hook /usr/share/apport/package-hooks/source_ubuntu-advantage-tools.py crashed:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/apport/report.py", line 218, in _run_hook
    symb['add_info'](report, ui)
  File "/usr/share/apport/package-hooks/source_ubuntu-advantage-tools.py", line 14, in add_info
    collect_logs(cfg, output_dir)
  File "/usr/lib/python3/dist-packages/uaclient/actions.py", line 221, in collect_logs
    content = system.load_file(f)
  File "/usr/lib/python3/dist-packages/uaclient/system.py", line 350, in load_file
    return content.decode("utf-8")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte
..

*** Send problem report to the developers?

After the problem report has been sent, please fill out the form in the
automatically opened web browser.

What would you like to do? Your options are:
  S: Send report (4.7 KB)
  V: View report
  K: Keep report file for sending later or copying to somewhere else
  I: Cancel and ignore future crashes of this program version
  C: Cancel
Please choose (S/V/K/I/C):

Thanks

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Renan Rodrigo (renanrodrigo)]

Hello, Seth,
Thanks for reporting this bug.

We could not reproduce this bug - do you happen to know, from the files we are trying to collect, which one has non-utf8 characters?

Regardless, this is recognized as a bug and a fix is on its way.

[Seth Arnold (seth-arnold)]

I'm having trouble even finding these files :(

Here's the snippet of the system's /usr/share/apport/package-hooks/source_ubuntu-advantage-tools.py :

        auto_include_log_files = [
            "cloud-id.txt",
            "cloud-id.txt-error",
            "ua-status.json",
            "ua-status.json-error",
            "livepatch-status.txt",
            "livepatch-status.txt-error",

I can't find these with locate; I used find to look for ua-status.json over the whole filesystem just in case locate missed it for some reason:

root@hunt:/var# find / -name '*ua-status*'
find: ‘/run/user/1000/gvfs’: Permission denied
root@hunt:/var#

root@hunt:/var# find / -name '*livepatch-status*'
find: ‘/run/user/1000/gvfs’: Permission denied
root@hunt:/var#

So I'm afraid I really don't know what exact data causes the trouble.

Thanks

[Robie Basak (racb)]

Hello Seth, or anyone else affected,

Accepted ubuntu-advantage-tools into kinetic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubuntu-advantage-tools/27.11.3~22.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-kinetic to verification-done-kinetic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-kinetic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Grant Orndorff (orndorffgrant)]

I have tested the version of ubuntu-advantage-tools in -proposed and can confirm that it fixes this issue.

Attaching the log of the script I used to test this and marking verification done.

[Launchpad Janitor (janitor)]

This bug was fixed in the package ubuntu-advantage-tools - 27.11.3~22.10.1

---------------
ubuntu-advantage-tools (27.11.3~22.10.1) kinetic; urgency=medium

  * New upstream release 27.11.3: (LP: #1993006)
    - d/postinst: remove the Ubuntu Pro beta apt message and set up the
      configurable flag for "APT news" instead
    - collect-logs: do not fail if a file cannot be read (LP: #1991858)
    - config: add a flag to disable "APT news" (LP: 1992026)
    - messaging: add announcement of "APT news" to apt output
    - messaging: only show "APT news" when using apt binary (GH: #2288)
    - version: use /run instead of /tmp for version file (GH: #2294)

 -- Grant Orndorff <email address hidden> Fri, 14 Oct 2022 16:55:27 -0400

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for ubuntu-advantage-tools has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package ubuntu-advantage-tools - 27.11.3~22.04.1

---------------
ubuntu-advantage-tools (27.11.3~22.04.1) jammy; urgency=medium

  * Backport new upstream release: (LP: #1993006) to jammy

 -- Grant Orndorff <email address hidden> Tue, 25 Oct 2022 12:46:23 -0400