uncaught python exception : UnicodeDecodeError: 'utf-8' codec can't decode byte; invalid continuation byte

Bug #1766337 reported by Laurent Bonnaud on 2018-04-23
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
apport (Ubuntu)
High
Brian Murray
Bionic
High
Brian Murray

Bug Description

[Impact]
Ubuntu crash reports and package installation failures may not include DpkgTerminalLog.txt if there is a UnicodeDecodeError trying to read the dpkg log file.

[Test Case]
Because the fix is simple and its complicated to have a dpkg log file with unicode in it we'll just test that truncating of a dpkg log file still works.

1) Install 2vcard (so we have a recent dpkg log entry)
2) Run update-manager
3) Kill it with signal 11 e.g. kill -11 $PID of update-manager
4) Run ubuntu-bug /var/crash/_usr_bin_update-manager.1000.crash
5) Observe DpkgTerminalLog.txt only contains a log entry regarding the installation of 2vcard and matches the content in /var/log/apt/term.log.

[Regression Potential]
Its possible that the code to replace the DpkgTerminalLog contents of the report is wrong but the test case above ensures that it isn't.

[Original Description]
Hi,

here is a python exception I got while reporting a bug:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/apport/report.py", line 198, in _run_hook
    symb['add_info'](report, ui)
  File "/usr/share/apport/general-hooks/ubuntu.py", line 89, in add_info
    check_attachment_for_errors(report, attachment)
  File "/usr/share/apport/general-hooks/ubuntu.py", line 208, in check_attachment_for_errors
    trim_dpkg_log(report)
  File "/usr/share/apport/general-hooks/ubuntu.py", line 508, in trim_dpkg_log
    report['DpkgTerminalLog'] = '\n'.join([str(line.decode('utf-8')) for line in lines])
  File "/usr/share/apport/general-hooks/ubuntu.py", line 508, in <listcomp>
    report['DpkgTerminalLog'] = '\n'.join([str(line.decode('utf-8')) for line in lines])
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe2 in position 3070: invalid continuation byte

It can also be seen here:

  https://launchpadlibrarian.net/367002573/HookError_ubuntu.txt

ProblemType: BugDistroRelease: Ubuntu 18.04
Package: apport 2.20.9-0ubuntu6
Uname: Linux 4.16.3-041603-generic x86_64
ApportLog:

ApportVersion: 2.20.9-0ubuntu6
Architecture: amd64
CurrentDesktop: MATE
Date: Mon Apr 23 20:13:24 2018
PackageArchitecture: all
ProcEnviron:
 TERM=xterm
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bashSourcePackage: apport
UpgradeStatus: No upgrade log present (probably fresh install)

tags: added: id-5ae35ea55ac10e9203cade8d
Changed in apport (Ubuntu):
status: New → In Progress
importance: Undecided → High
assignee: nobody → Brian Murray (brian-murray)
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package apport - 2.20.10-0ubuntu1

---------------
apport (2.20.10-0ubuntu1) cosmic; urgency=medium

  * New upstream release:
    - backends/packaging-apt-dpkg.py: when using a permanent sandbox better
      check for the package being already available thereby reducing quantity
      of downloads.
    - apport/report.py: Use timeout parameter in subprocess to limit how long
      we'll wait for gdb to return information as gdb can hang with some core
      files.
    - data/apport: Fix PEP8 issues
    - apport/ui.py: Include ProblemType in reports which are updated as
      package hooks may expect the report to have a ProblemType. (LP: #1766794)
    - test/test_ui.py: modify run_crash_kernel test to account for the fact
      that linux-image-$kvers-$flavor is now built from the linux-signed
      source package on amd64 and ppc64el. (LP: #1766740)
  * data/general-hooks/ubuntu.py: Fix UnicodeDecodeError when processing
    DpkgTerminalLog. (LP: #1766337)
  * data/general-hooks/ubuntu.py: Don't display a messy error if python or
    python3 is not installed. (LP: #1769262)
  * debian/apport.links: source_linux-signed.py -> source_linux.py package
    hook (LP: #1766740)
  * data/package-hooks/source_linux.py: Redirect bugs filed about the
    linux-signed source package to linux. (LP: #1766740)

 -- Brian Murray <email address hidden> Thu, 10 May 2018 09:31:34 -0700

Changed in apport (Ubuntu):
status: In Progress → Fix Released

Thank you for the fix!

Changed in apport (Ubuntu Bionic):
status: New → In Progress
importance: Undecided → High
assignee: nobody → Brian Murray (brian-murray)
Changed in apport (Ubuntu Bionic):
status: In Progress → Triaged
assignee: Brian Murray (brian-murray) → nobody
Changed in apport (Ubuntu Bionic):
assignee: nobody → Brian Murray (brian-murray)
status: Triaged → In Progress
description: updated

Hello Laurent, or anyone else affected,

Accepted apport into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/apport/2.20.9-0ubuntu7.6 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in apport (Ubuntu Bionic):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-bionic
Brian Murray (brian-murray) wrote :

DpkgTerminalLog.txt still appears in an update-manager crash report using the version of apport from bionic-proposed.

bdmurray@clean-bionic-amd64:~$ grep -A4 DpkgTerminal /var/crash/_usr_bin_update-manager.1000.crash
DpkgTerminalLog.txt:
 Log started: 2019-03-05 09:00:54
(Reading database ... 166173 files and directories currently installed.)
 Removing apport-noui (2.20.9-0ubuntu7.6) ...
 Failed to stop apport-noui.path: Unit apport-noui.path not loaded.

bdmurray@clean-bionic-amd64:~$ apt-cache policy apport
apport:
  Installed: 2.20.9-0ubuntu7.6
  Candidate: 2.20.9-0ubuntu7.6

tags: added: verification-done verification-done-bionic
removed: verification-needed verification-needed-bionic
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package apport - 2.20.9-0ubuntu7.6

---------------
apport (2.20.9-0ubuntu7.6) bionic; urgency=medium

  * data/general-hooks/ubuntu.py: Fix UnicodeDecodeError when processing
    DpkgTerminalLog. (LP: #1766337)
  * debian/apport-autoreport.service: Since this calls whoopsie-upload-all
    which fails if whoopsie isn't running, it should have a relationship with
    whoopsie.service. (LP: #1787729)

 -- Brian Murray <email address hidden> Wed, 06 Feb 2019 10:46:53 -0800

Changed in apport (Ubuntu Bionic):
status: Fix Committed → Fix Released

The verification of the Stable Release Update for apport has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers