apport hook fails in add_info with TypeError: 'str' does not support the buffer interface

[Jean-Baptiste Lallement]

Impact
------
Some apport package hooks are not able to check the contents of log files for errors, e.g. looking for hard disk errors in ubiquity install logs, so we may be getting crash reports that we don't want.

Test Case
---------
⟫ ubuntu-bug ubiquity
dpkg-query: aucun paquet ne correspond à ubiquity
ERROR: hook /usr/share/apport/package-hooks/source_ubiquity.py crashed:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/apport/report.py", line 197, in _run_hook
    symb['add_info'](report, ui)
  File "/usr/share/apport/package-hooks/source_ubiquity.py", line 61, in add_info
    if 'Buffer I/O error on device' in syslog:
TypeError: 'str' does not support the buffer interface

With the version of the package in the archive you'll see the above Traceback, with the version of proposed you will not.

Regression Potential
--------------------
The change uses a try, except clause to do the decoding of the log file so there should be little chance of a regression.

ProblemType: BugDistroRelease: Ubuntu 15.04
Package: ubiquity (not installed)
ProcVersionSignature: Ubuntu 3.19.0-14.14-generic 3.19.3
Uname: Linux 3.19.0-14-generic x86_64
ApportVersion: 2.17.2-0ubuntu1
Architecture: amd64
CurrentDesktop: Unity
Date: Tue Apr 21 10:15:56 2015
InstallationDate: Installed on 2013-09-03 (594 days ago)
InstallationMedia: Ubuntu 13.10 "Saucy Salamander" - Alpha amd64 (20130902)SourcePackage: ubiquity
UpgradeStatus: No upgrade log present (probably fresh install)

[Brian Murray (brian-murray)]

From the research I've done, it looks like this is a result of the following change:

 $ bzr log -r1369.34.1105 -p
------------------------------------------------------------
revno: 1369.34.1105
fixes bug: https://launchpad.net/bugs/1370259
committer: Martin Pitt <email address hidden>
branch nick: apport
timestamp: Thu 2015-01-08 11:20:16 +0100
message:
  * hookutils, attach_root_command_outputs(): Fix UnicodeDecodeError crash for non-textual values. (LP: #1370259)
diff:
=== modified file 'NEWS'
--- NEWS 2014-12-19 06:43:26 +0000
+++ NEWS 2015-01-08 10:20:16 +0000
@@ -1,6 +1,11 @@
 This file summarizes the major and interesting changes for each release. For a
 detailed list of changes, please see ChangeLog.

+2.15.2 (UNRELEASED)
+------------------
+ * hookutils, attach_root_command_outputs(): Fix UnicodeDecodeError crash for
+ non-textual values. (LP: #1370259)
+
 2.15.1 (2014-12-19)
 -------------------
  * Robustify report.test_get_timestamp test.

=== modified file 'apport/hookutils.py'
--- apport/hookutils.py 2014-04-29 17:34:35 +0000
+++ apport/hookutils.py 2015-01-08 10:20:16 +0000
@@ -465,7 +465,7 @@
         # now read back the individual outputs
         for keyname in command_map:
             try:
- with open(os.path.join(workdir, keyname)) as f:
+ with open(os.path.join(workdir, keyname), 'rb') as f:
                     buf = f.read().strip()
             except IOError:
                 # this can happen if the user dismisses authorization in

This then makes the files added by the ubiquity hook bytes, which when then can't check for strings in.

[Martin Pitt (pitti)]

Fixed in http://bazaar.launchpad.net/~apport-hackers/apport/trunk/revision/3081 . This now reverts back to the pre-2.15.2 behaviour of returing str, unless the value is not UTF-8. Some package hooks might get confused by this, so we should not SRU this right away. Or at least we need to audit the usage of attach_root_command_outputs() that its results are not assumed to be bytes.

[Brian Murray (brian-murray)]

I looked at the package hooks I have installed to see how they use attach_root_command_outputs() and if any post-processing is done.

source_unity-system-compositor.py - adds files
source_upstart.py - adds files
source_update-manager.py - adds files
source_ubiquity.py - searches files added
source_lightdm.py - adds files
source_ubuntu-release-upgrader.py - adds files
source_synaptic.py - adds files
source_xorg.py - adds files
source_plymouth.py - adds files
source_shim.py -adds files
mountall.py - adds files
source_debian-installer.py - adds files

I think this will be okay to SRU.

[Martin Pitt (pitti)]

Great, thanks. I only did a few spot checks yesterday and they looked unaffected too.

[Martin Pitt (pitti)]

I cherry-picked the commit into the xenial branch, queueing for the next SRU.

[Brian Murray (brian-murray)]

This change was uploaded to Yakkety and Zesty, so marking as Fix Released.

[Brian Murray (brian-murray)]

This seems like something worth fixing for the 16.04.2 point release, so I'll just SRU this now.

[Chris J Arges (arges)]

Hello Jean-Baptiste, or anyone else affected,

Accepted apport into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/apport/2.20.1-0ubuntu2.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Brian Murray (brian-murray)]

I utilized the version of the package from -proposed and did not receive a crash.

 $ PYTHONPATH='' ubuntu-bug ubiquity
/usr/share/apport/apport-gtk:16: PyGIWarning: Wnck was imported without specifying a version first. Use gi.require_version('Wnck', '3.0') before import to ensure that the right version gets loaded.
  from gi.repository import GLib, Wnck, GdkX11, Gdk
dpkg-query: no packages found matching ubiquity

 $ apt-cache policy apport
apport:
  Installed: 2.20.1-0ubuntu2.2
  Candidate: 2.20.1-0ubuntu2.2
  Version table:
 *** 2.20.1-0ubuntu2.2 500
        500 http://archive.ubuntu.com/ubuntu xenial-proposed/main amd64 Packages
        500 http://archive.ubuntu.com/ubuntu xenial-proposed/main i386 Packages
        100 /var/lib/dpkg/status
     2.20.1-0ubuntu2.1 500
        500 http://192.168.10.7/ubuntu xenial-updates/main amd64 Packages
        500 http://192.168.10.7/ubuntu xenial-updates/main i386 Packages
     2.20.1-0ubuntu2 500
        500 http://192.168.10.7/ubuntu xenial/main amd64 Packages
        500 http://192.168.10.7/ubuntu xenial/main i386 Packages

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for apport has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package apport - 2.20.1-0ubuntu2.2

---------------
apport (2.20.1-0ubuntu2.2) xenial; urgency=medium

  [ Martin Pitt ]
  * hookutils, attach_root_command_outputs(): Return str again, like before
    2.15.2. (LP: #1446537)

 -- Brian Murray <email address hidden> Tue, 29 Nov 2016 10:17:51 -0800