apport fails to collect nvram specific information -- attach_root_command_outputs() does not work for binary data

Bug #1370259 reported by bugproxy on 2014-09-16
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
apport (Ubuntu)
Undecided
Martin Pitt
Utopic
Undecided
Unassigned

Bug Description

---Problem Description---
apport fails to collect nvram specific information

---uname output---
3.16.0-10-generic

Machine Type = POWER8

---Steps to Reproduce---

1) Install Ubuntu 14.10 in Power non-virtualized environment.
2) Install apport package
3) Run following command to verify if apport collects power specific information.

   apport-cli -f -p linux-generic --save=/home/sachin/linux.apport

The command fails with following error :

root@lep8d:/home/sachin# apport-cli -f -p linux-generic --save=/home/sachin/linux.apport

*** Collecting problem information

The collected information can be sent to the developers to improve the
application. This might take a few minutes.
.......tar: Removing leading `/' from member names
....ERROR: hook /usr/share/apport/general-hooks/powerpc.py crashed:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/apport/report.py", line 197, in _run_hook
    symb['add_info'](report, ui)
  File "/usr/share/apport/general-hooks/powerpc.py", line 72, in add_info
    'nvram': 'cat /dev/nvram',
  File "/usr/lib/python3/dist-packages/apport/hookutils.py", line 469, in attach_root_command_outputs
    buf = f.read().strip()
  File "/usr/lib/python3.4/codecs.py", line 313, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb5 in position 1: invalid start byte
...
root@lep8d:/home/sachin#

Userspace tool common name: apport

Following version of apport is installed.

root@lep8d:/home/sachin# dpkg --list | grep apport
ii apport 2.14.7-0ubuntu1 all automatically generate crash reports for debugging
ii apport-symptoms 0.20 all symptom scripts for apport
ii python-apport 2.14.7-0ubuntu1 all Python library for Apport crash report handling
ii python3-apport 2.14.7-0ubuntu1 all Python 3 library for Apport crash report handling
root@lep8d:/home/sachin#

Looking at the code we are doing a cat /dev/nvram to the apport output file which doesn't sounds right to me. I would think using the function add_tar() creating a tar file and copying the content as a ascii file at end of apport output file more appropriate.
One dummy question remaining, how getting the tar file from the ascii output file if we need to ? ( that's also true to validate the content of the DeviceTree tar file).

Would suggest following patch:
--- powerpc.py 2014-09-10 10:28:18.824002770 -0400
+++ powerpc.py.old 2014-09-04 06:33:16.000000000 -0400
@@ -69,8 +69,8 @@
                 'cpu_runmode': 'ppc64_cpu --run-mode',
                 'cpu_freq': 'ppc64_cpu --frequency',
                 'cpu_dscr': 'ppc64_cpu --dscr',
+ 'nvram': 'cat /dev/nvram',
             })
- add_tar(report, '/dev/nvram', 'nvram.tar')
         attach_file_if_exists(report, '/var/log/platform')

     if ispSeries and not isPowerKVM:

thanks
 diff -urN powerpc.py.old powerpc.py
--- powerpc.py.old 2014-09-04 06:33:16.000000000 -0400
+++ powerpc.py 2014-09-10 10:28:18.824002770 -0400
@@ -69,8 +69,8 @@
                 'cpu_runmode': 'ppc64_cpu --run-mode',
                 'cpu_freq': 'ppc64_cpu --frequency',
                 'cpu_dscr': 'ppc64_cpu --dscr',
- 'nvram': 'cat /dev/nvram',
             })
+ add_tar(report, '/dev/nvram', 'nvram.tar')
         attach_file_if_exists(report, '/var/log/platform')

     if ispSeries and not isPowerKVM:

Initial Launchpad bug id #1336462 implemented feature support

bugproxy (bugproxy) on 2014-09-16
tags: added: architecture-ppc64le bugnameltc-115470 severity-high targetmilestone-inin1410
Luciano Chavez (lnx1138) wrote :

Please ignore the first diff. The second diff is the correct one. Thanks.

affects: ubuntu → apport (Ubuntu)
Thierry FAUCK (thierry-j) wrote :

As fix is available changed status in Launchpad as I don't know how to do it from bugzilla

Changed in apport (Ubuntu):
status: New → Fix Committed
Thierry FAUCK (thierry-j) wrote :

 diff -urN powerpc.py.old powerpc.py
--- powerpc.py.old 2014-09-04 06:33:16.000000000 -0400
+++ powerpc.py 2014-09-10 10:28:18.824002770 -0400
@@ -69,8 +69,8 @@
                 'cpu_runmode': 'ppc64_cpu --run-mode',
                 'cpu_freq': 'ppc64_cpu --frequency',
                 'cpu_dscr': 'ppc64_cpu --dscr',
- 'nvram': 'cat /dev/nvram',
             })
+ add_tar(report, '/dev/nvram', 'nvram.tar')
         attach_file_if_exists(report, '/var/log/platform')

     if ispSeries and not isPowerKVM:

Changed in apport (Ubuntu):
assignee: nobody → Canonical Taco Screen Team (canonical-taco-screeners)

------- Comment From <email address hidden> 2014-10-16 04:15 EDT-------
Still not fixed with latest builds.

root@lep8d:~# dpkg -l | grep apport
iF apport 2.14.7-0ubuntu7 all automatically generate crash reports for debugging
ii apport-gtk 2.14.7-0ubuntu6 all GTK+ frontend for the apport crash report system
ii apport-symptoms 0.20 all symptom scripts for apport
ii python-apport 2.14.7-0ubuntu6 all Python library for Apport crash report handling
ii python3-apport 2.14.7-0ubuntu7 all Python 3 library for Apport crash report handling
root@lep8d:~# apport-cli -f -p linux-generic --save=/home/sachin/linux.apport

*** Collecting problem information

The collected information can be sent to the developers to improve the
application. This might take a few minutes.
...tar: Removing leading `/' from member names
....ERROR: hook /usr/share/apport/general-hooks/powerpc.py crashed:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/apport/report.py", line 197, in _run_hook
symb['add_info'](report, ui)
File "/usr/share/apport/general-hooks/powerpc.py", line 72, in add_info
'nvram': 'cat /dev/nvram',
File "/usr/lib/python3/dist-packages/apport/hookutils.py", line 469, in attach_root_command_outputs
buf = f.read().strip()
File "/usr/lib/python3.4/codecs.py", line 313, in decode
(result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb5 in position 1: invalid start byte
...
root@lep8d:~#

Martin, this appears to be a regression introduced by the hook added for bug #1336462. Could you check if you think the proposed patch is suitable?

Changed in apport (Ubuntu):
assignee: Canonical Taco Screen Team (canonical-taco-screeners) → Martin Pitt (pitti)
status: Fix Committed → Triaged
bugproxy (bugproxy) on 2014-12-12
tags: added: targetmilestone-inin1504
removed: targetmilestone-inin1410
Martin Pitt (pitti) wrote :

Sorry for the late answer, this fell through the cracks. Did you actually test that this works? I. e. can a normal user actually read /dev/nvram? Currently the "cat /dev/nvram" is ran as root via attach_root_command_outputs(), and requiring root was the main reason why this "cat" approach was taken in the first place.

Does /dev/nvram actually contain text information and is this a locale/encoding problem? Or is it binary data, and unrelated to encodings?

With 14.10's apport, instead of cat the hook could call "base64 /dev/nvram" or a similar command which provides plaintext. But otherwise I think it would be better to fix attach_root_command_outputs() to get along with binary outputs.

> One dummy question remaining, how getting the tar file from the ascii output file if we need to ?

You can use "apport-unpack" to decode the apport report file into individual files (named by keys), and then just use tar to extract them further.

summary: - apport fails to collect nvram specific information
+ apport fails to collect nvram specific information --
+ attach_root_command_outputs() does not work for binary data
Martin Pitt (pitti) wrote :

Fixed in trunk in http://bazaar.launchpad.net/~apport-hackers/apport/trunk/revision/2890

I tested it with /dev/nvram on amd64 (I just saw that this also exists there) and it works now. If you want you can apply this simple fix inline in /usr/lib/python3/dist-packages/apport/hookutils.py for testing, but I'll upload a stable release update for this too.

Changed in apport (Ubuntu):
status: Triaged → Fix Committed
Martin Pitt (pitti) wrote :

SRU uploaded, this now needs the SRU team to review/approve. Could you please test the package in -proposed once it arrives, to verify? Thanks!

Changed in apport (Ubuntu Utopic):
status: New → In Progress
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package apport - 2.15.1-0ubuntu2

---------------
apport (2.15.1-0ubuntu2) vivid; urgency=medium

  * Add systemd units for apport-noui.
  * Merge from trunk:
    - hookutils, attach_root_command_outputs(): Fix UnicodeDecodeError crash
      for non-textual values. (LP: #1370259)
 -- Martin Pitt <email address hidden> Thu, 08 Jan 2015 17:33:12 +0100

Changed in apport (Ubuntu):
status: Fix Committed → Fix Released

Hello bugproxy, or anyone else affected,

Accepted apport into utopic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/apport/2.14.7-0ubuntu8.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in apport (Ubuntu Utopic):
status: In Progress → Fix Committed
tags: added: verification-needed

------- Comment From <email address hidden> 2015-01-12 08:59 EDT-------
Latest apport package works correctly.

root@ubuntu:~# apport-cli -f -p linux-generic --save=/home/linux.apport

*** Collecting problem information

The collected information can be sent to the developers to improve the
application. This might take a few minutes.
..........................................tar: Removing leading `/' from member names
.........

root@ubuntu:~# dpkg -l | grep apport
ii apport 2.14.7-0ubuntu8.1 all automatically generate crash reports for debugging
ii apport-gtk 2.14.7-0ubuntu8 all GTK+ frontend for the apport crash report system
ii apport-symptoms 0.20 all symptom scripts for apport
ii python3-apport 2.14.7-0ubuntu8.1 all Python 3 library for Apport crash report handling
root@ubuntu:~#

Martin Pitt (pitti) wrote :

Thanks for testing!

tags: added: verification-done
removed: verification-needed
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package apport - 2.14.7-0ubuntu8.1

---------------
apport (2.14.7-0ubuntu8.1) utopic-proposed; urgency=medium

  * hookutils, attach_root_command_outputs(): Fix UnicodeDecodeError crash for
    non-textual values. Cherry-picked from trunk r2890. (LP: #1370259)
 -- Martin Pitt <email address hidden> Thu, 08 Jan 2015 11:23:53 +0100

Changed in apport (Ubuntu Utopic):
status: Fix Committed → Fix Released

The verification of the Stable Release Update for apport has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers