[Ubuntu 18.04][sosreport] caught exception in sosreport at plugin method "logs.collect()"

[bugproxy]

== Comment: #0 - ABDUL HALEEM <email address hidden> - 2018-03-16 01:59:41 ==
Problem:
------------
While running sosreport on ubuntu 18.04 powerpc machine it got exception when file is large

  Running 27/60: logs... caught exception in plugin method "logs.collect()"
writing traceback to sos_logs/logs-plugin-errors.txt
  Running 30/60: lxd...

[plugin:lxd] command 'lxc list' timed out after 300s
[plugin:lxd] command 'lxc profile list' timed out after 300s

Machine : Witherspoon bare-metal DD2.2
 sosreport 3.5-1ubuntu2

uname -a
Linux ltciofvtr-spoon1 4.15.0-12-generic #13-Ubuntu SMP Wed Mar 7 21:37:03 UTC 2018 ppc64le
ppc64le ppc64le GNU/Linux

# free -h
              total used free shared buff/cache available
Mem: 503G 7.1G 493G 21M 2.4G 493G
Swap: 2.0G 0B 2.0G

- - - steps to reproduce - - -
 sosreport

sosreport (version 3.5)

This command will collect system configuration and diagnostic
information from this Ubuntu system. An archive containing the collected
information will be generated in /tmp/sos.2_u6n8p3.

For more information on Ubuntu visit:

  http://www.ubuntu.com/

The generated archive may contain data considered sensitive and its
content should be reviewed by the originating organization before being
passed to any third party.

No changes will be made to system configuration.

Press ENTER to continue, or CTRL-C to quit.

Please enter your first initial and last name [ltciofvtr-spoon1]:
Please enter the case id that you are generating this report for []:

 Setting up archive ...
 Setting up plugins ...

 Running plugins. Please wait ...

  Running 27/60: logs... caught exception in plugin method "logs.collect()"
writing traceback to sos_logs/logs-plugin-errors.txt
  Running 30/60: lxd...

[plugin:lxd] command 'lxc list' timed out after 300s
[plugin:lxd] command 'lxc profile list' timed out after 300s
[plugin:lxd] command 'lxc image list' timed out after 300s
  Running 60/60: xfs...
Creating compressed archive...

Your sosreport has been generated and saved in:
  /tmp/sosreport-ltciofvtr-spoon1-20180316023240.tar.xz

The checksum is: 8a2d4464d0ee94a306f6529a6a92f4e6

Please send this file to your support representative.

== Comment: #6 - ABDUL HALEEM <email address hidden> - 2018-04-27 00:18:45 ==
Machine is available only for today for debugging and I am still able to reproduce this bug on 4.15.0-17-generic

Please mirror this as soon as possible.

root@ltciofvtr-spoon1:~# sosreport

sosreport (version 3.5)

This command will collect system configuration and diagnostic
information from this Ubuntu system. An archive containing the collected
information will be generated in /tmp/sos.7debi_it.

For more information on Ubuntu visit:

  http://www.ubuntu.com/

The generated archive may contain data considered sensitive and its
content should be reviewed by the originating organization before being
passed to any third party.

No changes will be made to system configuration.

Press ENTER to continue, or CTRL-C to quit.

Please enter your first initial and last name [ltciofvtr-spoon1]:
Please enter the case id that you are generating this report for []:

 Setting up archive ...
 Setting up plugins ...
 Running plugins. Please wait ...

  Running 29/63: logs... caught exception in plugin method "logs.collect()"
writing traceback to sos_logs/logs-plugin-errors.txt
  Running 57/63: systemd...

== Comment: #8 - Urvashi Jawere <email address hidden> - 2018-04-27 06:46:57 ==
Hi Abdul,

I tried with a small fix and I was able to create the sosreport. Python 3 doesn't have decode anymore.
Attaching the patch fix with the bug.

== Comment: #9 - Urvashi Jawere <email address hidden> - 2018-04-27 06:47:33 ==

== Comment: #10 - Urvashi Jawere <email address hidden> - 2018-04-27 06:50:53 ==
After fix, I was able to collect sosreport from machine-

root@ltciofvtr-spoon1:/usr/share/sosreport/sos/plugins# sosreport

sosreport (version 3.5)

This command will collect system configuration and diagnostic
information from this Ubuntu system. An archive containing the collected
information will be generated in /tmp/sos.4i288ell.

For more information on Ubuntu visit:

  http://www.ubuntu.com/

The generated archive may contain data considered sensitive and its
content should be reviewed by the originating organization before being
passed to any third party.

No changes will be made to system configuration.

Press ENTER to continue, or CTRL-C to quit.

Please enter your first initial and last name [ltciofvtr-spoon1]: try
Please enter the case id that you are generating this report for []: bugfix

 Setting up archive ...
 Setting up plugins ...
 Running plugins. Please wait ...

  Running 63/63: xfs...
Creating compressed archive...

Your sosreport has been generated and saved in:
  /tmp/sosreport-try.bugfix-20180427071050.tar.xz

The checksum is: 03df61167b547a242ce3f18b588f3c47

Please send this file to your support representative.

root@ltciofvtr-spoon1:/usr/share/sosreport/sos/plugins# ls

== Comment: #11 - Urvashi Jawere <email address hidden> - 2018-04-30 01:32:55 ==

[bugproxy (bugproxy)]

Default Comment by Bridge

[bugproxy (bugproxy)]

Default Comment by Bridge

[Eric Desrochers (slashd)]

I was able to reproduce on powerpc ARCH by creating a fake "kern.log" of 1GB size.

Running 27/57: logs...
caught exception in plugin method "logs.collect()"
writing traceback to sos_logs/logs-plugin-errors.txt

# sos_logs/logs-plugin-errors.txt
Traceback (most recent call last):
  File "/usr/share/sosreport/sos/sosreport.py", line 1300, in collect
    plug.collect()
  File "/usr/share/sosreport/sos/plugins/__init__.py", line 877, in collect
    self._collect_strings()
  File "/usr/share/sosreport/sos/plugins/__init__.py", line 860, in _collect_strings
    (content.splitlines()[0]).decode('utf8', 'ignore'))
AttributeError: 'str' object has no attribute 'decode'

[Eric Desrochers (slashd)]

Reproducible (same behaviour) on amd64 ARCH as well ^

caught exception in plugin method "logs.collect()"
writing traceback to sos_logs/logs-plugin-errors.txt

[Eric Desrochers (slashd)]

Did you consider submitting your proposal patch to sosreport upstream [https://github.com/sosreport/sos] to get the credit for it ?

Once merged upstream, I can submit it to debian and then upload into Ubuntu (cosmic and stable) in that same order.

Regards,
Eric

[Eric Desrochers (slashd)]

Can you also reproduce using the following argument :

--all-logs
              Tell plugins to collect all possible log data ignoring any size limits and including logs in non-default locations. This option may significantly increase the size of reports.

[Eric Desrochers (slashd)]

It passes just fine for me if using "--all-logs" with the same reproducer.

$ sosreport --all-logs
  Running 27/57: logs...
  Running 28/57: lsbrelease...
  Running 29/57: lvm2...
  Running 30/57: lxd...

[Eric Desrochers (slashd)]

I will handle the upstream work on your behalf.

I have reported the situation upstream:
https://github.com/sosreport/sos/issues/1301

- Eric

[Eric Desrochers (slashd)]

I found out that the bug already exist LP[1] and Upstream[2]

[1] - Launchpad:
-------------------------------
https://bugs.launchpad.net/ubuntu/+source/sosreport/+bug/1761442
-------------------------------

[2] - Upstream:
-------------------------------
https://github.com/sosreport/sos/pull/1267
https://github.com/sosreport/sos/pull/1267/commits/bf8433945aeced7c0d73686ec83d826cc33bbd28

Looks like a duplicate of #1267 - fix is in the review queue, should land next week with any luck.

We're planning to make a 3.5.1 interim release shortly to help distro packagers who cannot wait until 3.6 final is out - this fix will be included in that tag.
-------------------------------

[Eric Desrochers (slashd)]

A colleague of mine (the one who own the original LP and upstream bug) and myself will upload the change in Ubuntu as soon as it got release upstream.

- Eric

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-06-13 06:51 EDT-------
(In reply to comment #15)
> Did you consider submitting your proposal patch to sosreport upstream
> [https://github.com/sosreport/sos] to get the credit for it ?
>
> Once merged upstream, I can submit it to debian and then upload into Ubuntu
> (cosmic and stable) in that same order.
>
> Regards,
> Eric

ok, Eric, will do this