[sync][sru] Backport of sosreport v3.6

Bug #1775195 reported by Eric Desrochers on 2018-06-05
20
This bug affects 1 person
Affects Status Importance Assigned to Milestone
sosreport (Debian)
Fix Released
Unknown
sosreport (Ubuntu)
Medium
Eric Desrochers
Trusty
Wishlist
Eric Desrochers
Xenial
Wishlist
Eric Desrochers
Bionic
Wishlist
Eric Desrochers
Cosmic
Wishlist
Eric Desrochers

Bug Description

[Impact]

sosreport 3.6 has been released in June 2018 with further enhancements in core sosreport functionality.

I already did the request for debian :
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=900818

and 3.6 has been uploaded in Debian since then (Thanks to caribou):
sosreport | 3.6-1 | testing
sosreport | 3.6-1 | unstable

and I have synced sosreport (Debian->Ubuntu Disco).

It would be great to have sosreport v3.6 SRU'd in supported stable releases once official release upstream, considering the fact that the release (especially LTSes) will be supported for a couple of years still:

[https://www.ubuntu.com/about/release-cycle]
Ubuntu 16.04 | support until 2023
Ubuntu 18.04 | support until 2024

sosreport is widely use by Canonical support team to troubleshoot UA customer and other vendors and community users. These improvement will benefit all of them.

sosreport 3.6 contains a number of enhancements, new features, and bug fixes. (See "Release Note" below)

Just like we did for v3.5 (LP: #1734983)

[Test Case]

 * Install sosreport
 * Run sosreport
   - sosreport plugins are separated by subject (juju, MAAS, grub,zfs, ipsec, ...) and allow the capability to detect (based on file and package) if it exist and/or installed and then only run the necessary plugins based on the detection made.

[Things identified during our testing]

 * [minor] juju plugin detection doesn't work if installed as a snap:
   https://github.com/sosreport/sos/issues/1475
   Fix already submitted upstream. This is not a new thing, juju in
   sosreport have always been made for .deb so far. It's not a
   regression introduce in 3.6, and IMHO not a blocker for the current
   SRU. It could be patch later on when the upstream discussion/approval
   will be done.

 * [major] Certain plugins may or may not substitute sensible
   information such as password. Upstream fix found and applied :
   https://github.com/sosreport/sos/commit/b96bdab
   - d/p/fix-string-substitution-method.patch

 * [minor] kernel plugin dont collect some tracing instance files
   https://github.com/sosreport/sos/commit/d6379b5
   LP: #1803735
   - d/p/dont-collect-some-tracing-instance-files.patch

[Regression Potential]

 * Regression risk is low, as long as the core functionnality works (Package manager, ...)

 * autopkgtest reveal no regressions.

 * We did some dogfooding on sosreport, but we can't test each individual plugins and scenarios one by one, that would be impossible but we have tested the ones we considered important and Ubuntu related (canonical-livepatch, MAAS, juju, openstack, ...). Plugin bug is an eventuality, but they are usually easy to fix and the impact will be isolated to the plugin itself or section of the plugin. If a plugin has a bug the worst that could happen is that this particular plugin won't collect system logs, debug information or anything this plugin was instructed to do, usually won't affect other plugin to perform as expected and/or sosreport to complete.

[Other Info]

[v3.6 Release notes]
# https://github.com/sosreport/sos/releases/tag/3.6

[Original Description]

sosreport 3.6 has been released in June 2018 with further enhancements in core sosreport functionality.

A sync can be done since all Ubuntu patches has been integrated in 3.6

I already did the request for debian :
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=900818

and 3.6 has been uploaded in Debian since then (Thanks to caribou):
sosreport | 3.6-1 | testing
sosreport | 3.6-1 | unstable

Now, it would be great to have sosreport v3.6 in Development Release & SRU'd in all supported stable release once official release upstream, considering the fact that the release will contain a number of enhancements, new features, and bug fixes and that sosreport is widely use by Canonical support team to troubleshoot UA customer.

Just like we did for v3.5.

[v3.6 Release notes]

# https://github.com/sosreport/sos/releases/tag/3.6
The sos team is pleased to announce the release of sos-3.6. This is a
significant release containing a number of enhancements, new features,
and bug fixes, including:

29 new plugins:
alternatives, ansible, btrfs, buildah, clear_containers, date,
elastic, fibrechannel, host, kata_containers, lustre, memcached,
mssql, networkmanager, nvme, omnipath_client, omnipath_manager,
opendaylight, openstack_octavia, ovirt_provider_ovn, ovn_central,
ovn_host, rear, release, runc, skydive, unpackaged, watchdog, wireless

User and policy defined command line presets

The ability to save and recall specific combinations of command
line parameters
Policy authors may define presets for specific situations, products
or other uses (e.g. "cantboot", "rhel", "openshift" etc.).
Size limits for external commands

Certain commands produce large volumes of data, inflating report
size (e.g. journalctl): the command collection interface now allows
an arbitrary size limit to be applied, which includes memory used
during the run (reducing sosreport's peak memory usage).
Automatic file and command size limits

Plugins that do not specify an explicit size limit for files or
commands are now subject to the default value (specified with the
--log-size command line option).
Plugin authors may override this behaviour if needed
Concurrent plugin execution

Plugins are now run in parallel using a thread pool
Reduces runtime by up to 50% (workload dependent)
Command line --threads option to set the number of threads to
use, or to disable parallel execution
New profiles (including containers and the Apache webserver)

major enhancements to core features and existing plugins:

better package manager version information
policy support for detecting package managed files
fixed exit status propagation
deprecated optparse replaced with argparse
simplified and improved SoSOptions interface
better error handling during interactive prompting
allow journal collection by identifier
allow collection of journal message catalogs
support for collecting binary file data
more fine-grained system plugins (date etc.)
policy defined report file name patterns
more human-readable report file names by default
increased default log size (25MiB vs. 10MiB)
support for forbidden path lists and forbid logging
support for enabling plugins by kernel module name
support for enabling plugins by executable name
support for collecting eBPF (bpftool) data
support for device information via add_udev_info()
support for detecting and reporting unpackaged binaries
optional collection of the RPMDB
improved archive compression level and multithreading
default log size increased from 10MiB to 25MiB
improved debug logging and ENOSPC handling
major updates to the IPA plugin
major updates to the Docker plugin
string decoding fixes
DNF and Yum module support
OpenShift 3.10 support
Python3 fixes

Eric Desrochers (slashd) on 2018-06-05
tags: added: sts
Changed in sosreport (Ubuntu):
importance: Undecided → Wishlist
Eric Desrochers (slashd) on 2018-06-05
Changed in sosreport (Ubuntu Bionic):
importance: Undecided → Wishlist
Changed in sosreport (Ubuntu Xenial):
importance: Undecided → Wishlist
Changed in sosreport (Ubuntu Trusty):
importance: Undecided → Wishlist
Changed in sosreport (Ubuntu Cosmic):
importance: Wishlist → Medium
description: updated
Changed in sosreport (Debian):
status: Unknown → New
Eric Desrochers (slashd) on 2018-06-14
Changed in sosreport (Ubuntu Cosmic):
assignee: nobody → Eric Desrochers (slashd)
status: New → In Progress
Eric Desrochers (slashd) wrote :

Quick update....

sosreport v3.6, has been push in sid.

# https://buildd.debian.org/
Oct 03, 22:11:00 sosreport 3.6-1 on mips in sid: successful (all arches)
# https://buildd.debian.org/status/fetch.php?pkg=sosreport&ver=3.6-1&arch=mips&stamp=1538604660

#rmadison -u debian sosreport
...
sosreport | 3.6-1 | unstable | source, amd64, arm64, i386, ppc64el, s390x

Thanks to Caribou !

Eric Desrochers (slashd) on 2018-10-03
description: updated
Eric Desrochers (slashd) on 2018-10-12
Changed in sosreport (Ubuntu Cosmic):
assignee: Eric Desrochers (slashd) → Louis Bouchard (louis)
Eric Desrochers (slashd) on 2018-11-05
Changed in sosreport (Ubuntu):
assignee: Louis Bouchard (louis) → Eric Desrochers (slashd)
Changed in sosreport (Ubuntu Cosmic):
assignee: Louis Bouchard (louis) → nobody
Eric Desrochers (slashd) on 2018-11-05
description: updated
summary: - sosreport v3.6
+ [sync] sosreport v3.6
Changed in sosreport (Ubuntu Cosmic):
status: In Progress → New
importance: Medium → Wishlist
Download full text (3.1 KiB)

This bug was fixed in the package sosreport - 3.6-1
Sponsored for Eric Desrochers (slashd)

---------------
sosreport (3.6-1) unstable; urgency=medium

  * New 3.6 upstream release.
    major enhancements to core features and existing plugins:

    - better package manager version information
    - policy support for detecting package managed files
    - fixed exit status propagation
    - deprecated optparse replaced with argparse
    - simplified and improved SoSOptions interface
    - better error handling during interactive prompting
    - allow journal collection by identifier
    - allow collection of journal message catalogs
    - support for collecting binary file data
    - more fine-grained system plugins (date etc.)
    - policy defined report file name patterns
       - more human-readable report file names by default
    - increased default log size (25MiB vs. 10MiB)
    - support for forbidden path lists and forbid logging
    - support for enabling plugins by kernel module name
    - support for enabling plugins by executable name
    - support for collecting eBPF (bpftool) data
    - support for device information via add_udev_info()
    - support for detecting and reporting unpackaged binaries
    - optional collection of the RPMDB
    - improved archive compression level and multithreading
    - default log size increased from 10MiB to 25MiB
    - improved debug logging and ENOSPC handling
    - major updates to the IPA plugin
    - major updates to the Docker plugin
    - string decoding fixes
    - DNF and Yum module support
    - OpenShift 3.10 support
    - Python3 fixes

  Further release information and tarballs are available at:
    https://github.com/sosreport/sos/releases/tag/3.6

  * Debian specific modifications :
    - Omit Python2 only unittests
    - Avoid distutils in lstopo plugin

  * Former debian modifications now fixed upstream:
    - d/p/0002-reset-to-expected-variable-order-for-packagemanager.patch:
    - d/p/0003-fix-name-containers-is-not-defined.patch:
    - Fix-string-decoding-for-debug-log-output.patch

 -- Louis Bouchard <email address hidden> Mon, 24 Sep 2018 09:24:07 +0000

sosreport (3.5-1ubuntu4) cosmic; urgency=medium

  * d/p/Fix-string-decoding-for-debug-log-output.patch:
    Fix bug in _collect_strings that causes error trying to str.decode()
    (LP: #1761442)

 -- Dan Streetman <email address hidden> Tue, 05 Jun 2018 10:52:56 -0400

sosreport (3.5-1ubuntu3) bionic; urgency=medium

  * Avoid using distutils during runtime.

 -- Matthias Klose <email address hidden> Thu, 12 Apr 2018 10:48:13 +0200

sosreport (3.5-1ubuntu2) bionic; urgency=medium

  * d/p/0003-fix-name-containers-is-not-defined.patch:
    "containers" is an unknown variable, "insp" is the correct one.

 -- Eric Desrochers <email address hidden> Mon, 15 Jan 2018 18:39:54 -0500

sosreport (3.5-1ubuntu1) bionic; urgency=medium

  * d/p/0002-reset-to-expected-variable-order-for-packagemanager.patch:
    Fixing severe regression where many "packages = <PKG_NAME>" lines are no
    longer working in Debian/Ubuntu. Preventing automatic package detection.

 -- Eric Desrochers <email address hidden> Tue, 05 Dec 2017 1...

Read more...

Changed in sosreport (Ubuntu):
status: In Progress → Fix Released
Eric Desrochers (slashd) on 2018-11-05
Changed in sosreport (Ubuntu Cosmic):
status: New → In Progress
assignee: nobody → Eric Desrochers (slashd)
summary: - [sync] sosreport v3.6
+ [sync][sru]sosreport v3.6
Changed in sosreport (Debian):
status: New → Fix Released
Eric Desrochers (slashd) on 2018-11-09
description: updated
description: updated
description: updated
Eric Desrochers (slashd) on 2018-11-09
description: updated
description: updated
Eric Desrochers (slashd) on 2018-11-09
Changed in sosreport (Ubuntu Bionic):
assignee: nobody → Eric Desrochers (slashd)
Eric Desrochers (slashd) on 2018-11-09
Changed in sosreport (Ubuntu Xenial):
assignee: nobody → Eric Desrochers (slashd)
Changed in sosreport (Ubuntu Trusty):
assignee: nobody → Eric Desrochers (slashd)
Changed in sosreport (Ubuntu Bionic):
status: New → In Progress
Changed in sosreport (Ubuntu Xenial):
status: New → In Progress
Changed in sosreport (Ubuntu Trusty):
status: New → In Progress
description: updated
Eric Desrochers (slashd) on 2018-11-09
description: updated

I ran sosreport -a on bionic with maas 2.4, corosync, pacemaker and postgres all in a very unhappy state. The output looks good. During execution I recieved the following:

[plugin:pacemaker] crm_from parameter 'True' is not a valid date: using default

A couple of minor notes from the output:

1. In sos_commands/maas/maas-region-admin_dumpdata:

WARNING: The maas-region-admin command is deprecated and will be removed in a future version. From now on please use 'maas-region' instead.

2. In sos_commands/pacemaker it looks like the pcs commands were run but pcs was not installed

I don't believe any of these are issues just notes to clean up at some point.

Eric Desrochers (slashd) on 2018-11-09
description: updated
Eric Desrochers (slashd) on 2018-11-09
description: updated
Chris Newcomer (cnewcomer) wrote :

Testing results:

LXD containers:
Xenial: ran and created an output file that looked fine. It outputted a single seemingly harmless error during run: [plugin:kvm] debugfs not mounted and mount attempt failed
Bionic: ran with no errors, output file appeared fine
Cosmic: ran with no errors, output file appeared fine

Bare metal Bionic: ran with no errors, output file appeared fine

KVM VM Xenial with MAAS: ran with no errors, output file appeared fine
KVM VM Xenial with snap juju: ran with no errors, juju plugin did not run. Reported in https://github.com/sosreport/sos/issues/1475 and fixed in https://github.com/sosreport/sos/pull/1476
KVM VM Xenial with package juju: ran with no errors. Since it ran with sudo, it collected the juju status data, but it changed the controllers.yaml file to root-owned. This is a known juju problem and documented at: https://bugs.launchpad.net/juju/+bug/1747357
KVM VM with Xenial running live-patch: ran with no errors, output file appeared fine

There are no outstanding issues that I have found that should hold this up.

Eric Desrochers (slashd) on 2018-11-12
description: updated
Eric Desrochers (slashd) on 2018-11-12
description: updated
Eric Desrochers (slashd) on 2018-11-12
description: updated
description: updated
description: updated
David Coronel (davecore) wrote :

Ran test in Xenial LXD container:

# dpkg -l | grep sosreport
ii sosreport 3.6-1ubuntu0.16.04.1 amd64 Set of tools to gather troubleshooting data from a system

# apt-cache policy sosreport
sosreport:
  Installed: 3.6-1ubuntu0.16.04.1
  Candidate: 3.6-1ubuntu0.16.04.1
  Version table:
 *** 3.6-1ubuntu0.16.04.1 500
        500 http://ppa.launchpad.net/slashd/srusosreport/ubuntu xenial/main amd64 Packages
        100 /var/lib/dpkg/status
     3.5-1~ubuntu16.04.3 500
        500 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages
     3.2+git276-g7da50d6-3ubuntu1 500
        500 http://archive.ubuntu.com/ubuntu xenial/main amd64 Packages

# sosreport -a
[...]
 Setting up archive ...
 Setting up plugins ...
caught exception in plugin method "juju.setup()"
writing traceback to sos_logs/juju-plugin-errors.txt
[plugin:kvm] debugfs not mounted and mount attempt failed
mysql.dbdump: database user name and password must be supplied
 Running plugins. Please wait ...

  Finishing plugins [Running: lxd]
  Finished running plugins
[...]

This is my primary Landscape quickstart server running in a LXD container.

I'm seeing non-sanitised password in the Landscape service.conf file:

$ cat sosreport-xeniallandscape-20181112170402/etc/landscape/service.conf | grep pass
password = b64:MC1[removed for Launchpad paste]==
password = b64:M1B[removed for Launchpad paste]==
store_password = b64:bWp[removed for Launchpad paste]==

I had just seen a previous sosreport with version 3.5-1~ubuntu16.04.3 that was sanitising properly:

$ cat sosreport-[removed for Launchpad]-20181112140259/etc/landscape/service.conf | grep password
password = [********]
password = [********]
store_password = [********]

We should investigate this further.

Brian Murray (brian-murray) wrote :

I'm setting this to Incomplete based on comment #6 and the possibility of a password being leaked.

Changed in sosreport (Ubuntu Cosmic):
status: In Progress → Incomplete
Eric Desrochers (slashd) on 2018-11-13
description: updated
description: updated
Eric Desrochers (slashd) wrote :

Hi Dave,

You are right, after a bisection from upstream code, I found the necessary fix to address this situation:
https://github.com/sosreport/sos/commit/b96bdab03f06408e162b1733b20e8ba9fbf8e012

I'll re-package, and would appreciate you to test again.

description: updated
description: updated
Eric Desrochers (slashd) on 2018-11-13
description: updated
Eric Desrochers (slashd) wrote :

I have re-package w/ the upstream fix to address the substitution method failure :

Can you give it a try and let me know the outcome ?

sudo add-apt-repository ppa:slashd/lp1775195
sudo apt-get update

Regards,
Eric

Eric Desrochers (slashd) wrote :

On my side, it looks good now using the package I have just built found on "ppa:slashd/lp1775195"

$ sosreport -o landscape
$ cd /tmp
$ tar Jxvf sosreport-lds-20181113210218.tar.xz
$ grep -i password sosreport-lds-20181113210218/etc/landscape/service.conf
password = [********]
password = [********]
store_password = [********]

Changed in sosreport (Ubuntu Cosmic):
status: Incomplete → In Progress
David Coronel (davecore) wrote :

Hi Eric,

I confirm with the sosreport in ppa:slashd/lp1775195 I no longer see the passwords anymore:

# grep password sosreport-xeniallandscape-20181114150420/etc/landscape/service.conf
password = [********]
password = [********]
store_password = [********]

I couldn't find any other issues with this new sosreport package. LGTM.

Eric Desrochers (slashd) wrote :

Fix has been push into Disco.
sosreport v3.6, is now waiting in the upload queue for C/B/X.

Thanks all for your testing !
I will need testing again during the testing phase in $RELEASE-proposed.

- Eric

Changed in sosreport (Ubuntu Trusty):
status: In Progress → Won't Fix

Hello Eric, or anyone else affected,

Accepted sosreport into cosmic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/sosreport/3.6-1ubuntu0.18.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-cosmic to verification-done-cosmic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-cosmic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in sosreport (Ubuntu Cosmic):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-cosmic
Changed in sosreport (Ubuntu Bionic):
status: In Progress → Fix Committed
tags: added: verification-needed-bionic
Brian Murray (brian-murray) wrote :

Hello Eric, or anyone else affected,

Accepted sosreport into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/sosreport/3.6-1ubuntu0.18.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in sosreport (Ubuntu Xenial):
status: In Progress → Fix Committed
tags: added: verification-needed-xenial
Brian Murray (brian-murray) wrote :

Hello Eric, or anyone else affected,

Accepted sosreport into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/sosreport/3.6-1ubuntu0.16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[VERIFICATION BIONIC]

I ran sosreport '3.6-1ubuntu0.18.04.1' inside a baremetal AND a container without any issue.

The test was run with '-a, --alloptions'[1] AND '--all-logs'[2]

[1] - Boolean that set to True for all enabled plugins.
[2] - Tell sosreport to collect all possible log data ignoring any size limits.

Report:
* All the mandatory directory and files (mention in the autopkgtest are present) has been collected and contain valid data.
* All 0 size files found in the tarball are related to thing which doesn't raise any red flag to me. For instance the network or wireless plugin still tries to run network command iwconfig,
ifenslave, even if not installed by default w/ Bionic.

0 ./sos_commands/wireless/iwconfig
0 ./sos_commands/networking/ifenslave_-a

# Output :
sosreport -a --all-logs

sosreport (version 3.6)

This command will collect system configuration and diagnostic
information from this Ubuntu system. An archive containing the collected
information will be generated in /tmp/sos.5a8o_cos.

For more information on Ubuntu visit:

  http://www.ubuntu.com/

The generated archive may contain data considered sensitive and its
content should be reviewed by the originating organization before being
passed to any third party.

No changes will be made to system configuration.

Press ENTER to continue, or CTRL-C to quit.

Please enter the case id that you are generating this report for []:

 Setting up archive ...
 Setting up plugins ...
 Running plugins. Please wait ...

  Finishing plugins [Running: ubuntu]
  Finished running plugins
Creating compressed archive...

Your sosreport has been generated and saved in:
  /tmp/sosreport-nuc7-20181115191906.tar.xz

The checksum is: 20e6b4aa11d572357e1887da9e650d29

Please send this file to your support representative.

Eric Desrochers (slashd) wrote :

[VERIFICATION BIONIC]
# Part 2

I also took another look at the sensible data substitution, and works as expected :

etc/ssl/openssl.cnf:# input_password = ******
etc/ssl/openssl.cnf:# output_password = ******
etc/ssl/openssl.cnf:challengePassword = ******
etc/ssl/openssl.cnf:challengePassword_min = ******
etc/ssl/openssl.cnf:challengePassword_max = ******

Eric Desrochers (slashd) on 2018-11-15
description: updated
Eric Desrochers (slashd) wrote :

[VERIFICATION XENIAL]

I ran sosreport '3.6-1ubuntu0.18.04.1' inside a baremetal AND a container without any issue.

The test was run with '-a, --alloptions'[1] AND '--all-logs'[2]

[1] - Boolean that set to True for all enabled plugins.
[2] - Tell sosreport to collect all possible log data ignoring any size limits.

Report:
* All the mandatory directory and files (mention in the autopkgtest are present) has been collected and contain valid data.
* All 0 size files found in the tarball are related to thing which doesn't raise any red flag to me.
* Sensible data substitution works as expected:

etc/ssl/openssl.cnf:# input_password = ******
etc/ssl/openssl.cnf:# output_password = ******
etc/ssl/openssl.cnf:challengePassword = ******
etc/ssl/openssl.cnf:challengePassword_min = ******
etc/ssl/openssl.cnf:challengePassword_max = ******

The only error I found only inside a container was the following :

$ grep ERROR sos_logs/sos.log
2018-11-15 20:24:09,005 ERROR: [plugin:kvm] debugfs not mounted and mount attempt failed

sosreport complained because debugfs wasn't mounted. It's probably related to unprivileges container. It's definitely not a SRU blocker at this point.

Inside a KVM guest, no problem

Output :
sosreport -a --all-logs

sosreport (version 3.6)

This command will collect system configuration and diagnostic
information from this Ubuntu system. An archive containing the collected
information will be generated in /tmp/sos.w0oo29nm.

For more information on Ubuntu visit:

  http://www.ubuntu.com/

The generated archive may contain data considered sensitive and its
content should be reviewed by the originating organization before being
passed to any third party.

No changes will be made to system configuration.

Press ENTER to continue, or CTRL-C to quit.

Please enter the case id that you are generating this report for []:

 Setting up archive ...
 Setting up plugins ...
 Running plugins. Please wait ...

  Finishing plugins [Running: ubuntu]
  Finished running plugins
Creating compressed archive...

Your sosreport has been generated and saved in:
  /tmp/sosreport-ubuntu-20181115154754.tar.xz

The checksum is: 42bd6fc5bde919a89ad0e2de3d0f5079

Please send this file to your support representative.

Eric Desrochers (slashd) wrote :

Correction I for my above statement ^ I I ran sosreport '3.6-1ubuntu0.16.04.1' for Xenial.

Eric Desrochers (slashd) wrote :

[VERIFICATION COSMIC]

I ran sosreport '3.6-1ubuntu0.18.10.1' inside a baremetal AND a container without any issue.

The test was run with '-a, --alloptions'[1] AND '--all-logs'[2]

[1] - Boolean that set to True for all enabled plugins.
[2] - Tell sosreport to collect all possible log data ignoring any size limits.

Report:
* All the mandatory directory and files (mention in the autopkgtest are present) has been collected and contain valid data.

* All 0 size files found in the tarball are related to thing which doesn't raise any red flag to me. For instance the network or wireless plugin still tries to run network command iwconfig,
ifenslave, even if not installed by default w/ Bionic.

0 ./sos_commands/wireless/iwconfig
0 ./sos_commands/networking/ifenslave_-a

* Sensible data substitution works as expected:

etc/ssl/openssl.cnf:# input_password = ******
etc/ssl/openssl.cnf:# output_password = ******
etc/ssl/openssl.cnf:challengePassword = ******
etc/ssl/openssl.cnf:challengePassword_min = ******
etc/ssl/openssl.cnf:challengePassword_max = ******

Output:
 sosreport -a --all-logs

sosreport (version 3.6)

This command will collect system configuration and diagnostic
information from this Ubuntu system. An archive containing the collected
information will be generated in /tmp/sos.sib3fe4u.

For more information on Ubuntu visit:

  http://www.ubuntu.com/

The generated archive may contain data considered sensitive and its
content should be reviewed by the originating organization before being
passed to any third party.

No changes will be made to system configuration.

Press ENTER to continue, or CTRL-C to quit.

Please enter the case id that you are generating this report for []:

 Setting up archive ...
 Setting up plugins ...
 Running plugins. Please wait ...

  Starting 8/55 btrfs [Running: apparmor apt block btrfs]
  Finishing plugins [Running: systemd]
  Finished running plugins
Creating compressed archive...

Your sosreport has been generated and saved in:
  /tmp/sosreport-ubuntu-20181115215323.tar.xz

The checksum is: 7055bd410bb11830fc631f06371e3dbe

Please send this file to your support representative.

Chris Newcomer (cnewcomer) wrote :
Download full text (6.7 KiB)

[ VERIFICATION BIONIC ]
Bare metal:
$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.1 LTS"

$ dpkg -l | grep sosreport
ii sosreport 3.6-1ubuntu0.18.04.1 amd64 Set of tools to gather troubleshooting data from a system
$ sudo sosreport

sosreport (version 3.6)

This command will collect system configuration and diagnostic
information from this Ubuntu system. An archive containing the collected
information will be generated in /tmp/sos.ojmc_spl.

For more information on Ubuntu visit:

  http://www.ubuntu.com/

The generated archive may contain data considered sensitive and its
content should be reviewed by the originating organization before being
passed to any third party.

No changes will be made to system configuration.

Press ENTER to continue, or CTRL-C to quit.

Please enter the case id that you are generating this report for []: 00001111

 Setting up archive ...
 Setting up plugins ...
 Running plugins. Please wait ...

  Starting 1/64 acpid [Running: acpid] Starting 2/64 anacron [Running: anacron] Starting 3/64 apparmor [Running: anacron apparmor] Starting 4/64 apport [Running: anacron apparmor apport] Starting 5/64 apt [Running: anacron apparmor apport apt] Starting 6/64 ata [Running: apparmor apport apt ata] Starting 7/64 block [Running: apparmor apport apt block] Starting 8/64 boot [Running: apparmor apt block boot] Starting 9/64 btrfs [Running: apparmor apt block btrfs] Starting 10/64 canonical_livepatch [Running: apparmor apt block canonical_live Starting 11/64 ceph [Running: apt block canonical_livepatch ceph] Starting 12/64 cgroups [Running: apt block ceph cgroups] Starting 13/64 cron [Running: apt block cgroups cron] Starting 14/64 crypto [Running: apt block cgroups crypto] Starting 15/64 cups [Running: apt block cgroups cups] Starting 16/64 date [Running: apt block cgroups date] Starting 17/64 dbus [Running: apt block cgroups dbus] Starting 18/64 devicemapper [Running: apt block cgroups devicemapper] Starting 19/64 devices [Running: apt block cgroups devices] Starting 20/64 dpkg [Running: apt block cgroups dpkg] Starting 21/64 filesys [Running: apt block cgroups filesys] Starting 22/64 hardware [Running: apt cgroups filesys hardware] Starting 23/64 host [Running: apt cgroups hardware host] Starting 24/64 i18n [Running: apt cgroups host i18n] Starting 25/64 java [Running: apt cgroups host java] Starting 26/64 kernel [Running: apt cgroups host kernel] Starting 27/64 kvm [Running: a...

Read more...

Bryan Quigley (bryanquigley) wrote :

Eric, unfortunately - I'm getting a hang in kernel now - I installed rasdaemon to debug a customer case... between when I tested last...

I believe it's fixed by https://github.com/sosreport/sos/commit/d6379b5ba0f381ea8ec2403b9985100a946a5866

Chris Newcomer (cnewcomer) wrote :
Download full text (6.7 KiB)

[ VERIFICATION XENIAL ]
libvirt VM:

$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04.5 LTS"

$ dpkg -l | grep sosreport
ii sosreport 3.6-1ubuntu0.16.04.1 amd64 Set of tools to gather troubleshooting data from a system

$ sudo sosreport

sosreport (version 3.6)

This command will collect system configuration and diagnostic
information from this Ubuntu system. An archive containing the collected
information will be generated in /tmp/sos.it_zntr5.

For more information on Ubuntu visit:

  http://www.ubuntu.com/

The generated archive may contain data considered sensitive and its
content should be reviewed by the originating organization before being
passed to any third party.

No changes will be made to system configuration.

Press ENTER to continue, or CTRL-C to quit.

Please enter the case id that you are generating this report for []: 00001111

 Setting up archive ...
 Setting up plugins ...
 Running plugins. Please wait ...

  Starting 1/64 acpid [Running: acpid] Starting 2/64 apache [Running: apache] Starting 3/64 apparmor [Running: apache apparmor] Starting 4/64 apport [Running: apache apparmor apport] Starting 5/64 apt [Running: apache apparmor apport apt] Starting 6/64 ata [Running: apache apparmor apt ata] Starting 7/64 block [Running: apache apparmor apt block] Starting 8/64 boot [Running: apparmor apt block boot] Starting 9/64 btrfs [Running: apparmor apt block btrfs] Starting 10/64 ceph [Running: apt block btrfs ceph] Starting 11/64 cgroups [Running: apt block ceph cgroups] Starting 12/64 cron [Running: apt block cgroups cron] Starting 13/64 crypto [Running: apt block cgroups crypto] Starting 14/64 date [Running: apt block cgroups date] Starting 15/64 dbus [Running: apt block cgroups dbus] Starting 16/64 devicemapper [Running: apt block cgroups devicemapper] Starting 17/64 devices [Running: apt block cgroups devices] Starting 18/64 dpkg [Running: apt block cgroups dpkg] Starting 19/64 filesys [Running: apt block cgroups filesys] Starting 20/64 hardware [Running: apt cgroups filesys hardware] Starting 21/64 host [Running: apt cgroups filesys host] Starting 22/64 i18n [Running: apt cgroups host i18n] Starting 23/64 iscsi [Running: apt cgroups host iscsi] Starting 24/64 java [Running: apt cgroups iscsi java] Starting 25/64 kernel [Running: apt cgroups java kernel] Starting 26/64 last [Running: apt cgroups kernel last] Starting 27/64 libraries [Running: apt cgr...

Read more...

Chris Newcomer (cnewcomer) wrote :

[ VERIFICATION COSMIC ]
libvirt VM:

$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.10
DISTRIB_CODENAME=cosmic
DISTRIB_DESCRIPTION="Ubuntu 18.10"

$ dpkg -l | grep sosreport
ii sosreport 3.6-1ubuntu0.18.10.1 amd64 Set of tools to gather troubleshooting data from a system

$ sudo sosreport

sosreport (version 3.6)

This command will collect system configuration and diagnostic
information from this Ubuntu system. An archive containing the collected
information will be generated in /tmp/sos.3wnd3fwh.

For more information on Ubuntu visit:

  http://www.ubuntu.com/

The generated archive may contain data considered sensitive and its
content should be reviewed by the originating organization before being
passed to any third party.

No changes will be made to system configuration.

Press ENTER to continue, or CTRL-C to quit.

Please enter the case id that you are generating this report for []: 00001111

 Setting up archive ...
 Setting up plugins ...
 Running plugins. Please wait ...

  Finishing plugins [Running: ubuntu]
  Finished running plugins
Creating compressed archive...

Your sosreport has been generated and saved in:
  /tmp/sosreport-test2.00001111-20181116002912.tar.xz

The checksum is: d09c4ddd96bc6d4b88e7aa1a5ca5b663

Please send this file to your support representative.

Eric Desrochers (slashd) wrote :

Bryan,

How did you manage to trigger the hang ? and on which release(s) ?

Bryan Quigley (bryanquigley) wrote :

I'm on 18.04 (I would assume this would not affect older releases)
I previously installed rasdaemon and started it. That's all it took to trigger it on Fedora - https://github.com/sosreport/sos/pull/1445

Bryan Quigley (bryanquigley) wrote :

(I would assume this would not affect older releases ... as they use mcelog not rasdaemon.)

Eric Desrochers (slashd) wrote :

I was able to reproduce the situation and did a kernel bisection to confirm, and it reveals that you were right :

$ git bisect log
...
# first fixed commit: [d6379b5ba0f381ea8ec2403b9985100a946a5866] [kernel] dont collect some tracing instance files

I test the above commit on Cosmic and sosreport no longer hangs at the kernel plugin when rasdaemon is installed.

I'll push a the fix during the week-end or next week.

Eric Desrochers (slashd) wrote :

[d6379b5ba0f381ea8ec2403b9985100a946a5866] fix has been push into disco, stable release will follow soon.

Thanks Bryan.

- Eric

Eric Desrochers (slashd) wrote :

Bug #1803735 has been reported to track the kernel plugin bug.

Eric Desrochers (slashd) on 2018-11-16
description: updated
Eric Desrochers (slashd) on 2018-11-16
tags: added: sos36

Hello Eric, or anyone else affected,

Accepted sosreport into cosmic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/sosreport/3.6-1ubuntu0.18.10.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-cosmic to verification-done-cosmic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-cosmic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Łukasz Zemczak (sil2100) wrote :

Hello Eric, or anyone else affected,

Accepted sosreport into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/sosreport/3.6-1ubuntu0.18.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Łukasz Zemczak (sil2100) wrote :

Hello Eric, or anyone else affected,

Accepted sosreport into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/sosreport/3.6-1ubuntu0.16.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Bryan Quigley (bryanquigley) ,

Today's new upload should address the bug you experienced. (I have tested myself pre-upload)
Could you give it a try as well ? and provide feedback ?

- Eric

Eric Desrochers (slashd) on 2018-11-22
tags: added: verification-done-bionic verification-done-cosmic verification-done-xenial
removed: verification-needed-bionic verification-needed-cosmic verification-needed-xenial
Eric Desrochers (slashd) on 2018-11-22
description: updated
description: updated
description: updated
summary: - [sync][sru]sosreport v3.6
+ [sync][sru] Backport of sosreport v3.6

The verification of the Stable Release Update for sosreport has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package sosreport - 3.6-1ubuntu0.18.04.2

---------------
sosreport (3.6-1ubuntu0.18.04.2) bionic; urgency=medium

  * d/p/dont-collect-some-tracing-instance-files.patch: (LP: #1803735)
    - Ignoring some kernel files as the copy of them make sosreport
      hangs in situation where kernel tracing events is in used.

sosreport (3.6-1ubuntu0.18.04.1) bionic; urgency=medium

  * New 3.6 upstream release. (LP: #1775195)
    major enhancements to core features and existing plugins:

    - better package manager version information
    - policy support for detecting package managed files
    - fixed exit status propagation
    - deprecated optparse replaced with argparse
    - simplified and improved SoSOptions interface
    - better error handling during interactive prompting
    - allow journal collection by identifier
    - allow collection of journal message catalogs
    - support for collecting binary file data
    - more fine-grained system plugins (date etc.)
    - policy defined report file name patterns
    - more human-readable report file names by default
    - increased default log size (25MiB vs. 10MiB)
    - support for forbidden path lists and forbid logging
    - support for enabling plugins by kernel module name
    - support for enabling plugins by executable name
    - support for collecting eBPF (bpftool) data
    - support for device information via add_udev_info()
    - support for detecting and reporting unpackaged binaries
    - optional collection of the RPMDB
    - improved archive compression level and multithreading
    - default log size increased from 10MiB to 25MiB
    - improved debug logging and ENOSPC handling
    - major updates to the IPA plugin
    - major updates to the Docker plugin
    - string decoding fixes
    - DNF and Yum module support
    - OpenShift 3.10 support
    - Python3 fixes

  Further release information and tarballs are available at:
    https://github.com/sosreport/sos/releases/tag/3.6

  * Debian specific modifications :
    - Omit Python2 only unittests
    - Avoid distutils in lstopo plugin

  * Former debian modifications now fixed upstream:
    - d/p/0002-reset-to-expected-variable-order-for-packagemanager.patch
    - d/p/0003-fix-name-containers-is-not-defined.patch
    - Fix-string-decoding-for-debug-log-output.patch

  * Cherry-pick upstream patch:
    - d/p/fix-string-substitution-method.patch:
      Fix plugin string substitution methods to fail.

 -- Eric Desrochers <email address hidden> Fri, 16 Nov 2018 11:11:27 -0500

Changed in sosreport (Ubuntu Bionic):
status: Fix Committed → Fix Released
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package sosreport - 3.6-1ubuntu0.16.04.2

---------------
sosreport (3.6-1ubuntu0.16.04.2) xenial; urgency=medium

  * d/p/dont-collect-some-tracing-instance-files.patch: (LP: #1803735)
    - Ignoring some kernel files as the copy of them make sosreport
      hangs in situation where kernel tracing events is in used.

sosreport (3.6-1ubuntu0.16.04.1) xenial; urgency=medium

  * New 3.6 upstream release. (LP: #1775195)
    major enhancements to core features and existing plugins:

    - better package manager version information
    - policy support for detecting package managed files
    - fixed exit status propagation
    - deprecated optparse replaced with argparse
    - simplified and improved SoSOptions interface
    - better error handling during interactive prompting
    - allow journal collection by identifier
    - allow collection of journal message catalogs
    - support for collecting binary file data
    - more fine-grained system plugins (date etc.)
    - policy defined report file name patterns
    - more human-readable report file names by default
    - increased default log size (25MiB vs. 10MiB)
    - support for forbidden path lists and forbid logging
    - support for enabling plugins by kernel module name
    - support for enabling plugins by executable name
    - support for collecting eBPF (bpftool) data
    - support for device information via add_udev_info()
    - support for detecting and reporting unpackaged binaries
    - optional collection of the RPMDB
    - improved archive compression level and multithreading
    - default log size increased from 10MiB to 25MiB
    - improved debug logging and ENOSPC handling
    - major updates to the IPA plugin
    - major updates to the Docker plugin
    - string decoding fixes
    - DNF and Yum module support
    - OpenShift 3.10 support
    - Python3 fixes

  Further release information and tarballs are available at:
    https://github.com/sosreport/sos/releases/tag/3.6

  * Debian specific modifications :
    - Omit Python2 only unittests
    - Avoid distutils in lstopo plugin

  * Former debian modifications now fixed upstream:
    - d/p/0002-reset-to-expected-variable-order-for-packagemanager.patch
    - d/p/0003-fix-name-containers-is-not-defined.patch
    - Fix-string-decoding-for-debug-log-output.patch

  * Cherry-pick upstream patch:
    - d/p/fix-string-substitution-method.patch:
      Fix plugin string substitution methods to fail.

 -- Eric Desrochers <email address hidden> Fri, 16 Nov 2018 11:13:16 -0500

Changed in sosreport (Ubuntu Xenial):
status: Fix Committed → Fix Released
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package sosreport - 3.6-1ubuntu0.18.10.2

---------------
sosreport (3.6-1ubuntu0.18.10.2) cosmic; urgency=medium

  * d/p/dont-collect-some-tracing-instance-files.patch: (LP: #1803735)
    - Ignoring some kernel files as the copy of them make sosreport
      hangs in situation where kernel tracing events is in used.

sosreport (3.6-1ubuntu0.18.10.1) cosmic; urgency=medium

  * New 3.6 upstream release. (LP: #1775195)
    major enhancements to core features and existing plugins:

    - better package manager version information
    - policy support for detecting package managed files
    - fixed exit status propagation
    - deprecated optparse replaced with argparse
    - simplified and improved SoSOptions interface
    - better error handling during interactive prompting
    - allow journal collection by identifier
    - allow collection of journal message catalogs
    - support for collecting binary file data
    - more fine-grained system plugins (date etc.)
    - policy defined report file name patterns
    - more human-readable report file names by default
    - increased default log size (25MiB vs. 10MiB)
    - support for forbidden path lists and forbid logging
    - support for enabling plugins by kernel module name
    - support for enabling plugins by executable name
    - support for collecting eBPF (bpftool) data
    - support for device information via add_udev_info()
    - support for detecting and reporting unpackaged binaries
    - optional collection of the RPMDB
    - improved archive compression level and multithreading
    - default log size increased from 10MiB to 25MiB
    - improved debug logging and ENOSPC handling
    - major updates to the IPA plugin
    - major updates to the Docker plugin
    - string decoding fixes
    - DNF and Yum module support
    - OpenShift 3.10 support
    - Python3 fixes

  Further release information and tarballs are available at:
    https://github.com/sosreport/sos/releases/tag/3.6

  * Debian specific modifications :
    - Omit Python2 only unittests
    - Avoid distutils in lstopo plugin

  * Former debian modifications now fixed upstream:
    - d/p/0002-reset-to-expected-variable-order-for-packagemanager.patch
    - d/p/0003-fix-name-containers-is-not-defined.patch
    - Fix-string-decoding-for-debug-log-output.patch

  * Cherry-pick upstream patch:
    - d/p/fix-string-substitution-method.patch:
      Fix plugin string substitution methods to fail.

 -- Eric Desrochers <email address hidden> Fri, 16 Nov 2018 10:52:59 -0500

Changed in sosreport (Ubuntu Cosmic):
status: Fix Committed → Fix Released
Bryan Quigley (bryanquigley) wrote :

Reopening Trusty as it will be supported in ESM for a while yet.

Changed in sosreport (Ubuntu Trusty):
status: Won't Fix → New
Changed in sosreport (Ubuntu Trusty):
status: New → In Progress
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.