[sru] sos upstream 4.4

Bug #1986611 reported by nikhil kshirsagar
16
This bug affects 1 person
Affects Status Importance Assigned to Milestone
sosreport (Ubuntu)
Fix Released
Medium
nikhil kshirsagar
Bionic
Fix Released
Medium
nikhil kshirsagar
Focal
Fix Released
Medium
nikhil kshirsagar
Jammy
Fix Released
Medium
nikhil kshirsagar

Bug Description

[IMPACT]

The sos team is pleased to announce the release of sos-4.4. This release includes a number of performance and stability fixes, as well as several developer-focused quality of life changes to various APIs.

[TEST PLAN]

Documentation for Special Cases:
https://wiki.ubuntu.com/SosreportUpdates

[WHERE PROBLEMS COULD OCCUR]

The changes in sosreport are described below. There are a number of changes related to report, plugins and policy. The changes in sosreport content may break existing tooling, so that will need extensive testing.

Global Changes
  Added a Georgian translation

Report Changes
  Fixed a bug where a plugin's default timeout was not being correctly
  evaluated.
  Improved binary file detection by leveraging python3-magic. SoS now has
  a dependency on this package/module.
  Encryption of the final archive now supports the use of the
  SOSENCRYPTKEY or SOSENCRYPTPASS environment variables to allow users to
  not pass these values via command line options.
  The --list-plugins output will now report actual, not default, option
  values for plugin options.
  The locale used for collections has been updated from C to C.UTF-8
  Loop devices are now more accurately enumerated on host systems.
  Network devices are now enumerated and saved to the network key for the
  devices dict handed to plugins.

Plugin Changes
  Added 2 new plugins: collectl, shmcli
  Container IDs may now be used with Plugin.container_exists(), not just
  container names.
  This also means that Plugin.exec_cmd() can now function with IDs.
  The container-based collection paths for several OSP-related plugins
  has been updated.
  Plugins may now define a default set of environment variables for their
  collections.
  The scsi plugin will now collect SCSI persistent reserve commands.
  The openshift plugin may now reference a local kubeconfig file for
  authentication, via the openshift.kubeconfig option.
  The no-oc option for the openshift plugin has been renamed to with-api
  to be more obvious about what collections it controls.
  The openshift plugin now does not collect oc command output by default
  (with-api is defaulted to False).
  The yum and dnf plugins have been merged. Only the dnf plugin remains.
  The fibrechannel plugin will now collect HBA device logs.
  The stratis plugin has been updated to collect stratis 3.x commands.
  The ceph_mgr plugin will now collect ceph orchestrator commands.
  The hpssm plugin now iterates over multiple slots for collections.
  Binary files will no longer be tailed by add_copy_spec() when their
  size exceeds the specified sizelimit.
  add_blockdev_cmd() has been refactored into add_device_cmd() which will
  be used for device command iteration, only now not limited to block
  devices.
  add_device_cmd() may be used to iterate over devices enumerated by sos
  during initialization(e.g. block, ethernet, network, and more), or
  manually compiled lists of devices.
  Plugins no longer log string content of tailed collections or manually
  compiled strings.

Policy Changes
  Fixed a bug in the Red Hat policy that would incorrectly override a
  provided --upload-url to Red Hat's SFTP server if incorrect credentials
  were given.
  Fixed usage of GenericPolicy so that it can actually be used to collect
  minimal reports from distributions without explicit sos policies.

Collect Changes
  New cluster profile: rhosp
  Note: This profile is designed for use with Red Hat OpenStack
  environments, and as such makes some assumptions about the deployment.
  It may work for non-Red Hat environments that are similarly deployed,
  but that is not guaranteed.
  Greatly improved robustness and integration with OpenShift Container
  Platform to provide consistent report collections and cleanup.
  Refactored node enumeration for pacemaker clusters. Users should now
  see more accurate and consistent node lists from these clusters.
  By default, the ocp cluster will now attempt to use a well-known stock
  kubeconfig file present in order to enable collections from the oc API.
  A new with-api option for the ocp cluster has been added to control
  whether or not any node is used to collect API data from the cluster.
  Fixed a bug where the --image option would be ignored for OCP cluster
  collections using containerized hosts.
  The use of exit() is now standardized and should be used in all cases
  where collect needs to stop execution.

Clean|Mask Changes
  Nothing in the /etc/sos/cleaner directory will be captured in sos
  reports now.
  Parsers will now pre-generate regex lists for matching obfuscation
  patterns, rather than creating them on the fly.
  As such, performance should be noticeably improved for most use cases.
  Fixed a bug where sos would obfuscate the tmpdir path of the final
  results, which would result in a "No such file or directory" error when
  trying to perform cleanup.
  Individual parsers (e.g. the hostname parser or ip parser) may be
  selectively disabled now via the --disable-parsers option.

Help Changes
  sos help was relatively unchanged this release. Various plugins,
  collect profiles, and other sections of the project were given extended
  sos help information alongside other improvements and fixes to those
  components.

[OTHER INFORMATION]

Regression could occur at core functionality, which may prevent sos (or its subcommand to work. I consider this regression type as 'low'. That is generally well tested, and we would find a problem at an early stage during the verification phase if it is the case.

On the other end, regression could happen and are some kind of expected at plugins levels. As of today, sos has more than 300 plugins. It is nearly impossible to test them all.

If a regression is found in a plugin, it is rarely affecting sos core functionalities nor other plugins. So mainly the impact would be limited to that plugin. The impact being that the plugin can't or partially can collect the information that it is instructed to gather.

A 3rd party vendor would then ask user/customer to collect the information manually for that particular plugins.

Plugins are segmented by services and/or applications (e.g. openstack_keystone, bcache, system, logs, ...) in order to collect things accordingly to the plugin detected or intentionally requested for.

Sosreport plugins philosophy is to (as much as possible) maintain backward compatibility when updating a plugin. The risk that an ancient version of a software has been dropped, is unlikely, unless it was intended to be that way for particular reasons. Certain plugin also support the DEB installation way and the snap one (MAAS, LXD, ...) so all Ubuntu standard installation types are covered.

Release note:
https://github.com/sosreport/sos/releases/tag/4.4

Changed in sosreport (Ubuntu Focal):
assignee: nobody → nikhil kshirsagar (nkshirsagar)
Revision history for this message
nikhil kshirsagar (nkshirsagar) wrote :
description: updated
Changed in sosreport (Ubuntu):
assignee: nobody → nikhil kshirsagar (nkshirsagar)
Changed in sosreport (Ubuntu Bionic):
assignee: nobody → nikhil kshirsagar (nkshirsagar)
Changed in sosreport (Ubuntu Jammy):
assignee: nobody → nikhil kshirsagar (nkshirsagar)
Changed in sosreport (Ubuntu):
importance: Undecided → Medium
Changed in sosreport (Ubuntu Bionic):
importance: Undecided → Medium
Changed in sosreport (Ubuntu Jammy):
importance: Undecided → Medium
Changed in sosreport (Ubuntu Focal):
importance: Undecided → Medium
Revision history for this message
nikhil kshirsagar (nkshirsagar) wrote :
tags: added: sts
tags: added: seg
removed: sts
Revision history for this message
Eric Desrochers (slashd) wrote :

[Kinetic sponsor]

Hi Nikhil,

I have sponsored your kinetic patch.

Small modification:
- Change the version from 4.4-1ubuntu3 to 4.4-1ubuntu0 in d/changelog.

Changed in sosreport (Ubuntu):
status: New → In Progress
Revision history for this message
Eric Desrochers (slashd) wrote :

[Note for other uploads]

Upstream mentions python-magic should be equal or above 0.4.20.

See here:
https://github.com/sosreport/sos/pull/2897/commits/9125518a4a6b7522e9987760ec734d98446d3395

...
+ python_magic>=0.4.20
...

Some Ubuntu stable version doesn't have that level of python3-magic. It may work anyway, but worth investigating a little bit more before.

# rmadison:
 python3-magic | 2:0.4.15-1 | bionic | all
 python3-magic | 2:0.4.15-3 | focal | all
 python3-magic | 2:0.4.24-2 | jammy | all
 python3-magic | 2:0.4.26-2 | kinetic | all

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package sosreport - 4.4-1ubuntu0

---------------
sosreport (4.4-1ubuntu0) kinetic; urgency=medium

  * New 4.4 upstream. (LP: #1986611)

  * For more details, full release note is available here:
    - https://github.com/sosreport/sos/releases/tag/4.4

  * Former patches, now fixed:
    - d/p/0002-fix-setup-py.patch
    - d/p/0003-mention-sos-help-in-sos-manpage.patch

  * d/control:
   - Add 'python3-magic' as part of the build depends.

  * Remaining patches:
    - d/p/0001-debian-change-tmp-dir-location.patch

 -- Nikhil Kshirsagar <email address hidden> Wed, 17 Aug 2022 08:38:30 +0000

Changed in sosreport (Ubuntu):
status: In Progress → Fix Released
Revision history for this message
nikhil kshirsagar (nkshirsagar) wrote (last edit ):

4.3->4.4 focal debdiff

Revision history for this message
nikhil kshirsagar (nkshirsagar) wrote (last edit ):

4.3->4.4 bionic debdiff

Revision history for this message
nikhil kshirsagar (nkshirsagar) wrote (last edit ):

uploading 4.4 jammy debdiff

Revision history for this message
nikhil kshirsagar (nkshirsagar) wrote :

Thank you for sponsoring the patch into Kinetic Eric!

For bionic and focal, we decided to remove the python3-magic call for binary file detection, so for eg. this patch is added for bionic (and a similar one for focal) to revert to the older way. We still keep that function (file_is_binary() ) in utilities as is the case in 4.4, but the way it does binary file detection is reverted to 4.3 approach. Accordingly, we removed the magic stuff from requirements.txt and setup.py too. Please do let me know if you think it should remain in there for reasons we may have overlooked! We have let it remain though in snap/snapcraft.yaml and sos.spec.

~~~
Description: Revert to old way of binary file detection
 Reverting to older (4.3) way of detecting binary files
 because sos 4.4 uses python3-magic version 0.4.20 which is
 not present in bionic.
 .
 sosreport (4.4-1ubuntu0.18.04.1) bionic; urgency=medium
 .
   * New 4.4 upstream. (LP: #1986611)
 .
   * For more details, full release note is available here:
     - https://github.com/sosreport/sos/releases/tag/4.4
 .
   * Former patches, now fixed:
     - d/p/0002-fix-setup-py.patch
     - d/p/0003-mention-sos-help-in-sos-manpage.patch
 .
   * Remaining patches:
     - d/p/0001-debian-change-tmp-dir-location.patch
 .
   * New patches:
     - d/p/0002-revert-to-old-style-binary-file-detection.patch
Author: Nikhil Kshirsagar <email address hidden>
Bug-Ubuntu: https://bugs.launchpad.net/bugs/1986611

--- sosreport-4.4.orig/requirements.txt
+++ sosreport-4.4/requirements.txt
@@ -2,5 +2,4 @@ pycodestyle>=2.4.0
 coverage>=4.0.3
 Sphinx>=1.3.5
 pexpect>=4.0.0
-python_magic>=0.4.20
 pyyaml
--- sosreport-4.4.orig/setup.py
+++ sosreport-4.4/setup.py
@@ -107,7 +107,7 @@ setup(
     ],
     cmdclass=cmdclass,
     command_options=command_options,
- requires=['pexpect', 'python_magic', 'pyyaml']
+ requires=['pexpect', 'pyyaml']
     )

--- sosreport-4.4.orig/sos/utilities.py
+++ sosreport-4.4/sos/utilities.py
@@ -19,7 +19,6 @@ import tempfile
 import threading
 import time
 import io
-import magic

 from contextlib import closing
 from collections import deque
@@ -75,17 +74,14 @@ def file_is_binary(fname):
     :returns: True if binary, else False
     :rtype: ``bool``
     """
- try:
- _ftup = magic.detect_from_filename(fname)
- _mimes = ['text/', 'inode/']
- return (
- _ftup.encoding == 'binary' and not
- any(_ftup.mime_type.startswith(_mt) for _mt in _mimes)
- )
- except Exception:
- # if for some reason this check fails, don't blindly remove all files
- # but instead rely on other checks done by the component
- return False
+ with open(fname, 'tr') as tfile:
+ try:
+ # when opened as above (tr), reading binary content will raise
+ # an exception
+ tfile.read(1)
+ return False
+ except UnicodeDecodeError:
+ return True

 def find(file_pattern, top_dir, max_depth=None, path_pattern=None):
~~~

Revision history for this message
Chris Halse Rogers (raof) wrote : Please test proposed package

Hello nikhil, or anyone else affected,

Accepted sosreport into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/sosreport/4.4-1ubuntu0.18.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in sosreport (Ubuntu Bionic):
status: New → Fix Committed
tags: added: verification-needed verification-needed-bionic
Revision history for this message
Chris Halse Rogers (raof) wrote :

Hello nikhil, or anyone else affected,

Accepted sosreport into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/sosreport/4.4-1ubuntu0.20.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in sosreport (Ubuntu Focal):
status: New → Fix Committed
tags: added: verification-needed-focal
Revision history for this message
Chris Halse Rogers (raof) wrote :

Hello nikhil, or anyone else affected,

Accepted sosreport into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/sosreport/4.4-1ubuntu1.22.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in sosreport (Ubuntu Jammy):
status: New → Fix Committed
tags: added: verification-needed-jammy
Revision history for this message
Ubuntu SRU Bot (ubuntu-sru-bot) wrote : Autopkgtest regression report (sosreport/4.4-1ubuntu0.18.04.1)

All autopkgtests for the newly accepted sosreport (4.4-1ubuntu0.18.04.1) for bionic have finished running.
The following regressions have been reported in tests triggered by the package:

sosreport/4.4-1ubuntu0.18.04.1 (arm64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/bionic/update_excuses.html#sosreport

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

tags: added: verification-done-bionic verification-done-focal verification-done-jammy
removed: verification-needed verification-needed-bionic verification-needed-focal verification-needed-jammy
Revision history for this message
nikhil kshirsagar (nkshirsagar) wrote :

I have tested the 4.4 proposed packages on bionic/focal/jammy and added the verification-done tags. (https://pastebin.canonical.com/p/DSmzHM5vgF/ has some testing details, and the https://wiki.ubuntu.com/SosreportUpdates procedure was followed to test the releases)

The one autopkgtest failure (on bionic+arm64) does not appear to be a real issue, and seems like a badtest to me. Manually running the upstream test suite on 4.4 did not show me any avocado test failures.

-Nikhil.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package sosreport - 4.4-1ubuntu1.22.04.1

---------------
sosreport (4.4-1ubuntu1.22.04.1) jammy; urgency=medium

  * New 4.4 upstream. (LP: #1986611)

  * For more details, full release note is available here:
    - https://github.com/sosreport/sos/releases/tag/4.4

  * Former patches, now fixed:
    - d/p/0002-fix-setup-py.patch
    - d/p/0003-mention-sos-help-in-sos-manpage.patch

  * d/control:
   - Add 'python3-magic' as part of the build depends.

  * Remaining patches:
    - d/p/0001-debian-change-tmp-dir-location.patch

 -- Nikhil Kshirsagar <email address hidden> Wed, 17 Aug 2022 05:59:06 +0000

Changed in sosreport (Ubuntu Jammy):
status: Fix Committed → Fix Released
Revision history for this message
Chris Halse Rogers (raof) wrote : Update Released

The verification of the Stable Release Update for sosreport has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package sosreport - 4.4-1ubuntu0.18.04.1

---------------
sosreport (4.4-1ubuntu0.18.04.1) bionic; urgency=medium

  * New 4.4 upstream. (LP: #1986611)

  * For more details, full release note is available here:
    - https://github.com/sosreport/sos/releases/tag/4.4

  * Former patches, now fixed:
    - d/p/0002-fix-setup-py.patch
    - d/p/0003-mention-sos-help-in-sos-manpage.patch

  * Remaining patches:
    - d/p/0001-debian-change-tmp-dir-location.patch

  * New patches:
    - d/p/0002-revert-to-old-style-binary-file-detection.patch

 -- Nikhil Kshirsagar <email address hidden> Wed, 17 Aug 2022 03:50:42 +0000

Changed in sosreport (Ubuntu Bionic):
status: Fix Committed → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package sosreport - 4.4-1ubuntu0.20.04.1

---------------
sosreport (4.4-1ubuntu0.20.04.1) focal; urgency=medium

  * New 4.4 upstream. (LP: #1986611)

  * For more details, full release note is available here:
    - https://github.com/sosreport/sos/releases/tag/4.3

  * New patches:
    - d/p/0002-revert-to-old-style-binary-file-detection.patch

  * Former patches, now fixed:
    - d/p/0002-fix-setup-py.patch
    - d/p/0003-mention-sos-help-in-sos-manpage.patch

  * Remaining patches:
    - d/p/0001-debian-change-tmp-dir-location.patch

 -- Nikhil Kshirsagar <email address hidden> Wed, 17 Aug 2022 04:21:47 +0000

Changed in sosreport (Ubuntu Focal):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.