[sru] sos upstream 4.2

[Eric Desrochers]

[IMPACT]

The sos team is pleased to announce the release of sos-4.2. This release contains numerous improvements and bug fixes to several components within sos, including an overhaul to the project's test suite and infrastructure.

[TEST PLAN]

Documentation for Special Cases:
https://wiki.ubuntu.com/SosreportUpdates

[WHERE PROBLEM COULD OCCUR]

Regression could occur at core functionality, which may prevent sos (or its subcommand to work. I consider this regression type as 'low'. That is generally well tested, and we would find a problem at an early stage during the verification phase if it is the case.

On the other end, regression could happen and are some kind of expected at plugins levels. As of today, sos has more than 300 plugins. It is nearly impossible to test them all.

If a regression is found in a plugin, it is rarely affecting sos core functionalities nor other plugins. So mainly the impact would be limited to that plugin. The impact being that the plugin can't or partially can collect the information that it is instructed to gather.

A 3rd party vendor would then ask user/customer to collect the information manually for that particular plugins.

Plugins are segmented by services and/or applications (e.g. openstack_keystone, bcache, system, logs, ...) in order to collect things accordingly to the plugin detected or intentionally requested for.

Sosreport plugins philosophy is to (as much as possible) maintain backward compatibility when updating a plugin. The risk that an ancient version of a software has been dropped, is unlikely, unless it was intended to be that way for particular reasons. Certain plugin also support the DEB installation way and the snap one (MAAS, LXD, ...) so all Ubuntu standard installation types are covered.

[OTHER INFORMATION]

Release note:
https://github.com/sosreport/sos/releases/tag/4.2

Debian RFS:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=994564

[Eric Desrochers (slashd)]

# Note

Currently facing a problem with dh_auto_test: while running test w/ avocado-framework.

Avocado is not packaged in Ubuntu/Debian, and requires to be installed via 'pip3 install avocado-framework' which doesn't seem to be accepted by the builder.

Tried severals scenarios such as:

# d/rules:
override_dh_auto_test:
        pip3 install avocado-framework
 export PYTHONPATH=tests/
 avocado run -t stageone tests/{cleaner,collect,report,vendor}_tests

but got the following error at build time from my PPA:

debian/rules override_dh_auto_test
make[1]: Entering directory '/<<PKGBUILDDIR>>'
pip3 install avocado-framework
WARNING: The directory '/sbuild-nonexistent/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f5a4cd153d0>: Failed to establish a new connection: [Errno -2] Name or service not known')': /simple/avocado-framework/
WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f5a4cc68cd0>: Failed to establish a new connection: [Errno -2] Name or service not known')': /simple/avocado-framework/
WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f5a4cc680a0>: Failed to establish a new connection: [Errno -2] Name or service not known')': /simple/avocado-framework/
WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f5a4cc685b0>: Failed to establish a new connection: [Errno -2] Name or service not known')': /simple/avocado-framework/
WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f5a4cc68760>: Failed to establish a new connection: [Errno -2] Name or service not known')': /simple/avocado-framework/
ERROR: Could not find a version that satisfies the requirement avocado-framework
ERROR: No matching distribution found for avocado-framework

Still under investigation.

I might have no other option but to package avocado-framework and add it as a Build-Depends, as I'm not a fan of disabling dh_auto_test: .

- Eric

[Eric Desrochers (slashd)]

While I figure out the 'avocado' portion on the test build.

I have a package ready, produces by skipping the tests at build time that can be found here:

sudo add-apt-repository ppa:slashd/sos42test
sudo apt-get update

[Eric Desrochers (slashd)]

Other possible option is to revert the avocado change that has been made between 4.1 and 4.2 and continue the former way by reverting the necessary commits....

Example:
e325f38e [tests] use Avocado "recursive" behavior by default
f568fb21 [tests] Start using avocado for test suite

[Eric Desrochers (slashd)]

I'll evaluate how feasible avocado-framework can be "debianized" for impish (AKA active devel release), and test.

So that sos4.2 will have the necessary bits for its testing exactly like upstream does.
Then, I'll review what can be done for stable release. I'm afraid I won't be able to introduce a new package in stable release like I can do in devel.

Stay tuned !!

[Eric Desrochers (slashd)]

Still working on making the 'testsuite' working with avocado-framework.
impish reaches a few freezes at this point.

I'll continue the work, but will likely wait until 'k' become official devel release.

As I'll have to push avocado-framework as well as a new package.

[Eric Desrochers (slashd)]

I'll probably drop 'avocado-framework' testing for now and keep the nosetest.

override_dh_auto_test:
    nosetests3 -v --with-cover --cover-package=sos tests/unittests

[Steve Langasek (vorlon)]

Hello Eric, or anyone else affected,

Accepted sosreport into impish-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/sosreport/4.2-1ubuntu0.21.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-impish to verification-done-impish. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-impish. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Brian Murray (brian-murray)]

Hello Eric, or anyone else affected,

Accepted sosreport into hirsute-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/sosreport/4.2-1ubuntu0.21.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-hirsute to verification-done-hirsute. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-hirsute. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted sosreport (4.2-1ubuntu0.21.04.1) for hirsute have finished running.
The following regressions have been reported in tests triggered by the package:

sosreport/4.2-1ubuntu0.21.04.1 (s390x, amd64, ppc64el)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/hirsute/update_excuses.html#sosreport

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Launchpad Janitor (janitor)]

This bug was fixed in the package sosreport - 4.2-1ubuntu2

---------------
sosreport (4.2-1ubuntu2) jammy; urgency=medium

  * d/p/0006-report-check-for-symlink-before-rmtree.patch:
    - Fixing --estimate-only option by checking if the dirs
      are also symlink before performing rmtree() method so
      that unlink() method can be used instead.

 -- Eric Desrochers <email address hidden> Tue, 19 Oct 2021 14:20:44 -0400

[Łukasz Zemczak (sil2100)]

Hello Eric, or anyone else affected,

Accepted sosreport into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/sosreport/4.2-1ubuntu0.20.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted sosreport (4.2-1ubuntu0.20.04.1) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

sosreport/4.2-1ubuntu0.20.04.1 (ppc64el, amd64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#sosreport

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Eric Desrochers (slashd)]

[VERIFICATION HIRSUTE/IMPISH]

Since theses 2 releases are non-LTS. They are less exposed to various context & environment types such as: Openstack, Ceph, k8s, JuJu, MAAS, Landscape, ...

It's safe to say that non-LTS should be limited to general sosreport run, most likely in the Desktop area.

For that reasons, I don't judge we need to spend time testing non-LTS for such context/environments that typically belongs to LTS version.

The autopkgtest (simple.sh) was run successfully, and it already does a lot of general sosreport runs testing. I also tested sosreport myself on H/I and didn't see any problem.

With that being said, if someone is keen to do more test, please feel free to do so, but as the maintainer, I consider the verification done for H/I.

- Eric

[Eric Desrochers (slashd)]

Let's focus our time testing Focal/20.04LTS, which can cover various areas:
Openstack, Ceph, k8s, MAAS, juju, Landscape, ... (to only name those)

- Eric

[Łukasz Zemczak (sil2100)]

Ok, I think the rationale seems sane. Let's proceed.

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for sosreport has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package sosreport - 4.2-1ubuntu0.21.04.1

---------------
sosreport (4.2-1ubuntu0.21.04.1) hirsute; urgency=medium

  * New 4.2 upstream release. (LP: #1941745)
    - This release contains numerous improvements
      and bug fixes to several components within sos,
      including an overhaul to the project's test suite
      and infrastructure.

  * For more details, full release note is available here:
    - https://github.com/sosreport/sos/releases/tag/4.2

  * Remaining patches:
   - d/p/0001-debian-change-tmp-dir-location.patch

  * New patches:
    - d/p/0002-report-implement_estimate-only.patch
    - d/p/0003-ceph-add-support-for-containerized-ceph-setup.patch
    - d/p/0004-ceph-split-plugin-by-components.patch
    - d/p/0005-openvswitch-get-userspace-datapath-implementations.patch
    - d/p/0006-report-check-for-symlink-before-rmtree.patch

  * Former patches, now fixed:
    - d/p/0002-clean-prevent-parsing-ubuntu-user.patch
    - d/p/0003-ubuntu-policy-fix-upload.patch
    - d/p/0004-chrony-configuration-can-now-be-fragmented.patch
    - d/p/0005-global-drop-plugin-version.patch
    - d/p/0006-networking-check-presence-of-devlink.patch
    - d/p/0007-sosnode-avoid-checksum-cleanup-if-no-archive.patch

  * d/control:
   - Add 'python3-coverage' as part of the build depends.

  * d/rules:
   - Fix misplaced and duplicated sos.conf file in /usr/config.

 -- Eric Desrochers <email address hidden> Sun, 24 Oct 2021 13:16:22 -0400

[Launchpad Janitor (janitor)]

This bug was fixed in the package sosreport - 4.2-1ubuntu0.21.10.1

---------------
sosreport (4.2-1ubuntu0.21.10.1) impish; urgency=medium

  * New 4.2 upstream release. (LP: #1941745)
    - This release contains numerous improvements
      and bug fixes to several components within sos,
      including an overhaul to the project's test suite
      and infrastructure.

  * For more details, full release note is available here:
    - https://github.com/sosreport/sos/releases/tag/4.2

  * Remaining patches:
   - d/p/0001-debian-change-tmp-dir-location.patch

  * New patches:
    - d/p/0002-report-implement_estimate-only.patch
    - d/p/0003-ceph-add-support-for-containerized-ceph-setup.patch
    - d/p/0004-ceph-split-plugin-by-components.patch
    - d/p/0005-openvswitch-get-userspace-datapath-implementations.patch
    - d/p/0006-report-check-for-symlink-before-rmtree.patch

  * Former patches, now fixed:
    - d/p/0002-clean-prevent-parsing-ubuntu-user.patch
    - d/p/0003-ubuntu-policy-fix-upload.patch
    - d/p/0004-chrony-configuration-can-now-be-fragmented.patch
    - d/p/0005-global-drop-plugin-version.patch
    - d/p/0006-networking-check-presence-of-devlink.patch
    - d/p/0007-sosnode-avoid-checksum-cleanup-if-no-archive.patch

  * d/control:
   - Add 'python3-coverage' as part of the build depends.

  * d/rules:
   - Fix misplaced and duplicated sos.conf file in /usr/config.

 -- Eric Desrochers <email address hidden> Wed, 20 Oct 2021 12:38:22 -0400

[Edward Hope-Morley (hopem)]

I've verified version 4.2-1ubuntu0.20.04.1 from focal-proposed and it looks good. I generated a sos with 4.1-1ubuntu0.20.04.3 and the newer version and diffed between them and the contents look sane.

[Launchpad Janitor (janitor)]

This bug was fixed in the package sosreport - 4.2-1ubuntu0.20.04.1

---------------
sosreport (4.2-1ubuntu0.20.04.1) focal; urgency=medium

  * New 4.2 upstream release. (LP: #1941745)
    - This release contains numerous improvements
      and bug fixes to several components within sos,
      including an overhaul to the project's test suite
      and infrastructure.

  * For more details, full release note is available here:
    - https://github.com/sosreport/sos/releases/tag/4.2

  * Remaining patches:
   - d/p/0001-debian-change-tmp-dir-location.patch

  * New patches:
    - d/p/0002-report-implement_estimate-only.patch
    - d/p/0003-ceph-add-support-for-containerized-ceph-setup.patch
    - d/p/0004-ceph-split-plugin-by-components.patch
    - d/p/0005-openvswitch-get-userspace-datapath-implementations.patch
    - d/p/0006-report-check-for-symlink-before-rmtree.patch

  * Former patches, now fixed:
    - d/p/0002-clean-prevent-parsing-ubuntu-user.patch
    - d/p/0003-ubuntu-policy-fix-upload.patch
    - d/p/0004-networking-check-presence-of-devlink.patch

  * d/control:
   - Add 'python3-coverage' as part of the build depends.

  * d/rules:
   - Fix misplaced and duplicated sos.conf file in /usr/config.

 -- Eric Desrochers <email address hidden> Mon, 01 Nov 2021 09:02:20 -0400

[Arif Ali (arif-ali)]

bionic was never released, so marking it invalid as we had 4.4