[FFe][sync][sru] sos upstream 4.1

[Eric Desrochers]

[IMPACT]

The sos team is pleased to announce the release of sos-4.1. This is the first minor version update to the 4.x line which introduced a new design and project direction. Included in this release are numerous bug fixes and feature enhancements focused on increasing the stability and reliability of the previous major version update.

https://github.com/sosreport/sos/releases/tag/4.1

It will also address and close the following reported LP:
(LP: #1910264)
(LP: #1906302)
(LP: #1913284)
(LP: #1913583)
(LP: #1913581)
(LP: #1915072)

[TEST PLAN]

Documentation for Special Cases:
https://wiki.ubuntu.com/SosreportUpdates

[WHERE PROBLEM COULD OCCUR]

Regression could occur at core functionality which may prevent sos (or its subcommand to work. I consider this regression type as 'low'. That is generally well tested, and we would find a problem at early stage during the verification phase if it is the case.

On the other end, regression could happen and are some kind of expected at plugins levels. As of today, sos has more than 300 plugins. It is nearly impossible to test them all.

If a regression is found in a plugin, it is rarely affecting sos core functionalities nor other plugins. So mainly the impact would be limited to that plugin. The impact being that the plugin can't or partially can collect the information that it is instructed to gather.

A 3rd party vendor would then ask user/customer to collect the information manually for that particular plugins.

Plugins are segmentated by services and/or applications (e.g. openstack_keystone, bcache, system, logs, ...) in order to collect things accordingly to the plugin detected or intentionally requested for.

[OTHER INFORMATION]

Debian RFS:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=984805

Upload queue:
https://mentors.debian.net/package/sosreport/

[Eric Desrochers (slashd)]

sos 4.1-1 uploaded in Debian mentors upload queue.

[Eric Desrochers (slashd)]

Quick updates:

* Debian didn't approve the upload yet.
* Hirsute is in freeze schedule at the moment. I'll contact sil2100 and see what are our option other that wait for hirsute to become stable (if any other option)

[Łukasz Zemczak (sil2100)]

Knowing how thorough the sosreport testing is, I'm fine to get this in now - but not later than the Beta Freeze. FFe approved, upload ASAP!

[Eric Desrochers (slashd)]

[VERIFICATION HIRSUTE]

Pre upload verification have been successful, testing with the upstream simple.sh script

# simple.sh

Summary

 failures false time 2 Size 0.00MiB List plugins
 failures false time 1 Size 0.00MiB List presets
 failures false time 2 Size 0.00MiB List profiles
 failures false time 5 Size 0.00MiB --build -t1 -o host,kernel,filesys,hardware,date,logs
 failures false time 16 Size 3.59MiB -vvv
 failures false time 15 Size 3.62MiB -e opencl -v -k kernel.with-timer,libraries.ldconfigv --no-postproc
 failures false time 3 Size 11.82KiB --no-report --label TEST -o hardware
 failures false time 14 Size 4.21MiB --log-size 0 --no-env-vars -z gzip
 failures false time 31 Size 3.33MiB --mask
Everything worked!

I had to add "isolation-machine" to d/t/controls as simple.sh interact with kernel, modules and all.

[Eric Desrochers (slashd)]

[LINTIAN]

LINTIAN testing doesn't reveal anything important:

I: sosreport source: patch-not-forwarded-upstream debian/patches/0001-debian-change-tmp-dir-location.patch
X: sosreport source: debian-watch-does-not-check-gpg-signature
P: sosreport source: silent-on-rules-requiring-root
X: sosreport source: upstream-metadata-file-is-missing

[Eric Desrochers (slashd)]

Uploaded into Hirsute.

[Eric Desrochers (slashd)]

Sorry I have unintentionally omitted to add the (LP: #1917894) in d/changelog.

One can find the changelog here:
https://launchpad.net/ubuntu/+source/sosreport/4.1-1ubuntu1

- Eric

[Eric Desrochers (slashd)]

Fix Released in Hirsute:

-----
sosreport (4.1-1ubuntu1) hirsute; urgency=medium

  * New 4.1 upstream minor release.
    - https://github.com/sosreport/sos/releases/tag/4.1

  * d/test/simple.sh:
    - Update the script from upstream
    - Modify the script to use /tmp as a target, instead
      of /var/tmp.

  * d/test/control:
    - Adding isolation-machine as simple.sh wants to
      interact with the kernel.

  * Former patches:
    - d/p/0002-fix-dict-order-py38-incompatibility.patch
    - d/p/0003-sosclean-fix-handling-of-filepath-with-archive-name.patch
    - d/p/0004-sosclean-fix-tarball-skipping-regex.patch
    - d/p/0005-ceph-collect-balancer-and-pg-autoscale-status.patch
    - d/p/0006-rabbitmq-add-info-on-maybe-stuck-processes.patch
    - d/p/0007-rabbitmq-add-10sec-timeout-to-call-to-maybestuck.patch
    - d/p/0008-networking-include-ip-neigh-and-rule-info.patch
    - d/p/0009-conntrack-add-conntrack-info.patch
    - d/p/0010-conntrack-gather-per-namespace-data.patch
    - d/p/0011-ceph-include-time-sync-status-for-ceph-mon.patch
    - d/p/0012-apt-move-unattended-upgrades-log-collection.patch
    - d/p/0013-bcache-add-a-new-plugin-for-bcache.patch
    - d/p/0014-k8s-add-cdk-master-auth-webhook-to-journal.patch
    - d/p/0015-k8s-fix-cdk-related-file-paths.patch
    - d/p/0016-systemd-prefer-resolvectl-over-systemd-resolve.patch
    - d/p/0017-ovn-extend-information.patch
    - d/p/0018-ua-prefer-new-ua-cmd-over-the-deprecated-one.patch
    - d/p/0019-ovn-fix-sbctl-cmd-execution.patch

  * Remaining patches:
   - d/p/0001-debian-change-tmp-dir-location.patch

  * New patches:
   - d/p/0002-clean-prevent-parsing-ubuntu-user.patch
-----

[Eric Desrochers (slashd)]

Uploaded in Groovy (now waiting for SRU approval).

Next step when I have some time: Focal & Bionic.

- Eric

[Łukasz Zemczak (sil2100)]

Hello Eric, or anyone else affected,

Accepted sosreport into groovy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/sosreport/4.1-1ubuntu0.20.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-groovy to verification-done-groovy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-groovy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Eric Desrochers (slashd)]

[VERIFICATION GROOVY]

* "simple.sh" reveal that everything is working as expected as part of the autopkgtest.
* Did some functionnal sosreport testing and everything worked as expected as well.

* No errors reported in sos_logs/
* Both "sosreport" (legacy) and "sos" (new binary) binaries works.
* 'sos' subcommands work as expected (clean, collect, ...)

* User obfuscation (new feature in 'sos clean') report no issue. I have catch one issue that I was able to fixed upstream pre-released via : https://github.com/sosreport/sos/commit/70d4e1a1e901ada4e84005a93c2d23f6116aa2b5

All look good from my side, but more feedbacks are welcome.

[Eric Desrochers (slashd)]

Uploaded in Focal/20.04LTS. Waiting for SRU approval.

[Eric Desrochers (slashd)]

Uploaded in Bionic/18.04LTS. Waiting for SRU approval.

Please take extra care during Bionic testing, as it is passing from 3.9 to 4.1 as opposed to other where it passes from 4.0 to 4.1.

- Eric

[Łukasz Zemczak (sil2100)]

Hello Eric, or anyone else affected,

Accepted sosreport into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/sosreport/4.1-1ubuntu0.20.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Łukasz Zemczak (sil2100)]

Hello Eric, or anyone else affected,

Accepted sosreport into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/sosreport/4.1-1ubuntu0.18.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted sosreport (4.1-1ubuntu0.20.04.1) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

sosreport/4.1-1ubuntu0.20.04.1 (amd64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#sosreport

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted sosreport (4.1-1ubuntu0.18.04.1) for bionic have finished running.
The following regressions have been reported in tests triggered by the package:

sosreport/4.1-1ubuntu0.18.04.1 (s390x, arm64, i386, ppc64el, amd64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/bionic/update_excuses.html#sosreport

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Eric Desrochers (slashd)]

[Focal regression / autopkgtest]

After a few retries, the test passed. I had the same failure at the first run with Hirsute.
http://autopkgtest.ubuntu.com/packages/s/sosreport/focal/amd64

[Eric Desrochers (slashd)]

[Bionic regression / autopkgtest]

simple.sh is looking if it doesn't load module by happenstance by comparing the list before and after, and Bionic w/ 4.15 kernel reveal that 'devlink' is added along the way. I was able to isolate it to the networking plugin:

root@srv:~# lsmod | grep -i devlink
root@srv:~# sos report -o networking

sosreport (version 4.1)

.....

Your sosreport has been generated and saved in:
 /tmp/sosreport-srv-2021-03-31-tvjgeyv.tar.xz

 Size 29.75KiB
 Owner root
 sha256 72d49c26463f8fe42f11696dbc69aaa6eaf4e621551d88822b72a6f27ed7a136

Please send this file to your support representative.

More precisely to this specific code line:
https://github.com/sosreport/sos/blob/master/sos/report/plugins/networking.py#L109

# Manual reproducer:
root@srv:~# lsmod | grep -i devlink
root@srv:~# devlink dev
root@srv:~# lsmod | grep -i devlink
devlink 45056 0

I can reproduce the behaviour every time only on 4.15 but not on more recent kernel (5.4).

I highly suspect that this is due to following kernel configuration:

config-4.15.0-140-generic:CONFIG_MAY_USE_DEVLINK=m
config-5.4.0-70-generic:CONFIG_NET_DEVLINK=y

With that being said, it is safe to ignore the failure here for now as it has nothing to do with sosreport. I already addressed it with sos upstream. And I'll try work on mitigation/workaround in sosreport eventually, maybe using SoSpredicate or some sort.

- Eric

[Arif Ali (arif-ali)]

Tested on bionic, focal, groovy and hirsute, specifically the --clean and --mask options

Ran these commands, and all came out

sos report --mask --all-logs -a --batch
sos report --clean --all-logs -a --batch

sosreport --mask --all-logs -a --batch
sosreport --mask --all-logs -a --batch

sos report --all-logs -a --batch
sos clean sosreport-as1-maas-node-01-2021-04-08-xqdpdbp.tar.xz --batch

The only thing that sosreport doesn't obfuscate are the journal files, but I think that is expected as these are binary files.

I have opened a separate issue on this in the upstream project [1], but I think this was expected

[1] https://github.com/sosreport/sos/issues/2478

[Eric Desrochers (slashd)]

It was indeed expected, Thanks Arif.

[David Negreira (dnegreira)]

My tests were mainly focused on Openstack hence I have run the tests on Bionic and Focal only, I have tested the following application within an OpenStack environment:

* ceph-mon
* ceph-osd
* ceph-radosgw
* cinder
* cinder-ceph
* glance
* keystone
* mysql
* neutron-api
* neutron-gateway
* neutron-openvswitch
* nova-cloud-controller
* nova-compute
* octavia
* openstack-dashboard
* placement
* rabbitmq-server

I have run the following command on each of the applications:
sos report -a --all-logs

They all returned the expected application data as well as the logs, no errors found.

[Eric Desrochers (slashd)]

Thanks David !

[Eric Desrochers (slashd)]

Waiting for 2 more bugs verification #1913583 #1915072, and I should be good to ask SRU team to promote it into -updates.

[Eric Desrochers (slashd)]

[Testing note][Minor regression]

--upload is no longer working with sos 4.1.

There is an ongoing discussion upstream to address the situation.

I don't judge sos inability to upload an archive via its cmd important enough to be a blocker.
One can do the upload separately, just like it was before --upload option didn't exist.

--upload is a convenience more than a core functionality here.

With that being said, as said we are working at addressing it upstream as we speak, and I will SRU the fix in sos asap. Once the package is released and upstream commit merged.

I have reported LP: #1923209 in order to keep track of it.

- Eric

[Launchpad Janitor (janitor)]

This bug was fixed in the package sosreport - 4.1-1ubuntu0.20.10.1

---------------
sosreport (4.1-1ubuntu0.20.10.1) groovy; urgency=medium

  * New 4.1 upstream minor release. (LP: #1917894)
    - https://github.com/sosreport/sos/releases/tag/4.1

  * d/tests/simple.sh:
    - Update the script from upstream
    - Modify the script to use /tmp as a target, instead
      of /var/tmp.

  * d/tests/control:
    - Adding isolation-machine as simple.sh wants to
      interact with the kernel.

  * Former patches, now fixed:
    - d/p/0002-fix-dict-order-py38-incompatibility.patch
    - d/p/0003-sosclean-fix-handling-of-filepath-with-archive-name.patch
    - d/p/0004-sosclean-fix-tarball-skipping-regex.patch
    - d/p/0005-ceph-collect-balancer-and-pg-autoscale-status.patch
    - d/p/0006-rabbitmq-add-info-on-maybe-stuck-processes.patch
    - d/p/0007-rabbitmq-add-10sec-timeout-to-call-to-maybestuck.patch
    - d/p/0008-networking-include-ip-neigh-and-rule-info.patch
    - d/p/0009-conntrack-add-conntrack-info.patch
    - d/p/0010-conntrack-gather-per-namespace-data.patch
    - d/p/0011-ceph-include-time-sync-status-for-ceph-mon.patch
    - d/p/0012-apt-move-unattended-upgrades-log-collection.patch
    - d/p/0013-bcache-add-a-new-plugin-for-bcache.patch
    - d/p/0014-k8s-add-cdk-master-auth-webhook-to-journal.patch
    - d/p/0015-k8s-fix-cdk-related-file-paths.patch
    - d/p/0016-systemd-prefer-resolvectl-over-systemd-resolve.patch
    - d/p/0017-ovn-extend-information.patch
    - d/p/0018-ua-prefer-new-ua-cmd-over-the-deprecated-one.patch
    - d/p/0019-ovn-fix-sbctl-cmd-execution.patch

  * Remaining patches:
   - d/p/0001-debian-change-tmp-dir-location.patch

  * New patches:
   - d/p/0002-clean-prevent-parsing-ubuntu-user.patch

  * Fixing the following LP bugs:
   - (LP: #1910264)
   - (LP: #1906302)
   - (LP: #1913284)
   - (LP: #1913583)
   - (LP: #1913581)
   - (LP: #1915072)

 -- Eric Desrochers <email address hidden> Tue, 23 Mar 2021 19:56:28 +0000

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for sosreport has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package sosreport - 4.1-1ubuntu0.20.04.1

---------------
sosreport (4.1-1ubuntu0.20.04.1) focal; urgency=medium

  * New 4.1 upstream minor release. (LP: #1917894)
    - https://github.com/sosreport/sos/releases/tag/4.1

  * d/tests/simple.sh:
    - Update the script from upstream
    - Modify the script to use /tmp as a target, instead
      of /var/tmp.

  * d/tests/control:
    - Adding isolation-machine as simple.sh wants to
      interact with the kernel.

  * Former patches, now fixed:
    - d/p/0002-fix-dict-order-py38-incompatibility.patch
    - d/p/0003-sosclean-fix-handling-of-filepath-with-archive-name.patch
    - d/p/0004-sosclean-fix-tarball-skipping-regex.patch
    - d/p/0005-ceph-collect-balancer-and-pg-autoscale-status.patch
    - d/p/0006-rabbitmq-add-info-on-maybe-stuck-processes.patch
    - d/p/0007-rabbitmq-add-10sec-timeout-to-call-to-maybestuck.patch
    - d/p/0008-networking-include-ip-neigh-and-rule-info.patch
    - d/p/0009-conntrack-add-conntrack-info.patch
    - d/p/0010-conntrack-gather-per-namespace-data.patch
    - d/p/0011-ceph-include-time-sync-status-for-ceph-mon.patch
    - d/p/0012-apt-move-unattended-upgrades-log-collection.patch
    - d/p/0013-bcache-add-a-new-plugin-for-bcache.patch
    - d/p/0014-k8s-add-cdk-master-auth-webhook-to-journal.patch
    - d/p/0015-k8s-fix-cdk-related-file-paths.patch
    - d/p/0016-systemd-prefer-resolvectl-over-systemd-resolve.patch
    - d/p/0017-ovn-extend-information.patch
    - d/p/0018-ua-prefer-new-ua-cmd-over-the-deprecated-one.patch
    - d/p/0019-ovn-fix-sbctl-cmd-execution.patch

  * Remaining patches:
   - d/p/0001-debian-change-tmp-dir-location.patch

  * New patches:
   - d/p/0002-clean-prevent-parsing-ubuntu-user.patch

  * Fixing the following LP bugs:
   - (LP: #1910264)
   - (LP: #1906302)
   - (LP: #1913284)
   - (LP: #1913583)
   - (LP: #1913581)
   - (LP: #1915072)

 -- Eric Desrochers <email address hidden> Thu, 25 Mar 2021 19:02:37 +0000

[Launchpad Janitor (janitor)]

This bug was fixed in the package sosreport - 4.1-1ubuntu0.18.04.1

---------------
sosreport (4.1-1ubuntu0.18.04.1) bionic; urgency=medium

  * New 4.1 upstream minor release. (LP: #1917894)
    - https://github.com/sosreport/sos/releases/tag/4.1

  * d/tests/*:
    - Remove obsolete scripts

  * d/tests/simple.sh:
    - Update the script from upstream to match sos-4.1
    - Modify the script to use /tmp as a target, instead
      of sos default /var/tmp.

  * d/tests/control:
    - Adding isolation-machine as simple.sh wants to
      interact with the kernel.

  * New config file location now under /etc/sos/sos.conf
    - The old config (/etc/sos.conf) contents will not be
      removed nor carried over after update. Users will
      have to modify the new file instead (as needed).

  * Former patches, now fixed:
    - d/p/0001-lshw-command.patch
    - d/p/0002-lds-substitute-oidc-conf.patch
    - d/p/0003-kvm-change-trigger-to-dev-kvm.patch
    - d/p/0004-maas-add-snap-support.patch
    - d/p/0005-conntrack-add-conntrack-info.patch
    - d/p/0006-conntrack-gather-per-namespace-data.patch
    - d/p/0007-networking-include-ns-ip-neigh-and-ip-rule-info.patch

  * New patches:
   - d/p/0001-debian-change-tmp-dir-location.patch
   - d/p/0002-clean-prevent-parsing-ubuntu-user.patch

  * Fixing the following LP bugs:
   - (LP: #1910264)
   - (LP: #1906302)
   - (LP: #1913284)
   - (LP: #1913583)
   - (LP: #1913581)
   - (LP: #1915072)

 -- Eric Desrochers <email address hidden> Fri, 26 Mar 2021 13:47:44 +0000