[sru] sos upstream 4.5.6

[nikhil kshirsagar]

[IMPACT]

The sos team is pleased to announce the release of sos-4.5.6 This release focused on expanding support in policies for more and multiple package managers.

[TEST PLAN]

Documentation for Special Cases:
https://wiki.ubuntu.com/SosreportUpdates

[WHERE PROBLEMS COULD OCCUR]

The changes in sosreport are described below. There are a number of changes related to report, plugins and policy. The changes in sosreport content may break existing tooling, so that will need extensive testing.

Global Changes
snaps are now automatically built and pushed with new releases.
Fixed an incompatibility issue with python-3.12 runtimes.

Report Changes
Added a new ceph profile for use in those types of environments.

Plugin Changes
Fixed an issue with the enablement of ceph plugins possibly be erroneously enabled.
Fixed an issue with multiline regexes raising exceptions on python 3.11 runtimes.
The lustre plugin now collects more debug parameters.
The ssh plugin now controls collection of user ~/.ssh/config files via the userconfs option.
The default for the option is set to true/on/enabled, preserving previous behavior.

Cleaner Changes
Obfuscated files will now have their permissions set to match the source file.
Refined the regex used to extract MAC addresses, and will now match underscore separated addresses.

For full information on the changes contained in this release, please refer to the Git commit logs. Further release information and tarballs are available at:

https://github.com/sosreport/sos/releases/tag/4.5.6

Please report any problems to the sos-devel mailing list, or the GitHub issue tracker:

https://github.com/sosreport/sos/issues/

[nikhil kshirsagar (nkshirsagar)]

debdiff for mantic and some testing for mantic - https://pastebin.canonical.com/p/gHysvvw4zX/

[Launchpad Janitor (janitor)]

This bug was fixed in the package sosreport - 4.5.6-0ubuntu1

---------------
sosreport (4.5.6-0ubuntu1) mantic; urgency=medium

  * New 4.5.6 upstream. (LP: #2028327)

  * For more details, full release note is available here:
    - https://github.com/sosreport/sos/releases/tag/4.5.6

  * Former patches, now fixed:
    - d/p/0002-regex-flags.patch

  * Remaining patches:
    - d/p/0001-debian-change-tmp-dir-location.patch

 -- Nikhil Kshirsagar <email address hidden> Fri, 21 Jul 2023 08:04:22 +0000

[Andreas Hasenack (ahasenack)]

Note that this test has been consistently failing:

######### DONE WITH -vvv #########
!!! TEST FAILED: new kernel modules loaded during execution !!!
tls

...
     test_normal_report: new kernel modules loaded during execution

Would be great if this could be fixed too.

[Andreas Hasenack (ahasenack)]

Hello nikhil, or anyone else affected,

Accepted sosreport into lunar-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/sosreport/4.5.6-0ubuntu1~23.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-lunar to verification-done-lunar. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-lunar. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Andreas Hasenack (ahasenack)]

Hello nikhil, or anyone else affected,

Accepted sosreport into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/sosreport/4.5.6-0ubuntu1~22.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Andreas Hasenack (ahasenack)]

Hello nikhil, or anyone else affected,

Accepted sosreport into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/sosreport/4.5.6-0ubuntu1~20.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted sosreport (4.5.6-0ubuntu1~20.04.1) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

sosreport/4.5.6-0ubuntu1~20.04.1 (ppc64el)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#sosreport

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[nikhil kshirsagar (nkshirsagar)]

I cannot reproduce the ip address mask failure that the autopkgtest failure flags,

tested on a ppc64le machine,

https://pastebin.canonical.com/p/DfZFYmvxKY/

I have restarted the test on focal to check if it is an infra issue.

[nikhil kshirsagar (nkshirsagar)]

Hi Andreas,

I don't think the mask autopkgtest failure is a blocker. It only is seen on ppc64le and when I tried on that architecture I could not reproduce it (earlier comment)

I've restarted that test on focal in case its an infra issue.

As for the failure due to the loading of the tls module, that is a real issue. It appears sos has been causing loading of the "tls" module for a long time now, in fact all the way back to the 4.2-1ubuntu2 version jammy onwards and it started when we moved to python 3.10. For eg - https://autopkgtest.ubuntu.com/packages/sosreport/jammy/amd64 shows the first failure on 2022-01-14,

4.2-1ubuntu2 python3-defaults/3.10.1-0ubuntu1 2022-01-14 15:03:07 UTC 0h 06m 52s - fail log   artifacts   ♻
4.2-1ubuntu2 python3-defaults/3.9.7-4 2021-11-07 06:17:56 UTC 0h 03m 04s - pass log   artifacts

We've finally realized this after you flagged it, so thank you! It seems we never got these autopkgtest failures reported on the SRU bugs for each sos release due to the fact that they were no longer regressions since they were happening since some time. However I am not sure why we never got autopkgtest results (even if it was a "no regressions" report for any release other than focal on the sru bugs for sos releases.)

I will open an upstream issue for this, and we've also tracked it down to the ubuntu plugin.

# lsmod | grep tls
# sos report -o ubuntu

sosreport (version 4.5.3)

This command will collect system configuration and diagnostic
information from this Ubuntu system.

For more information on Canonical visit:

        Community Website : https://www.ubuntu.com/
        Commercial Support : https://www.canonical.com

The generated archive may contain data considered sensitive and its
content should be reviewed by the originating organization before being
passed to any third party.

No changes will be made to system configuration.

Press ENTER to continue, or CTRL-C to quit.

Optionally, please enter the case id that you are generating this report for []:

 Setting up archive ...
 Setting up plugins ...
 Running plugins. Please wait ...

  Starting 1/1 ubuntu [Running: ubuntu]

  Finished running plugins
Creating compressed archive...

Your sosreport has been generated and saved in:
    /tmp/sosreport-nikhil-XXXX-dolysxz.tar.xz

 Size 9.52KiB
 Owner root
 sha256 0e693d420391ef1c20503a90b706f440bfb5882bcabda3981a8a93cf26296718

Please send this file to your support representative.

#
# lsmod | grep tls
tls 147456 0

The command the triggers it is the "ua status" command,

# lsmod | grep tls
# ua status

No Ubuntu Pro services are available to this system.

For a list of all Ubuntu Pro services, run 'pro status --all'

This machine is not attached to an Ubuntu Pro subscription.
See https://ubuntu.com/pro

Supported livepatch kernels are listed here: https://ubuntu.com/security/livepatch/docs/kernels
# lsmod | grep tls
tls 147456 0

Given that this is such an old issue, we still think the release should go ahead, but with this documented somewhere, so we ...

[Andreas Hasenack (ahasenack)]

> It seems we never got these autopkgtest failures reported on the SRU bugs for each sos release due to the
> fact that they were no longer regressions since they were happening since some time.

Correct, since it's not a new failure, it wasn't a blocker. And as you realized, it's easy to miss them because of that :)

For this SRU, the tls failure is not blocking indeed. But the ppc64el one should be addressed one way or the other. It looks odd, though, like a failure of the test script:

338s !!! TEST FAILED: IP address not obfuscated in all places !!!
338s grep: /tmp/sosreport/_test/*: No such file or directory

Unless that directory should really have content (I haven't checked the test code).

What you could do is trigger a migration-reference/0 test run. If it also fails with the previous version, than it was not caused by this update.

[nikhil kshirsagar (nkshirsagar)]

bionic marked wont-fix, we will push it to the ESM queue instead as discussed.

[Andreas Hasenack (ahasenack)]

https://ubuntu-archive-team.ubuntu.com/proposed-migration/focal/update_excuses.html#sosreport is clean now.

[nikhil kshirsagar (nkshirsagar)]

I have tested bionic, focal, jammy and lunar packages on openstack and ceph nodes. https://pastebin.canonical.com/p/8qQJ4hqHXX/

The testing looks good and there are no regressions.

Existing issues like the mask error on ppc64le and the TLS module loading issue by the ubuntu plugin ( https://github.com/sosreport/sos/issues/3326) are to be investigated and fixed in future releases.

Neither of these issues are SRU blocker (since retest test failed with the migration-reference/0 trigger), so we are good to release focal, jammy and lunar packages to updates.

We will spend time investigating the mask issue for PPC64EL and also fix the typo in the autopkgtest to print the problematic IP address for our next release.

I am marking the verification done flags.

[nikhil kshirsagar (nkshirsagar)]

> For this SRU, the tls failure is not blocking indeed. But the ppc64el one should be addressed one way or > the other. It looks odd, though, like a failure of the test script:

> 338s !!! TEST FAILED: IP address not obfuscated in all places !!!
> 338s grep: /tmp/sosreport/_test/*: No such file or directory

> Unless that directory should really have content (I haven't checked the test code).

The "no such file or directory" is a bug in the test, the path for grep should be /tmp/sosreport_test/* so that line of code has a "/", and will be fixed in the next release. Pasting the test for reference,

# test using mask
test_mask () {
 <snip>
        # only tests first interface
        ip_addr=$(ip route show default | awk '/default/ {print $3}')
        if [ "$(grep -rI $ip_addr /tmp/sosreport_test/*)" ]; then
            add_failure "IP address not obfuscated in all places"
            echo "$(grep -rI $ip_addr /tmp/sosreport/_test/*)" <=======
        fi
        update_failures
    fi
    update_summary "$cmd"
}

[nikhil kshirsagar (nkshirsagar)]

The "module load" autopkgtest failure ( https://autopkgtest.ubuntu.com/results/autopkgtest-jammy/jammy/amd64/s/sosreport/20220114_150307_778d8@/log.gz ) has been debugged as an issue related to openssl version used in python 3.10, and can be reproduced by this simple python code - rmmod tls and then run,

#!/usr/bin/env python3
import socket
import ssl
ctx = ssl.create_default_context()
sock = socket.create_connection(("example.com", 443))
sock = ctx.wrap_socket(sock, server_hostname="example.com")
print(sock)

This will cause the tls module to be left loaded after the code executes.

Another simple reproducer, -

$ sudo rmmod tls; echo "pre"; lsmod | grep tls; echo "user"; wget https://canonical.com -q -o /dev/null; lsmod | grep tls; sudo wget https://canonical.com -q -o /dev/null; echo "post"; lsmod | grep tls;

The openssl change https://github.com/openssl/openssl/commit/50ec750567e056fcecff2344c2d9044d81cc731b came with openssl 3.0

fixed in sosreport 4.5.7 onwards by a predicate - https://github.com/sosreport/sos/issues/3326

[nikhil kshirsagar (nkshirsagar)]

bionic handled through https://bugs.launchpad.net/ubuntu-pro/+bug/2030838

[Andreas Hasenack (ahasenack)]

Thanks for the detailed test execution logs in that pastebin. I saved it and attached it to the bug, because I think pastebins can be lost. And because I think it was behind a login wall. I don't think there is anything private in it.

[Andreas Hasenack (ahasenack)]

A few comments:

a) it's hard to map the test plan from https://wiki.ubuntu.com/SosreportUpdates to this very large pastebin. I would suggest to mark it clearly in the test logs where each step from that wiki page is being done. It's fine to remove or abbreviate command outputs, as long as the important bits stay there.

b) Please separate the tests for each ubuntu release, it's very easy to get lost in that large pastebin, specially when searching for a specific test: "in which ubuntu release was it made? Scroll up until you find the name"

c) I'm not finding some of the tests from the plan, in the pastebin. Specifically, I'm looking for lunar tests, and I didn't find these:
- sos report:
  - Validate that sosreport obfuscates sensible information for plugins instructed to do so such as:
landscape plugin, should obfuscate password(s) and secret-token from config file.
or any plugins (sos/plugins/) exercising the do_file_sub() method.
  - Inspect for 0 size file(s) within the archive and use common sense if legit or not (e.g. "command is not found" can be avoided for instance)

- sos clean/mask: only found one execution for it, and I think it's jammy. Where are the others? Furthermore, I didn't see these checks:
  - Make sure it generates a default_mapping file inside /etc/sos/cleaner/ (at first run)
  - Tarball accompanied sha256 checksum (just found the tarball)
  - sos clean execution logs (Not to share, keep it private)

- sos collect: no run whatsoever for any ubuntu release

I know these tests are time consuming, but it's what is documented in the wiki, and I'm just checking that. If some must be skipped for some reason, I think it should be justified. Or perhaps the dep8 test does these now?

[Arif Ali (arif-ali)]

sos collects results

focal: https://paste.ubuntu.com/p/JFHtzR47hF/
jammy: https://paste.ubuntu.com/p/5vRmmk33DH/
lunar: https://paste.ubuntu.com/p/STh3jGJrJV/

All worked as expected

[nikhil kshirsagar (nkshirsagar)]

In addition to the earlier tests (https://pastebin.canonical.com/p/8qQJ4hqHXX/) - I tested sos report, --all-logs, mask/clean on containers, VMs and bare-metal today. Everything looked good.

In containers -

lunar - https://pastebin.canonical.com/p/Pt337tBTD7/
jammy - https://pastebin.canonical.com/p/NdXcP3hRWg/
focal - https://pastebin.canonical.com/p/MP4kzX56NF/

In VMs -

lunar - https://pastebin.canonical.com/p/zTbdjhhk5c/
jammy - https://pastebin.canonical.com/p/VQYmyxBPsf/
focal - https://pastebin.canonical.com/p/SZd2PFYDjP/

bare-metal - general sos and clean tests.

jammy - https://pastebin.canonical.com/p/ms3XfVg5BF/
lunar - https://pastebin.canonical.com/p/fttWcGnzXz/
focal - https://pastebin.canonical.com/p/6DTBR9bJSN/

simple.sh runs on all 3 distros - https://pastebin.canonical.com/p/3yVpNrdx3F/

[Andreas Hasenack (ahasenack)]

The focal and lunar sos collect report has this error/warning about the version:

  Connection to 10.0.1.125 closed.:check_sos_version] Error checking sos version: Invalid version: '4.5.6-0ubuntu1~20.04.1'

and

  Connection to 10.202.214.78 closed.:check_sos_version] Error checking sos version: Invalid version: '4.5.6-0ubuntu1~23.04.1'

This warning is not present in the jammy run. Not a blocker, but curious. Maybe related to python pep440? https://peps.python.org/pep-0440/

[Launchpad Janitor (janitor)]

This bug was fixed in the package sosreport - 4.5.6-0ubuntu1~23.04.1

---------------
sosreport (4.5.6-0ubuntu1~23.04.1) lunar; urgency=medium

  * New 4.5.6 upstream. (LP: #2028327)

  * For more details, full release note is available here:
    - https://github.com/sosreport/sos/releases/tag/4.5.6

  * d/control:
   - Add 'python3-magic' as part of the runtime depends.
   - Add 'python3-pexpect' as part of the build depends.

  * d/rules:
    - skip running unittests/policy_tests.py due to the avocado dependency

  * Remaining patches:
    - d/p/0001-debian-change-tmp-dir-location.patch

 -- Nikhil Kshirsagar <email address hidden> Mon, 24 Jul 2023 04:30:32 +0000

[Andreas Hasenack (ahasenack)]

The verification of the Stable Release Update for sosreport has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package sosreport - 4.5.6-0ubuntu1~22.04.1

---------------
sosreport (4.5.6-0ubuntu1~22.04.1) jammy; urgency=medium

  * New 4.5.6 upstream. (LP: #2028327)

  * For more details, full release note is available here:
    - https://github.com/sosreport/sos/releases/tag/4.5.6

  * d/control:
   - Add 'python3-magic' as part of the runtime depends.
   - Add 'python3-pexpect' as part of the build depends.
   - Add 'python3-magic' as part of the build depends.

  * d/rules:
    - skip running unittests/policy_tests.py due to the avocado dependency

  * Remaining patches:
    - d/p/0001-debian-change-tmp-dir-location.patch

 -- Nikhil Kshirsagar <email address hidden> Tue, 25 Jul 2023 10:20:24 +0000

[Launchpad Janitor (janitor)]

This bug was fixed in the package sosreport - 4.5.6-0ubuntu1~20.04.1

---------------
sosreport (4.5.6-0ubuntu1~20.04.1) focal; urgency=medium

  * New 4.5.6 upstream. (LP: #2028327)

  * For more details, full release note is available here:
    - https://github.com/sosreport/sos/releases/tag/4.5.6

  * d/control:
   - Add 'python3-pexpect' as part of the build depends.

  * d/rules:
    - skip running unittests/policy_tests.py due to the avocado dependency

  * New patches:
    - d/p/0002-debian-remove-magic-stderr.patch (LP: #2028666)

  * Former patches, now fixed:
    - d/p/0002-revert-to-old-style-binary-file-detection.patch

  * Remaining patches:
    - d/p/0001-debian-change-tmp-dir-location.patch

 -- Nikhil Kshirsagar <email address hidden> Tue, 25 Jul 2023 10:52:47 +0000

[nikhil kshirsagar (nkshirsagar)]

18.04 release is on hold due to https://bugs.launchpad.net/ubuntu/+source/sosreport/+bug/2038648

[Mauricio Faria de Oliveira (mfo)]

18.04 Pro will receive an update to later series, not 4.5.x.