[sync][sru] sos upstream 4.0

[Eric Desrochers]

[Impact]
The sos team is pleased to announce the release of sos-4.0. This is a major version release that represents a significant change to the sos project, new features, and bug fixes.

https://github.com/sosreport/sos/releases/tag/4.0

[Test Case]
* Install sosreport 4.0-1* package
 ** Test new main binary 'sos' and it's former binary still present 'sosreport' (e.g. sosreport -a --all-logs & sos report -a --all-logs)
  ** Test new features:
    *** sos-collect (Collect sosreport from multiple node simultaneously)
    *** sos-clean || sos-mask (It's aim is to scrub potentially sensitive information from sosreports in a consistent manner, beyond the obfuscation done by plugins already.) Replacement of the former project called 'soscleaner' which no longer deliver development.

[Regression Potential]

Main binary formerly known as 'sosreport' has been officially renamed to 'sos' but 'sosreport' still exist and has been carried over for now to help to transition for package maintainer. No impact nor behavioural change for existing user. They are welcome to use the new binary or the former one as they wish.

-rwxr-xr-x 1 root root 1057 Aug 25 16:24 /usr/bin/sosreport
-rwxr-xr-x 1 root root 596 Aug 25 16:24 /usr/bin/sos

New functionalities has been added which could lead to possible bugs ,since they are fairly new addition, if not already caught by the travis/autopkg tests but if a problem arise it will be limited to the components itself and won't affect the main behaviour (which by the way remain the same, which is to collect a tarball using 'sosreport' or/and now 'sos report' subcommand.

To resume, main known existing behaviour remain the same but some new features which are limited/independent to each other has been added to offer new features such as :

* sos collect is a new sub command in this release, and is an integration of the standalone sos-collector project, with the aim being to collect sosreports from multiple systems simultaneously. Note that this sub-command requires python3-pexpect to be available. If the module is not available, sos collect will abort with an appropriate error message

* sos clean, also available as sos mask, is a newly added sub-command in this release and is an implementation of the standalone soscleaner project. Its aim is to scrub potentially sensitive information from sosreports in a consistent manner, beyond the obfuscation done by plugins already.

https://wiki.ubuntu.com/SosreportUpdates

sos4.0 now enforces archives to be stored in /var/tmp.
A debian patch has been made to continue to store in /tmp (current behaviour) to stay align with current behaviour but also because debian's systemd implementation doesn't provide a tmpfiles.d cleaner for /var/tmp. So /tmp remains the default "tmp-dir" location.

During the testing, I found a py38 incompatibility related to dictionnary order, that I fixed upstream and cherry-pick for Ubuntu release having py38 found in : d/p/0002-fix-dict-order-py38-incompatibility.patch.

Upstream commit:
https://github.com/sosreport/sos/pull/2207/commits/86dfd99931ac0199895cf5e41711fc9b1ab6feaf

The old config (/etc/sos.conf) contents will not be carried over after update. Users will have to modify the new file (/etc/sos/sos.conf) instead (if needed).

[Other Information]

https://github.com/sosreport/sos/releases/tag/4.0

[Original Description]
https://github.com/sosreport/sos/releases/tag/4.0

Major changes
Sos has been redesigned to provide functionality beyond the well known report data collection usage. It has been updated to provide
more functionality via sub-commands.

A new sos binary has replaced the former sosreport binary as the main entry point for the utility.

sos report is now used to generate sosreport tarballs. A sosreport binary is maintained as a redirection point and will now invoke sos report.
sos collect formally brings sos-collector into the main sos project, and is used to collect sosreports from multiple nodes simultaneously. A sos-collector binary is maintained as a redirection point and will invoke sos collect.
This means the standalone sos-collector utility will no longer be independently developed.
sos clean formally brings soscleaner-like functionality into the main sos project. This sub command will perform further data obfuscation on reports, such as scrubbing IP addresses, domain names, and user-provided keywords. See below for more information.
/etc/sos.conf has been moved to /etc/sos/sos.conf, and the layout of the config file has changed:

The general section has been renamed to global, and may be used to specify options that are available to all sos commands and sub-commands.
Each sub-command will have its own section, e.g. sos report will load options from global and from report.
Sos is now a Python3-only utility. Python2 is no longer supported in any capacity.

Dropped use of make, building/installing sos from source should now exclusively be done via setuptools

Report
sos will now generate metadata and save it in sos_reports/manifest.json.

7 new plugins: nvmetcli, drbd, openstack_designate, pmem, containers_common, hyperv, freeipmi

The nfsserver plugin has been merged into the nfs plugin

sos may now be used to collect data from within a container, rather than aborting if that container was not configured to allow sos to collect information from the host

Added support for Container-Optimized OS (COS)

Dropped Mac OSX support

Dropped bzip2 compression support

Users may now use the --clean or --mask option to process a report-being-generated through sos clean at runtime.

Size limits will now apply to add_copy_spec() calls that target a directory

The openshift plugin has been re-written to be used for Openshift Container Platform 4

Significantly expanded the amount of API resources the openstack_octavia plugin will collect

The networking plugin will no longer execute ethtool -e against NICs using the bnx2x driver

The logs plugin will now capture journal information correctly when logs are stored in-memory only

The systemd plugin will no longer collect systemd-resolve if the service is not running

The openvswitch plugin has been significantly updated to pull more meaningful data, and now supports OpenFlow 1.4 and 1.5

Plugin API changes
The command execution/collection methods have been overhauled:

add_cmd_output() should continue to be used to specify commands that should be executed during the collection phase
exec_cmd() should now be used to execute commands and retrieve output during setup(), but will not save that output to the archive
collect_cmd_output() should be used to execute commands, retrieve output during setup() and will save that output to the archive
get_command_output() has been removed
A new add_device_cmd() method is available to facilitate easier iteration of commands over a set of devices

add_blockdev_cmd() has been added to facilitate iteration of commands of storage devices
A container runtime abstraction has been added that aims to standardize the discovery of a container runtime in use (e.g. docker, podman) and the retrieval of data from the runtime across plugins

Collect
sos collect is a new sub command in this release, and is an integration of the standalone sos-collector project, with the aim being to collect sosreports from multiple systems simultaneously. Note that this sub-command requires python3-pexpect to be available. If the module is not available, sos collect will abort with an appropriate error message

Compared to the standalone project, enhancements include:

collect is now supported on all distributions that sos report supports (i.e. any distribution with a Policy defined)
The --insecure-sudo option has been renamed to --nopasswd-sudo
--threads in the context of the number of nodes to simultaneously connect to has been renamed to jobs
Fixed a bug where a local node would be displayed for collection even when --no-local was used
Cleaner
sos clean, also available as sos mask, is a newly added sub-command in this release and is an implementation of the standalone soscleaner project. Its aim is to scrub potentially sensitive information from sosreports in a consistent manner, beyond the obfuscation done by plugins already.

Support for ipv4 address/network obfuscation. Note that this will attempt to preserve topological relationships between discovered addresses
Support for hostname, and domain name obfuscation.
Support for user-provided keyword obfuscations
Users may either use the --clean or --mask flag to sos report to obfuscate a report being generated, or may use sos (clean|mask) $archive to obfuscate an already existing report.
Using the former will result in a single obfuscated report archive, while the latter approach will result in two; an obfuscated archive and the un-obfuscated original.
For full information on the changes contained in this release, please refer to the Git commit logs. Further release information and tarballs are available at:

https://github.com/sosreport/sos/releases/tag/4.0

Please report any problems to the sos-devel mailing list, or the GitHub issue tracker:

https://github.com/sosreport/sos/issues/

The team would like to thank everyone who contributed fixes, new features, testing, and feedback for this release.

[Eric Desrochers (slashd)]

During the Debian packaging, I found a bug that I fixed in:
https://github.com/sosreport/sos/commit/1d7bab6c7ce3f78758113ca3cdf3e9fa1762df24

I also divert from upstream by instructing sos.conf to save sosreport contents into /tmp instead of now enforce location /var/tmp in order to keep same behaviour and because no tmpfiles.d cleanup directive in Debian's systemd implementation is in place at the moment for /var/tmp.

[Eric Desrochers (slashd)]

This bug was fixed in the package sosreport - 4.0-1
Sponsored for Eric Desrochers (slashd)

---------------
sosreport (4.0-1) unstable; urgency=medium

  * New 4.0 upstream release.

    Release information and tarballs are available at:
    - https://github.com/sosreport/sos/releases/tag/4.0

  * Other specific modifications:
    - d/p/0001-debian-change-tmp-dir-location.patch
    - d/p/0002-fix-dict-order-py38-incompatiblity.patch

  * d/control: Maintainer change from Louis to Eric.

 -- Eric Desrochers <email address hidden> Wed, 19 Aug 2020 22:49:24 +0000

[Eric Desrochers (slashd)]

"4.0-1~ubuntu0.20.04.1" has been uploaded in the focal upload queue, waiting for approval to start building in focal-proposed where more verification will happen across Canonical support team.

- Eric

[Arif Ali (arif-ali)]

A suggestion on bringing sos 4.0 to bionic

Leave the original sosreport as it is, so that it doesn't break anything for any users

Introduce a new package called sosreport4 which packages sos 4.0. This would will then give the opportunity to use the new version of sos so that newer data can be collected as well as being able to support the likes of "sos clean" and "sos collect"

Similar how the packaging of python is done, we have python and python3

[Eric Desrochers (slashd)]

@arif-ali,

I like the idea.

Let me talk with an Ubuntu archive admin.

Most likely sosreport4 would have to go in universe at first and then go over the MIR process if need be.

I also like to have sosreport4 in Universe and sosreport in Main, since sosreport4 is still experimental ish.

Let's see what an Admin Archive think of that first.

[Eric Desrochers (slashd)]

That's also going to involve more package work:

* Does sosreport and sosreport4 needs to break/conflict ? or can co-exist ?
I think it would make more sense to have one or the other.

* Make sure sosreport and sosreport4 follow the same upgrade path if one upgrade from focal to groovy. Note that groovy only has sosreport package which default to 4.0

To summarise the idea:

Groovy (Currently in place as we speak):
- binary package name: sosreport | upstream version: 4.0

Focal (arif-ali proposal):
- binary package name: sosreport | upstream version: 3.0
- binary package name: sosreport4 | upstream version: 4.0

Bionic (arif-ali proposal):
- binary package name: sosreport | upstream version: 3.0
- binary package name: sosreport4 | upstream version: 4.0

[Eric Desrochers (slashd)]

As discussed the above proposal might require a meta package call 'sosreport4' in Groovy to ease the upgrade path.

Again let's wait for a Ubuntu archive admin to look at this.

[Eric Desrochers (slashd)]

Considering that "sos report" (AKA soreport) doesn't have any feature/behavioural breaks (just usual plugins update/addition/...), and that 4.X focuses more on addition such as "sos clean" and "sos collect", I start to think it's not worth the maintenance of 2 splitting things out.

After a discussion with sil2100 (to get a neutral opinion) he agreed with the above statement.

- Eric

[Łukasz Zemczak (sil2100)]

Hello Eric, or anyone else affected,

Accepted sosreport into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/sosreport/4.0-1~ubuntu0.20.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Łukasz Zemczak (sil2100)]

Approved sosreport 4.0 for focal. One thing I want to make sure people testing this are aware of is this snippet from the regression potential section:

"The old config (/etc/sos.conf) contents will not be carried over after update. Users will have to modify the new file (/etc/sos/sos.conf) instead (if needed)."

Normally we'd proceed with providing maintscripts that would handle the mv_conffile for the location change, but here it might not be required. But in case we get a lot of complaints, we can always revisit.

[Eric Desrochers (slashd)]

[VERIFICATION FOCAL]

[sos report (new main binary) / sosreport (former binary name)]

Work as usual. It has been tested with simple.sh script, which runs various/several sos scenarios/arguments/...).

[sos clean]

Obfuscation works as expect.

Take a bit of time to complete, but nothing preventing the release of the package for now, spending most of its time at obfuscating data.

Verification includes:
    - Ipv4 address/network obfuscation : OK

3: wlp58s0 inet 100.0.0.1/24 brd 100.0.0.255 scope global dynamic noprefixroute wlp58s0\ obfuscatedword0 7000
8sec preferred_lft 70008sec

#mapping file:
    "ip_map": {
        ...
        "10.10.30.123/24": "100.0.0.1/24",
        ...
}

    - Host name, and domain name obfuscation: OK

$ cat path_to_obfuscated_sosreport/hostname
host0

$ cat path_to_obfuscated_sosreport/var/log/*
Aug 28 08:57:14 host0 kernel: [ 472.635702] usb usb3: New USB device strings: Mfr=3, Product=2, SerialNumber=1
Aug 28 08:57:14 host0 kernel: [ 472.635710] usb usb3: Product: xHCI Host Controller

# mapping file:

    "hostname_map": {
        "thinkpad": "host0"

    - User-provided keyword obfuscations: OK

3: wlp58s0 inet 100.0.0.1/24 brd 100.0.0.255 scope global dynamic noprefixroute wlp58s0\ obfuscatedword0 7000
8sec preferred_lft 70008sec

#mapping file:
    "keyword_map": {
        "valid_lft": "obfuscatedword0"
    }

[sos collect & sos mask (alias)]

sos collector has been tested using SSH key authorization
No problem reported, and it works as expected.

[Eric Desrochers (slashd)]

[VERIFICATION FOCAL] [PART 2]

No ERROR nor WARNING found during my testing on container, VM and physical machine (laptop/workstation)

[Eric Desrochers (slashd)]

I would like to see testing and feedbacks from others now.

[Jose Delarosa (jdelaros1)]

Test report:

- Tested collection of logs, all looks OK.
- Tested obfuscation, extracted obfuscated sosreport and checked *some* of the obfuscated files, all looks OK.
- I obfuscated the original sosreport again, this time I provided a keyword (--keyword ubuntu) to obfuscate and did it OK. It added the word 'ubuntu' to the mapfile:
    "keyword_map": {
        "ubuntu": "obfuscatedword0"
    }

Possible issues:
1) On the 2nd obfuscated sosreport, I still noticed 'ubuntu' in some files:
- installed-debs
- sos_commands/dpkg/dpkg_-l
- sos_commands/snappy/snap_--version

I realize this was a trick word and not a real-world example, but wanted to point it out.

2) I started from scratch, I created a new report, obfuscated it again, this time I did NOT pass any keywords, but it still obfuscated the word 'ubuntu' (and reported it in the mapfile). Is there a cache or something I need to clean up from the previous run? If so, probably needs to be documented.

Jose

[Eric Desrochers (slashd)]

Thanks Jose.

1) There is a list of file to skip obfuscation:
https://github.com/sosreport/sos/blob/master/sos/cleaner/obfuscation_archive.py#L56-L77

2) Yes, sos keeps track of obfuscation directive in /etc/sos/cleaner/default_mapping.
Most likely you still have the 'ubuntu' keyword saved in there.

- Eric

[Arif Ali (arif-ali)]

Initial Test

* sos report

Ran through the simple.sh as per the document. Had to change all the /var/tmp to /tmp, and then all went through without a problem

sos report passed, and didn't have any problems

same test using sosreport instead, as that is what customers would be used to

Ran all my commands with "-a --batch --case-id=0123456"

* sos clean

tested sos report with --mask option

after extracting, I still have one file, which still shows the hostname

sosreport-host0-0123456-2020-09-10-qbfomdn/sos_commands/lvm2/lvmdump_-d_.tmp.sos.csjtzsye.sosreport-focal-test-0123456-2020-09-10-qbfomdn.sos_commands.lvm2.lvmdump:Creating dump directory: /tmp/sos.csjtzsye/sosreport-focal-test-0123456-2020-09-10-qbfomdn/sos_commands/lvm2/lvmdump

same issue if you use sos clean on top of the sosreport tarball

[Eric Desrochers (slashd)]

@arif,

lvmdump obfuscation issue as well as other hostname unobfuscated bit found) has been reported upstream: https://github.com/sosreport/sos/issues/2236

I personally don't consider this a major blocker for the actual release, because one still has to validate and check the content before passing it along to any 3rd party.

sos clean message:

"
Note that this utility provides a best-effort approach to data obfuscation, but
it does not guarantee that such obfuscation provides complete coverage of all
such data in the archive, or that any obfuscation is provided to data that does
not fit the description above.
"

- Eric

[Eric Desrochers (slashd)]

....

"
Users should review any resulting data and/or archives generated or processed by
this utility for remaining sensitive content before being passed to a third
party.
"

[Łukasz Zemczak (sil2100)]

Since from my understanding the sos clean command is a new feature in this release, I don't think we should block the release because of it being imperfect. As long as all the issues have been reported and made their way upstream, I think we are good to go as is.

[Launchpad Janitor (janitor)]

This bug was fixed in the package sosreport - 4.0-1~ubuntu0.20.04.1

---------------
sosreport (4.0-1~ubuntu0.20.04.1) focal; urgency=medium

  * New 4.0 upstream release. (LP: #1892275)

    Release information and tarballs are available at:
    - https://github.com/sosreport/sos/releases/tag/4.0

  * Other specific modifications:
    - d/p/0001-debian-change-tmp-dir-location.patch
    - d/p/0002-fix-dict-order-py38-incompatibility.patch

  * d/control: Maintainer change from Louis to Eric.

 -- Eric Desrochers <email address hidden> Thu, 27 Aug 2020 13:59:51 +0000

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for sosreport has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Robie Basak (racb)]

Reviewing the Bionic SRU, I see some changes between Focal and the Bionic backport:

debhelper compat moved back to 11
Removal of setuptools, sphinx and pexpect from Python dependencies

...but I don't see any mention of these changes in the changelog.

Were these changes intentional? Presumably the debhelper change was, but what about the Python dependency changes? Are they going to cause an accidental regression in some corner case? Please could you explain why the changes are needed, and mention them in the changelog?

[Eric Desrochers (slashd)]

I ask SRU team to drop 'sosreport' in Bionic's upload queue, as I'd like to add more things to the package after re-consideration.

Will re-submit soon.

- Eric

[Eric Desrochers (slashd)]

Settting Bionic to 'Won't fix' as I will focus effort on new 4.1 release via LP: #1917894