Update sosreport to 3.9

[Eric Desrochers]

[Impact]

sosreport 3.9 is now released.

It would be great to find sosreport v3.9 in supported stable releases, and active development release considering the fact that the releasea (especially LTSes) are going to be supported for a couple of years still.

Sosreport is widely use by Canonical support team to troubleshoot UA (Ubuntu Advantage) customer, other vendors and community users.

Just like we did for :
- v3.5 (LP: #1734983)
- v3.6 (LP: #1775195)

[Test Case]

* Install sosreport
* Test the new upload functionality and check if the archive is found in BrickFTP under uploads/ as it should. (e.g. sosreport -a --all-logs --upload)
* Test new --all-logs behaviour
* Make sure the new naming pattern doesn't break any current Canonical mechanism (Brickftp scripts, ...). If yes evaluate if easy fixable before considering reverting back to 'legacy' pattern.
* Run sosreport in different customer scenarios:
server, desktop, cloud, hypervisor, instance (container, vm), physical server, ...
* Extract archive and look at the content, look for 0 size file (and use common sense if legit or not)
* Look under "sos_reports" for full report.
* Look under "sos_logs" for warnings/errors.
  $ grep -v "INFO:" sos_logs/sos.log
* Run "simple.sh": A quick port of the travis tests to bash. Generating various type of sosreports collection.

[Regression Potential]

Sosreport has 292 plugins that are all configured differently and configured to run under certains conditions. We can't test all possible scenarios. All we can do is idenfity the most common, important and Ubuntu/Canonical related one and test them (e.g. Openstack*, juju, MAAS, kernel, ...) . With that being said, it is definitely possible that certains plugins may not work as expected, but the risk will be very low (e.g not collecting the desired informations) and isolated to this specific plugin. It shouldn't affect the other plugins nor core functionnalities of sosreport.

[Other Informations]

* Release note:
https://github.com/sosreport/sos/releases/tag/3.9

[Original Description]

A new version of sosreport upstream (v3.9) will be released soon.
This bug is to track activities to update sosreport in Ubuntu stable + Active Devel release.

[Nick Niehoff (nniehoff)]

I have tested:

vanilla Focal instance
Focal with openvswitch
bionic juju machine
bionic openstack nova-cloud-controller (openstack plugins need some work)
bionic ceph (new plugin additions look good)
bionic maas 2.6 ... commissioning-results is not a valid MAAS command plugin needs work

[Eric Desrochers (slashd)]

My today's upload of sosreport 3.9-1 has been accepted in Debian unstable.

[Eric Desrochers (slashd)]

This bug was fixed in the package sosreport - 3.9-1
Sponsored for Eric Desrochers (slashd)

---------------
sosreport (3.9-1) unstable; urgency=medium

  * New 3.9 upstream release.
    - Improved human-readable archive naming and support for archive labels
    - Improved reporting of archive output and properties
    - Support for automatic uploading of report archives via FTP and HTTPS
    - Automatic PATH support on Ubuntu distributions
    - Improved policy performance
    - Improved service status collection API
    - 9 new plugins: cloud_init, convert2rhel, ebpf, fwupd, login, nginx,
      nvidia, openstack_tripleo
    - 6 obsolete plugins removed or merged into other plugins:
      katello, last, mrggrid, mrgmessg, satellite
    - Significant updates to 14 plugins: dlm, dnf, ceph, foreman, gluster,
      gnocchi, juju, kubernetes, logs, maas, networking, openvswitch,
      python, plugins
    - The openswan plugin was renamed to libreswan to reflect the active
      upstream project name
    - Updated Red Hat presets and new Cloud Forms preset
    - Updates to networking plugin namespace handling
    - Updates to the OVN plugins (ovn_central, ovn_host)
    - Kernel eBPF data consolidated in a single plugin

  Further release information and tarballs are available at:
    https://github.com/sosreport/sos/releases/tag/3.9

  * Other specific modifications:
    - d/p/0001-unittest-py3-fix.patch

 -- Eric Desrochers <email address hidden> Sun, 16 Feb 2020 01:35:52 +0000

[Eric Desrochers (slashd)]

A discussion is in progress about what Canonical should use in term of pattern naming.

For now debian policy uses 'friendly' which the Ubuntu policy inherit.

Examples:
legacy - 'sosreport-tux.123456-20171224185433.tar.xz'
friendly - 'sosreport-tux-mylabel-123456-2017-12-24-ezcfcop.tar.xz'

The concerns is about 'friendly' mode not providing the timestamp as oppose to its predecessor the 'legacy' mode.

https://github.com/sosreport/sos/blob/master/sos/policies/__init__.py#L534-L565

Potential options:

a) Add time in current friendly format
- It's a trivial change, but this will cause additional length for sosreport archive name.

b) Do a 'UBUNTU SAUCE' in the sosreport Ubuntu packages by enforcing the 'legacy' mode in Ubuntu policy for now.
- The 'legacy' mode is still available and working.

c) Leave it as is and drop the idea of timestamp

....

[Eric Desrochers (slashd)]

Having the timestamp is helpful in situaiton where more than 1 sosreport is generated within the same day for the same host.

[Eric Desrochers (slashd)]

The downside of not using 'legacy' is that we won't take benefit of the label

[Eric Desrochers (slashd)]

We will keep the 'friendly' mode for now (not diverting from upstream), and we can re-evaluate and do a 'UBUNTU SAUCE' patch if it becomes a annoyance for us.

Upstream has been pretty decisive about the naming pattern and they don't want to disrupt it again:
https://github.com/sosreport/sos/issues/1963

[Eric Desrochers (slashd)]

Uploaded in B/E.

[Łukasz Zemczak (sil2100)]

Hello Eric, or anyone else affected,

Accepted sosreport into eoan-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/sosreport/3.9-1ubuntu0.19.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-eoan to verification-done-eoan. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-eoan. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Łukasz Zemczak (sil2100)]

Hello Eric, or anyone else affected,

Accepted sosreport into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/sosreport/3.9-1ubuntu0.18.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Jose Delarosa (jdelaros1)]

Eric,

100% in favor of adding a time stamp (hhmmss) to the sosreport file (as in the legacy mode). I have seen many cases where we ask customers to upload 5-6 sosreports of the same host the same day. Not having an exact generation time in the file name will make it impossible to manage.

[John Lewis (jlewis-johnlewis-deactivatedaccount)]

Tried it out on a juju controller in lxd works fine
Tried it on a bionic cluster (bundles made with stsstack-bundles generate-bundle.sh and no options) - first time I ran through it go stuck on nova-compute and I had to ctrl+c. Ran again, no issues.
Have tried --upload with --all-logs and no problem uploading 777MB
Particularly happy a customer can run with a cli such as "sudo sosreport -a --case-id 00263859 --upload" and it now requires no other user interaction to attach to the case! Nice one Eric!

[Bryan Quigley (bryanquigley)]

A quick script to run on a node to verify multiple different options:
wget https://raw.githubusercontent.com/sosreport/sos/master/tests/simple.sh
sudo bash simple.sh /usr/bin/python3 /usr/bin/sosreport

there is one possible known bug with it, in regards to lxd: https://github.com/sosreport/sos/issues/1921

Running ii sosreport 3.9-1ubuntu0.19.10.1 on Eoan. All checks out. Manual checking as well.

[Eric Desrochers (slashd)]

[VERIFICATION BIONIC]

* --upload works as expected

* Canonical internal file-mover script still works as expected with 'friendly' naming pattern. Working using both 'legacy' (< sosreport v3.6) and/or 'friendly (> sosreport v3.9). Since I'm still expecting a certain ratio of customer using v3.6.

* I ran sosreport by hand multiples time without ERROR (VM, lxd containers, physical machines)

* simple.sh[0] script didn't report error (minus the already known github bug 1921)

[0] - A quick port of the travis tests to bash. Generating various type of sosreports collection.

[Eric Desrochers (slashd)]

[VERIFICATION EOAN]

* --upload works as expected

* Canonical internal file-mover script still works as expected with 'friendly' naming pattern. Working using both 'legacy' (< sosreport v3.6) and/or 'friendly (> sosreport v3.9). Since I'm still expecting a certain ratio of customer using v3.6.

* I ran sosreport by hand multiples time without ERROR (VM, lxd containers, physical machines)

* simple.sh[0] script didn't report error at all.
...
### end stdout
### start extraction
### stop extraction
######### DONE WITH --batch --verbose --no-postproc #########
Everything worked!

[0] - A quick port of the travis tests to bash. Generating various type of sosreports collection.

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for sosreport has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package sosreport - 3.9-1ubuntu0.19.10.1

---------------
sosreport (3.9-1ubuntu0.19.10.1) eoan; urgency=medium

  * New 3.9 upstream release (LP: #1862830)
    - Improved human-readable archive naming and support for archive labels
    - Improved reporting of archive output and properties
    - Support for automatic uploading of report archives via FTP and HTTPS
    - Automatic PATH support on Ubuntu distributions
    - Improved policy performance
    - Improved service status collection API
    - 9 new plugins: cloud_init, convert2rhel, ebpf, fwupd, login, nginx,
      nvidia, openstack_tripleo
    - 6 obsolete plugins removed or merged into other plugins:
      katello, last, mrggrid, mrgmessg, satellite
    - Significant updates to 14 plugins: dlm, dnf, ceph, foreman, gluster,
      gnocchi, juju, kubernetes, logs, maas, networking, openvswitch,
      python, plugins
    - The openswan plugin was renamed to libreswan to reflect the active
      upstream project name
    - Updated Red Hat presets and new Cloud Forms preset
    - Updates to networking plugin namespace handling
    - Updates to the OVN plugins (ovn_central, ovn_host)
    - Kernel eBPF data consolidated in a single plugin

  Further release information and tarballs are available at:
    https://github.com/sosreport/sos/releases/tag/3.9

  * Other specific modifications:
    - d/p/0001-unittest-py3-fix.patch

 -- Eric Desrochers <email address hidden> Thu, 20 Feb 2020 20:43:21 +0000

[Pedro Principeza (pprincipeza)]

Tested the collection of sosreport on O7K Units (Nova, Cinder, Neutron-*), everything working smoothly. Will test with Microstack soon.

[Łukasz Zemczak (sil2100)]

Ah, I saw the verification but then missed it because of the tag not being set! Duh. Releasing bionic.

[Launchpad Janitor (janitor)]

This bug was fixed in the package sosreport - 3.9-1ubuntu0.18.04.1

---------------
sosreport (3.9-1ubuntu0.18.04.1) bionic; urgency=medium

  * New 3.9 upstream release (LP: #1862830)
    - Improved human-readable archive naming and support for archive labels
    - Improved reporting of archive output and properties
    - Support for automatic uploading of report archives via FTP and HTTPS
    - Automatic PATH support on Ubuntu distributions
    - Improved policy performance
    - Improved service status collection API
    - 9 new plugins: cloud_init, convert2rhel, ebpf, fwupd, login, nginx,
      nvidia, openstack_tripleo
    - 6 obsolete plugins removed or merged into other plugins:
      katello, last, mrggrid, mrgmessg, satellite
    - Significant updates to 14 plugins: dlm, dnf, ceph, foreman, gluster,
      gnocchi, juju, kubernetes, logs, maas, networking, openvswitch,
      python, plugins
    - The openswan plugin was renamed to libreswan to reflect the active
      upstream project name
    - Updated Red Hat presets and new Cloud Forms preset
    - Updates to networking plugin namespace handling
    - Updates to the OVN plugins (ovn_central, ovn_host)
    - Kernel eBPF data consolidated in a single plugin

  Further release information and tarballs are available at:
    https://github.com/sosreport/sos/releases/tag/3.9

  * Other specific modifications:
    - d/p/0001-unittest-py3-fix.patch

 -- Eric Desrochers <email address hidden> Thu, 20 Feb 2020 20:53:25 +0000

[Dan Hill (hillpd)]

Tested sosreport collection on a bionic-rocky mimic ceph cluster. Everything worked as expected.

Regarding comments #8, #12: upstream adds 8 random characters to the filename in 'friendly' mode. Practically, this should prevent all collisions. If we need to approximate legacy formatting for some reason, the label can be used to include the missing timestamp: `--label=$(date +%T%Z)`

[Łukasz Zemczak (sil2100)]

Hello Eric, or anyone else affected,

Accepted sosreport into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/sosreport/3.9-1ubuntu0.16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Eric Desrochers (slashd)]

[VERIFICATION XENIAL]

General testing using the uptream travis test converted into a bash script (AKA simple.sh) reported everything to work well:

.....
### end stdout
### start extraction
### stop extraction
######### DONE WITH --batch --verbose --no-postproc #########
Everything worked!

[Launchpad Janitor (janitor)]

This bug was fixed in the package sosreport - 3.9-1ubuntu0.16.04.1

---------------
sosreport (3.9-1ubuntu0.16.04.1) xenial; urgency=medium

  * New 3.9 upstream release (LP: #1862830)
    - Improved human-readable archive naming and support for archive labels
    - Improved reporting of archive output and properties
    - Support for automatic uploading of report archives via FTP and HTTPS
    - Automatic PATH support on Ubuntu distributions
    - Improved policy performance
    - Improved service status collection API
    - 9 new plugins: cloud_init, convert2rhel, ebpf, fwupd, login, nginx,
      nvidia, openstack_tripleo
    - 6 obsolete plugins removed or merged into other plugins:
      katello, last, mrggrid, mrgmessg, satellite
    - Significant updates to 14 plugins: dlm, dnf, ceph, foreman, gluster,
      gnocchi, juju, kubernetes, logs, maas, networking, openvswitch,
      python, plugins
    - The openswan plugin was renamed to libreswan to reflect the active
      upstream project name
    - Updated Red Hat presets and new Cloud Forms preset
    - Updates to networking plugin namespace handling
    - Updates to the OVN plugins (ovn_central, ovn_host)
    - Kernel eBPF data consolidated in a single plugin

  Further release information and tarballs are available at:
    https://github.com/sosreport/sos/releases/tag/3.9

  * Other specific modifications:
    - d/p/0001-unittest-py3-fix.patch

 -- Eric Desrochers <email address hidden> Wed, 11 Mar 2020 13:07:26 -0400