[sru] bug in do_file_private_sub for the juju plugin

[nikhil kshirsagar]

[IMPACT]

As described in https://github.com/sosreport/sos/issues/3261 , the flags=re.IGNORECASE argument cannot be passed with a compiled pattern. This affects sos upstream 4.5.4 and 4.5.3.

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/sos/report/__init__.py", line 1459, in postproc
    plug.postproc()
  File "/usr/lib/python3/dist-packages/sos/report/plugins/juju.py", line 69, in postproc
    self.do_file_private_sub(agents_path)
  File "/usr/lib/python3/dist-packages/sos/report/plugins/__init__.py", line 1257, in do_file_private_sub
    self.do_file_sub(path, _certmatch, replace)
  File "/usr/lib/python3/dist-packages/sos/report/plugins/__init__.py", line 1287, in do_file_sub
    result, replacements = re.subn(regexp, subst, content,
  File "/usr/lib/python3.8/re.py", line 221, in subn
    return _compile(pattern, flags).subn(repl, string, count)
  File "/usr/lib/python3.8/re.py", line 299, in _compile
    raise ValueError(
ValueError: cannot process flags argument with a compiled pattern

The issue was introduced by this commit - https://github.com/sosreport/sos/commit/bb43ac3b0dddcecf9bf34e489490296e15a2addc

It is fixed by https://github.com/sosreport/sos/pull/3263

The patch is taken for the 4.5.4 sos release in https://bugs.launchpad.net/ubuntu/+source/sosreport/+bug/2022915, as d/p/0002-regex-flags.patch, pasted below for reference,

Description: Fix do_file_private_sub() bug related to regex flags
Origin: upstream
Bug: https://github.com/sosreport/sos/issues/3261
Applied-Upstream: https://github.com/sosreport/sos/pull/3263
---
This patch header follows DEP-3: http://dep.debian.net/deps/dep3/
Index: sos-4.5.4/sos/report/plugins/__init__.py
===================================================================
--- sos-4.5.4.orig/sos/report/plugins/__init__.py
+++ sos-4.5.4/sos/report/plugins/__init__.py
@@ -1274,7 +1274,12 @@ class Plugin():
         """
         try:
             path = self._get_dest_for_srcpath(srcpath)
- pattern = regexp.pattern if hasattr(regexp, "pattern") else regexp
+ if hasattr(regexp, "pattern"):
+ pattern = regexp.pattern
+ flags = regexp.flags | re.IGNORECASE
+ else:
+ pattern = regexp
+ flags = re.IGNORECASE
  self._log_debug("substituting scrpath '%s'" % srcpath)
  self._log_debug("substituting '%s' for '%s' in '%s'"
                             % (subst, pattern, path))
@@ -1284,8 +1289,8 @@ class Plugin():
  content = readable.read()
  if not isinstance(content, str):
      content = content.decode('utf8', 'ignore')
- result, replacements = re.subn(regexp, subst, content,
- flags=re.IGNORECASE)
+ result, replacements = re.subn(pattern, subst, content,
+ flags=flags)
  if replacements:
      self.archive.add_string(result, srcpath)
  else:

[TEST PLAN]

The 4.5.4 ppa's built with this additional fix are tested as per the process described https://wiki.ubuntu.com/SosreportUpdates and verfied to have fixed the issue.

[WHERE PROBLEMS COULD OCCUR]

As this bug is in a method invoked by multiple plugins, verification also would need to performed in environments that ensures those plugins are enabled, particularly on other distributions. In our testing, we have found no issues, but there might be some disabled plugins that use that method with assumptions that are made invalid by the patch. However we have found no evidence so far of such occurences.

[nikhil kshirsagar (nkshirsagar)]

Fixed upstream, and also built on top of the upstraem 4.5.4 sos release for ubuntu 4.5.4 release - https://bugs.launchpad.net/ubuntu/+source/sosreport/+bug/2022915

[David Negreira (dnegreira)]

I have verified that the issue does not happen on 4.5.4 PPA release [0] and the issue is fixed.

Before the fix we would see an extra file from with the juju error on sosreport under sos_logs/ folder, as well as an ERROR on sos_logs/sos.log

[0] https://launchpad.net/~nkshirsagar/+archive/ubuntu/focal-sos-4.5.4-sru
### Output ###
ls */sos_logs/
sosreport-juju-43d0aa-f-sos-t-0-2023-06-24-oikuwgu/sos_logs/:
sos.log ui.log

sosreport-juju-43d0aa-f-sos-t-1-2023-06-24-gitiaxq/sos_logs/:
sos.log ui.log

sosreport-juju-43d0aa-f-sos-t-10-2023-06-24-vksjbrs/sos_logs/:
sos.log ui.log

sosreport-juju-43d0aa-f-sos-t-11-2023-06-24-miqwnah/sos_logs/:
sos.log ui.log

sosreport-juju-43d0aa-f-sos-t-12-2023-06-24-ildlhxd/sos_logs/:
sos.log ui.log

sosreport-juju-43d0aa-f-sos-t-13-2023-06-24-cxrvvvz/sos_logs/:
sos.log ui.log

sosreport-juju-43d0aa-f-sos-t-14-2023-06-24-nqxieks/sos_logs/:
sos.log ui.log

sosreport-juju-43d0aa-f-sos-t-15-2023-06-24-bihuyia/sos_logs/:
sos.log ui.log

sosreport-juju-43d0aa-f-sos-t-16-2023-06-24-yhsnwir/sos_logs/:
sos.log ui.log

sosreport-juju-43d0aa-f-sos-t-17-2023-06-24-cvothqw/sos_logs/:
sos.log ui.log

sosreport-juju-43d0aa-f-sos-t-2-2023-06-24-waeusgt/sos_logs/:
sos.log ui.log

sosreport-juju-43d0aa-f-sos-t-3-2023-06-24-phxflms/sos_logs/:
sos.log ui.log

sosreport-juju-43d0aa-f-sos-t-4-2023-06-24-kfclnxe/sos_logs/:
sos.log ui.log

sosreport-juju-43d0aa-f-sos-t-5-2023-06-24-hmkekbt/sos_logs/:
sos.log ui.log

sosreport-juju-43d0aa-f-sos-t-6-2023-06-24-cwcjkqk/sos_logs/:
sos.log ui.log

sosreport-juju-43d0aa-f-sos-t-7-2023-06-24-jnrzyux/sos_logs/:
sos.log ui.log

sosreport-juju-43d0aa-f-sos-t-8-2023-06-24-khhfxui/sos_logs/:
sos.log ui.log

sosreport-juju-43d0aa-f-sos-t-9-2023-06-24-vqepwsx/sos_logs/:
sos.log ui.log
---
ubuntu@dnegreira-bastion:~/sosreportesting/proposedfiles/focal$ grep -r ERR */sos_logs/ | wc -l
0

####

[Steve Langasek (vorlon)]

Hello nikhil, or anyone else affected,

Accepted sosreport into lunar-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/sosreport/4.5.4ubuntu0.23.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-lunar to verification-done-lunar. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-lunar. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Steve Langasek (vorlon)]

Hello nikhil, or anyone else affected,

Accepted sosreport into kinetic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/sosreport/4.5.4ubuntu0.22.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-kinetic to verification-done-kinetic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-kinetic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Steve Langasek (vorlon)]

Hello nikhil, or anyone else affected,

Accepted sosreport into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/sosreport/4.5.4ubuntu0.22.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Steve Langasek (vorlon)]

Hello nikhil, or anyone else affected,

Accepted sosreport into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/sosreport/4.5.4ubuntu0.20.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted sosreport (4.5.4ubuntu0.20.04.1) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

sosreport/4.5.4ubuntu0.20.04.1 (amd64, ppc64el, s390x)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#sosreport

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Łukasz Zemczak (sil2100)]

The version of sosreport in the proposed pocket of Kinetic that was purported to fix this bug report has been removed because one or more bugs that were to be fixed by the upload have failed verification and been in this state for more than 10 days.

[Andreas Hasenack (ahasenack)]

This is being superseeded by 4.5.6, which does not mention this bug in its changes file. I guess that's ok, since the bug was only present in 4.5.3 and 4.5.4, and neither was released to ubuntu.

[Andreas Hasenack (ahasenack)]

4.5.6 was released to L, J, F.

[Brian Murray (brian-murray)]

Ubuntu 23.04 (Lunar Lobster) has reached end of life, so this bug will not be fixed for that specific release.