[SRU] openscap 1.2.15 1.2.16 and 1.2.17 not able to build complianceascode

[Marina Latini]

[Impact]
The openscap versions 1.2.15 1.2.16 and 1.2.17 are missing the patch https://github.com/OpenSCAP/openscap/commit/bbcbffcf6f901cb67ca5645307d170a32504a491.patch provided via https://github.com/OpenSCAP/openscap/pull/1324

Without this patch openscap isn't able to build ComplianceAsCode (https://github.com/ComplianceAsCode/content).

This bugreport is the Ubuntu equivalent of https://github.com/ComplianceAsCode/content/issues/9718 and the patch https://github.com/OpenSCAP/openscap/commit/bbcbffcf6f901cb67ca5645307d170a32504a491.patch fixes the problem.

The patch can be applied to all the mentioned versions.

The affected Ubuntu versions are:
18.04, 20.04, 22.04 and 22.10

The Ubuntu Security Team also needs this fix to better provide support on ComplianceAsCode and Ubuntu Security Guide
(USG: https://launchpad.net/usg)

[Test Case]
Here is some instructions how to test it:
$ git clone https://github.com/ComplianceAsCode/content
$ cd content
$ ./build_product -j4 ubuntu1604 -o'5.11'

You will probably need to install some packages to be able to build it, for more information please check documentation for accurate version:
$ sudo apt-get install -y cmake make expat libopenscap8 libxml2-utils ninja-build python3-jinja2 python3-yaml xsltproc shellcheck ninja-build yamllint ansible-lint build-essential

As the above testing plan might be too burdensome, a better way to do it
is by running:
$ oscap xccdf eval --profile standard ssg-ubuntu1604-ds.xml

The `ssg-ubuntu1604-ds.xml` file used is attached to this bug.

[Where problems could occur]
The issue doesn't allow the build of USG/ComplianceAsCode for other platforms, such as ubuntu1604, rhel7 and others.

[Other Info]

The issue affects all releases from Bionic to Kinetic.

[Eduardo Barretto (ebarretto)]

Hi Marina,

Thanks for reaching out. May I ask which product are you trying to build?
Ubuntu should be buildable as of now

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "Allow-DS-session-to-continue-without-remote-resource.patch" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

[Marina Latini (marina-dnb-latini)]

Hello Eduardo,

I'm trying to build the scap-security-guide (ComplianceAsCode 0.1.64) on Ubuntu 18.04, Ubuntu 20.04 and Ubuntu 22.04.

On the 3 Ubuntu versions, the failure is always the same. The scap-security-guide uses openscap but, if the OVAL CVE/RPM data are not available, the build will fail.

ComplianceAsCode on version 0.1.63 was building fine.

This is one example of failure due to missing remote resources (but there are more).
All xcddf generate fix with embedded remote resources fail.

oscap xccdf generate fix --skip-valid --benchmark-id xccdf_org.ssgproject.content_benchmark_UBUNTU-XENIAL --profile xccdf_org.ssgproject.content_profile_anssi_np_nt28_restrictive --template urn:xccdf:fix:script:sh ./ssg-ubuntu1604-ds.xml

WARNING: Datastream component 'scap_org.open-scap_cref_-ubuntu-security-oval-com.ubuntu.xenial.cve.oval.xml' points out to the remote 'https://people.canonical.com/~ubuntu-security/oval/com.ubuntu.xenial.cve.oval.xml'. Use '--fetch-remote-resources' option to download it.
WARNING: Skipping 'https://people.canonical.com/~ubuntu-security/oval/com.ubuntu.xenial.cve.oval.xml' file which is referenced from datastream
OpenSCAP Error: Could not extract scap_org.open-scap_cref_ssg-ubuntu1604-xccdf-1.2.xml with all dependencies from datastream. [../../../src/DS/ds_sds_session.c:211]

If the --fetch-remote-resources option is not provided, the resources pointed by the components won't be downloaded. The provided patch allows the scan to continue without remote components.
The result of rules which reference the missing remote resource will be 'notchecked'.

The PR https://github.com/OpenSCAP/openscap/pull/1324 was done after 1.2.16 release and made available in openscap 1.2.18 and 1.3.1, so Ubuntu 20.04 but also Ubuntu 22.04 missed it.

For Ubuntu 18.04, that has openscap 1.2.15, the patch applies too (just with offset).

I hope this clarifies better the issue.

[Eduardo Barretto (ebarretto)]

Hi Marina,

Thanks for the information!
Yes, we are aware of such issues when building other products in Ubuntu, thanks for mentioning that this is an issue for building the ubuntu1604 product.

As this is a bug fix it must go through the SRU process: https://wiki.ubuntu.com/StableReleaseUpdates#Procedure

I will be doing a SRU for openscap soon, whenever I get this PR merged in openscap upstream: https://github.com/OpenSCAP/openscap/pull/1901

If you want me to do both yours and mine in just one request I can, but do note that this might take some time.

If you want to fix this as soon as possible, then please proceed with the SRU process and the SRU team will take a look at it.

Thanks again!

[Marina Latini (marina-dnb-latini)]

Hi Eduardo,

feel free to move forward with the SRU whenever you can. I'm not really familiar with this process and the fix isn't extremely urgent for me.

Please, let me know if there's anything else I can do.
Thanks a lot for your help and support!

[Eduardo Barretto (ebarretto)]

FYI, I've created this: https://bugs.launchpad.net/ubuntu/+source/openscap/+bug/2004476
to include both changes as promised

[Timo Aaltonen (tjaalton)]

missing SRU header, please add

[Eduardo Barretto (ebarretto)]

no need to, this is being handled in the bug described in comment #7

[Andreas Hasenack (ahasenack)]

After reading https://bugs.launchpad.net/ubuntu/+source/openscap/+bug/2004476, and adding a comment there, I agree with Timo. I think you should make this bug here a normal SRU bug, with the template, test plan, regression potential, etc. There is nothing wrong with fixing multiple bugs in one upload, but mixing two different fixes in one bug (where there is another one already open and with a discussion) is confusing.

[Łukasz Zemczak (sil2100)]

The "Where problems could occur" section could need some more looking into. That being said, I think the change itself is safe enough for me not to block on this.

[Łukasz Zemczak (sil2100)]

Hello Marina, or anyone else affected,

Accepted openscap into kinetic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/openscap/1.2.17-0.1ubuntu7.22.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-kinetic to verification-done-kinetic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-kinetic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Łukasz Zemczak (sil2100)]

Hello Marina, or anyone else affected,

Accepted openscap into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/openscap/1.2.17-0.1ubuntu7.22.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Łukasz Zemczak (sil2100)]

Hello Marina, or anyone else affected,

Accepted openscap into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/openscap/1.2.15-1ubuntu0.4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Łukasz Zemczak (sil2100)]

Hello Marina, or anyone else affected,

Accepted openscap into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/openscap/1.2.16-2ubuntu3.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Eduardo Barretto (ebarretto)]

I can confirm that with openscap in -proposed I was able to build other platforms in ComplianceAsCode.

Marina, if you could also confirm it that would be much appreciated.

[Andreas Hasenack (ahasenack)]

Eduardo, can you please add a test verification for the specific ubuntu releases? I don't know which one you tested in comment #16, and yet you flipped the verification tag for all ubuntu releases.

[Eduardo Barretto (ebarretto)]

I tested them all.

[Eduardo Barretto (ebarretto)]

Test results on Focal:

With current archive version:
$ dpkg -l | grep openscap
ii libopenscap8 1.2.16-2ubuntu3.2 amd64 Set of libraries enabling integration of the SCAP line of standards

$ oscap xccdf eval --profile standard ssg-ubuntu1604-ds.xml
WARNING: Datastream component 'scap_org.open-scap_cref_-ubuntu-security-oval-com.ubuntu.xenial.cve.oval.xml' points out to the remote 'https://people.canonical.com/~ubuntu-security/oval/com.ubuntu.xenial.cve.oval.xml'. Use '--fetch-remote-resources' option to download it.
WARNING: Skipping 'https://people.canonical.com/~ubuntu-security/oval/com.ubuntu.xenial.cve.oval.xml' file which is referenced from datastream
OpenSCAP Error: Could not extract scap_org.open-scap_cref_ssg-ubuntu1604-xccdf.xml with all dependencies from datastream. [../../../src/DS/ds_sds_session.c:211]

With proposed version:
$ dpkg -l | grep openscap
ii libopenscap8 1.2.16-2ubuntu3.3 amd64 Set of libraries enabling integration of the SCAP line of standards

$ oscap xccdf eval --profile standard ssg-ubuntu1604-ds.xml
WARNING: Datastream component 'scap_org.open-scap_cref_-ubuntu-security-oval-com.ubuntu.xenial.cve.oval.xml' points out to the remote 'https://people.canonical.com/~ubuntu-security/oval/com.ubuntu.xenial.cve.oval.xml'. Use '--fetch-remote-resources' option to download it.
WARNING: Skipping 'https://people.canonical.com/~ubuntu-security/oval/com.ubuntu.xenial.cve.oval.xml' file which is referenced from datastream
WARNING: Skipping ./-ubuntu-security-oval-com.ubuntu.xenial.cve.oval.xml file which is referenced from XCCDF content
Title Ensure /home Located On Separate Partition
Rule xccdf_org.ssgproject.content_rule_partition_for_home
Result notapplicable
.... This goes on, I won't paste the whole log.

[Eduardo Barretto (ebarretto)]

Jammy testing results:
With current archive version:
$ dpkg -l | grep openscap
ii libopenscap8 1.2.17-0.1ubuntu7 amd64 Set of libraries enabling integration of the SCAP line of standards

$ oscap xccdf eval --profile standard ssg-ubuntu1604-ds.xml
WARNING: Datastream component 'scap_org.open-scap_cref_-ubuntu-security-oval-com.ubuntu.xenial.cve.oval.xml' points out to the remote 'https://people.canonical.com/~ubuntu-security/oval/com.ubuntu.xenial.cve.oval.xml'. Use '--fetch-remote-resources' option to download it.
WARNING: Skipping 'https://people.canonical.com/~ubuntu-security/oval/com.ubuntu.xenial.cve.oval.xml' file which is referenced from datastream
OpenSCAP Error: Could not extract scap_org.open-scap_cref_ssg-ubuntu1604-xccdf.xml with all dependencies from datastream. [../../../src/DS/ds_sds_session.c:211]

With proposed version:
$ dpkg -l | grep openscap
ii libopenscap8 1.2.17-0.1ubuntu7.22.04.1 amd64 Set of libraries enabling integration of the SCAP line of standards

$ oscap xccdf eval --profile standard ssg-ubuntu1604-ds.xml
WARNING: Datastream component 'scap_org.open-scap_cref_-ubuntu-security-oval-com.ubuntu.xenial.cve.oval.xml' points out to the remote 'https://people.canonical.com/~ubuntu-security/oval/com.ubuntu.xenial.cve.oval.xml'. Use '--fetch-remote-resources' option to download it.
WARNING: Skipping 'https://people.canonical.com/~ubuntu-security/oval/com.ubuntu.xenial.cve.oval.xml' file which is referenced from datastream
WARNING: Skipping ./-ubuntu-security-oval-com.ubuntu.xenial.cve.oval.xml file which is referenced from XCCDF content
Title Ensure /home Located On Separate Partition
Rule xccdf_org.ssgproject.content_rule_partition_for_home
Result notapplicable
...

[Eduardo Barretto (ebarretto)]

Bionic testing results:
With current archive version:
$ dpkg -l | grep openscap
ii libopenscap8 1.2.15-1ubuntu0.3 amd64 Set of libraries enabling integration of the SCAP line of standards

$ oscap xccdf eval --profile standard ssg-ubuntu1604-ds.xml
WARNING: Datastream component 'scap_org.open-scap_cref_-ubuntu-security-oval-com.ubuntu.xenial.cve.oval.xml' points out to the remote 'https://people.canonical.com/~ubuntu-security/oval/com.ubuntu.xenial.cve.oval.xml'. Use '--fetch-remote-resources' option to download it.
WARNING: Skipping 'https://people.canonical.com/~ubuntu-security/oval/com.ubuntu.xenial.cve.oval.xml' file which is referenced from datastream
OpenSCAP Error: Could not extract scap_org.open-scap_cref_ssg-ubuntu1604-xccdf.xml with all dependencies from datastream. [../../../src/DS/ds_sds_session.c:211]

With proposed version:
$ dpkg -l | grep openscap
ii libopenscap8 1.2.15-1ubuntu0.4 amd64 Set of libraries enabling integration of the SCAP line of standards

$ oscap xccdf eval --profile standard ssg-ubuntu1604-ds.xml
WARNING: Datastream component 'scap_org.open-scap_cref_-ubuntu-security-oval-com.ubuntu.xenial.cve.oval.xml' points out to the remote 'https://people.canonical.com/~ubuntu-security/oval/com.ubuntu.xenial.cve.oval.xml'. Use '--fetch-remote-resources' option to download it.
WARNING: Skipping 'https://people.canonical.com/~ubuntu-security/oval/com.ubuntu.xenial.cve.oval.xml' file which is referenced from datastream
WARNING: Skipping ./-ubuntu-security-oval-com.ubuntu.xenial.cve.oval.xml file which is referenced from XCCDF content
Title Ensure /home Located On Separate Partition
Rule xccdf_org.ssgproject.content_rule_partition_for_home
Result notapplicable
...

[Eduardo Barretto (ebarretto)]

Kinetic testing results:
With current archive version:
$ dpkg-query -l | grep openscap
ii libopenscap8 1.2.17-0.1ubuntu7 amd64 Set of libraries enabling integration of the SCAP line of standards

$ oscap xccdf eval --profile standard ssg-ubuntu1604-ds.xml
WARNING: Datastream component 'scap_org.open-scap_cref_-ubuntu-security-oval-com.ubuntu.xenial.cve.oval.xml' points out to the remote 'https://people.canonical.com/~ubuntu-security/oval/com.ubuntu.xenial.cve.oval.xml'. Use '--fetch-remote-resources' option to download it.
WARNING: Skipping 'https://people.canonical.com/~ubuntu-security/oval/com.ubuntu.xenial.cve.oval.xml' file which is referenced from datastream
OpenSCAP Error: Could not extract scap_org.open-scap_cref_ssg-ubuntu1604-xccdf.xml with all dependencies from datastream. [../../../src/DS/ds_sds_session.c:211]

With proposed version:
$ dpkg-query -l | grep openscap
ii libopenscap8 1.2.17-0.1ubuntu7.22.10.1 amd64 Set of libraries enabling integration of the SCAP line of standards

$ oscap xccdf eval --profile standard ssg-ubuntu1604-ds.xml
WARNING: Datastream component 'scap_org.open-scap_cref_-ubuntu-security-oval-com.ubuntu.xenial.cve.oval.xml' points out to the remote 'https://people.canonical.com/~ubuntu-security/oval/com.ubuntu.xenial.cve.oval.xml'. Use '--fetch-remote-resources' option to download it.
WARNING: Skipping 'https://people.canonical.com/~ubuntu-security/oval/com.ubuntu.xenial.cve.oval.xml' file which is referenced from datastream
WARNING: Skipping ./-ubuntu-security-oval-com.ubuntu.xenial.cve.oval.xml file which is referenced from XCCDF content
Title Ensure /home Located On Separate Partition
Rule xccdf_org.ssgproject.content_rule_partition_for_home
Result notapplicable
...

[Andreas Hasenack (ahasenack)]

I verified the test results and am satisfied that they show the executed planned test case, and that the results are correct.

The package built correctly in all architectures and Ubuntu releases it was meant for.

There are no DEP8 regressions.

There is no SRU freeze ongoing at the moment.

There is no halted phasing on the previous update.

[Andreas Hasenack (ahasenack)]

Note that there are no uploads for trusty or xenial, but this doesn't block the sru for the other (later) releases.

[Launchpad Janitor (janitor)]

This bug was fixed in the package openscap - 1.2.17-0.1ubuntu7.22.10.1

---------------
openscap (1.2.17-0.1ubuntu7.22.10.1) kinetic; urgency=medium

  * Make dpkg version comparison less strict for epoch digit. (LP: #2004476)
    - d/p/debian-epoch-less-strict.patch: oval_cmp_evr_string:
      Make epoch comparison less restrict for dpkg.
  * Allow build of ComplianceAsCode and USG projects for platforms that use
    remote resources. (LP: #2002551)
    - d/p/allow-DS-session-to-continue-without-remote-resource.patch:
      Allow DS session to continue without remote resource.

 -- Eduardo Barretto <email address hidden> Tue, 31 Jan 2023 13:16:05 +0100

[Andreas Hasenack (ahasenack)]

The verification of the Stable Release Update for openscap has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package openscap - 1.2.17-0.1ubuntu7.22.04.1

---------------
openscap (1.2.17-0.1ubuntu7.22.04.1) jammy; urgency=medium

  * Make dpkg version comparison less strict for epoch digit. (LP: #2004476)
    - d/p/debian-epoch-less-strict.patch: oval_cmp_evr_string:
      Make epoch comparison less restrict for dpkg.
  * Allow build of ComplianceAsCode and USG projects for platforms that use
    remote resources. (LP: #2002551)
    - d/p/allow-DS-session-to-continue-without-remote-resource.patch:
      Allow DS session to continue without remote resource.

 -- Eduardo Barretto <email address hidden> Tue, 31 Jan 2023 13:18:40 +0100

[Launchpad Janitor (janitor)]

This bug was fixed in the package openscap - 1.2.16-2ubuntu3.3

---------------
openscap (1.2.16-2ubuntu3.3) focal; urgency=medium

  * Make dpkg version comparison less strict for epoch digit. (LP: #2004476)
    - d/p/debian-epoch-less-strict.patch: oval_cmp_evr_string:
      Make epoch comparison less restrict for dpkg.
  * Allow build of ComplianceAsCode and USG projects for platforms that use
    remote resources. (LP: #2002551)
    - d/p/allow-DS-session-to-continue-without-remote-resource.patch:
      Allow DS session to continue without remote resource.

 -- Eduardo Barretto <email address hidden> Tue, 31 Jan 2023 13:27:43 +0100

[Launchpad Janitor (janitor)]

This bug was fixed in the package openscap - 1.2.15-1ubuntu0.4

---------------
openscap (1.2.15-1ubuntu0.4) bionic; urgency=medium

  * Make dpkg version comparison less strict for epoch digit. (LP: #2004476)
    - d/p/debian-epoch-less-strict.patch: oval_cmp_evr_string:
      Make epoch comparison less restrict for dpkg.
  * Allow build of ComplianceAsCode and USG projects for platforms that use
    remote resources. (LP: #2002551)
    - d/p/allow-DS-session-to-continue-without-remote-resource.patch:
      Allow DS session to continue without remote resource.

 -- Eduardo Barretto <email address hidden> Tue, 31 Jan 2023 13:38:45 +0100