oscap with com.ubuntu.xenial.cve.oval.xml wrongly reports many unpatched (and unknown) non-installed packages on Ubuntu Xenial 16.04.1 LTS

[Norbert]

Steps to reproduce:
1. Download OVAL definitions
  cd /tmp
  wget https://people.canonical.com/~ubuntu-security/oval/com.ubuntu.xenial.cve.oval.xml

2. Install OpenSCAP
  2a. from official repository
sudo apt-get install libopenscap8
  2b. build from sources
    (see my comment 27 on https://answers.launchpad.net/ubuntu/+source/openscap/+question/242354)

3. Check system and open report
  oscap oval eval --results /tmp/results-xenial.xml --report /tmp/report-xenial.html /tmp/com.ubuntu.xenial.cve.oval.xml
  firefox /tmp/report-xenial.html

Expected results:
  Fully upgraded system should have
    0 Non-Compliant/Vulnerable/Unpatched
    0 Unknown
  OVAL scanning results.

Actual results:
  Fully upgraded system has
    1531 Non-Compliant/Vulnerable/Unpatched
    1690 Unknown
  OVAL scanning results.

Notes:
  1. 'oscap oval eval' command works normally on Ubuntu 12.04 LTS and 14.04 LTS with openscap from https://github.com/OpenSCAP/openscap. It correctly reports 0 "Non-Compliant/Vulnerable/Unpatched", 0 "Unknown" OVAL scanning results on fully upgraded systems.
  2. Error about "com.ubuntu.xenial.cve.oval.xml" was reported on mail-list (https://lists.ubuntu.com/archives/ubuntu-hardened/2016-October/000870.html) with no result.

ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: libopenscap8 1.2.8-1
ProcVersionSignature: Ubuntu 4.4.0-59.80-generic 4.4.35
Uname: Linux 4.4.0-59-generic i686
ApportVersion: 2.20.1-0ubuntu2.5
Architecture: i386
CurrentDesktop: MATE
Date: Mon Jan 23 20:22:42 2017
InstallationDate: Installed on 2016-10-08 (107 days ago)
InstallationMedia: Ubuntu-MATE 16.04.1 LTS "Xenial Xerus" - Release i386 (20160719)
SourcePackage: openscap
UpgradeStatus: No upgrade log present (probably fresh install)

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Johan Ryberg (jryberg)]

It's still the same issue, I downloaded OVAL definitions last week and also today.

I got thousands of unknown and unpatched packages, most are not even installed.

Tested with Redhat OVAL (on a Redhat system) and it works perfectly fine (I just wanted to verify that I used correct syntax).

[Seth Arnold (seth-arnold)]

Norbert, Johan, have either of you done enough investigation to know if the issue is the openscap that is in the archives or is the issue in the generated oval XML?

Thanks

[Johan Ryberg (jryberg)]

Hi Seth,

I have not verified the XML or tried to compile from source. I tried to reach out to ubuntu-hardened email list before without any response.

I'm just trying to get some help where I can to have this issue resolved.

OVAL/CVE scanning within the Ubuntu community must be very very low, or everyone are using other tools.

// Johan

[Johan Ryberg (jryberg)]

I have compiled from source and both the maintained package and direct from GitHub behaves the same.

Build steps:
 sudo apt-get install build-essential cmake libqt4-dev libxslt1-dev libcurl4-openssl-dev libz-dev autoconf libtool libpcre3-dev asciidoctor git checkinstall libgcrypt-dev pkg-config

 cd /tmp
 git clone https://github.com/OpenSCAP/openscap
 cd openscap
 git checkout maint-1.2
 ./autogen.sh
 ./configure --prefix /usr --disable-python --disable-util-oscap-docker
 make -j4
 sudo checkinstall make install # specify version 1.2.14

Cred: https://answers.launchpad.net/ubuntu/+source/openscap/+question/242354

[Norbert (nrbrtx)]

I made test with Xenial amd64 again on system with all updates:

   cd /tmp
   wget https://people.canonical.com/~ubuntu-security/oval/com.ubuntu.xenial.cve.oval.xml
   sudo apt-get install libopenscap8
   oscap oval eval --results /tmp/results-xenial.xml --report /tmp/report-xenial.html /tmp/com.ubuntu.xenial.cve.oval.xml
   #...
   #Evaluation done.
   firefox /tmp/report-xenial.html

OVAL shema is 5.11.1, it reports 354 unpatched problems.

[Norbert (nrbrtx)]

I made test with openscap 1.2.14 from Github on Xenial amd64.

   sudo apt-get purge libopenscap8
   cd /tmp
   sudo apt-get install build-essential cmake libqt4-dev libxslt1-dev libcurl4-openssl-dev libz-dev autoconf libtool libpcre3-dev asciidoctor git checkinstall libgcrypt-dev
   git clone https://github.com/OpenSCAP/openscap
   cd openscap
   git checkout maint-1.2
   ./autogen.sh
   ./configure --prefix /usr --disable-python --disable-util-oscap-docker
   make -j4
   sudo checkinstall make install # specified version 1.2.14
   oscap oval eval --results /tmp/results-xenial.xml --report /tmp/report-xenial.html /tmp/com.ubuntu.xenial.cve.oval.xml
   #...
   #Evaluation done.
   firefox /tmp/report-xenial.html

OVAL shema is 5.11.1, it reports 0 unpatched problems. It's great.
So we really need SRU for openscap package (see bug 1658529).
openscap-workbench is needed to be packaged too (see bug 1658492).

[Tyler Hicks (tyhicks)]

Norbert, the expected result is not to have zero unpatched CVEs. That's unrealistic and I'm afraid that openscap from github is giving you the wrong results. For instance, as of this writing, CVE-2015-5180 against glibc is unfixed so it should not be reporting zero unpatched CVEs.

[Tyler Hicks (tyhicks)]

Was this bug fixed with the oval_lib.py change, as mentioned in https://lists.ubuntu.com/archives/ubuntu-hardened/2017-July/000940.html, being committed?

[Norbert (nrbrtx)]

With ocsap from github CVE-2015-5180 is marked Unknown.
The full statistics is:
  Non-Compliant/Vulnerable/Unpatched = 0,
  Compliant/Non-Vulnerable/Patched = 1988,
  Error = 0,
  Unknown = 6389,
  Other = 1.

With oscap from official repository:
  Non-Compliant/Vulnerable/Unpatched = 354 (11 high, 229 medium, 102 low, 12 negligible),
  Compliant/Non-Vulnerable/Patched = 6829,
  Error = 0,
  Unknown = 1194,
  Other = 1.
Here some CVEs contain references to Android, Qualcomm, aarch64, PuTTY (and WinSCP), but I do not understand this (I'm using amd64 laptop).
I have Wireshark installed, oscap reports CVEs in it. I removed it. Unpatched decreased to 264, Patched increased to 6919. So wireshark has 90 unpatched CVEs. But openscap from github does not change values.

I'll use oscap from official package libopenscap8.

As answer to your comment 10 I can say that CVEs 2012-2150, 2017-8386, 2014-8111 (https://lists.ubuntu.com/archives/ubuntu-hardened/2017-July/000940.html) are marked fixed in my results. I read this conversation and understood it.

So It seems that you are right.
Thank you, Tyler!
This bug is fixed.

[Tyler Hicks (tyhicks)]

On 07/10/2017 12:44 PM, Norbert wrote
> So It seems that you are right.
> Thank you, Tyler!
> This bug is fixed.

Great to hear! Jesus Linares deserves the credit for the bug fix.

[Tyler Hicks (tyhicks)]

Fixed in ubuntu-cve-tracker by http://bazaar.launchpad.net/~ubuntu-security/ubuntu-cve-tracker/master/revision/12856

Marking the openscap task as in Invalid since it was a problem in the OVAL content itself.