Unit test failure with OpenSSL 1.1.1

Bug #1771506 reported by Thomas Goirand on 2018-05-16
26
This bug affects 3 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Low
Corey Bryant
Ubuntu Cloud Archive
Status tracked in Stein
Queens
High
Unassigned
Rocky
High
Unassigned
Stein
High
Unassigned
nova (Ubuntu)
High
Unassigned
Bionic
High
Unassigned
Cosmic
High
Unassigned
Disco
High
Unassigned

Bug Description

Hi,

Building the Nova Queens package with OpenSSL 1.1.1 leads to unit test problems. This was reported to Debian at: https://bugs.debian.org/898807

The new openssl 1.1.1 is currently in experimental [0]. This package failed to build against this new package [1] while it built fine against the openssl version currently in unstable [2]. Could you please have a look?

FAIL: nova.tests.unit.virt.xenapi.test_xenapi.XenAPIDiffieHellmanTestCase.test_encrypt_newlines_inside_message
|nova.tests.unit.virt.xenapi.test_xenapi.XenAPIDiffieHellmanTestCase.test_encrypt_newlines_inside_message
|----------------------------------------------------------------------
|_StringException: pythonlogging:'': {{{2018-05-01 20:48:09,960 WARNING [oslo_config.cfg] Config option key_manager.api_class is deprecated. Use option key_manager.backend instead.}}}
|
|Traceback (most recent call last):
| File "/<<PKGBUILDDIR>>/nova/tests/unit/virt/xenapi/test_xenapi.py", line 1592, in test_encrypt_newlines_inside_message
| self._test_encryption('Message\nwith\ninterior\nnewlines.')
| File "/<<PKGBUILDDIR>>/nova/tests/unit/virt/xenapi/test_xenapi.py", line 1577, in _test_encryption
| enc = self.alice.encrypt(message)
| File "/<<PKGBUILDDIR>>/nova/virt/xenapi/agent.py", line 432, in encrypt
| return self._run_ssl(text).strip('\n')
| File "/<<PKGBUILDDIR>>/nova/virt/xenapi/agent.py", line 428, in _run_ssl
| raise RuntimeError(_('OpenSSL error: %s') % err)
|RuntimeError: OpenSSL error: *** WARNING : deprecated key derivation used.
|Using -iter or -pbkdf2 would be better.

It looks like due to additional message on stderr.

[0] https://<email address hidden>
[1] https://breakpoint.cc/openssl-rebuild/2018-05-03-rebuild-openssl1.1.1-pre6/attempted/nova_17.0.0-4_amd64-2018-05-01T20%3A39%3A38Z
[2] https://breakpoint.cc/openssl-rebuild/2018-05-03-rebuild-openssl1.1.1-pre6/successful/nova_17.0.0-4_amd64-2018-05-02T18%3A46%3A36Z

jichenjc (jichenjc) wrote :

seems some key deprecated ? can we check [0] above to know which of follow param lead to error?
best way would be within 1.1.1 env to consturct a command string and try it ..

|RuntimeError: OpenSSL error: *** WARNING : deprecated key derivation used.
|Using -iter or -pbkdf2 would be better.

    def _run_ssl(self, text, decrypt=False):
        cmd = ['openssl', 'aes-128-cbc', '-A', '-a', '-pass',
               'pass:%s' % self._shared, '-nosalt']
        if decrypt:
            cmd.append('-d')
        out, err = utils.execute(*cmd,
                                 process_input=encodeutils.safe_encode(text))
        if err:
            raise RuntimeError(_('OpenSSL error: %s') % err)
        return out

tags: added: xen
tags: added: testing
melanie witt (melwitt) on 2018-08-08
Changed in nova:
importance: Undecided → Low
status: New → Confirmed
Corey Bryant (corey.bryant) wrote :

I'm hitting this as well now that we have openssl 1.1.1 in cosmic-proposed. This is affecting rocky and above for ubuntu. Unfortunately this is preventing our unit tests from running successfully for an 18.0.1 release. To recreate:

lxc launch ubuntu-daily:cosmic c1
lxc exec c1 /bin/bash
root@c1:~# cat >> /etc/apt/sources.list << EOF
deb http://archive.ubuntu.com/ubuntu cosmic-proposed main restricted
deb http://archive.ubuntu.com/ubuntu cosmic-proposed universe
EOF
root@c1:~# sudo apt update
root@c1:~# sudo apt dist-upgrade --yes
root@c1:~# apt policy openssl # should be at openssl 1.1.1-1ubuntu2
root@c1:~# sudo apt install python-dev git gcc tox --yes
root@c1:~# git clone https://github.com/openstack/nova
root@c1:~# cd nova
root@c1:~/nova# tox -e py27 # results in failures: https://paste.ubuntu.com/p/3W39Vy87Sy/

By any chance can the importance of this bug be increased?

Gábor Antal (gabor.antal) wrote :

I also hit this bug! Any news since then?

Dimitri John Ledkov (xnox) wrote :

This is not just a testing issue, it means that xenapi will not able to talk to xen agent at runtime, with openssl 1.1.1 binary.

Since openssl binary is executed, it's a bit hard to determine if it failed or not. As it generates genuine errors and warning in stderr.

In this case the password derivation function has been deprecated in OpenSSL but it still works. I don't know what xen api agent can or cannot accept, thus I don't think it is safe to upgrade the openssl command to use stronger key derivation. Instead, we should whitelist the harmless warning and not treat it as an error.

I do not believe the string is translated in OpenSSL upstream.

Please see the attached path.

It would be copyright canonical, with OpenStack CLA signed. But i'm not sure when I will have time to submit this patch upstream properly.

Fix proposed to branch: master
Review: https://review.openstack.org/635533

Changed in nova:
assignee: nobody → Corey Bryant (corey.bryant)
status: Confirmed → In Progress
Corey Bryant (corey.bryant) wrote :

@xnox, thanks for the patch. I've submitted it to the upstream master branch. Once that lands I'll start backporting to stable branches and Ubuntu.

Changed in nova (Ubuntu Bionic):
status: New → Triaged
importance: Undecided → High
Changed in nova (Ubuntu Cosmic):
importance: Undecided → High
status: New → Triaged
Changed in nova (Ubuntu Disco):
importance: Undecided → High
status: New → Triaged
tags: added: patch

Reviewed: https://review.opendev.org/635533
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=1da71fa4ab1d7d0f580cd5cbc97f2dfd2e1c378a
Submitter: Zuul
Branch: master

commit 1da71fa4ab1d7d0f580cd5cbc97f2dfd2e1c378a
Author: Corey Bryant <email address hidden>
Date: Thu Feb 7 10:12:54 2019 -0500

    xenapi/agent: Change openssl error handling

    Prior to this patch, if the openssl command returned a zero exit code
    and wrote details to stderr, nova would raise a RuntimeError exception.
    This patch changes the behavior to only raise a RuntimeError exception
    when openssl returns a non-zero exit code. Regardless of the exit code
    a warning will always be logged with stderr details if stderr is not
    None. Note that processutils.execute will now raise a
    processutils.ProcessExecutionError exception for any non-zero exit code
    since we are passing check_exit_code=True, which we convert to a
    Runtime error.

    Thanks to Dimitri John Ledkov <email address hidden> and Eric Fried
    <email address hidden> for helping with this patch.

    Change-Id: I212ac2b5ccd93e00adb7b9fe102fcb70857c6073
    Partial-Bug: #1771506

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package nova - 2:19.0.0-0ubuntu4

---------------
nova (2:19.0.0-0ubuntu4) eoan; urgency=medium

  * d/p/xenapi-agent-change-openssl-error-handling.patch: Cherry-picked from
    upstream to ensure xenapi agent only raises a RuntimeError exception
    when openssl returns a non-zero exit code (LP: #1771506).

 -- Corey Bryant <email address hidden> Wed, 01 May 2019 17:10:47 -0400

Changed in nova (Ubuntu):
status: Triaged → Fix Released
Corey Bryant (corey.bryant) wrote :

New versions of nova with this fix have been uploaded to eoan, disco, cosmic, and bionic. Stable release uploads are awaiting review from the SRU team [1].

[1]
https://launchpad.net/ubuntu/disco/+queue?queue_state=1&queue_text=nova
https://launchpad.net/ubuntu/cosmic/+queue?queue_state=1&queue_text=nova
https://launchpad.net/ubuntu/bionic/+queue?queue_state=1&queue_text=nova

An upload of nova to disco-proposed has been rejected from the upload queue for the following reason: "The .changes file doesn't incorporate changes in 2:19.0.0-0ubuntu2.1 please reupload.".

Hello Thomas, or anyone else affected,

Accepted nova into disco-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/nova/2:19.0.0-0ubuntu2.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-disco to verification-done-disco. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-disco. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in nova (Ubuntu Disco):
status: Triaged → Fix Committed
tags: added: verification-needed verification-needed-disco
Changed in nova (Ubuntu Cosmic):
status: Triaged → Fix Committed
tags: added: verification-needed-cosmic
Brian Murray (brian-murray) wrote :

Hello Thomas, or anyone else affected,

Accepted nova into cosmic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/nova/2:18.1.0-0ubuntu2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-cosmic to verification-done-cosmic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-cosmic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in nova (Ubuntu Bionic):
status: Triaged → Fix Committed
tags: added: verification-needed-bionic
Brian Murray (brian-murray) wrote :

Hello Thomas, or anyone else affected,

Accepted nova into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/nova/2:17.0.9-0ubuntu2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Corey Bryant (corey.bryant) wrote :

Hello Thomas, or anyone else affected,

Accepted nova into stein-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:stein-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-stein-needed to verification-stein-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-stein-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

tags: added: verification-stein-needed
Corey Bryant (corey.bryant) wrote :

Hello Thomas, or anyone else affected,

Accepted nova into rocky-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:rocky-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-rocky-needed to verification-rocky-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-rocky-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

tags: added: verification-rocky-needed
Corey Bryant (corey.bryant) wrote :

Hello Thomas, or anyone else affected,

Accepted nova into queens-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:queens-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-queens-needed to verification-queens-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-queens-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

tags: added: verification-queens-needed
Łukasz Zemczak (sil2100) wrote :

Hello Thomas, or anyone else affected,

Accepted nova into disco-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/nova/2:19.0.0-0ubuntu2.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-disco to verification-done-disco. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-disco. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Łukasz Zemczak (sil2100) wrote :

Hello Thomas, or anyone else affected,

Accepted nova into cosmic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/nova/2:18.1.0-0ubuntu3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-cosmic to verification-done-cosmic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-cosmic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Łukasz Zemczak (sil2100) wrote :

Hello Thomas, or anyone else affected,

Accepted nova into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/nova/2:17.0.9-0ubuntu3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Dimitri John Ledkov (xnox) wrote :

Bionic is great, previously it would fail nova.tests.unit.virt.xenapi.test_xenapi.XenAPIDiffieHellmanTestCase tests but now they pass.

Cosmic/Disco/Eoan are also correctly fixed at runtime, however the unittests that exercise this runtime issue are force skipped since the skip-openssl-1.1.1-tests.patch is still applied. We should drop skip-openssl-1.1.1-tests.patch from Cosmic/Disco/Eoan in the subsequent uploads. I've now uploaded dropping skip-openssl-1.1.1-tests.patch into Eoan.

Pass (with nitpicks on cosmic/disco).

tags: added: verification-done verification-done-bionic verification-done-cosmic verification-done-disco
removed: verification-needed verification-needed-bionic verification-needed-cosmic verification-needed-disco
Corey Bryant (corey.bryant) wrote :

@Dimitri, thanks very much. I've pushed changes to cosmic and disco branches to drop the skip-openssl-1.1.1-tests.patch patch and have built them successfully (locally) for disco, cosmic, bionic-stein, and bionic-rocky. I'm going to hold off on uploads just for that change as nova has a lot of churn and they'll get picked up on the next SRU.

tags: added: verification-rocky-done verification-stein-done
removed: verification-rocky-needed verification-stein-needed
Corey Bryant (corey.bryant) wrote :

This has also built successfully in the queens cloud archive.

tags: added: verification-queens-done
removed: verification-queens-needed
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers