Bug #1399498 “centos 7 unit test fails” : Bugs : OpenStack Compute (nova)

Revision history for this message

Davanum Srinivas (DIMS) (dims-v) wrote on 2014-12-08:

#1

Download full text (9.6 KiB)

Here's a bit more detail

[stack@dc8f7542d504 nova]$ tox -e py27 nova.tests.unit.test_crypto.X509Test.test_can_generate_x509
py27 develop-inst-noop: /opt/stack/nova
py27 runtests: PYTHONHASHSEED='0'
py27 runtests: commands[0] | find . -type f -name *.pyc -delete
py27 runtests: commands[1] | bash tools/pretty_tox.sh nova.tests.unit.test_crypto.X509Test.test_can_generate_x509
running testr
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-160} \
${PYTHON:-python} -m subunit.run discover -t ./ ${OS_TEST_PATH:-./nova/tests} --list
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-160} \
${PYTHON:-python} -m subunit.run discover -t ./ ${OS_TEST_PATH:-./nova/tests} --load-list /tmp/tmpR91GkK

{0} nova.tests.unit.test_crypto.X509Test.test_can_generate_x509 [5.409021s] ... FAILED

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):
      File "nova/tests/unit/test_crypto.py", line 50, in test_can_generate_x509
        project_cert_file, '-verbose', signed_cert_file)
      File "nova/utils.py", line 164, in execute
        return processutils.execute(*cmd, **kwargs)
      File "/opt/stack/nova/.tox/py27/lib/python2.7/site-packages/oslo_concurrency/processutils.py", line 224, in execute
        cmd=sanitized_cmd)
    ProcessExecutionError: Unexpected error while running command.
    Command: openssl verify -CAfile /tmp/tmpcHtkYL/tmp1Sfc6G/project -verbose /tmp/tmpcHtkYL/tmp1Sfc6G/signed
    Exit code: 2
    Stdout: u'/tmp/tmpcHtkYL/tmp1Sfc6G/signed: C = US, ST = California, O = OpenStack, OU = NovaDev, CN = fake-fake-2014-12-08T16:12:47Z\nerror 7 at 0 depth lookup:certificate signature failure\n'
    Stderr: u'140085465835424:error:0D0C50A1:asn1 encoding routines:ASN1_item_verify:unknown message digest algorithm:a_verify.c:179:\n'
    Traceback (most recent call last):
    _StringException: Empty attachments:
      stderr
      stdout

pythonlogging:'': {{{2014-12-08 16:12:46,482 INFO [248_add_expire_reservations_index] Skipped adding reservations_deleted_expire_idx because an equivalent index already exists.}}}

    Traceback (most recent call last):
      File "nova/tests/unit/test_crypto.py", line 50, in test_can_generate_x509
        project_cert_file, '-verbose', signed_cert_file)
      File "nova/utils.py", line 164, in execute
        return processutils.execute(*cmd, **kwargs)
      File "/opt/stack/nova/.tox/py27/lib/python2.7/site-packages/oslo_concurrency/processutils.py", line 224, in execute
        cmd=sanitized_cmd)
    ProcessExecutionError: Unexpected error while running command.
    Command: openssl verify -CAfile /tmp/tmpcHtkYL/tmp1Sfc6G/project -verbose /tmp/tmpcHtkYL/tmp1Sfc6G/signed
    Exit code: 2
    Stdout: u'/tmp/tmpcHtkYL/tmp1Sfc6G/signed: C = US, ST = California, O = OpenStack, OU = NovaDev, CN = fake-fake-2014-12-08T16:12:47Z\nerror 7 at 0 depth lookup:certificate signature failure\n'
    Stderr: u'140085465835424:error:0D0C50A1:asn1 encoding routines:ASN1_item_verify:unknown message digest algorithm:a_verify.c:179:\n'

Traceba...

Here's a bit more detail

[stack@dc8f7542d504 nova]$ tox -e py27 nova.tests.unit.test_crypto.X509Test.test_can_generate_x509
py27 develop-inst-noop: /opt/stack/nova
py27 runtests: PYTHONHASHSEED='0'
py27 runtests: commands[0] | find . -type f -name *.pyc -delete
py27 runtests: commands[1] | bash tools/pretty_tox.sh nova.tests.unit.test_crypto.X509Test.test_can_generate_x509
running testr
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-160} \
${PYTHON:-python} -m subunit.run discover -t ./ ${OS_TEST_PATH:-./nova/tests} --list
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-160} \
${PYTHON:-python} -m subunit.run discover -t ./ ${OS_TEST_PATH:-./nova/tests}  --load-list /tmp/tmpR91GkK

{0} nova.tests.unit.test_crypto.X509Test.test_can_generate_x509 [5.409021s] ... FAILED

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):
      File "nova/tests/unit/test_crypto.py", line 50, in test_can_generate_x509
        project_cert_file, '-verbose', signed_cert_file)
      File "nova/utils.py", line 164, in execute
        return processutils.execute(*cmd, **kwargs)
      File "/opt/stack/nova/.tox/py27/lib/python2.7/site-packages/oslo_concurrency/processutils.py", line 224, in execute
        cmd=sanitized_cmd)
    ProcessExecutionError: Unexpected error while running command.
    Command: openssl verify -CAfile /tmp/tmpcHtkYL/tmp1Sfc6G/project -verbose /tmp/tmpcHtkYL/tmp1Sfc6G/signed
    Exit code: 2
    Stdout: u'/tmp/tmpcHtkYL/tmp1Sfc6G/signed: C = US, ST = California, O = OpenStack, OU = NovaDev, CN = fake-fake-2014-12-08T16:12:47Z\nerror 7 at 0 depth lookup:certificate signature failure\n'
    Stderr: u'140085465835424:error:0D0C50A1:asn1 encoding routines:ASN1_item_verify:unknown message digest algorithm:a_verify.c:179:\n'
    Traceback (most recent call last):
    _StringException: Empty attachments:
      stderr
      stdout

pythonlogging:'': {{{2014-12-08 16:12:46,482 INFO [248_add_expire_reservations_index] Skipped adding reservations_deleted_expire_idx because an equivalent index already exists.}}}

Traceback (most recent call last):
      File "nova/tests/unit/test_crypto.py", line 50, in test_can_generate_x509
        project_cert_file, '-verbose', signed_cert_file)
      File "nova/utils.py", line 164, in execute
        return processutils.execute(*cmd, **kwargs)
      File "/opt/stack/nova/.tox/py27/lib/python2.7/site-packages/oslo_concurrency/processutils.py", line 224, in execute
        cmd=sanitized_cmd)
    ProcessExecutionError: Unexpected error while running command.
    Command: openssl verify -CAfile /tmp/tmpcHtkYL/tmp1Sfc6G/project -verbose /tmp/tmpcHtkYL/tmp1Sfc6G/signed
    Exit code: 2
    Stdout: u'/tmp/tmpcHtkYL/tmp1Sfc6G/signed: C = US, ST = California, O = OpenStack, OU = NovaDev, CN = fake-fake-2014-12-08T16:12:47Z\nerror 7 at 0 depth lookup:certificate signature failure\n'
    Stderr: u'140085465835424:error:0D0C50A1:asn1 encoding routines:ASN1_item_verify:unknown message digest algorithm:a_verify.c:179:\n'

Traceback (most recent call last):
    _StringException: Empty attachments:
      stderr
      stdout

pythonlogging:'': {{{2014-12-08 16:12:46,482 INFO [248_add_expire_reservations_index] Skipped adding reservations_deleted_expire_idx because an equivalent index already exists.}}}

Traceback (most recent call last):
      File "nova/tests/unit/test_crypto.py", line 50, in test_can_generate_x509
        project_cert_file, '-verbose', signed_cert_file)
      File "nova/utils.py", line 164, in execute
        return processutils.execute(*cmd, **kwargs)
      File "/opt/stack/nova/.tox/py27/lib/python2.7/site-packages/oslo_concurrency/processutils.py", line 224, in execute
        cmd=sanitized_cmd)
    ProcessExecutionError: Unexpected error while running command.
    Command: openssl verify -CAfile /tmp/tmpcHtkYL/tmp1Sfc6G/project -verbose /tmp/tmpcHtkYL/tmp1Sfc6G/signed
    Exit code: 2
    Stdout: u'/tmp/tmpcHtkYL/tmp1Sfc6G/signed: C = US, ST = California, O = OpenStack, OU = NovaDev, CN = fake-fake-2014-12-08T16:12:47Z\nerror 7 at 0 depth lookup:certificate signature failure\n'
    Stderr: u'140085465835424:error:0D0C50A1:asn1 encoding routines:ASN1_item_verify:unknown message digest algorithm:a_verify.c:179:\n'

Captured pythonlogging:
~~~~~~~~~~~~~~~~~~~~~~~
    2014-12-08 16:12:46,482 INFO [248_add_expire_reservations_index] Skipped adding reservations_deleted_expire_idx because an equivalent index already exists.

Slowest Tests
Test id                                                      Runtime (s)
-----------------------------------------------------------  -----------
nova.tests.unit.test_crypto.X509Test.test_can_generate_x509  5.409

==============================
Failed 1 tests - output below:
==============================

nova.tests.unit.test_crypto.X509Test.test_can_generate_x509
-----------------------------------------------------------

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):
      File "nova/tests/unit/test_crypto.py", line 50, in test_can_generate_x509
        project_cert_file, '-verbose', signed_cert_file)
      File "nova/utils.py", line 164, in execute
        return processutils.execute(*cmd, **kwargs)
      File "/opt/stack/nova/.tox/py27/lib/python2.7/site-packages/oslo_concurrency/processutils.py", line 224, in execute
        cmd=sanitized_cmd)
    ProcessExecutionError: Unexpected error while running command.
    Command: openssl verify -CAfile /tmp/tmpcHtkYL/tmp1Sfc6G/project -verbose /tmp/tmpcHtkYL/tmp1Sfc6G/signed
    Exit code: 2
    Stdout: u'/tmp/tmpcHtkYL/tmp1Sfc6G/signed: C = US, ST = California, O = OpenStack, OU = NovaDev, CN = fake-fake-2014-12-08T16:12:47Z\nerror 7 at 0 depth lookup:certificate signature failure\n'
    Stderr: u'140085465835424:error:0D0C50A1:asn1 encoding routines:ASN1_item_verify:unknown message digest algorithm:a_verify.c:179:\n'
    Traceback (most recent call last):
    _StringException: Empty attachments:
      stderr
      stdout

pythonlogging:'': {{{2014-12-08 16:12:46,482 INFO [248_add_expire_reservations_index] Skipped adding reservations_deleted_expire_idx because an equivalent index already exists.}}}

Traceback (most recent call last):
      File "nova/tests/unit/test_crypto.py", line 50, in test_can_generate_x509
        project_cert_file, '-verbose', signed_cert_file)
      File "nova/utils.py", line 164, in execute
        return processutils.execute(*cmd, **kwargs)
      File "/opt/stack/nova/.tox/py27/lib/python2.7/site-packages/oslo_concurrency/processutils.py", line 224, in execute
        cmd=sanitized_cmd)
    ProcessExecutionError: Unexpected error while running command.
    Command: openssl verify -CAfile /tmp/tmpcHtkYL/tmp1Sfc6G/project -verbose /tmp/tmpcHtkYL/tmp1Sfc6G/signed
    Exit code: 2
    Stdout: u'/tmp/tmpcHtkYL/tmp1Sfc6G/signed: C = US, ST = California, O = OpenStack, OU = NovaDev, CN = fake-fake-2014-12-08T16:12:47Z\nerror 7 at 0 depth lookup:certificate signature failure\n'
    Stderr: u'140085465835424:error:0D0C50A1:asn1 encoding routines:ASN1_item_verify:unknown message digest algorithm:a_verify.c:179:\n'

Traceback (most recent call last):
    _StringException: Empty attachments:
      stderr
      stdout

pythonlogging:'': {{{2014-12-08 16:12:46,482 INFO [248_add_expire_reservations_index] Skipped adding reservations_deleted_expire_idx because an equivalent index already exists.}}}

Traceback (most recent call last):
      File "nova/tests/unit/test_crypto.py", line 50, in test_can_generate_x509
        project_cert_file, '-verbose', signed_cert_file)
      File "nova/utils.py", line 164, in execute
        return processutils.execute(*cmd, **kwargs)
      File "/opt/stack/nova/.tox/py27/lib/python2.7/site-packages/oslo_concurrency/processutils.py", line 224, in execute
        cmd=sanitized_cmd)
    ProcessExecutionError: Unexpected error while running command.
    Command: openssl verify -CAfile /tmp/tmpcHtkYL/tmp1Sfc6G/project -verbose /tmp/tmpcHtkYL/tmp1Sfc6G/signed
    Exit code: 2
    Stdout: u'/tmp/tmpcHtkYL/tmp1Sfc6G/signed: C = US, ST = California, O = OpenStack, OU = NovaDev, CN = fake-fake-2014-12-08T16:12:47Z\nerror 7 at 0 depth lookup:certificate signature failure\n'
    Stderr: u'140085465835424:error:0D0C50A1:asn1 encoding routines:ASN1_item_verify:unknown message digest algorithm:a_verify.c:179:\n'

Captured pythonlogging:
~~~~~~~~~~~~~~~~~~~~~~~
    2014-12-08 16:12:46,482 INFO [248_add_expire_reservations_index] Skipped adding reservations_deleted_expire_idx because an equivalent index already exists.

======
Totals
======
Run: 1 in 5.409021 sec.
 - Passed: 0
 - Skipped: 0
 - Failed: 1

==============
Worker Balance
==============
 - Worker 0 (1 tests) => 0:00:05.409021s
ERROR: InvocationError: '/bin/bash tools/pretty_tox.sh nova.tests.unit.test_crypto.X509Test.test_can_generate_x509'
______________________________________________________________________ summary _______________________________________________________________________
ERROR:   py27: commands failed
[stack@dc8f7542d504 nova]$ uname -a
Linux dc8f7542d504 3.13.0-29-generic #53-Ubuntu SMP Wed Jun 4 21:00:20 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
[stack@dc8f7542d504 nova]$ cat /etc/*release*
CentOS Linux release 7.0.1406 (Core)
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CentOS Linux release 7.0.1406 (Core)
CentOS Linux release 7.0.1406 (Core)
cpe:/o:centos:centos:7

Revision history for this message

Joe Gordon (jogo) wrote on 2014-12-10:

#2

This looks like an issue with missing dependencies. We try to keep a record of what packages are needed to make the unit tests pass here http://docs.openstack.org/developer/nova/devref/development.environment.html#linux-systems

So once we find the missing package we should update that document http://git.openstack.org/cgit/openstack/nova/tree/doc/source/devref/development.environment.rst

Changed in nova:
status:	New → Confirmed
importance:	Undecided → Low
tags:	added: low-hanging-fruit

Revision history for this message

Mike Bayer (zzzeek) wrote on 2015-01-06:

#3

is this really a packaging thing? From what I can tell, this is a redhat/fedora patch to openssl and is just their behavior now:

https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/7.0_Release_Notes/Known-Issues-Networking.html#varl-Red_Hat_Enterprise_Linux-7.0_Release_Notes-Networking-openssl_component_BZ1062656

https://lists.fedoraproject.org/pipermail/scm-commits/Week-of-Mon-20131111/1144043.html

https://bugzilla.redhat.com/show_bug.cgi?id=1062656#c12

etc.

I would think the solution is to just add these environment variables to tox.ini.

Yasuyuki (yasuyuki) on 2015-02-03

Changed in nova:
assignee:	nobody → Yasuyuki (yasuyuki)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-02-05: Fix proposed to nova (master)

#4

Fix proposed to branch: master
Review: https://review.openstack.org/153339

Changed in nova:
assignee:	Yasuyuki (yasuyuki) → Davanum Srinivas (DIMS) (dims-v)
status:	Confirmed → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-02-09: Fix merged to nova (master)

#5

Reviewed: https://review.openstack.org/153339
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=f4495de5a04b03bbd6773b6b059ea0341a2d0aea
Submitter: Jenkins
Branch: master

commit f4495de5a04b03bbd6773b6b059ea0341a2d0aea
Author: Davanum Srinivas <email address hidden>
Date: Thu Feb 5 13:57:04 2015 -0500

Make tests use sha256 as openssl default digest algorithm

    The tests previously used md5, which is considered broken,
    and distros are starting to disable this in their openssl
    builds.

Make the tests use sha256 as the default as a long term
sane alternative that should work on all distros.

This will fix Centos 7, and future proof the tests.

Closes-Bug: #1399498
Change-Id: Ic6cc92e47a318d789db3c3c98c67948eefb51fc2

Changed in nova:
status:	In Progress → Fix Committed

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-02-24: Fix proposed to nova (stable/juno)

#6

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/158816

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-02-24: Fix proposed to nova (stable/icehouse)

#7

Fix proposed to branch: stable/icehouse
Review: https://review.openstack.org/158895

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-03-10: Fix merged to nova (stable/juno)

#8

Reviewed: https://review.openstack.org/158816
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=b3f80bd23855fa49420f5b66698d3e88e965e808
Submitter: Jenkins
Branch: stable/juno

commit b3f80bd23855fa49420f5b66698d3e88e965e808
Author: Davanum Srinivas <email address hidden>
Date: Thu Feb 5 13:57:04 2015 -0500

Make tests use sha256 as openssl default digest algorithm

    The tests previously used md5, which is considered broken,
    and distros are starting to disable this in their openssl
    builds.

Make the tests use sha256 as the default as a long term
sane alternative that should work on all distros.

This will fix Centos 7, and future proof the tests.

    Closes-Bug: #1399498
    Change-Id: Ic6cc92e47a318d789db3c3c98c67948eefb51fc2
    (cherry picked from commit f4495de5a04b03bbd6773b6b059ea0341a2d0aea)

tags:	added: in-stable-juno
tags:	added: in-stable-icehouse

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-03-10: Fix merged to nova (stable/icehouse)

#9

Reviewed: https://review.openstack.org/158895
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=e27596102087984d2b8c9b74ef0e340c372d0b77
Submitter: Jenkins
Branch: stable/icehouse

commit e27596102087984d2b8c9b74ef0e340c372d0b77
Author: Davanum Srinivas <email address hidden>
Date: Thu Feb 5 13:57:04 2015 -0500

Make tests use sha256 as openssl default digest algorithm

    The tests previously used md5, which is considered broken,
    and distros are starting to disable this in their openssl
    builds.

Make the tests use sha256 as the default as a long term
sane alternative that should work on all distros.

This will fix Centos 7, and future proof the tests.

    Closes-Bug: #1399498
    Change-Id: Ic6cc92e47a318d789db3c3c98c67948eefb51fc2
    (cherry picked from commit f4495de5a04b03bbd6773b6b059ea0341a2d0aea)

Thierry Carrez (ttx) on 2015-03-20

Changed in nova:
milestone:	none → kilo-3
status:	Fix Committed → Fix Released

Thierry Carrez (ttx) on 2015-04-30

Changed in nova:
milestone:	kilo-3 → 2015.1.0

	Status	Importance	Assigned to	Milestone
OpenStack Compute (nova)	Fix Released	Low	Davanum Srinivas (DIMS)	OpenStack Compute (nova) 2015.1.0 "kilo"
Icehouse	Fix Released	Undecided	Unassigned	OpenStack Compute (nova) 2014.1.4
Juno	Fix Released	Low	Artom Lifshitz	OpenStack Compute (nova) 2014.2.3

OpenStack Compute (nova)

centos 7 unit test fails

Bug Description

Other bug subscribers

Remote bug watches