nova.tests.unit.test_crypto.RevokeCertsTest.test_revoke_cert_project_not_found_chdir_fails fails locally since ~9/11

[Matt Riedemann]

I rebased my local master branch around 9/11 and I've been seeing this failing consistently locally:

nova.tests.unit.test_crypto.RevokeCertsTest.test_revoke_cert_project_not_found_chdir_fails
------------------------------------------------------------------------------------------

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):
      File "nova/tests/unit/test_crypto.py", line 168, in test_revoke_cert_project_not_found_chdir_fails
        2, 'test_file')
      File "/home/mriedem/git/nova/.tox/py27/local/lib/python2.7/site-packages/testtools/testcase.py", line 422, in assertRaises
        self.assertThat(our_callable, matcher)
      File "/home/mriedem/git/nova/.tox/py27/local/lib/python2.7/site-packages/testtools/testcase.py", line 433, in assertThat
        mismatch_error = self._matchHelper(matchee, matcher, message, verbose)
      File "/home/mriedem/git/nova/.tox/py27/local/lib/python2.7/site-packages/testtools/testcase.py", line 483, in _matchHelper
        mismatch = matcher.match(matchee)
      File "/home/mriedem/git/nova/.tox/py27/local/lib/python2.7/site-packages/testtools/matchers/_exception.py", line 108, in match
        mismatch = self.exception_matcher.match(exc_info)
      File "/home/mriedem/git/nova/.tox/py27/local/lib/python2.7/site-packages/testtools/matchers/_higherorder.py", line 62, in match
        mismatch = matcher.match(matchee)
      File "/home/mriedem/git/nova/.tox/py27/local/lib/python2.7/site-packages/testtools/testcase.py", line 414, in match
        reraise(*matchee)
      File "/home/mriedem/git/nova/.tox/py27/local/lib/python2.7/site-packages/testtools/matchers/_exception.py", line 101, in match
        result = matchee()
      File "/home/mriedem/git/nova/.tox/py27/local/lib/python2.7/site-packages/testtools/testcase.py", line 969, in __call__
        return self._callable_object(*self._args, **self._kwargs)
      File "nova/crypto.py", line 227, in revoke_cert
        raise exception.RevokeCertFailure(project_id=project_id)
    nova.exception.RevokeCertFailure: Failed to revoke certificate for 2

The last changes to test_crypto were made on 9/4:

https://review.openstack.org/#/c/191604/

Given that's related to processutils in oslo.concurrency, I'm wondering if there was a regression in 2.6.0, released on 9/8:

https://pypi.python.org/pypi/oslo.concurrency/2.6.0

[Matt Riedemann (mriedem)]

Even running the test by itself fails so it's not some concurrency issue.

[Matt Riedemann (mriedem)]

mriedem@ubuntu:~$ source git/nova/.tox/py27/bin/activate
(py27)mriedem@ubuntu:~$ pip show cryptography
You are using pip version 7.0.3, however version 7.1.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
---
Metadata-Version: 2.0
Name: cryptography
Version: 1.0.1
Summary: cryptography is a package which provides cryptographic recipes and primitives to Python developers.
Home-page: https://github.com/pyca/cryptography
Author: The cryptography developers
Author-email: <email address hidden>
License: BSD or Apache License, Version 2.0
Location: /home/mriedem/git/nova/.tox/py27/lib/python2.7/site-packages
Requires: setuptools, enum34, ipaddress, pyasn1, six, idna, cffi
(py27)mriedem@ubuntu:~$
(py27)mriedem@ubuntu:~$
(py27)mriedem@ubuntu:~$ pip show paramiko
You are using pip version 7.0.3, however version 7.1.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
---
Metadata-Version: 2.0
Name: paramiko
Version: 1.15.2
Summary: SSH2 protocol library
Home-page: https://github.com/paramiko/paramiko/
Author: Jeff Forcier
Author-email: <email address hidden>
License: LGPL
Location: /home/mriedem/git/nova/.tox/py27/lib/python2.7/site-packages
Requires: ecdsa, pycrypto
(py27)mriedem@ubuntu:~$

[Matt Riedemann (mriedem)]

This change:

https://review.openstack.org/#/c/191604/20/nova/tests/unit/test_crypto.py

Basically made the unit test non-deterministic. The test expects you to hit an OSError and then raise ProjectNotFound but I'm actually hitting the ProcessExecutionError, I'm not really sure why. I have the same versions of openssl locally as in community infra:

ii openssl 1.0.1f-1ubuntu2.15 i386 Secure Sockets Layer toolkit - cryptographic utility
ii python-openssl 0.13-2ubuntu6 i386 Python 2 wrapper around the OpenSSL library

From a gate run on 9/11:

ii openssl 1.0.1f-1ubuntu2.15 amd64 Secure Sockets Layer toolkit - cryptographic utility
ii python-openssl 0.13-2ubuntu6 amd64 Python 2 wrapper around the OpenSSL library

[Matt Riedemann (mriedem)]

I think I have an idea of the issue. The revoke method calls ca_folder to get the path:

def ca_folder(project_id=None):
    if CONF.use_project_ca and project_id:
        return os.path.join(CONF.ca_path, 'projects', project_id)
    return CONF.ca_path

And by default CONF.use_project_ca is False, which would mean use the static CONF.ca_path, which may exist, and then chdir wouldn't fail with OSError and then the test fails.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/223203

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/223203
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=f8f822afafbc58258bfe0653b21be86a5e648305
Submitter: Jenkins
Branch: master

commit f8f822afafbc58258bfe0653b21be86a5e648305
Author: Matt Riedemann <email address hidden>
Date: Mon Sep 14 09:04:51 2015 -0700

    Make test_revoke_cert_project_not_found_chdir_fails deterministic

    Commit b64cebe2eb71a6c2275f6af76839a0b36cf56cbd removed the mock on
    os.chdir to always fail with OSError in the test.

    The revoke_cert method is using the ca_folder method to get the path and
    that checks CONF.use_project_ca which is by default False, so the
    project_id isn't even used in the path we're changing to before running
    the command, it just uses the global static CONF.ca_path. If that path
    exists when this test runs, os.chdir won't raise OSError and the
    assertion in the test will fail.

    This change makes the test deterministic by (1) setting
    use_project_ca=True and (2) using a random project_id uuid.

    Closes-Bug: #1495547

    Change-Id: I0dc6574b6644aa09028a07503a3d0b82e68e69d2