unit test nova.tests.auth_unittest.AuthTestCase.test_209_can_generate_x509 failing on Mac OS X

[Joseph Heck]

While setting up to do cleanup work on Nova for the pylint/pep8 blueprint, I brought down and enabled everything I could find to make the tests all function on Nova. I wrote up the components and steps, publishing in the wiki at http://wiki.openstack.org/HackingNovaMacOSX.

With redis, out of the box everything works *except* nova.tests.auth_unittest.AuthTestCase.test_209_can_generate_x509

Here's the stack trace that I receive:

[ERROR]: nova.tests.auth_unittest.AuthTestCase.test_209_can_generate_x509

Traceback (most recent call last):
  File "/Users/heckj/Documents/code/nova/nova/test.py", line 222, in run
    d = self._maybeInlineCallbacks(testMethod)
  File "/Users/heckj/Documents/code/nova/nova/test.py", line 182, in _maybeInlineCallbacks
    g = f()
  File "/Users/heckj/Documents/code/nova/nova/tests/auth_unittest.py", line 162, in test_209_can_generate_x509
    signed_cert = X509.load_cert_string(cert_str)
  File "/Users/heckj/Documents/code/nova/.nova-venv/lib/python2.6/site-packages/M2Crypto/X509.py", line 655, in load_cert_string
    return load_cert_bio(bio, format)
  File "/Users/heckj/Documents/code/nova/.nova-venv/lib/python2.6/site-packages/M2Crypto/X509.py", line 639, in load_cert_bio
    raise X509Error(Err.get_error())
M2Crypto.X509.X509Error: 140735090166816:error:0906D06C:PEM routines:PEM_read_bio:no start line:/SourceCache/OpenSSL098/OpenSSL098-32/src/crypto/pem/pem_lib.c:650:Expecting: CERTIFICATE

Digging into to the test and code a little bit, it appears that the cert_str getting returned is blank, and hence we're hitting an error when trying to use it. Under the covers, the code appears to be doing its work calling openssl directly and getting pieces back. I have 1.0.0a installed (through MacPorts) - just happened to have it. Didn't see it as an explicit dependency on anything earlier than finding it in the code.

Would be happy to help debug, but I'm very unfamiliar with both this code base and the crypto space.

-joe

[justinsb (justin-fathomdb)]

I replied on IRC, but I think I missed you...

One possibility if the openssl program is not returning what you expect, is that there's little or no error checking of spawned processes. I have a patch that will hopefully get merged soon that error checks spawned processes:
https://code.launchpad.net/~justin-fathomdb/nova/check-subprocess-exit-code

It might be worth trying to run the openssl command manually, or merging check-subprocess-exit-code into a copy of your branch, to see if that sheds any light on the problem...

[Joseph Heck (heckj)]

Thanks Justin - I'll give it a shot!

-joe

On Aug 9, 2010, at 12:51 PM, justinsb wrote:

> I replied on IRC, but I think I missed you...
>
> One possibility if the openssl program is not returning what you expect, is that there's little or no error checking of spawned processes. I have a patch that will hopefully get merged soon that error checks spawned processes:
> https://code.launchpad.net/~justin-fathomdb/nova/check-subprocess-exit-code
>
> It might be worth trying to run the openssl command manually, or merging
> check-subprocess-exit-code into a copy of your branch, to see if that
> sheds any light on the problem...
>
> --
> unit test nova.tests.auth_unittest.AuthTestCase.test_209_can_generate_x509 failing on Mac OS X
> https://bugs.launchpad.net/bugs/615518
> You received this bug notification because you are a direct subscriber
> of the bug.
>
> Status in OpenStack Compute (Nova): New
>
> Bug description:
> While setting up to do cleanup work on Nova for the pylint/pep8 blueprint, I brought down and enabled everything I could find to make the tests all function on Nova. I wrote up the components and steps, publishing in the wiki at http://wiki.openstack.org/HackingNovaMacOSX.
>
> With redis, out of the box everything works *except* nova.tests.auth_unittest.AuthTestCase.test_209_can_generate_x509
>
> Here's the stack trace that I receive:
>
> [ERROR]: nova.tests.auth_unittest.AuthTestCase.test_209_can_generate_x509
>
> Traceback (most recent call last):
> File "/Users/heckj/Documents/code/nova/nova/test.py", line 222, in run
> d = self._maybeInlineCallbacks(testMethod)
> File "/Users/heckj/Documents/code/nova/nova/test.py", line 182, in _maybeInlineCallbacks
> g = f()
> File "/Users/heckj/Documents/code/nova/nova/tests/auth_unittest.py", line 162, in test_209_can_generate_x509
> signed_cert = X509.load_cert_string(cert_str)
> File "/Users/heckj/Documents/code/nova/.nova-venv/lib/python2.6/site-packages/M2Crypto/X509.py", line 655, in load_cert_string
> return load_cert_bio(bio, format)
> File "/Users/heckj/Documents/code/nova/.nova-venv/lib/python2.6/site-packages/M2Crypto/X509.py", line 639, in load_cert_bio
> raise X509Error(Err.get_error())
> M2Crypto.X509.X509Error: 140735090166816:error:0906D06C:PEM routines:PEM_read_bio:no start line:/SourceCache/OpenSSL098/OpenSSL098-32/src/crypto/pem/pem_lib.c:650:Expecting: CERTIFICATE
>
> Digging into to the test and code a little bit, it appears that the cert_str getting returned is blank, and hence we're hitting an error when trying to use it. Under the covers, the code appears to be doing its work calling openssl directly and getting pieces back. I have 1.0.0a installed (through MacPorts) - just happened to have it. Didn't see it as an explicit dependency on anything earlier than finding it in the code.
>
> Would be happy to help debug, but I'm very unfamiliar with both this code base and the crypto space.
>
> -joe
>
> To unsubscribe from this bug, go to:
> https://bugs.launchpad.net/nova/+bug/615518/+subscribe

[Joseph Heck (heckj)]

Using Justinb's branch merged over, I get an updated error with an unexpected exit code narrowing this down to invoking the openssl commands:

[ERROR]: nova.tests.auth_unittest.AuthTestCase.test_209_can_generate_x509

Traceback (most recent call last):
  File "/Users/heckj/Documents/code/nova/nova/test.py", line 222, in run
    d = self._maybeInlineCallbacks(testMethod)
  File "/Users/heckj/Documents/code/nova/nova/test.py", line 182, in _maybeInlineCallbacks
    g = f()
  File "/Users/heckj/Documents/code/nova/nova/tests/auth_unittest.py", line 154, in test_209_can_generate_x509
    _key, cert_str = self.manager._generate_x509_cert('test1', 'testproj')
  File "/Users/heckj/Documents/code/nova/nova/auth/manager.py", line 730, in _generate_x509_cert
    signed_cert = crypto.sign_csr(csr, pid)
  File "/Users/heckj/Documents/code/nova/nova/crypto.py", line 116, in sign_csr
    return _sign_csr(csr_text, FLAGS.ca_path)
  File "/Users/heckj/Documents/code/nova/nova/crypto.py", line 134, in _sign_csr
    utils.runthis("Signing cert: %s", "openssl ca -batch -out %s/outbound.crt -config ./openssl.cnf -infiles %s/inbound.csr" % (tmpfolder, tmpfolder))
  File "/Users/heckj/Documents/code/nova/nova/utils.py", line 106, in runthis
    % (exit_code, cmd))
exceptions.Exception: Unexpected exit code: 1 from cmd: openssl ca -batch -out /var/folders/eo/eodZkg3ZFqeUxobvY1jb+++++TI/-Tmp-/tmpx_CdiL/outbound.crt -config ./openssl.cnf -infiles /var/folders/eo/eodZkg3ZFqeUxobvY1jb+++++TI/-Tmp-/tmpx_CdiL/inbound.csr

[Vish Ishaya (vishvananda)]

It runs fine on my mac with version
OpenSSL 0.9.8l 5 Nov 2009
perhaps there is a compatibility issue with the newer version of openssl?

Vish

On Mon, Aug 9, 2010 at 6:06 PM, Joseph Heck <email address hidden>wrote:

> Using Justinb's branch merged over, I get an updated error with an
> unexpected exit code narrowing this down to invoking the openssl
> commands:
>
>
> [ERROR]: nova.tests.auth_unittest.AuthTestCase.test_209_can_generate_x509
>
> Traceback (most recent call last):
> File "/Users/heckj/Documents/code/nova/nova/test.py", line 222, in run
> d = self._maybeInlineCallbacks(testMethod)
> File "/Users/heckj/Documents/code/nova/nova/test.py", line 182, in
> _maybeInlineCallbacks
> g = f()
> File "/Users/heckj/Documents/code/nova/nova/tests/auth_unittest.py", line
> 154, in test_209_can_generate_x509
> _key, cert_str = self.manager._generate_x509_cert('test1', 'testproj')
> File "/Users/heckj/Documents/code/nova/nova/auth/manager.py", line 730, in
> _generate_x509_cert
> signed_cert = crypto.sign_csr(csr, pid)
> File "/Users/heckj/Documents/code/nova/nova/crypto.py", line 116, in
> sign_csr
> return _sign_csr(csr_text, FLAGS.ca_path)
> File "/Users/heckj/Documents/code/nova/nova/crypto.py", line 134, in
> _sign_csr
> utils.runthis("Signing cert: %s", "openssl ca -batch -out
> %s/outbound.crt -config ./openssl.cnf -infiles %s/inbound.csr" % (tmpfolder,
> tmpfolder))
> File "/Users/heckj/Documents/code/nova/nova/utils.py", line 106, in
> runthis
> % (exit_code, cmd))
> exceptions.Exception: Unexpected exit code: 1 from cmd: openssl ca -batch
> -out
> /var/folders/eo/eodZkg3ZFqeUxobvY1jb+++++TI/-Tmp-/tmpx_CdiL/outbound.crt
> -config ./openssl.cnf -infiles
> /var/folders/eo/eodZkg3ZFqeUxobvY1jb+++++TI/-Tmp-/tmpx_CdiL/inbound.csr
>
> --
> unit test nova.tests.auth_unittest.AuthTestCase.test_209_can_generate_x509
> failing on Mac OS X
> https://bugs.launchpad.net/bugs/615518
> You received this bug notification because you are a member of Nova
> Bugs, which is subscribed to OpenStack Compute (nova).
>
> Status in OpenStack Compute (Nova): New
>
> Bug description:
> While setting up to do cleanup work on Nova for the pylint/pep8 blueprint,
> I brought down and enabled everything I could find to make the tests all
> function on Nova. I wrote up the components and steps, publishing in the
> wiki at http://wiki.openstack.org/HackingNovaMacOSX.
>
> With redis, out of the box everything works *except*
> nova.tests.auth_unittest.AuthTestCase.test_209_can_generate_x509
>
> Here's the stack trace that I receive:
>
> [ERROR]: nova.tests.auth_unittest.AuthTestCase.test_209_can_generate_x509
>
> Traceback (most recent call last):
> File "/Users/heckj/Documents/code/nova/nova/test.py", line 222, in run
> d = self._maybeInlineCallbacks(testMethod)
> File "/Users/heckj/Documents/code/nova/nova/test.py", line 182, in
> _maybeInlineCallbacks
> g = f()
> File "/Users/heckj/Documents/code/nova/nova/tests/auth_unittest.py", line
> 162, in test_209_can_generate_x509
> signed_cert = X509.load_cert_string(cert_str)
> File
> "/Users/heckj/Documents/code/nova/.nova-venv/lib/python2.6/site-packages/M2Crypto/X509.py",
> l...

[Joseph Heck (heckj)]

Vish is right on - the bug is related to the version of OpenSSL. 1.0.0a must have some different responses that the code isn't anticipating. This should likely be called out as an explicit dependency and the code updated to check/fail on version correctly. What that means in terms of the code I'm not sure though...

[Jay Pipes (jaypipes)]

Do we have any update on this? Do we have a specific version of OpenSSL specified in the install instructions for MacOSX? Do we need to have this specified? Marking Incomplete until we have a follow-up.

[Joseph Heck (heckj)]

The version that functions correctly is OpenSSL 0.9.8l 5 , installed with MacOS 10.6 as a base element. The version that doesn't function is OpenSSL 1.0.0a (installed via MacPorts while getting other MacPorts pieces).

[Jay Pipes (jaypipes)]

OK, thx for the update Joseph. Sounds like this is a documentation bug for now. This limitation should be addressed in the installation docs. Assigning to Anne.

[Anne Gentle (annegentle)]

Added notes to these wiki pages:
http://wiki.openstack.org/HackingNovaMacOSX
http://wiki.openstack.org/NovaInstall