[neutron-lbaas][7.0] Periodic job fails

[Dmitry Teselkin]

Periodic job periodic-neutron-lbaas-python27-mos70 fails:

-----
04:45:01 neutron_lbaas.tests.unit.common.tls_utils.test_cert_parser.TestTLSParseUtils.test_read_private_key
04:45:01 --------------------------------------------------------------------------------------------------
04:45:01
04:45:01 Captured traceback:
04:45:01 ~~~~~~~~~~~~~~~~~~~
04:45:01 Traceback (most recent call last):
04:45:01 _StringException: Empty attachments:
04:45:01 pythonlogging:''
04:45:01 pythonlogging:'neutron.api.extensions'
04:45:01 stderr
04:45:01 stdout
04:45:01
04:45:01 Traceback (most recent call last):
04:45:01 File "neutron_lbaas/tests/unit/common/tls_utils/test_cert_parser.py", line 238, in test_read_private_key
04:45:01 ENCRYPTED_PKCS8_CRT_KEY)
04:45:01 File "/home/jenkins/workspace/periodic-neutron-lbaas-python27-mos70/.tox/py27/local/lib/python2.7/site-packages/testtools/testcase.py", line 420, in assertRaises
04:45:01 self.assertThat(our_callable, matcher)
04:45:01 File "/home/jenkins/workspace/periodic-neutron-lbaas-python27-mos70/.tox/py27/local/lib/python2.7/site-packages/testtools/testcase.py", line 431, in assertThat
04:45:01 mismatch_error = self._matchHelper(matchee, matcher, message, verbose)
04:45:01 File "/home/jenkins/workspace/periodic-neutron-lbaas-python27-mos70/.tox/py27/local/lib/python2.7/site-packages/testtools/testcase.py", line 481, in _matchHelper
04:45:01 mismatch = matcher.match(matchee)
04:45:01 File "/home/jenkins/workspace/periodic-neutron-lbaas-python27-mos70/.tox/py27/local/lib/python2.7/site-packages/testtools/matchers/_exception.py", line 108, in match
04:45:01 mismatch = self.exception_matcher.match(exc_info)
04:45:01 File "/home/jenkins/workspace/periodic-neutron-lbaas-python27-mos70/.tox/py27/local/lib/python2.7/site-packages/testtools/matchers/_higherorder.py", line 62, in match
04:45:01 mismatch = matcher.match(matchee)
04:45:01 File "/home/jenkins/workspace/periodic-neutron-lbaas-python27-mos70/.tox/py27/local/lib/python2.7/site-packages/testtools/testcase.py", line 412, in match
04:45:01 reraise(*matchee)
04:45:01 File "/home/jenkins/workspace/periodic-neutron-lbaas-python27-mos70/.tox/py27/local/lib/python2.7/site-packages/testtools/matchers/_exception.py", line 101, in match
04:45:01 result = matchee()
04:45:01 File "/home/jenkins/workspace/periodic-neutron-lbaas-python27-mos70/.tox/py27/local/lib/python2.7/site-packages/testtools/testcase.py", line 965, in __call__
04:45:01 return self._callable_object(*self._args, **self._kwargs)
04:45:01 File "neutron_lbaas/common/tls_utils/cert_parser.py", line 64, in _read_privatekey
04:45:01 return crypto.load_privatekey(crypto.FILETYPE_PEM, privatekey_pem, cb)
04:45:01 Error: [('PEM routines', 'PEM_READ_BIO_PRIVATEKEY', 'bad password read')]
-----

The reason is the pyOpenSSL==0.13 that was used during the test. That version is used because it's available through system repositories. However, tests need more recent version, e.g. pyOpenSSL==0.15.1 fixes the problem [3].

Links
====
[1] https://jenkins.mosi.mirantis.net/job/periodic-neutron-lbaas-python27-mos70/
[2] http://logs.mosi.mirantis.net/periodic/periodic-neutron-lbaas-python27-mos70/06740bd/
[3] https://review.fuel-infra.org/#/c/8961/

[Dmitry Teselkin (teselkin-d)]

PyOpenSSL==0.14 pinned in mos-requirements.
We need to update system package too, probably.

[Ilya Shakhat (shakhat)]

Upstream bug: https://bugs.launchpad.net/neutron/+bug/1470856

[Ilya Shakhat (shakhat)]

Unit test failure detects a real bug in the code of pyOpenSSL library version 0.13.x. The issue can be fixed by upgrading pyOpenSSL to 0.14 or higher. Re-assigning the issue to MOS Packaging team

[Igor Yozhikov (iyozhikov)]

Update of any dependent project/component can lead to unknown results in common or specific functionality in different parts of OpenStack.
In this case upgrade of python-openssl from 0.13 to 0.14 will require check whole bunch of packages that used it as dependency.
apt-cache rdepends python-openssl
python-openssl
Reverse Depends:
  python-glance
  telepathy-gabble-tests
  python-freeipa
  mercurial-common
  python-werkzeug
  python-swift
  python-glance
  python-openssl:i386
  sabnzbdplus
  zorp
  w3af-console
  tryton-server
  tpclient-pywx
  telepathy-gabble-tests
  tahoe-lafs
  syncevolution-http
  spikeproxy
  shinken-module-poller-nrpe
  servefile
  roundup
  rhn-client-tools
  python-u1db
  python-tp-netlib
  python-swauth
  python-scrapy
  python-rhn
  python-netlib
  python-freeipa
  python-foolscap
  python-epsilon
  python-cyclone
  python-apns-client
  pymsnt
  pyicqt
  pyaimt
  nordugrid-arc-acix-index
  nordugrid-arc-acix-core
  nordugrid-arc-acix-cache
  mitmproxy
  mercurial-common
  maptransfer-server
  ltsp-controlaula
  ltsp-cluster-agent
  lottanzb
  ganeti
  gajim
  flumotion
  epoptes
  emesene
  deluge-common
  controlaula
  certmaster
  cardstories
  calendarserver
  python-werkzeug
  python-ubuntu-sso-client
  python-twisted-words
  python-twisted-mail
  python-twisted-core
  python-swift
  python-paste
  python-openssl-doc
  python-openssl-dbg
  python-glanceclient
  python-glance
  python-gevent

[Igor Yozhikov (iyozhikov)]

If this change required due some kind of broken functionality - impact should be described and analysed.

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/10448
Submitter: mos-infra-ci <>
Branch: openstack-ci/fuel-7.0/2015.1.0

Commit: 98da84be741c4195d75b1a4beb19929e344e5697
Author: Ilya Shakhat <email address hidden>
Date: Fri Aug 14 13:49:54 2015

Correct private key reading error under pyOpenSSL 0.13

Code that reads private key relies on pyOpenSSL 0.14 or later. Older
versions of library throw a different type of exception upon decryption
failure. Handle that exception and re-raise a good one.

Change-Id: Ie20e202bfc129d26c5885f6084dbc21bb2373cf2
Closes-Bug: #1470856
Closes-Bug: #1471151

[Anna Babich (ababich)]

The latest builds after merged fix for the job were checked, they didn't fail due to reported cause, and tests passed.

Have some builds failed by another cause which is not related to the verified one:
03:11:36 Slave went offline during the build
03:11:36 ERROR: Disconnected by anonymous : Offline due to Gearman request
03:11:36 Build step 'Execute shell' marked build as failure
03:11:36 [SCP] Connecting to static.mosi.mirantis.net
03:11:37 [SCP] No workspace found, files cannot be copied. Probably an error communicating with slave.
03:11:37 [SCP] No workspace found, files cannot be copied. Probably an error communicating with slave.
03:11:37 [SCP] No workspace found, files cannot be copied. Probably an error communicating with slave.
03:11:37 [SCP] No workspace found, files cannot be copied. Probably an error communicating with slave.
03:11:37 [SCP] No workspace found, files cannot be copied. Probably an error communicating with slave.
03:11:37 [PostBuildScript] - Execution post build scripts.
03:11:37 ERROR: Build step failed with exception
03:11:37 java.lang.NullPointerException: no workspace from node hudson.slaves.DumbSlave[bare-trusty-mos-local-586510] which is computer hudson.slaves.SlaveComputer@4c5d3343 and has channel null