barbican_tempest_plugin.tests.scenario.test_volume_encryption.VolumeEncryptionTest fails on Invalid Volume

Bug #1744151 reported by wes hayutin on 2018-01-18
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
High
Arx Cruz

Bug Description

https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset017-master/3c1453c/undercloud/home/jenkins/tempest/tempest.html.gz

traceback-1: {{{
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/tempest/lib/common/utils/test_utils.py", line 84, in call_and_ignore_notfound_exc
    return func(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/tempest/lib/services/volume/v2/volumes_client.py", line 103, in delete_volume
    resp, body = self.delete(url)
  File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 310, in delete
    return self.request('DELETE', url, extra_headers, headers, body)
  File "/usr/lib/python2.7/site-packages/tempest/lib/services/volume/base_client.py", line 38, in request
    method, url, extra_headers, headers, body, chunked)
  File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 668, in request
    self._error_checker(resp, resp_body)
  File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 779, in _error_checker
    raise exceptions.BadRequest(resp_body, resp=resp)
tempest.lib.exceptions.BadRequest: Bad request
Details: {u'message': u'Invalid volume: Volume status must be available or error or error_restoring or error_extending or error_managing and must not be migrating, attached, belong to a group, have snapshots or be disassociated from snapshots after volume transfer.', u'code': 400}
}}}

traceback-2: {{{
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 880, in wait_for_resource_deletion
    raise exceptions.TimeoutException(message)
tempest.lib.exceptions.TimeoutException: Request timed out
Details: (VolumeEncryptionTest:_run_cleanups) Failed to delete volume aabc6f14-27cf-4a6d-8aa6-143c773ac819 within the required time (500 s).
}}}

traceback-3: {{{
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/tempest/lib/services/volume/v2/types_client.py", line 88, in delete_volume_type
    resp, body = self.delete("types/%s" % volume_type_id)
  File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 310, in delete
    return self.request('DELETE', url, extra_headers, headers, body)
  File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 668, in request
    self._error_checker(resp, resp_body)
  File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 779, in _error_checker
    raise exceptions.BadRequest(resp_body, resp=resp)
tempest.lib.exceptions.BadRequest: Bad request
Details: {u'message': u'Target volume type is still in use.', u'code': 400}
}}}

Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/tempest/common/utils/__init__.py", line 89, in wrapper
    return f(*func_args, **func_kwargs)
  File "/usr/lib/python2.7/site-packages/barbican_tempest_plugin/tests/scenario/test_volume_encryption.py", line 115, in test_encrypted_cinder_volumes_cryptsetup
    self.attach_detach_volume(server, volume, keypair)
  File "/usr/lib/python2.7/site-packages/barbican_tempest_plugin/tests/scenario/test_volume_encryption.py", line 66, in attach_detach_volume
    private_key=keypair['private_key']
  File "/usr/lib/python2.7/site-packages/barbican_tempest_plugin/tests/scenario/manager.py", line 348, in create_timestamp
    private_key=private_key)
  File "/usr/lib/python2.7/site-packages/barbican_tempest_plugin/tests/scenario/manager.py", line 438, in get_remote_client
    self._log_console_output()
AttributeError: 'VolumeEncryptionTest' object has no attribute '_log_console_output'

https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset017-master/3c1453c/subnode-2/var/log/extra/errors.txt.gz#_2018-01-18_03_46_40_141

2018-01-18 03:23:25.698 ERROR /var/log/extra/docker/containers/ceilometer_agent_central/log/ceilometer/central.log: 25 ERROR ceilometer.nova_client
2018-01-18 03:46:40.141 ERROR /var/log/extra/docker/containers/openstack-cinder-volume-docker-0/log/cinder/cinder-api.log: 19 ERROR cinder.db.sqlalchemy.api [req-45389489-aded-4737-8522-c48c3967e2b1 fd934e62b5d848d68c03c50543cb533c 72b4834462084c40aaef5c1184bf5574 - default default] VolumeType 1ac71189-400d-4003-8ff7-4db0aab4cf50 deletion failed, VolumeType in use.
2018-01-18 03:46:40.141 ERROR /var/log/containers/cinder/cinder-api.log: 19 ERROR cinder.db.sqlalchemy.api [req-45389489-aded-4737-8522-c48c3967e2b1 fd934e62b5d848d68c03c50543cb533c 72b4834462084c40aaef5c1184bf5574 - default default] VolumeType 1ac71189-400d-4003-8ff7-4db0aab4cf50 deletion failed, VolumeType in use.

Reviewed: https://review.openstack.org/536464
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart/commit/?id=1be05ec176e1f12ed5f12172680429b5f7932bf2
Submitter: Zuul
Branch: master

commit 1be05ec176e1f12ed5f12172680429b5f7932bf2
Author: Arx Cruz <email address hidden>
Date: Mon Jan 22 18:15:52 2018 +0100

    Reducing tempest_workers to 1 for fs017

    Usually, the concurrency is set to number of cpus / 2, however we are
    seeing parallelism issues with this particular featureset when tests are
    running in in parallel. So, let's test running the jobs without
    parallelism for now.

    Change-Id: Ic232f2395971bdef9d0ac48e0d7f9007919eac5c
    Related-Bug: #1744151

Arx Cruz (arxcruz) on 2018-01-23
Changed in tripleo:
assignee: nobody → Arx Cruz (arxcruz)
status: Triaged → Fix Released
wes hayutin (weshayutin) wrote :
Download full text (3.8 KiB)

New error in this test..

https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset017-master/b7194c8/tempest.html.gz

ft1.1: barbican_tempest_plugin.tests.scenario.test_volume_encryption.VolumeEncryptionTest.test_encrypted_cinder_volumes_cryptsetup[compute,id-cbc752ed-b716-4727-910f-956ccf965723,image,volume]_StringException: pythonlogging:'': {{{
2018-01-23 13:55:14,549 11004 DEBUG [barbican_tempest_plugin.tests.scenario.barbican_manager] Uploading certificate to barbican
2018-01-23 13:55:16,139 11004 INFO [tempest.lib.common.rest_client] Request (VolumeEncryptionTest:setUp): 500 POST http://192.168.24.14:9311/v1/secrets 1.589s
2018-01-23 13:55:16,140 11004 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
        Body: {"secret_type": "certificate", "name": "tempest-tempest-sec-1920057739", "algorithm": "rsa", "payload_content_type": "application/octet-stream", "payload_content_encoding": "base64", "expiration": "2018-01-28T13:55:14.549460", "payload": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUNTVENDQWJLZ0F3SUJBZ0lVVnJHRVkzMjQxMWlLUjlCakcvUEh6K3F2eUIwd0RRWUpLb1pJaHZjTkFRRUwKQlFBd1lqRUxNQWtHQTFVRUJoTUNWVk14Q3pBSkJnTlZCQWdNQWtOQk1SWXdGQVlEVlFRSERBMVRZVzRnUm5KaApibU5wYzJOdk1STXdFUVlEVlFRS0RBcE5lU0JEYjIxd1lXNTVNUmt3RndZRFZRUUREQkJVWlhOMElFTmxjblJwClptbGpZWFJsTUI0WERURTRNREV5TXpFek5UVXhORm9YRFRFNE1ESXdNakV6TlRVeE5Gb3dZakVMTUFrR0ExVUUKQmhNQ1ZWTXhDekFKQmdOVkJBZ01Ba05CTVJZd0ZBWURWUVFIREExVFlXNGdSbkpoYm1OcGMyTnZNUk13RVFZRApWUVFLREFwTmVTQkRiMjF3WVc1NU1Sa3dGd1lEVlFRRERCQlVaWE4wSUVObGNuUnBabWxqWVhSbE1JR2RNQTBHCkNTcUdTSWIzRFFFQkFRVUFBNEdMQURDQmh3S0JnUUMzWU1ZMFdQL0lNTUYxUHRtbzF0c2dFYkJFb0lSNitHdjQKTXBvR21WOThwOHBmdStIVmxLazlwTWFlYkhjOFRtanBMTnhKeVdoL252N3Iybkw0ZGxFOVNQUHVsRzZnNzN0TgpQTVZWeDFPTWlZcmZYNktFMjJPNVJKbGVSNTF1OWprNkhhRm9HQzg0TnF5TDNhSGtzMVpKM2tjaFRGZHlaMDRnCk01cDJGV1ZsTndJQkF6QU5CZ2txaGtpRzl3MEJBUXNGQUFPQmdRQk5jOG5wWnhuY1RtREVPeWNCeHp3WHJ1TEYKaTUxcFpaNkJ2cVNpb0w3VitLTXJ2MVZnYmM0cURIWm1kWXBtRnArYzhMQTF1UXgwaXkreXB2eThmTFhnVTBUQgp5Rm1zWUlhMWJBWlBYRU1sbW1wYU8vZWVBQ0ZTWG1ONmViaFRYRTlUaWFDUGxVQnhLUDhlVDJVK1lKcWpmTFJCCmtZTWljQU5QSTVHZVRJZVQwUT09Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K"}
    Response - Headers: {'status': '500', u'content-length': '131', 'content-location': 'http://192.168.24.14:9311/v1/secrets', u'server': 'Apache', u'connection': 'close', u'date': 'Tue, 23 Jan 2018 13:55:14 GMT', u'content-type': 'application/json', u'x-openstack-request-id': 'req-6416adbe-4eab-4954-98ab-7b3ec6bce7fb'}
        Body: {"code": 500, "description": "Secret creation failure seen - please contact site administrator.", "title": "Internal Server Error"}
}}}

Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/barbican_tempest_plugin/tests/scenario/barbican_manager.py", line 55, in setUp
    self.signing_certificate
  File "/usr/lib/python2.7/site-packages/barbican_tempest_plugin/tests/scenario/barbican_manager.py", line 127, in _store_cert
    payload=cert_b64
  File "/usr/lib/python2.7/site-packages/barbican_tempest_plugin/services/key_manager/json/secret_client.py", line 37, in...

Read more...

Changed in tripleo:
status: Fix Released → In Progress
Arx Cruz (arxcruz) wrote :

This error seems to be a real issue! I would contact some barbican guys.

Alan Pevec (apevec) wrote :
Download full text (4.2 KiB)

The stack trace is in
https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset017-master/b7194c8/subnode-2/var/log/containers/httpd/barbican-api/barbican_wsgi_main_error.log.txt.gz

[Tue Jan 23 13:55:16.998828 2018] [:error] [pid 22] 2018-01-23 13:55:16.995 22 ERROR barbican.api.controllers [req-21ab4f6f-cc46-43e9-b8bb-c8be5695d356 956af803575c49d8a8616287474c66a9 - - default default] Secret creation failure seen - please contact site administrator.: AttributeError: 'NoneType' object has no attribute 'to_dict_fields'
[Tue Jan 23 13:55:16.998853 2018] [:error] [pid 22] 2018-01-23 13:55:16.995 22 ERROR barbican.api.controllers Traceback (most recent call last):
[Tue Jan 23 13:55:16.998865 2018] [:error] [pid 22] 2018-01-23 13:55:16.995 22 ERROR barbican.api.controllers File "/usr/lib/python2.7/site-packages/barbican/api/controllers/__init__.py", line 103, in handler
[Tue Jan 23 13:55:16.998877 2018] [:error] [pid 22] 2018-01-23 13:55:16.995 22 ERROR barbican.api.controllers return fn(inst, *args, **kwargs)
[Tue Jan 23 13:55:16.998894 2018] [:error] [pid 22] 2018-01-23 13:55:16.995 22 ERROR barbican.api.controllers File "/usr/lib/python2.7/site-packages/barbican/api/controllers/__init__.py", line 89, in enforcer
[Tue Jan 23 13:55:16.998906 2018] [:error] [pid 22] 2018-01-23 13:55:16.995 22 ERROR barbican.api.controllers return fn(inst, *args, **kwargs)
[Tue Jan 23 13:55:16.998917 2018] [:error] [pid 22] 2018-01-23 13:55:16.995 22 ERROR barbican.api.controllers File "/usr/lib/python2.7/site-packages/barbican/api/controllers/__init__.py", line 151, in content_types_enforcer
[Tue Jan 23 13:55:16.998929 2018] [:error] [pid 22] 2018-01-23 13:55:16.995 22 ERROR barbican.api.controllers return fn(inst, *args, **kwargs)
[Tue Jan 23 13:55:16.998940 2018] [:error] [pid 22] 2018-01-23 13:55:16.995 22 ERROR barbican.api.controllers File "/usr/lib/python2.7/site-packages/barbican/api/controllers/secrets.py", line 441, in on_post
[Tue Jan 23 13:55:16.998952 2018] [:error] [pid 22] 2018-01-23 13:55:16.995 22 ERROR barbican.api.controllers transport_key_id=data.get('transport_key_id'))
[Tue Jan 23 13:55:16.998963 2018] [:error] [pid 22] 2018-01-23 13:55:16.995 22 ERROR barbican.api.controllers File "/usr/lib/python2.7/site-packages/barbican/plugin/resources.py", line 100, in store_secret
[Tue Jan 23 13:55:16.998974 2018] [:error] [pid 22] 2018-01-23 13:55:16.995 22 ERROR barbican.api.controllers project_id=project_model.id)
[Tue Jan 23 13:55:16.998985 2018] [:error] [pid 22] 2018-01-23 13:55:16.995 22 ERROR barbican.api.controllers File "/usr/lib/python2.7/site-packages/barbican/plugin/interface/secret_store.py", line 539, in _check_plugins_configured
[Tue Jan 23 13:55:16.998997 2018] [:error] [pid 22] 2018-01-23 13:55:16.995 22 ERROR barbican.api.controllers return plugin_related_function(self, *args, **kwargs)
[Tue Jan 23 13:55:16.999008 2018] [:error] [pid 22] 2018-01-23 13:55:16.995 22 ERROR barbican.api.controllers File "/usr/lib/python2.7/site-packages/barbican/plugin/interface/secret_store.py", line 573, in get_plugin_store
[Tue Jan 23 13:55:16.999020 2018] [:error] [pid...

Read more...

Reviewed: https://review.openstack.org/536399
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/commit/?id=8eedcb4bf23899f3df8acdbf3eae022b5710d11a
Submitter: Zuul
Branch: master

commit 8eedcb4bf23899f3df8acdbf3eae022b5710d11a
Author: John Trowbridge <email address hidden>
Date: Mon Jan 22 10:49:51 2018 -0500

    Add VolumeEncryptionTest to skip list

    This test is failing when trying to clean up, because the volume
    is still in use. There is a bug open to address the actual issue,
    but we should skip the test in the meantime to not block promotion.

    Change-Id: I2f58b28552e20352ee69efbc9cbac52d5da7c4f4
    Related-Bug: 1744151

Rafael Folco (rafaelfolco) wrote :
tags: added: ci tempest
removed: promotion-blocker
Changed in tripleo:
milestone: queens-3 → queens-rc1
Alex Schultz (alex-schultz) wrote :

Since this is no longer blocking things, dropping the priority and adding tech-debt as a reminder that we added it to the skip list

Changed in tripleo:
importance: Critical → High
milestone: queens-rc1 → rocky-1
tags: added: tech-debt

Reviewed: https://review.openstack.org/549852
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=1b767be691ff315c84d3595454a5f904e71e6115
Submitter: Zuul
Branch: stable/queens

commit 1b767be691ff315c84d3595454a5f904e71e6115
Author: Ade Lee <email address hidden>
Date: Thu Jan 25 00:40:02 2018 -0500

    Add step to run secret_store_sync before instance startup

    Closes-Bug: 1744151
    Change-Id: Ibe4e5023b8773ee70ee914118628acc5c149f379
    (cherry picked from commit 86793eea52a91a8a2f59eeaa7b05cbd7e00e4659)

tags: added: in-stable-queens
Arx Cruz (arxcruz) on 2018-03-12
Changed in tripleo:
status: In Progress → Fix Released

This issue was fixed in the openstack/tripleo-heat-templates 8.0.0 release.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers