tripleo

barbican_tempest_plugin.tests.scenario.test_volume_encryption.VolumeEncryptionTest fails on Invalid Volume

Bug #1744151 reported by wes hayutin
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
High
Arx Cruz
tripleo rocky-1

Bug Description

https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset017-master/3c1453c/undercloud/home/jenkins/tempest/tempest.html.gz

traceback-1: {{{
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/tempest/lib/common/utils/test_utils.py", line 84, in call_and_ignore_notfound_exc
    return func(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/tempest/lib/services/volume/v2/volumes_client.py", line 103, in delete_volume
    resp, body = self.delete(url)
  File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 310, in delete
    return self.request('DELETE', url, extra_headers, headers, body)
  File "/usr/lib/python2.7/site-packages/tempest/lib/services/volume/base_client.py", line 38, in request
    method, url, extra_headers, headers, body, chunked)
  File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 668, in request
    self._error_checker(resp, resp_body)
  File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 779, in _error_checker
    raise exceptions.BadRequest(resp_body, resp=resp)
tempest.lib.exceptions.BadRequest: Bad request
Details: {u'message': u'Invalid volume: Volume status must be available or error or error_restoring or error_extending or error_managing and must not be migrating, attached, belong to a group, have snapshots or be disassociated from snapshots after volume transfer.', u'code': 400}
}}}

traceback-2: {{{
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 880, in wait_for_resource_deletion
    raise exceptions.TimeoutException(message)
tempest.lib.exceptions.TimeoutException: Request timed out
Details: (VolumeEncryptionTest:_run_cleanups) Failed to delete volume aabc6f14-27cf-4a6d-8aa6-143c773ac819 within the required time (500 s).
}}}

traceback-3: {{{
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/tempest/lib/services/volume/v2/types_client.py", line 88, in delete_volume_type
    resp, body = self.delete("types/%s" % volume_type_id)
  File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 310, in delete
    return self.request('DELETE', url, extra_headers, headers, body)
  File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 668, in request
    self._error_checker(resp, resp_body)
  File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 779, in _error_checker
    raise exceptions.BadRequest(resp_body, resp=resp)
tempest.lib.exceptions.BadRequest: Bad request
Details: {u'message': u'Target volume type is still in use.', u'code': 400}
}}}

Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/tempest/common/utils/__init__.py", line 89, in wrapper
    return f(*func_args, **func_kwargs)
  File "/usr/lib/python2.7/site-packages/barbican_tempest_plugin/tests/scenario/test_volume_encryption.py", line 115, in test_encrypted_cinder_volumes_cryptsetup
    self.attach_detach_volume(server, volume, keypair)
  File "/usr/lib/python2.7/site-packages/barbican_tempest_plugin/tests/scenario/test_volume_encryption.py", line 66, in attach_detach_volume
    private_key=keypair['private_key']
  File "/usr/lib/python2.7/site-packages/barbican_tempest_plugin/tests/scenario/manager.py", line 348, in create_timestamp
    private_key=private_key)
  File "/usr/lib/python2.7/site-packages/barbican_tempest_plugin/tests/scenario/manager.py", line 438, in get_remote_client
    self._log_console_output()
AttributeError: 'VolumeEncryptionTest' object has no attribute '_log_console_output'

https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset017-master/3c1453c/subnode-2/var/log/extra/errors.txt.gz#_2018-01-18_03_46_40_141

2018-01-18 03:23:25.698 ERROR /var/log/extra/docker/containers/ceilometer_agent_central/log/ceilometer/central.log: 25 ERROR ceilometer.nova_client
2018-01-18 03:46:40.141 ERROR /var/log/extra/docker/containers/openstack-cinder-volume-docker-0/log/cinder/cinder-api.log: 19 ERROR cinder.db.sqlalchemy.api [req-45389489-aded-4737-8522-c48c3967e2b1 fd934e62b5d848d68c03c50543cb533c 72b4834462084c40aaef5c1184bf5574 - default default] VolumeType 1ac71189-400d-4003-8ff7-4db0aab4cf50 deletion failed, VolumeType in use.
2018-01-18 03:46:40.141 ERROR /var/log/containers/cinder/cinder-api.log: 19 ERROR cinder.db.sqlalchemy.api [req-45389489-aded-4737-8522-c48c3967e2b1 fd934e62b5d848d68c03c50543cb533c 72b4834462084c40aaef5c1184bf5574 - default default] VolumeType 1ac71189-400d-4003-8ff7-4db0aab4cf50 deletion failed, VolumeType in use.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-quickstart-extras (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/536399

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-quickstart (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/536464

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-quickstart (master)

Reviewed: https://review.openstack.org/536464
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart/commit/?id=1be05ec176e1f12ed5f12172680429b5f7932bf2
Submitter: Zuul
Branch: master

commit 1be05ec176e1f12ed5f12172680429b5f7932bf2
Author: Arx Cruz <email address hidden>
Date: Mon Jan 22 18:15:52 2018 +0100

    Reducing tempest_workers to 1 for fs017

    Usually, the concurrency is set to number of cpus / 2, however we are
    seeing parallelism issues with this particular featureset when tests are
    running in in parallel. So, let's test running the jobs without
    parallelism for now.

    Change-Id: Ic232f2395971bdef9d0ac48e0d7f9007919eac5c
    Related-Bug: #1744151

Arx Cruz (arxcruz)
Changed in tripleo:
assignee: nobody → Arx Cruz (arxcruz)
status: Triaged → Fix Released
Revision history for this message
wes hayutin (weshayutin) wrote :
Download full text (3.8 KiB)

New error in this test..

https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset017-master/b7194c8/tempest.html.gz

ft1.1: barbican_tempest_plugin.tests.scenario.test_volume_encryption.VolumeEncryptionTest.test_encrypted_cinder_volumes_cryptsetup[compute,id-cbc752ed-b716-4727-910f-956ccf965723,image,volume]_StringException: pythonlogging:'': {{{
2018-01-23 13:55:14,549 11004 DEBUG [barbican_tempest_plugin.tests.scenario.barbican_manager] Uploading certificate to barbican
2018-01-23 13:55:16,139 11004 INFO [tempest.lib.common.rest_client] Request (VolumeEncryptionTest:setUp): 500 POST http://192.168.24.14:9311/v1/secrets 1.589s
2018-01-23 13:55:16,140 11004 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
        Body: {"secret_type": "certificate", "name": "tempest-tempest-sec-1920057739", "algorithm": "rsa", "payload_content_type": "application/octet-stream", "payload_content_encoding": "base64", "expiration": "2018-01-28T13:55:14.549460", "payload": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUNTVENDQWJLZ0F3SUJBZ0lVVnJHRVkzMjQxMWlLUjlCakcvUEh6K3F2eUIwd0RRWUpLb1pJaHZjTkFRRUwKQlFBd1lqRUxNQWtHQTFVRUJoTUNWVk14Q3pBSkJnTlZCQWdNQWtOQk1SWXdGQVlEVlFRSERBMVRZVzRnUm5KaApibU5wYzJOdk1STXdFUVlEVlFRS0RBcE5lU0JEYjIxd1lXNTVNUmt3RndZRFZRUUREQkJVWlhOMElFTmxjblJwClptbGpZWFJsTUI0WERURTRNREV5TXpFek5UVXhORm9YRFRFNE1ESXdNakV6TlRVeE5Gb3dZakVMTUFrR0ExVUUKQmhNQ1ZWTXhDekFKQmdOVkJBZ01Ba05CTVJZd0ZBWURWUVFIREExVFlXNGdSbkpoYm1OcGMyTnZNUk13RVFZRApWUVFLREFwTmVTQkRiMjF3WVc1NU1Sa3dGd1lEVlFRRERCQlVaWE4wSUVObGNuUnBabWxqWVhSbE1JR2RNQTBHCkNTcUdTSWIzRFFFQkFRVUFBNEdMQURDQmh3S0JnUUMzWU1ZMFdQL0lNTUYxUHRtbzF0c2dFYkJFb0lSNitHdjQKTXBvR21WOThwOHBmdStIVmxLazlwTWFlYkhjOFRtanBMTnhKeVdoL252N3Iybkw0ZGxFOVNQUHVsRzZnNzN0TgpQTVZWeDFPTWlZcmZYNktFMjJPNVJKbGVSNTF1OWprNkhhRm9HQzg0TnF5TDNhSGtzMVpKM2tjaFRGZHlaMDRnCk01cDJGV1ZsTndJQkF6QU5CZ2txaGtpRzl3MEJBUXNGQUFPQmdRQk5jOG5wWnhuY1RtREVPeWNCeHp3WHJ1TEYKaTUxcFpaNkJ2cVNpb0w3VitLTXJ2MVZnYmM0cURIWm1kWXBtRnArYzhMQTF1UXgwaXkreXB2eThmTFhnVTBUQgp5Rm1zWUlhMWJBWlBYRU1sbW1wYU8vZWVBQ0ZTWG1ONmViaFRYRTlUaWFDUGxVQnhLUDhlVDJVK1lKcWpmTFJCCmtZTWljQU5QSTVHZVRJZVQwUT09Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K"}
    Response - Headers: {'status': '500', u'content-length': '131', 'content-location': 'http://192.168.24.14:9311/v1/secrets', u'server': 'Apache', u'connection': 'close', u'date': 'Tue, 23 Jan 2018 13:55:14 GMT', u'content-type': 'application/json', u'x-openstack-request-id': 'req-6416adbe-4eab-4954-98ab-7b3ec6bce7fb'}
        Body: {"code": 500, "description": "Secret creation failure seen - please contact site administrator.", "title": "Internal Server Error"}
}}}

Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/barbican_tempest_plugin/tests/scenario/barbican_manager.py", line 55, in setUp
    self.signing_certificate
  File "/usr/lib/python2.7/site-packages/barbican_tempest_plugin/tests/scenario/barbican_manager.py", line 127, in _store_cert
    payload=cert_b64
  File "/usr/lib/python2.7/site-packages/barbican_tempest_plugin/services/key_manager/json/secret_client.py", line 37, in...

Read more...

Changed in tripleo:
status: Fix Released → In Progress
Revision history for this message
Arx Cruz (arxcruz) wrote :

This error seems to be a real issue! I would contact some barbican guys.

Revision history for this message
Alan Pevec (apevec) wrote :
Download full text (4.2 KiB)

The stack trace is in
https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset017-master/b7194c8/subnode-2/var/log/containers/httpd/barbican-api/barbican_wsgi_main_error.log.txt.gz

[Tue Jan 23 13:55:16.998828 2018] [:error] [pid 22] 2018-01-23 13:55:16.995 22 ERROR barbican.api.controllers [req-21ab4f6f-cc46-43e9-b8bb-c8be5695d356 956af803575c49d8a8616287474c66a9 - - default default] Secret creation failure seen - please contact site administrator.: AttributeError: 'NoneType' object has no attribute 'to_dict_fields'
[Tue Jan 23 13:55:16.998853 2018] [:error] [pid 22] 2018-01-23 13:55:16.995 22 ERROR barbican.api.controllers Traceback (most recent call last):
[Tue Jan 23 13:55:16.998865 2018] [:error] [pid 22] 2018-01-23 13:55:16.995 22 ERROR barbican.api.controllers File "/usr/lib/python2.7/site-packages/barbican/api/controllers/__init__.py", line 103, in handler
[Tue Jan 23 13:55:16.998877 2018] [:error] [pid 22] 2018-01-23 13:55:16.995 22 ERROR barbican.api.controllers return fn(inst, *args, **kwargs)
[Tue Jan 23 13:55:16.998894 2018] [:error] [pid 22] 2018-01-23 13:55:16.995 22 ERROR barbican.api.controllers File "/usr/lib/python2.7/site-packages/barbican/api/controllers/__init__.py", line 89, in enforcer
[Tue Jan 23 13:55:16.998906 2018] [:error] [pid 22] 2018-01-23 13:55:16.995 22 ERROR barbican.api.controllers return fn(inst, *args, **kwargs)
[Tue Jan 23 13:55:16.998917 2018] [:error] [pid 22] 2018-01-23 13:55:16.995 22 ERROR barbican.api.controllers File "/usr/lib/python2.7/site-packages/barbican/api/controllers/__init__.py", line 151, in content_types_enforcer
[Tue Jan 23 13:55:16.998929 2018] [:error] [pid 22] 2018-01-23 13:55:16.995 22 ERROR barbican.api.controllers return fn(inst, *args, **kwargs)
[Tue Jan 23 13:55:16.998940 2018] [:error] [pid 22] 2018-01-23 13:55:16.995 22 ERROR barbican.api.controllers File "/usr/lib/python2.7/site-packages/barbican/api/controllers/secrets.py", line 441, in on_post
[Tue Jan 23 13:55:16.998952 2018] [:error] [pid 22] 2018-01-23 13:55:16.995 22 ERROR barbican.api.controllers transport_key_id=data.get('transport_key_id'))
[Tue Jan 23 13:55:16.998963 2018] [:error] [pid 22] 2018-01-23 13:55:16.995 22 ERROR barbican.api.controllers File "/usr/lib/python2.7/site-packages/barbican/plugin/resources.py", line 100, in store_secret
[Tue Jan 23 13:55:16.998974 2018] [:error] [pid 22] 2018-01-23 13:55:16.995 22 ERROR barbican.api.controllers project_id=project_model.id)
[Tue Jan 23 13:55:16.998985 2018] [:error] [pid 22] 2018-01-23 13:55:16.995 22 ERROR barbican.api.controllers File "/usr/lib/python2.7/site-packages/barbican/plugin/interface/secret_store.py", line 539, in _check_plugins_configured
[Tue Jan 23 13:55:16.998997 2018] [:error] [pid 22] 2018-01-23 13:55:16.995 22 ERROR barbican.api.controllers return plugin_related_function(self, *args, **kwargs)
[Tue Jan 23 13:55:16.999008 2018] [:error] [pid 22] 2018-01-23 13:55:16.995 22 ERROR barbican.api.controllers File "/usr/lib/python2.7/site-packages/barbican/plugin/interface/secret_store.py", line 573, in get_plugin_store
[Tue Jan 23 13:55:16.999020 2018] [:error] [pid...

Read more...

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-quickstart-extras (master)

Reviewed: https://review.openstack.org/536399
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/commit/?id=8eedcb4bf23899f3df8acdbf3eae022b5710d11a
Submitter: Zuul
Branch: master

commit 8eedcb4bf23899f3df8acdbf3eae022b5710d11a
Author: John Trowbridge <email address hidden>
Date: Mon Jan 22 10:49:51 2018 -0500

    Add VolumeEncryptionTest to skip list

    This test is failing when trying to clean up, because the volume
    is still in use. There is a bug open to address the actual issue,
    but we should skip the test in the meantime to not block promotion.

    Change-Id: I2f58b28552e20352ee69efbc9cbac52d5da7c4f4
    Related-Bug: 1744151

Revision history for this message
Rafael Folco (rafaelfolco) wrote :

Verified 2 last runs passed with the VolumeEncryptionTest added to the skip list:

https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset017-master/f136132/tempest.html.gz
https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset017-master/fc07ecb/tempest.html.gz

Not closing this bug as we need to investigate the root cause for the failure.

tags: added: ci tempest
removed: promotion-blocker
Emilien Macchi (emilienm)
Changed in tripleo:
milestone: queens-3 → queens-rc1
Revision history for this message
Alex Schultz (alex-schultz) wrote :

Since this is no longer blocking things, dropping the priority and adding tech-debt as a reminder that we added it to the skip list

Changed in tripleo:
importance: Critical → High
milestone: queens-rc1 → rocky-1
tags: added: tech-debt
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (stable/queens)

Reviewed: https://review.openstack.org/549852
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=1b767be691ff315c84d3595454a5f904e71e6115
Submitter: Zuul
Branch: stable/queens

commit 1b767be691ff315c84d3595454a5f904e71e6115
Author: Ade Lee <email address hidden>
Date: Thu Jan 25 00:40:02 2018 -0500

    Add step to run secret_store_sync before instance startup

    Closes-Bug: 1744151
    Change-Id: Ibe4e5023b8773ee70ee914118628acc5c149f379
    (cherry picked from commit 86793eea52a91a8a2f59eeaa7b05cbd7e00e4659)

tags: added: in-stable-queens
Arx Cruz (arxcruz)
Changed in tripleo:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates 8.0.0

This issue was fixed in the openstack/tripleo-heat-templates 8.0.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.