Tempest Tests fail on test_deploy_app_with_volume_creation_from_backup test

[Alexander Tivelkov]

Tempest tests fail with the following error:

ft24.4: murano_tempest_tests.tests.scenario.application_catalog.test_cinder_volumes.TestCinderVolumes.test_deploy_app_with_volume_creation_from_backup[scenario,smoke]_StringException: Empty attachments:
  stderr
  stdout

pythonlogging:'': {{{
2016-10-19 11:13:26,872 16274 INFO [tempest.lib.common.rest_client] Request (TestCinderVolumes:test_deploy_app_with_volume_creation_from_backup): 202 POST http://127.0.0.1:8776/v1/5f0bfd588f2344a7a784b677dcd81b81/backups 0.150s
2016-10-19 11:13:26,873 16274 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'X-Auth-Token': '<omitted>', 'Accept': 'application/json', 'Content-Type': 'application/json'}
        Body: {"backup": {"volume_id": "3f464e77-0f3c-41ea-9d07-713c95088482", "force": true}}
    Response - Headers: {'connection': 'close', 'x-compute-request-id': 'req-adb96e1d-fc2f-4137-a73c-33a5baa3c42e', 'content-type': 'application/json', 'date': 'Wed, 19 Oct 2016 11:13:26 GMT', 'status': '202', 'content-location': 'http://127.0.0.1:8776/v1/5f0bfd588f2344a7a784b677dcd81b81/backups', 'x-openstack-request-id': 'req-adb96e1d-fc2f-4137-a73c-33a5baa3c42e', 'content-length': '346'}
        Body: {"backup": {"id": "00619e99-ddab-483a-ac60-ab55963fa606", "links": [{"href": "http://127.0.0.1:8776/v1/5f0bfd588f2344a7a784b677dcd81b81/backups/00619e99-ddab-483a-ac60-ab55963fa606", "rel": "self"}, {"href": "http://127.0.0.1:8776/5f0bfd588f2344a7a784b677dcd81b81/backups/00619e99-ddab-483a-ac60-ab55963fa606", "rel": "bookmark"}], "name": null}}
2016-10-19 11:13:26,889 16274 INFO [tempest.lib.common.rest_client] Request (TestCinderVolumes:test_deploy_app_with_volume_creation_from_backup): 200 GET http://127.0.0.1:8776/v1/5f0bfd588f2344a7a784b677dcd81b81/backups/00619e99-ddab-483a-ac60-ab55963fa606 0.016s
2016-10-19 11:13:26,889 16274 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'X-Auth-Token': '<omitted>', 'Accept': 'application/json', 'Content-Type': 'application/json'}
        Body: None
    Response - Headers: {'connection': 'close', 'x-compute-request-id': 'req-94af2c35-6fc4-4f66-96db-b0a66c691559', 'content-type': 'application/json', 'date': 'Wed, 19 Oct 2016 11:13:26 GMT', 'status': '200', 'content-location': 'http://127.0.0.1:8776/v1/5f0bfd588f2344a7a784b677dcd81b81/backups/00619e99-ddab-483a-ac60-ab55963fa606', 'x-openstack-request-id': 'req-94af2c35-6fc4-4f66-96db-b0a66c691559', 'content-length': '753'}
        Body: {"backup": {"status": "creating", "object_count": 0, "container": null, "name": null, "links": [{"href": "http://127.0.0.1:8776/v1/5f0bfd588f2344a7a784b677dcd81b81/backups/00619e99-ddab-483a-ac60-ab55963fa606", "rel": "self"}, {"href": "http://127.0.0.1:8776/5f0bfd588f2344a7a784b677dcd81b81/backups/00619e99-ddab-483a-ac60-ab55963fa606", "rel": "bookmark"}], "availability_zone": null, "created_at": "2016-10-19T11:13:26.000000", "description": null, "updated_at": "2016-10-19T11:13:26.000000", "data_timestamp": "2016-10-19T11:13:26.000000", "has_dependent_backups": false, "snapshot_id": null, "volume_id": "3f464e77-0f3c-41ea-9d07-713c95088482", "fail_reason": null, "is_incremental": false, "id": "00619e99-ddab-483a-ac60-ab55963fa606", "size": 1}}
2016-10-19 11:13:27,911 16274 INFO [tempest.lib.common.rest_client] Request (TestCinderVolumes:test_deploy_app_with_volume_creation_from_backup): 200 GET http://127.0.0.1:8776/v1/5f0bfd588f2344a7a784b677dcd81b81/backups/00619e99-ddab-483a-ac60-ab55963fa606 0.019s
2016-10-19 11:13:27,911 16274 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'X-Auth-Token': '<omitted>', 'Accept': 'application/json', 'Content-Type': 'application/json'}
        Body: None
    Response - Headers: {'connection': 'close', 'x-compute-request-id': 'req-fcd3d6b4-8adb-49d7-bbbd-a27060e1f5e9', 'content-type': 'application/json', 'date': 'Wed, 19 Oct 2016 11:13:27 GMT', 'status': '200', 'content-location': 'http://127.0.0.1:8776/v1/5f0bfd588f2344a7a784b677dcd81b81/backups/00619e99-ddab-483a-ac60-ab55963fa606', 'x-openstack-request-id': 'req-fcd3d6b4-8adb-49d7-bbbd-a27060e1f5e9', 'content-length': '755'}
        Body: {"backup": {"status": "creating", "object_count": 0, "container": null, "name": null, "links": [{"href": "http://127.0.0.1:8776/v1/5f0bfd588f2344a7a784b677dcd81b81/backups/00619e99-ddab-483a-ac60-ab55963fa606", "rel": "self"}, {"href": "http://127.0.0.1:8776/5f0bfd588f2344a7a784b677dcd81b81/backups/00619e99-ddab-483a-ac60-ab55963fa606", "rel": "bookmark"}], "availability_zone": "nova", "created_at": "2016-10-19T11:13:26.000000", "description": null, "updated_at": "2016-10-19T11:13:26.000000", "data_timestamp": "2016-10-19T11:13:26.000000", "has_dependent_backups": false, "snapshot_id": null, "volume_id": "3f464e77-0f3c-41ea-9d07-713c95088482", "fail_reason": null, "is_incremental": false, "id": "00619e99-ddab-483a-ac60-ab55963fa606", "size": 1}}
2016-10-19 11:13:28,925 16274 INFO [tempest.lib.common.rest_client] Request (TestCinderVolumes:test_deploy_app_with_volume_creation_from_backup): 200 GET http://127.0.0.1:8776/v1/5f0bfd588f2344a7a784b677dcd81b81/backups/00619e99-ddab-483a-ac60-ab55963fa606 0.013s
2016-10-19 11:13:28,925 16274 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'X-Auth-Token': '<omitted>', 'Accept': 'application/json', 'Content-Type': 'application/json'}
        Body: None
    Response - Headers: {'connection': 'close', 'x-compute-request-id': 'req-c5b88d1d-a99f-4b20-a851-a437110a4a2b', 'content-type': 'application/json', 'date': 'Wed, 19 Oct 2016 11:13:28 GMT', 'status': '200', 'content-location': 'http://127.0.0.1:8776/v1/5f0bfd588f2344a7a784b677dcd81b81/backups/00619e99-ddab-483a-ac60-ab55963fa606', 'x-openstack-request-id': 'req-c5b88d1d-a99f-4b20-a851-a437110a4a2b', 'content-length': '755'}
        Body: {"backup": {"status": "creating", "object_count": 0, "container": null, "name": null, "links": [{"href": "http://127.0.0.1:8776/v1/5f0bfd588f2344a7a784b677dcd81b81/backups/00619e99-ddab-483a-ac60-ab55963fa606", "rel": "self"}, {"href": "http://127.0.0.1:8776/5f0bfd588f2344a7a784b677dcd81b81/backups/00619e99-ddab-483a-ac60-ab55963fa606", "rel": "bookmark"}], "availability_zone": "nova", "created_at": "2016-10-19T11:13:26.000000", "description": null, "updated_at": "2016-10-19T11:13:26.000000", "data_timestamp": "2016-10-19T11:13:26.000000", "has_dependent_backups": false, "snapshot_id": null, "volume_id": "3f464e77-0f3c-41ea-9d07-713c95088482", "fail_reason": null, "is_incremental": false, "id": "00619e99-ddab-483a-ac60-ab55963fa606", "size": 1}}
2016-10-19 11:13:29,941 16274 INFO [tempest.lib.common.rest_client] Request (TestCinderVolumes:test_deploy_app_with_volume_creation_from_backup): 200 GET http://127.0.0.1:8776/v1/5f0bfd588f2344a7a784b677dcd81b81/backups/00619e99-ddab-483a-ac60-ab55963fa606 0.015s
2016-10-19 11:13:29,942 16274 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'X-Auth-Token': '<omitted>', 'Accept': 'application/json', 'Content-Type': 'application/json'}
        Body: None
    Response - Headers: {'connection': 'close', 'x-compute-request-id': 'req-5ed39980-b1e2-430d-a07d-04e3ecc0e3b6', 'content-type': 'application/json', 'date': 'Wed, 19 Oct 2016 11:13:29 GMT', 'status': '200', 'content-location': 'http://127.0.0.1:8776/v1/5f0bfd588f2344a7a784b677dcd81b81/backups/00619e99-ddab-483a-ac60-ab55963fa606', 'x-openstack-request-id': 'req-5ed39980-b1e2-430d-a07d-04e3ecc0e3b6', 'content-length': '755'}
        Body: {"backup": {"status": "creating", "object_count": 0, "container": null, "name": null, "links": [{"href": "http://127.0.0.1:8776/v1/5f0bfd588f2344a7a784b677dcd81b81/backups/00619e99-ddab-483a-ac60-ab55963fa606", "rel": "self"}, {"href": "http://127.0.0.1:8776/5f0bfd588f2344a7a784b677dcd81b81/backups/00619e99-ddab-483a-ac60-ab55963fa606", "rel": "bookmark"}], "availability_zone": "nova", "created_at": "2016-10-19T11:13:26.000000", "description": null, "updated_at": "2016-10-19T11:13:26.000000", "data_timestamp": "2016-10-19T11:13:26.000000", "has_dependent_backups": false, "snapshot_id": null, "volume_id": "3f464e77-0f3c-41ea-9d07-713c95088482", "fail_reason": null, "is_incremental": false, "id": "00619e99-ddab-483a-ac60-ab55963fa606", "size": 1}}
2016-10-19 11:13:30,955 16274 INFO [tempest.lib.common.rest_client] Request (TestCinderVolumes:test_deploy_app_with_volume_creation_from_backup): 200 GET http://127.0.0.1:8776/v1/5f0bfd588f2344a7a784b677dcd81b81/backups/00619e99-ddab-483a-ac60-ab55963fa606 0.012s
2016-10-19 11:13:30,955 16274 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'X-Auth-Token': '<omitted>', 'Accept': 'application/json', 'Content-Type': 'application/json'}
        Body: None
    Response - Headers: {'connection': 'close', 'x-compute-request-id': 'req-959148ea-90f8-4695-b3b5-4b71951464b5', 'content-type': 'application/json', 'date': 'Wed, 19 Oct 2016 11:13:30 GMT', 'status': '200', 'content-location': 'http://127.0.0.1:8776/v1/5f0bfd588f2344a7a784b677dcd81b81/backups/00619e99-ddab-483a-ac60-ab55963fa606', 'x-openstack-request-id': 'req-959148ea-90f8-4695-b3b5-4b71951464b5', 'content-length': '755'}
        Body: {"backup": {"status": "creating", "object_count": 0, "container": null, "name": null, "links": [{"href": "http://127.0.0.1:8776/v1/5f0bfd588f2344a7a784b677dcd81b81/backups/00619e99-ddab-483a-ac60-ab55963fa606", "rel": "self"}, {"href": "http://127.0.0.1:8776/5f0bfd588f2344a7a784b677dcd81b81/backups/00619e99-ddab-483a-ac60-ab55963fa606", "rel": "bookmark"}], "availability_zone": "nova", "created_at": "2016-10-19T11:13:26.000000", "description": null, "updated_at": "2016-10-19T11:13:26.000000", "data_timestamp": "2016-10-19T11:13:26.000000", "has_dependent_backups": false, "snapshot_id": null, "volume_id": "3f464e77-0f3c-41ea-9d07-713c95088482", "fail_reason": null, "is_incremental": false, "id": "00619e99-ddab-483a-ac60-ab55963fa606", "size": 1}}
2016-10-19 11:13:31,971 16274 INFO [tempest.lib.common.rest_client] Request (TestCinderVolumes:test_deploy_app_with_volume_creation_from_backup): 200 GET http://127.0.0.1:8776/v1/5f0bfd588f2344a7a784b677dcd81b81/backups/00619e99-ddab-483a-ac60-ab55963fa606 0.014s
2016-10-19 11:13:31,971 16274 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'X-Auth-Token': '<omitted>', 'Accept': 'application/json', 'Content-Type': 'application/json'}
        Body: None
    Response - Headers: {'connection': 'close', 'x-compute-request-id': 'req-64a3da4c-f94d-4973-91a7-47c3386092aa', 'content-type': 'application/json', 'date': 'Wed, 19 Oct 2016 11:13:31 GMT', 'status': '200', 'content-location': 'http://127.0.0.1:8776/v1/5f0bfd588f2344a7a784b677dcd81b81/backups/00619e99-ddab-483a-ac60-ab55963fa606', 'x-openstack-request-id': 'req-64a3da4c-f94d-4973-91a7-47c3386092aa', 'content-length': '755'}
        Body: {"backup": {"status": "creating", "object_count": 0, "container": null, "name": null, "links": [{"href": "http://127.0.0.1:8776/v1/5f0bfd588f2344a7a784b677dcd81b81/backups/00619e99-ddab-483a-ac60-ab55963fa606", "rel": "self"}, {"href": "http://127.0.0.1:8776/5f0bfd588f2344a7a784b677dcd81b81/backups/00619e99-ddab-483a-ac60-ab55963fa606", "rel": "bookmark"}], "availability_zone": "nova", "created_at": "2016-10-19T11:13:26.000000", "description": null, "updated_at": "2016-10-19T11:13:26.000000", "data_timestamp": "2016-10-19T11:13:26.000000", "has_dependent_backups": false, "snapshot_id": null, "volume_id": "3f464e77-0f3c-41ea-9d07-713c95088482", "fail_reason": null, "is_incremental": false, "id": "00619e99-ddab-483a-ac60-ab55963fa606", "size": 1}}
2016-10-19 11:13:32,988 16274 INFO [tempest.lib.common.rest_client] Request (TestCinderVolumes:test_deploy_app_with_volume_creation_from_backup): 200 GET http://127.0.0.1:8776/v1/5f0bfd588f2344a7a784b677dcd81b81/backups/00619e99-ddab-483a-ac60-ab55963fa606 0.015s
2016-10-19 11:13:32,988 16274 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'X-Auth-Token': '<omitted>', 'Accept': 'application/json', 'Content-Type': 'application/json'}
        Body: None
    Response - Headers: {'connection': 'close', 'x-compute-request-id': 'req-fc931782-cb5f-4baf-babc-a347b59824fd', 'content-type': 'application/json', 'date': 'Wed, 19 Oct 2016 11:13:32 GMT', 'status': '200', 'content-location': 'http://127.0.0.1:8776/v1/5f0bfd588f2344a7a784b677dcd81b81/backups/00619e99-ddab-483a-ac60-ab55963fa606', 'x-openstack-request-id': 'req-fc931782-cb5f-4baf-babc-a347b59824fd', 'content-length': '766'}
        Body: {"backup": {"status": "creating", "object_count": 0, "container": "volumebackups", "name": null, "links": [{"href": "http://127.0.0.1:8776/v1/5f0bfd588f2344a7a784b677dcd81b81/backups/00619e99-ddab-483a-ac60-ab55963fa606", "rel": "self"}, {"href": "http://127.0.0.1:8776/5f0bfd588f2344a7a784b677dcd81b81/backups/00619e99-ddab-483a-ac60-ab55963fa606", "rel": "bookmark"}], "availability_zone": "nova", "created_at": "2016-10-19T11:13:26.000000", "description": null, "updated_at": "2016-10-19T11:13:32.000000", "data_timestamp": "2016-10-19T11:13:26.000000", "has_dependent_backups": false, "snapshot_id": null, "volume_id": "3f464e77-0f3c-41ea-9d07-713c95088482", "fail_reason": null, "is_incremental": false, "id": "00619e99-ddab-483a-ac60-ab55963fa606", "size": 1}}
2016-10-19 11:13:34,002 16274 INFO [tempest.lib.common.rest_client] Request (TestCinderVolumes:test_deploy_app_with_volume_creation_from_backup): 200 GET http://127.0.0.1:8776/v1/5f0bfd588f2344a7a784b677dcd81b81/backups/00619e99-ddab-483a-ac60-ab55963fa606 0.013s
2016-10-19 11:13:34,003 16274 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'X-Auth-Token': '<omitted>', 'Accept': 'application/json', 'Content-Type': 'application/json'}
        Body: None
    Response - Headers: {'connection': 'close', 'x-compute-request-id': 'req-5df3c44a-55fe-45da-bda1-00578e041f35', 'content-type': 'application/json', 'date': 'Wed, 19 Oct 2016 11:13:34 GMT', 'status': '200', 'content-location': 'http://127.0.0.1:8776/v1/5f0bfd588f2344a7a784b677dcd81b81/backups/00619e99-ddab-483a-ac60-ab55963fa606', 'x-openstack-request-id': 'req-5df3c44a-55fe-45da-bda1-00578e041f35', 'content-length': '964'}
        Body: {"backup": {"status": "error", "object_count": 0, "container": "volumebackups", "name": null, "links": [{"href": "http://127.0.0.1:8776/v1/5f0bfd588f2344a7a784b677dcd81b81/backups/00619e99-ddab-483a-ac60-ab55963fa606", "rel": "self"}, {"href": "http://127.0.0.1:8776/5f0bfd588f2344a7a784b677dcd81b81/backups/00619e99-ddab-483a-ac60-ab55963fa606", "rel": "bookmark"}], "availability_zone": "nova", "created_at": "2016-10-19T11:13:26.000000", "description": null, "updated_at": "2016-10-19T11:13:33.000000", "data_timestamp": "2016-10-19T11:13:26.000000", "has_dependent_backups": false, "snapshot_id": null, "volume_id": "3f464e77-0f3c-41ea-9d07-713c95088482", "fail_reason": "Container PUT failed: http://127.0.0.1:8080/v1/AUTH_5f0bfd588f2344a7a784b677dcd81b81/volumebackups 403 Forbidden [first 60 chars of response] <html><h1>Forbidden</h1><p>Access was denied to this resourc", "is_incremental": false, "id": "00619e99-ddab-483a-ac60-ab55963fa606", "size": 1}}
}}}

Traceback (most recent call last):
  File "/opt/stack/new/murano/murano_tempest_tests/tests/scenario/application_catalog/test_cinder_volumes.py", line 369, in test_deploy_app_with_volume_creation_from_backup
    backup = self.create_backup(self.volume)
  File "/opt/stack/new/murano/murano_tempest_tests/tests/scenario/application_catalog/base.py", line 331, in create_backup
    backup['id'], 'available')
  File "tempest/common/waiters.py", line 224, in wait_for_backup_status
    raise lib_exc.VolumeBackupException(backup_id=backup_id)
tempest.lib.exceptions.VolumeBackupException: Volume backup 00619e99-ddab-483a-ac60-ab55963fa606 failed and is in ERROR status

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/388677

[OpenStack Infra (hudson-openstack)]

Change abandoned by Alexander Tivelkov (<email address hidden>) on branch: master
Review: https://review.openstack.org/388677

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/387993
Committed: https://git.openstack.org/cgit/openstack/murano/commit/?id=254202b98e6e3ec53750cde881b32196ca313ae5
Submitter: Jenkins
Branch: master

commit 254202b98e6e3ec53750cde881b32196ca313ae5
Author: Alexander Tivelkov <email address hidden>
Date: Wed Oct 19 15:45:16 2016 +0300

    Unblocking the gate

    Skipped test_deploy_app_with_volume_creation_from_backup test

    Tempest integration test is failing for volume creation from backup
    (reported as bug #1634876). To unblock the gate the appropriate test
    is skipped for now while the bug is being investigated.

    AgentListener unit test didn't properly override configuration, so in
    some circumstances it could be executed with the disabled murano agent
    which was leading to errors.

    At the same time AgentListener itself contained some legacy code
    leading to under-initialized state in case if the listener was
    disabled in config file.

    Related-bug: #1634876
    Change-Id: I5642b8017c61f77afb68463ba8eb3906eeae7225
    Closes-bug: #1634483

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/388693
Committed: https://git.openstack.org/cgit/openstack/murano/commit/?id=8783d531479c55c3cd36467dee1765ef7512c578
Submitter: Jenkins
Branch: master

commit 8783d531479c55c3cd36467dee1765ef7512c578
Author: Victor Ryzhenkin <email address hidden>
Date: Wed Oct 19 16:15:27 2016 +0300

    Move test with cinder backup creation to the IsolatedAdmin suite

    - Move test_deploy_app_with_volume_creation_from_backup to the
    IsolatedAdmin test suite
    - Make tests using cinder v2 version due tempest defaults
    - Unskip test_deploy_app_with_volume_creation_from_backup test

    Closes-Bug: #1634876

    Change-Id: I8b3ab7f000804f6b368d06ef877776b324774ad4

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/murano 3.1.0 release.