queens, tempest.api.volume multiple failures

[wes hayutin]

http://logs.rdoproject.org/openstack-periodic-24hr/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-queens/5744534/logs/tempest.html.gz

tempest.api.volume.test_volumes_snapshots.VolumesSnapshotTestJSON
tempest.api.volume.test_volumes_backup.VolumesBackupsV39Test
tempest.api.volume.test_volumes_backup.VolumesBackupsTest
tempest.api.volume.admin.test_volumes_backup.VolumesBackupsAdminTest
tempest.api.compute.images.test_images_oneserver_negative.ImagesOneServerNegativeTestJSON

 Body: {"backup": {"name": "tempest-VolumesBackupsAdminTest-Backup-1716457612", "volume_id": "ec769724-b9b5-4f43-917e-26c29c9246d3"}}
    Response - Headers: {'status': '503', u'content-length': '93', 'content-location': 'http://10.0.0.5:8776/v2/d36ae8f2a5724e0789fae231c5a76985/backups', u'x-compute-request-id': 'req-195aaebc-b25b-4943-a15d-2de7402c70b0', u'server': 'Apache', u'connection': 'close', u'date': 'Thu, 09 May 2019 08:18:23 GMT', u'content-type': 'application/json', u'x-openstack-request-id': 'req-195aaebc-b25b-4943-a15d-2de7402c70b0'}
        Body: {"serviceUnavailable": {"message": "Service cinder-backup could not be found.", "code": 503}}

Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/tempest/api/volume/admin/test_volumes_backup.py", line 131, in test_volume_backup_reset_status
    name=backup_name)
  File "/usr/lib/python2.7/site-packages/tempest/api/volume/base.py", line 164, in create_backup
    volume_id=volume_id, **kwargs)['backup']
  File "/usr/lib/python2.7/site-packages/tempest/lib/services/volume/v2/backups_client.py", line 36, in create_backup
    resp, body = self.post('backups', post_body)
  File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 279, in post
    return self.request('POST', url, extra_headers, headers, body, chunked)
  File "/usr/lib/python2.7/site-packages/tempest/lib/services/volume/base_client.py", line 38, in request
    method, url, extra_headers, headers, body, chunked)
  File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 668, in request
    self._error_checker(resp, resp_body)
  File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 849, in _error_checker
    resp=resp)
tempest.lib.exceptions.UnexpectedResponseCode: Unexpected response code received
Details: 503

[Marios Andreou (marios-b)]

problem is ongoing seeing it today 2019-05-17 at http://logs.rdoproject.org/openstack-periodic-24hr/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-queens/27b91e3/logs/tempest.html.gz

going to reach out to tempest team and see if we should temporarily skip

[Marios Andreou (marios-b)]

did some digging today so I know who to reach out to. I saw many errors in [1]
 but a lot of those are red-herrings.

The main interesting thing i found was rabbit - main log at [2] isn't too interesting but the sasl log at [3] has some "shutdown_error"... there are 6 of them where in a green job (green @ [4] for reference) there were only 3... not sure if that means something

Going to reach out to pidone for a look - grateful for any info about this if anyone has it add a comment thank you.

[1] http://logs.rdoproject.org/openstack-periodic-24hr/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-queens/27b91e3/logs/undercloud/var/log/extra/errors.txt.txt.gz
[2] http://logs.rdoproject.org/openstack-periodic-24hr/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020<email address hidden>
[3] http://logs.rdoproject.org/openstack-periodic-24hr/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020<email address hidden>
[4] http://logs.rdoproject.org/openstack-periodic-24hr/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020<email address hidden>

[Marios Andreou (marios-b)]

nope sorry comment #2 is wrong.

I am pointing to undercloud logs there. arxrcruz thinks this is related to cinder-backup service itself and we are currently discussing in #tripleo

[Marios Andreou (marios-b)]

16:40 < ykarel> marios_|ruck, arxcruz you should see chandan's effort https://review.opendev.org/#/c/658284