Master promotions are failing tempest test (fs020) tempest.api.volume.admin.test_volume_quotas.VolumeQuotasAdminTestJSON

[Ronelle Landy]

Since 08/15 master promotions have shown fs020 failing tempest test: tempest.api.volume.admin.test_volume_quotas.VolumeQuotasAdminTestJSON.test_quota_usage_after_volume_transfer with the following trace:

2019-08-15 17:52:18 | ==============================
2019-08-15 17:52:18 | Failed 1 tests - output below:
2019-08-15 17:52:18 | ==============================
2019-08-15 17:52:18 |
2019-08-15 17:52:18 | tempest.api.volume.admin.test_volume_quotas.VolumeQuotasAdminTestJSON.test_quota_usage_after_volume_transfer[id-8911036f-9d54-4720-80cc-a1c9796a8805]
2019-08-15 17:52:18 | -----------------------------------------------------------------------------------------------------------------------------------------------------
2019-08-15 17:52:18 |
2019-08-15 17:52:18 | Captured traceback:
2019-08-15 17:52:18 | ~~~~~~~~~~~~~~~~~~~
2019-08-15 17:52:18 | Traceback (most recent call last):
2019-08-15 17:52:18 | File "/usr/lib/python2.7/site-packages/tempest/api/volume/admin/test_volume_quotas.py", line 144, in test_quota_usage_after_volume_transfer
2019-08-15 17:52:18 | volume = self.create_volume()
2019-08-15 17:52:18 | File "/usr/lib/python2.7/site-packages/tempest/api/volume/base.py", line 136, in create_volume
2019-08-15 17:52:18 | volume['id'], wait_until)
2019-08-15 17:52:18 | File "/usr/lib/python2.7/site-packages/tempest/common/waiters.py", line 204, in wait_for_volume_resource_status
2019-08-15 17:52:18 | resource_name=resource_name, resource_id=resource_id)
2019-08-15 17:52:18 | tempest.exceptions.VolumeResourceBuildErrorException: volume 49bf773f-354e-4fa4-945a-7315c2356aae failed to build and is in ERROR status
2019

The full log is included below:

https://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-master/1ea6dd7/logs/undercloud/home/zuul/tempest.log.txt.gz#_2019-08-15_17_52_18

[Ronelle Landy (rlandy)]

http://logs.rdoproject.org/32/20832/25/check/periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-master/a73285d/ just passed.

[Ronelle Landy (rlandy)]

Marking this now as a promotion blocker:

There are two more logs:

logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-master/60010b1/logs/undercloud/home/zuul/tempest.log.txt.gz

http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-master/3a75cf3/logs/undercloud/home/zuul/tempest.log.txt.gz

[Ronelle Landy (rlandy)]

This is now blocking master promotions for fs020. The failures in periodic-tripleo-ci-centos-7-standalone-full-tempest-master are unrelated

[chandan kumar (chkumar246)]

From this tempest failue: https://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-master/60010b1/logs/undercloud/home/zuul/tempest.log.txt.gz#_2019-08-18_05_46_10
2019-08-18 05:46:10 | Traceback (most recent call last):
2019-08-18 05:46:10 | File "/usr/lib/python2.7/site-packages/tempest/common/utils/__init__.py", line 89, in wrapper
2019-08-18 05:46:10 | return f(*func_args, **func_kwargs)
2019-08-18 05:46:10 | File "/usr/lib/python2.7/site-packages/tempest/api/volume/test_volumes_snapshots.py", line 78, in test_snapshot_create_offline_delete_online
2019-08-18 05:46:10 | snapshot3 = self.create_snapshot(self.volume_origin['id'], force=True)
2019-08-18 05:46:10 | File "/usr/lib/python2.7/site-packages/tempest/api/volume/base.py", line 151, in create_snapshot
2019-08-18 05:46:10 | snapshot['id'], 'available')
2019-08-18 05:46:10 | File "/usr/lib/python2.7/site-packages/tempest/common/waiters.py", line 204, in wait_for_volume_resource_status
2019-08-18 05:46:10 | resource_name=resource_name, resource_id=resource_id)
2019-08-18 05:46:10 | tempest.exceptions.VolumeResourceBuildErrorException: snapshot e6306022-55dd-4b04-8a19-34ac600d0565 failed to build and is in ERROR status

In https://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-master/60010b1/logs/overcloud-controller-0/var/log/containers/cinder/cinder-scheduler.log.txt.gz?level=ERROR

2019-08-18 04:42:33.501 7 ERROR cinder.scheduler.flows.create_volume [req-0ae7845f-eb9b-4b61-94d9-fcf3a077c5f1 c2bba3cbcc3a4f67b3383bedd46e5f16 9465a2ac942b45d2a3b82b14aafb36ba - default default] Failed to run task cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create: No valid backend was found. No weighed backends available: NoValidBackend: No valid backend was found. No weighed backends available

[chandan kumar (chkumar246)]

The same error is seen on https://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-master/3a75cf3/logs/overcloud-controller-0/var/log/containers/cinder/cinder-scheduler.log.txt.gz?level=ERROR

[chandan kumar (chkumar246)]

2019-08-18 16:36:57.774 ERROR /var/log/containers/cinder/cinder-api.log: 18 ERROR cinder.volume.api [req-ab2b5c8c-38f6-4f62-b813-f06ea6709373 e4f340de0d7b4e3e84d66383305505c4 dafcf44183b241bcb523e6d19a740e5e - default default] Quota exceeded for dafcf44183b241bcb523e6d19a740e5e, tried to extend volume by 1G, (1G of 1G already consumed).: OverQuota: Quota exceeded for resources: ['gigabytes']

[chandan kumar (chkumar246)]

2019-08-18 04:44:43.521 ERROR /var/log/containers/cinder/cinder-scheduler.log: 7 ERROR cinder.scheduler.manager [req-445184e3-49de-411d-bb27-6a1c0f18f7fb 8d95eff08f3d4f058f231e7e31ecb3ae 9a015ea3920b4669921121fc4a40d928 - default default] Failed to schedule_create_snapshot: Cannot place snapshot e6306022-55dd-4b04-8a19-34ac600d0565 on hostgroup@tripleo_iscsi#tripleo_iscsi.: NoValidBackend: Cannot place snapshot e6306022-55dd-4b04-8a19-34ac600d0565 on hostgroup@tripleo_iscsi#tripleo_iscsi.

[chandan kumar (chkumar246)]

https://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-master/1ea6dd7/logs/overcloud-controller-0/var/log/containers/cinder/cinder-scheduler.log.txt.gz#_2019-08-15_16_50_12_180

2019-08-15 16:50:12.180 7 WARNING cinder.scheduler.filters.capacity_filter [req-48cfa5bc-d078-4554-bb71-d1e625e32ba8 e2248dae2278414aa6bd451aa7f826d3 efa95eba066748d28485322df6c61bda - default default] Insufficient free virtual space (-10.0GB) to accommodate thin provisioned 1GB volume on host hostgroup@tripleo_iscsi#tripleo_iscsi.

[Alan Bishop (alan-bishop)]

As the logs indicate, the 10G LVM backend is simply out of space. The fix is to increase CinderLVMLoopDeviceSize [1].

[1] https://opendev.org/openstack/tripleo-heat-templates/src/branch/master/deployment/cinder/cinder-volume-container-puppet.yaml#L95

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/677227

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/677227
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/commit/?id=00e7cf7b209e0fddbb60544d8e0b01de23cc2c97
Submitter: Zuul
Branch: master

commit 00e7cf7b209e0fddbb60544d8e0b01de23cc2c97
Author: Ronelle Landy <email address hidden>
Date: Mon Aug 19 10:56:01 2019 -0400

    Add overcloud deploy parameter to set CinderLVMLoopDeviceSize

    Master promotion jobs are failing due to cinder backend being
    out of space. This review adjusts the LVM backend up for master
    to avoid the 'out of space' problem.

    Change-Id: Ib46afdc3ff38b51dd1c608e910436188cd0e5472
    Closes-Bug: #1840451

[chandan kumar (chkumar246)]

We are still seeing this issue https://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-master/9c5f8af/logs/overcloud-controller-0/var/log/containers/cinder/cinder-scheduler.log.txt.gz?level=WARNING#_2019-08-21_04_27_15_820

2019-08-21 04:27:15.820 7 ERROR cinder.scheduler.flows.create_volume [req-e00f8245-ee03-41a2-90ba-6b4e72f6d029 02a3fb5af9ce4724815079720a772427 1eb7a5dad8d94573a9aa79f8330fa26b - default default] Failed to run task cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create: No valid backend was found. No weighed backends available: NoValidBackend: No valid backend was found. No weighed backends available
2019-08-21 04:30:14.861 7 WARNING cinder.scheduler.filters.capacity_filter [req-73a1d781-6909-4b6d-946d-d91317296d67 0ad8284ae2f34eab87a0495161461c35 b0190a0fee814aa58aab9401b7db472b - default default] Insufficient free virtual space (-12.0GB) to accommodate thin provisioned 1GB volume on host hostgroup@tripleo_iscsi#tripleo_iscsi.

[chandan kumar (chkumar246)]

two snapshot tests got failed now: https://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-master/9c5f8af/logs/undercloud/home/zuul/tempest.log.txt.gz#_2019-08-21_04_30_17

[Ronelle Landy (rlandy)]

This may be related or not - checking with abishop

[Ronelle Landy (rlandy)]

Opening a separate bug for the other tempest.api.volume.* failures.

This failure was fixed by increasing the CinderLVMLoopDeviceSize

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/678141