periodic standalone full-tempest train failing tempest.api.volume.test_volumes_snapshots.VolumesSnapshotTestJSON

[Marios Andreou]

Last 3 runs at [1][2][3] the periodic-tripleo-ci-centos-7-standalone-full-tempest-train fails during tempest run specifically
tempest.api.volume.test_volumes_snapshots.VolumesSnapshotTestJSON test_volume_from_snapshot and test_volume_from_snapshot_no_size with trace like:

        Traceback (most recent call last):
          File "/usr/lib/python2.7/site-packages/tempest/api/volume/test_volumes_snapshots.py", line 160, in test_volume_from_snapshot
            self._create_volume_from_snapshot(extra_size=1)
          File "/usr/lib/python2.7/site-packages/tempest/api/volume/test_volumes_snapshots.py", line 141, in _create_volume_from_snapshot
            size=size)
          File "/usr/lib/python2.7/site-packages/tempest/api/volume/base.py", line 136, in create_volume
            volume['id'], wait_until)
          File "/usr/lib/python2.7/site-packages/tempest/common/waiters.py", line 204, in wait_for_volume_resource_status
            resource_name=resource_name, resource_id=resource_id)
        tempest.exceptions.VolumeResourceBuildErrorException: volume f0ec40d9-3797-46c3-a83d-cf166e57ad76 failed to build and is in ERROR status

[1] https://logserver.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-standalone-full-tempest-train/0377fc9/logs/undercloud/var/log/tempest/stestr_results.html.gz
[2] https://logserver.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-standalone-full-tempest-train/3c758a6/logs/undercloud/var/log/tempest/stestr_results.html.gz
[3] https://logserver.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-standalone-full-tempest-train/c0f18fa/logs/undercloud/var/log/tempest/stestr_results.html.gz

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/709725

[Marios Andreou (marios-b)]

skip to unblock us until we get a proper fix for that

tripleo-quickstart-extras master: Add VolumesSnapshotTestJSON test_volume_from_snapshot train skiplist https://review.opendev.org/709725

[Marios Andreou (marios-b)]

so not sure we need the skip in comment #2 ... the very latest run of this just passed after the 3 previous fails https://logserver.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-standalone-full-tempest-train/11b1d53/

[Marios Andreou (marios-b)]

nop failed again there * https://logserver.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-standalone-full-tempest-train/a845b8d/logs/undercloud/var/log/tempest/stestr_results.html.gz
        * test_volume_from_snapshot_no_size[id-053d8870-8282-4fff-9dbb-99cb58bb5e0a]

[Martin Kopec (mkopec)]

test_volume_from_snapshot_no_size can't create a volume, if this happens all the time, there is for sure a problem in the infrastructure, feel free to skip the test until the issue is debugged more and fixed.

[Marios Andreou (marios-b)]

so we had a couple more green runs https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-7-standalone-full-tempest-train ... after those recent green runs train also promoted last night.

looking at the build history though the job doesn't appear to be completely stable - not sure if we still want to merge the skip or not. The error in this bug has been confirmed in multiple runs but we can't say it is consistent.

[Marios Andreou (marios-b)]

after comment #6 went and did some more digging... we definitely want the skip IMO. I went looking at the zuul builds and picking the failed... i quickly found these confirmed cases (last one is from 10th Feb so not new either)

        * https://logserver.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-standalone-full-tempest-train/45a7cfd/logs/undercloud/var/log/tempest/stestr_results.html.gz
            * test_volume_from_snapshot[id-677863d1-3142-456d-b6ac-9924f667a7f4]
            * test_volume_from_snapshot_no_size[id-053d8870-8282-4fff-9dbb-99cb58bb5e0a]

        * https://logserver.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-standalone-full-tempest-train/95982e1/logs/undercloud/var/log/tempest/stestr_results.html.gz
            * test_volume_from_snapshot[id-677863d1-3142-456d-b6ac-9924f667a7f4]
            * test_volume_from_snapshot_no_size[id-053d8870-8282-4fff-9dbb-99cb58bb5e0a]

        * https://logserver.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-standalone-full-tempest-train/3edda68/logs/undercloud/var/log/tempest/stestr_results.html.gz
            * test_volume_from_snapshot[id-677863d1-3142-456d-b6ac-9924f667a7f4]
            * test_volume_from_snapshot_no_size[id-053d8870-8282-4fff-9dbb-99cb58bb5e0a]

        * https://logserver.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-standalone-full-tempest-train/bb3bbd6/logs/undercloud/var/log/tempest/stestr_results.html.gz
            * test_volume_from_snapshot_no_size[id-053d8870-8282-4fff-9dbb-99cb58bb5e0a]

        * https://logserver.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-standalone-full-tempest-train/33ff494/logs/undercloud/var/log/tempest/stestr_results.html.gz
            * test_volume_from_snapshot[id-677863d1-3142-456d-b6ac-9924f667a7f4]
            * test_volume_from_snapshot_no_size[id-053d8870-8282-4fff-9dbb-99cb58bb5e0a]

        * https://logserver.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-standalone-full-tempest-train/a202c6b/logs/undercloud/var/log/tempest/stestr_results.html.gz
            * test_volume_from_snapshot[id-677863d1-3142-456d-b6ac-9924f667a7f4]
            * test_volume_from_snapshot_no_size[id-053d8870-8282-4fff-9dbb-99cb58bb5e0a]

        * https://logserver.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-standalone-full-tempest-train/98d63d7/logs/undercloud/var/log/tempest/stestr_results.html.gz
            * test_volume_from_snapshot[id-677863d1-3142-456d-b6ac-9924f667a7f4]
            * test_volume_from_snapshot_no_size[id-053d8870-8282-4fff-9dbb-99cb58bb5e0a]

[Marios Andreou (marios-b)]

and again latest run https://logserver.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-standalone-full-tempest-train/7183963/logs/undercloud/var/log/tempest/stestr_results.html.gz

test_volume_from_snapshot_no_size[id-053d8870-8282-4fff-9dbb-99cb58bb5e0a]

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/709725
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/commit/?id=85d2f5ff344701880508fcf76995f8719ccf7036
Submitter: Zuul
Branch: master

commit 85d2f5ff344701880508fcf76995f8719ccf7036
Author: Marios Andreou <email address hidden>
Date: Tue Feb 25 15:22:13 2020 +0200

    Add VolumesSnapshotTestJSON test_volume_from_snapshot train skiplist

    Adds the test_volume_from_snapshot test_volume_from_snapshot_no_size
    tests to the train skip lists to unblock related-bug.

    Related-Bug: 1864652
    Change-Id: Ia01a495a85f533ff16bd60347a4ec0ac100aa591

[Marios Andreou (marios-b)]

skip is now merged so lowering to high from critical as it no longer blocks

[wes hayutin (weshayutin)]

tracked in bugzilla https://bugzilla.redhat.com/1822305

[Marios Andreou (marios-b)]

This is an automated action. Bug status has been set to 'Incomplete' and target milestone has been removed due to inactivity. If you disagree please re-set these values and reach out to us on freenode #tripleo

[Launchpad Janitor (janitor)]

[Expired for tripleo because there has been no activity for 60 days.]

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/885248

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Ghanshyam <email address hidden>" on branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/885248
Reason: TrieplO project is retiring now, for details, please see https://review.opendev.org/c/openstack/governance/+/905145 or reach out to OpenStack TC.