Filter out failing tempest volume test on stable Ocata branch

[Bernard Cafarelli]

Per [0] "There is no statement about the level of testing and upgrades from Extended Maintenance are not supported within the Community."

In Ocata (currently in EM) and Pike (soon to be) branches, we see Zuul check failures from time to time on unstable tests, that require a few rechecks before the backport gets in.

For some issues it is better to fix the test/setup itself when it is easy (see [1] and [2] for recent examples), but for some failing tests (testing exotic cases or not directly related to networking), we should start filtering them out.

An initial example is tempest.api.volume.test_volumes_extend.VolumesExtendTest.test_volume_extend_when_volume_has_snapshot which fails regularly on ocata and often on pike:
http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:%5C%22test_volume_extend_when_volume_has_snapshot%5C%22%20AND%20project:%5C%22openstack%2Fneutron%5C%22%20AND%20build_status:%5C%22FAILURE%5C%22

We can use this bug to track later similar additions

[0] https://docs.openstack.org/project-team-guide/stable-branches.html#extended-maintenance
[1] https://bugs.launchpad.net/neutron/+bug/1821815 fixes cover jobs
[2] https://review.openstack.org/#/c/648046/ fixes a functional test failure (simple backport)

[Bence Romsics (bence-romsics)]

I understand the motivation to limit the CI maintenance effort as a stable branch get older. But how do you want to use this bug report? Do you want to keep it open indefinitely and refer to it from changes marking tests to be skipped or unstable? Won't we mix multiple distinct problems in the same tracking report then?

[Bernard Cafarelli (bcafarel)]

Ideally yes if new tests start to fail, the patch adding them to "exclude list" would be marked as related to this bug. I do not expect too many of them to pop in though (or it would mean the stable branch is really starting to get stale and should probably be EOLed).

That said, this is mostly to get initial feedback and get this "exclude list" in place (still looking how it can be done in gates) for failing volume test. Nothing against filling separate bugs later on for new test failures

[Elod Illes (elod-illes)]

During reviewing stable patches what I see is:

* some failures are either fixed on newer branches and the fix can be backported, or the problem still exists on master as well. In these cases this general bug report is not really needed I think.
* zuul/CI change related failures - these usually affect several projects, so those are more likely need and get a fix. (like in the example cover job fix)

For the "volume extend" example there's a bug report [1] already if I'm not mistaken as it turns up on other projects, too.

So in short, I think, there isn't / shouldn't be such a big number of failures that would need a "general bug report" on stable branches under Extended Maintenance. Also the number of rechecks is not that significantly different from the rechecks on master / maintained stable branches. I might be wrong, though :)

[1] https://bugs.launchpad.net/cinder/+bug/1796708

[Bernard Cafarelli (bcafarel)]

OK for the general tracking list feedback, let's scale down to filtering that volume test (not likely to be fixe in EM branches): if you set the logstash request in initial comment to 7d ago (aka first week of April), you will see around 60 hits failing on ocata (and 40-50 on pike, but that is another subject).

I have a tentative review to filter out this test in ocata branch, will update here (last approved cherry-picks on ocata are on recheck because of that test)

[Bernard Cafarelli (bcafarel)]

Suggested change: https://review.openstack.org/#/c/648239/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/648239
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=33c18a65f829295ca99cfb537e270986185f9ba4
Submitter: Zuul
Branch: stable/ocata

commit 33c18a65f829295ca99cfb537e270986185f9ba4
Author: Bernard Cafarelli <email address hidden>
Date: Wed Mar 27 21:40:09 2019 +0100

    Filter out failing tempest volume test on stable/ocata

    tempest.api.volume.test_volumes_extend.VolumesExtendTest.test_volume_extend_when_volume_has_snapshot
    has a high failure rate and will most probably be not fixe in an
    Extended Maintenance branch, so filter it out of run tempest tests for
    neutron stable/ocata tempest runs

    Based on the full-parallel tox target regex, this adds a blacklist file
    parameter with a single file listing excluded tests

    Change-Id: I1051a56a0c91fd40bc69e5a7295498b64f6ded96
    Related-Bug: #1821925