periodic standalone full-tempest-master failing neutron_tempest_plugin.scenario.test_security_groups

[Marios Andreou]

At [1][2] the periodic-tripleo-ci-centos-7-standalone-full-tempest-master fails running tempest specifically neutron_tempest_plugin.scenario.test_security_groups.NetworkSecGroupTest test_two_sec_groups with trace like:

        Traceback (most recent call last):
          File "/usr/lib/python2.7/site-packages/neutron_tempest_plugin/scenario/test_security_groups.py", line 188, in test_two_sec_groups
            name=ssh_secgrp_name)
          File "/usr/lib/python2.7/site-packages/neutron_tempest_plugin/services/network/json/network_client.py", line 145, in _create
            resp, body = self.post(uri, post_data)
          File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 283, in post
            return self.request('POST', url, extra_headers, headers, body, chunked)
          File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 687, in request
            self._error_checker(resp, resp_body)
          File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 808, in _error_checker
            raise exceptions.Conflict(resp_body, resp=resp)
        tempest.lib.exceptions.Conflict: Conflict with state of target resource
        Details: {u'message': u"Quota exceeded for resources: ['security_group'].", u'type': u'OverQuota', u'detail': u''}

Also seen at fs20 in [3]

[1] https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-standalone-full-tempest-master/1f0a53e/logs/undercloud/var/log/tempest/stestr_results.html.gz
[2] https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-standalone-full-tempest-master/29c440d/logs/undercloud/var/log/tempest/stestr_results.html.gz
[3] https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-master/e520cda/logs/undercloud/var/log/tempest/stestr_results.html.gz

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/709715

[Marios Andreou (marios-b)]

so first step is skip it to unblock us...

tripleo-quickstart-extras master: Add neutron_tempest_plugin.scenario test_two_sec_groups master skip https://review.opendev.org/709715

[Martin Kopec (mkopec)]

I'm not sure about skipping these tests is the right way, because:

neutron_tempest_plugin.scenario.test_security_groups.NetworkSecGroupTest failed cause a quota got exceeded for resources, that's not tempest's fault

tempest.scenario.test_volume_boot_pattern.TestVolumeBootPattern failed due to an ssh timeout, i see from time to time, i'm not sure if in this test, but I see the same failure from time to time and recheck always help, this shouldn't happen twice in a row, at least i haven't seen it happening twice in a row

are those two failing constantly now?

[Marios Andreou (marios-b)]

Looks related to https://bugs.launchpad.net/neutron/+bug/1864471

14:29 < ykarel> marios|ruck, is https://bugs.launchpad.net/neutron/+bug/1864471 known?
14:30 < ykarel> i see standalone full-tempest failing for same reason in last 4 run
14:30 < ykarel> https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-7-standalone-full-tempest-master

[Marios Andreou (marios-b)]

might be fixed by bump in defaults at https://review.opendev.org/709478 (https://bugs.launchpad.net/neutron/+bug/1864471)

[Marios Andreou (marios-b)]

15:14 < ykarel> fix in there job will not help our jobs
15:15 < marios|ruck|call> ykarel: ack i was doing 2 things same time dind't notice the change was in zuul.yaml layout... will check
                          after the call we can add to ours

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/710083

[Marios Andreou (marios-b)]

k, so we still have the skip on the table if needed until we get the fix but this seems to be a legit issue.

Thanks to ykarel, since [1] we need to bump our security groups quota. The default [2] is 10.
Just posted this to be able to control the value:

https://review.opendev.org/710083 tripleo-heat-templates master: Add NeutronSecurityGroupQuota to allow setting quota_security_group

I will pick it up tomorrow... if we do that ^^ then we can bump it for the job to 150 like they did in [3]

[1] https://review.opendev.org/#/c/682369
[2] https://github.com/openstack/neutron/blob/a388701ddfe628e9a5bd16a78422164799b11ef8/neutron/conf/quota.py#L30
[3] https://review.opendev.org/709478

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/710227

[Marios Andreou (marios-b)]

So we need these things:

1. Make security groups overrideable in tripleo-heat-templates:

https://review.opendev.org/#/c/710083/ Add NeutronSecurityGroupQuota to allow setting quota_security_group

2. Wire up standalone full tempest:

https://review.rdoproject.org/r/#/c/25418/ Adds env for neutron sec groups to standalone full tempest master

3. Wire up fs20:

https://review.opendev.org/#/c/710227/ Add neutron_sec_group_quota.yaml to fs 20 network_isolation_args

Testing at https://review.rdoproject.org/r/25420

[Marios Andreou (marios-b)]

14:06 < ykarel> marios|ruck, looks like issue should be fixed for us too with https://review.opendev.org/#/c/709691/4

We could still land the things in comment #10 .. especially #1 but #2/#3 aren't strictly needed (but won't hurt). The review at /#/c/709691 sets the quota to unlimited as far as i can quickly see/understand there...
The reviews at comment #10 set it to 150.

[Marios Andreou (marios-b)]

https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-7-standalone-full-tempest-master

ok looks like it's back to green with https://review.opendev.org/#/c/709691/4 (from https://bugs.launchpad.net/neutron/+bug/1864471 )

we can still merge some or all the things in https://bugs.launchpad.net/tripleo/+bug/1864646/comments/10 (realistically probably only #1) but they are no longer critical.

Marking bug as fix-released

[OpenStack Infra (hudson-openstack)]

Change abandoned by Marios Andreou (<email address hidden>) on branch: master
Review: https://review.opendev.org/709715
Reason: https://bugs.launchpad.net/tripleo/+bug/1864646/comments/12

[OpenStack Infra (hudson-openstack)]

Change abandoned by Marios Andreou (<email address hidden>) on branch: master
Review: https://review.opendev.org/710227
Reason: https://bugs.launchpad.net/tripleo/+bug/1864646/comments/11

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/710083
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=a7566ecc77d03fc268deb63eff277bbf2f436607
Submitter: Zuul
Branch: master

commit a7566ecc77d03fc268deb63eff277bbf2f436607
Author: Marios Andreou <email address hidden>
Date: Wed Feb 26 18:51:58 2020 +0200

    Add NeutronSecurityGroupQuota to allow setting quota_security_group

    As in related bug, we need to bump this to more than the default
    10. Also adds an environment file into ci/environments for use
    by the relevant jobs.

    Change-Id: Ia944be7be58575b59d2ff3a46da059e34bc8d0c2
    Related-Bug: 1864646