skipped neutron dns-integration related tempest test started failing with test_create_router_with_snat_explicit

[chandan kumar]

While comparing with passed http://logs.rdoproject.org/openstack-periodic-24hr/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-stein/cb0bcb7/logs/undercloud/home/zuul/tempest.log.txt.gz and

failed logs for fs020 stein http://logs.rdoproject.org/openstack-periodic-24hr/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-stein/69f2259/logs/undercloud/home/zuul/tempest.log.txt.gz

* neutron_tempest_plugin.scenario.test_dns_integration.DNSIntegrationTests) ... SKIPPED: Skipped because network extension: dns-integration is not enabled
* neutron_tempest_plugin.scenario.test_internal_dns.InternalDNSTest.test_dns_domain_and_name ... SKIPPED: Skipped because network extension: dns-integration is not enabled
* neutron_tempest_plugin.api.test_revisions.TestRevisions.test_update_dns_domain_bumps_revision ... SKIPPED: Skipped because network extension: dns-integration is not enabled

Above tests were skipped earlier due to missing dns-integration network, now it started failing on stein

Here is the most detailed logs:
http://logs.rdoproject.org/openstack-periodic-24hr/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-stein/69f2259/logs/undercloud/home/zuul/tempest.log.txt.gz

It also causes tempest.api.network.admin.test_routers.RoutersAdminTest.test_create_router_with_snat_explicit tests to fail which was passing earlier.

It appears to be linked with dns-integration network extension feature.

For current investigation it appears that
https://review.opendev.org/#/c/694031/ -> Enable tempest DNS tests -> in networking-ovn and

https://review.opendev.org/#/c/693707/ -> Remove Floating IP DNS record upon associated port deletion in openstack neutron might caused the issue.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/697043

[Slawek Kaplonski (slaweq)]

Test test_create_router_with_snat_explicit is failing on deletion of router. Error in neutron is like below:

2019-12-02 08:03:41.019 28 ERROR neutron.api.v2.resource [req-65833112-d708-47f3-8227-3261cfda5af2 686c30b80c1f4583ae8a5e1c84141d0e 5d9cee455cbe479d8348db77f44c7553 - default default] delete failed: No details.: InvalidRequestError: Could not refresh instance '<Router
at 0x7fc3ea9bca10>'
2019-12-02 08:03:41.019 28 ERROR neutron.api.v2.resource Traceback (most recent call last):
2019-12-02 08:03:41.019 28 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/site-packages/neutron/api/v2/resource.py", line 98, in resource
2019-12-02 08:03:41.019 28 ERROR neutron.api.v2.resource result = method(request=request, **args)
2019-12-02 08:03:41.019 28 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/site-packages/neutron/api/v2/base.py", line 567, in delete
2019-12-02 08:03:41.019 28 ERROR neutron.api.v2.resource return self._delete(request, id, **kwargs)
2019-12-02 08:03:41.019 28 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 139, in wrapped
2019-12-02 08:03:41.019 28 ERROR neutron.api.v2.resource setattr(e, '_RETRY_EXCEEDED', True)
2019-12-02 08:03:41.019 28 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-12-02 08:03:41.019 28 ERROR neutron.api.v2.resource self.force_reraise()
2019-12-02 08:03:41.019 28 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-12-02 08:03:41.019 28 ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb)
2019-12-02 08:03:41.019 28 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 135, in wrapped
2019-12-02 08:03:41.019 28 ERROR neutron.api.v2.resource return f(*args, **kwargs)
2019-12-02 08:03:41.019 28 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 154, in wrapper
2019-12-02 08:03:41.019 28 ERROR neutron.api.v2.resource ectxt.value = e.inner_exc
2019-12-02 08:03:41.019 28 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-12-02 08:03:41.019 28 ERROR neutron.api.v2.resource self.force_reraise()
2019-12-02 08:03:41.019 28 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-12-02 08:03:41.019 28 ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb)
2019-12-02 08:03:41.019 28 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 142, in wrapper
2019-12-02 08:03:41.019 28 ERROR neutron.api.v2.resource return f(*args, **kwargs)
2019-12-02 08:03:41.019 28 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 183, in wrapped
2019-12-02 08:03:41.019 28 ERROR neutron.api.v2.resource LOG.debug("Retry wrapper got retriable exception: %s", e)
2019-12-02 08:03:41.019 28 ERROR neutron.api.v2.resource File "/us...

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/697043
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/commit/?id=c2da1fb75f27ef3147160e9b961912e547101690
Submitter: Zuul
Branch: master

commit c2da1fb75f27ef3147160e9b961912e547101690
Author: Chandan Kumar (raukadah) <email address hidden>
Date: Tue Dec 3 15:38:40 2019 +0530

    [Stein] Updated neutron related dns tests to skip list

    Recently dns-integrationg got enabled in stein, 3 skipped
    tests started failing.

    These tests are currently skipped in master also and it needs
    to be fixed in master first then we can enable it.

    We are currently putting it in skiplist to unblock the stein
    promotion.

    Related-Bug: #1854899

    Change-Id: Ied6b5f973b2350c4b958fd049667e17d95eda5fd
    Signed-off-by: Chandan Kumar (raukadah) <email address hidden>

[Marios Andreou (marios-b)]

so the skip merged https://review.opendev.org/#/c/697043/ but we didn't get a periodic yet to verify. Chkumar is running that in https://review.rdoproject.org/r/#/c/23902/ so we can check it when it reports.

But we need to re-enable those asap and keep this bug open for that regardless if the job is green with the skip.

[chandan kumar (chkumar246)]

On test project, https://review.rdoproject.org/r/#/c/23902/ and tempest tests passed http://logs.rdoproject.org/02/23902/2/check/periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-stein/48354a1/logs/tempest.html.gz it appears to be a regression from the existing tests.

[chandan kumar (chkumar246)]

SInce we have promoted queens, test is moved to skip list, do we want to close this bug?

[Marios Andreou (marios-b)]

so to be clear and based on discussions with chkumar|ruck the tests being skipped here:

  - test: 'neutron_tempest_plugin.scenario.test_dns_integration.DNSIntegrationTests'
  - test: 'neutron_tempest_plugin.scenario.test_internal_dns.InternalDNSTest.test_dns_domain_and_name'
  - test: 'neutron_tempest_plugin.api.test_revisions.TestRevisions.test_update_dns_domain_bumps_revision'

were not previously running/passing and some further work is needed here to get them to a passing state.

Closing out this blocker bug as "Wont fix" for now ... if someone picks up these tests we can either update this or file a new bug.