periodic master standalone-full-tempest-api fails during tests- neutron_tempest_plugin.api.test_subnets.SubnetServiceTypeTestJSON

[Bhagyashri Shewale]

neutron_tempest_plugin.api.test_subnets.SubnetServiceTypeTestJSON.test_allocate_ips_are_from_correct_subnet test is failing on periodic master standalone-full-tempest-api

Error log:

ft1.1: neutron_tempest_plugin.api.test_subnets.SubnetServiceTypeTestJSON.test_allocate_ips_are_from_correct_subnet[id-7e0edb66-1bb2-4473-ab83-d039cddced0d]testtools.testresult.real._StringException: pythonlogging:'': {{{
2021-07-05 02:43:58,457 202751 INFO [tempest.lib.common.rest_client] Request (SubnetServiceTypeTestJSON:test_allocate_ips_are_from_correct_subnet): 409 POST http://192.168.24.3:9696/v2.0/subnets 0.859s
2021-07-05 02:43:58,457 202751 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
        Body: {"subnet": {"network_id": "17c71c80-dc23-4237-892b-93492871b6a9", "cidr": "192.168.1.0/24", "ip_version": 4, "service_types": ["test:type_1"], "gateway_ip": "192.168.1.1"}}
    Response - Headers: {'content-type': 'application/json', 'content-length': '232', 'x-openstack-request-id': 'req-d2bdaa6d-57ae-4c2e-8afb-f9c74dab531e', 'date': 'Mon, 05 Jul 2021 02:43:58 GMT', 'connection': 'close', 'status': '409', 'content-location': 'http://192.168.24.3:9696/v2.0/subnets'}
        Body: b'{"NeutronError": {"type": "IpAddressGenerationFailureNoMatchingSubnet", "message": "No valid service subnet for the given device owner, network 17c71c80-dc23-4237-892b-93492871b6a9, service type network:distributed.", "detail": ""}}'
}}}

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/neutron_tempest_plugin/api/test_subnets.py", line 91, in test_allocate_ips_are_from_correct_subnet
    cidr=str(cidr_1))
  File "/usr/lib/python3.6/site-packages/neutron_tempest_plugin/api/base.py", line 465, in create_subnet
    **kwargs)
  File "/usr/lib/python3.6/site-packages/neutron_tempest_plugin/services/network/json/network_client.py", line 145, in _create
    resp, body = self.post(uri, post_data)
  File "/usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 299, in post
    return self.request('POST', url, extra_headers, headers, body, chunked)
  File "/usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 703, in request
    self._error_checker(resp, resp_body)
  File "/usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 824, in _error_checker
    raise exceptions.Conflict(resp_body, resp=resp)
tempest.lib.exceptions.Conflict: Conflict with state of target resource
Details: {'type': 'IpAddressGenerationFailureNoMatchingSubnet', 'message': 'No valid service subnet for the given device owner, network 17c71c80-dc23-4237-892b-93492871b6a9, service type network:distributed.', 'detail': ''}

#links:

[1]: https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-standalone-full-tempest-api-master&pipeline=openstack-periodic-integration-main
[2]: https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-full-tempest-api-master/6d1e428/job-output.txt
[3]: https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-full-tempest-api-master/6d1e428/logs/undercloud/var/log/tempest/stestr_results.html.gz
[4]: https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-full-tempest-api-master/6d1e428/logs/undercloud/var/log/extra/errors.txt.gz

[Bhagyashri Shewale (bhagyashri-shewale)]

posted skiplist at https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/799432

[Slawek Kaplonski (slaweq)]

I think this should be already fixed in Neutron by https://review.opendev.org/c/openstack/neutron/+/797595

[chandan kumar (chkumar246)]

Since https://review.opendev.org/c/openstack/neutron/+/797595 is merged and already available in tripleo-ci-testing hash.

In order to test it,
1] We are reverting the skiplist review here: https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/799577 and running a testproject here: https://review.rdoproject.org/r/c/testproject/+/34399

I will update the bug once I have the result.

[chandan kumar (chkumar246)]

EDIT: https://review.opendev.org/c/openstack/neutron/+/797595 got merged on June 28th, 2021 and
Current tripleo-ci-testing hash for neutron
```
[delorean-component-network]
name=delorean-openstack-neutron-bf7a6061a061bfa08eb1ca96c0ac1bc2d5e580b9
baseurl=https://trunk.rdoproject.org/centos8/component/network/bf/7a/bf7a6061a061bfa08eb1ca96c0ac1bc2d5e580b9_534c9b30
enabled=1
gpgcheck=0
priority=1
```
Above hash pointing to https://opendev.org/openstack/neutron/commit/bf7a6061a061bfa08eb1ca96c0ac1bc2d5e580b9 which contains commits from 19hrs before, so above commit is already in tripleo-ci-testing hash.

So above testproject change https://review.rdoproject.org/r/c/testproject/+/34399 test results will confirm it.

[chandan kumar (chkumar246)]

based on the testproject https://review.rdoproject.org/r/c/testproject/+/34399 and https://logserver.rdoproject.org/99/34399/1/check/periodic-tripleo-ci-centos-8-standalone-full-tempest-api-master/c17638c/logs/undercloud/var/log/tempest/tempest_run.log.txt.gz
```
neutron_tempest_plugin.api.test_subnets.SubnetServiceTypeTestJSON.test_allocate_ips_are_from_correct_subnet[id-7e0edb66-1bb2-4473-ab83-d039cddced0d]
----------------------------------------------------------------------------------------------------------------------------------------------------

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):
      File "/usr/lib/python3.6/site-packages/neutron_tempest_plugin/api/test_subnets.py", line 91, in test_allocate_ips_are_from_correct_subnet
        cidr=str(cidr_1))
      File "/usr/lib/python3.6/site-packages/neutron_tempest_plugin/api/base.py", line 465, in create_subnet
        **kwargs)
      File "/usr/lib/python3.6/site-packages/neutron_tempest_plugin/services/network/json/network_client.py", line 145, in _create
        resp, body = self.post(uri, post_data)
      File "/usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 299, in post
        return self.request('POST', url, extra_headers, headers, body, chunked)
      File "/usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 703, in request
        self._error_checker(resp, resp_body)
      File "/usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 824, in _error_checker
        raise exceptions.Conflict(resp_body, resp=resp)
    tempest.lib.exceptions.Conflict: Conflict with state of target resource
    Details: {'type': 'IpAddressGenerationFailureNoMatchingSubnet', 'message': 'No valid service subnet for the given device owner, network 13fb40e3-9cb7-4e41-beb0-b762f3cb0306, service type network:distributed.', 'detail': ''}

Captured pythonlogging:
~~~~~~~~~~~~~~~~~~~~~~~
    2021-07-06 08:33:27,115 213754 INFO [tempest.lib.common.rest_client] Request (SubnetServiceTypeTestJSON:test_allocate_ips_are_from_correct_subnet): 409 POST http://192.168.24.3:9696/v2.0/subnets 0.873s
    2021-07-06 08:33:27,116 213754 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
            Body: {"subnet": {"network_id": "13fb40e3-9cb7-4e41-beb0-b762f3cb0306", "cidr": "192.168.1.0/24", "ip_version": 4, "service_types": ["test:type_1"], "gateway_ip": "192.168.1.1"}}
        Response - Headers: {'content-type': 'application/json', 'content-length': '232', 'x-openstack-request-id': 'req-b4ce6cfe-fda6-47db-bece-b016abd95a61', 'date': 'Tue, 06 Jul 2021 08:33:27 GMT', 'connection': 'close', 'status': '409', 'content-location': 'http://192.168.24.3:9696/v2.0/subnets'}
            Body: b'{"NeutronError": {"type": "IpAddressGenerationFailureNoMatchingSubnet", "message": "No valid service subnet for the given device owner, network 13fb40e3-9cb7-4e41-beb0-b762f3cb0306, service type network:distributed.", "detail": ""}}'

```

Apart from this patch https://review.opendev.org/c/openstack/neutron/+/7975...

[Slawek Kaplonski (slaweq)]

Ok, I think I know what is going on there. It's an issue with how OVN backend works in Neutron. I will try to fix either test or neutron code to make it working

[chandan kumar (chkumar246)]

As discussed with slawq on IRC
11:00:07 --> | Tengu (~<email address hidden>) has joined #oooq
11:02:11 chkumar|rover | slaweq: hello, please have a look at this bug https://bugs.launchpad.net/tripleo/+bug/1934622 context:
                          | https://review.opendev.org/c/openstack/neutron/+/797595 is already present in the tested hash, not sure
                          | do we need to make some changes somewhere else?
11:02:34 chkumar|rover | slaweq: https://logserver.rdoproject.org/99/34399/1/check/periodic-tripleo-ci-centos-8-standalone-full-te
                          | mpest-api-master/c17638c/logs/undercloud/var/log/tempest/tempest_run.log.txt.gz
11:02:43 chkumar|rover | slaweq: thanks :-)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
11:03:28 slaweq | chkumar|rover: looking
11:08:03 slaweq | chkumar|rover: ok, I think I understand now, it's different issue related to how ovn backend works in
                          | Neutron
11:08:25 slaweq | I will deploy dev vm and will try to fix that today or tomorrow
11:17:38

It seems to be a different issue.

[Slawek Kaplonski (slaweq)]

Fix proposed https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/799648

[chandan kumar (chkumar246)]

I have tested the above patch here: https://review.rdoproject.org/r/c/testproject/+/34399
and run is green: https://logserver.rdoproject.org/99/34399/2/check/periodic-tripleo-ci-centos-8-standalone-full-tempest-api-master/3360bed/logs/undercloud/var/log/tempest/tempest_run.log.txt.gz

```
{3} neutron_tempest_plugin.api.test_subnets.SubnetServiceTypeTestJSON.test_allocate_ips_are_from_correct_subnet [4.036313s] ... ok
```
Thank you Slawq for fixing it.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron-tempest-plugin train-last release.