neutron_tempest_plugin.scenario.test_trunk.TrunkTest.test_trunk_subport_lifecycle fails with Port already has an attached device

[Ananya Banerjee]

neutron_tempest_plugin.scenario.test_trunk.TrunkTest.test_trunk_subport_lifecycle is failing in periodic-tripleo-ci-centos-8-standalone-full-tempest-scenario-master with

    Response - Headers: {'date': 'Mon, 13 Sep 2021 18:30:12 GMT', 'server': 'Apache', 'content-length': '1695', 'openstack-api-version': 'compute 2.1', 'x-openstack-nova-api-version': '2.1', 'vary': 'OpenStack-API-Version,X-OpenStack-Nova-API-Version,Accept-Encoding', 'x-openstack-request-id': 'req-cbbe0384-f683-4bd9-990a-bbaceff70255', 'x-compute-request-id': 'req-cbbe0384-f683-4bd9-990a-bbaceff70255', 'connection': 'close', 'content-type': 'application/json', 'status': '200', 'content-location': 'http://192.168.24.3:8774/v2.1/servers/9da29ae5-2784-4cb2-8cb6-cd2ec19e7fbf'}
        Body: b'{"server": {"id": "9da29ae5-2784-4cb2-8cb6-cd2ec19e7fbf", "name": "tempest-server-test-2120448491", "status": "ACTIVE", "tenant_id": "f907072324a04823b5267ebfd078f139", "user_id": "81a2f96324174768a1aa435f2856272c", "metadata": {}, "hostId": "985e26b32ef2005c617cddf445634feda09e1eb51abe1b10032b9f9b", "image": {"id": "9add59d5-6458-46fc-b806-ad3b39a7ebfe", "links": [{"rel": "bookmark", "href": "http://192.168.24.3:8774/images/9add59d5-6458-46fc-b806-ad3b39a7ebfe"}]}, "flavor": {"id": "48b6ea74-8aeb-4086-99ac-c4a4d18398f6", "links": [{"rel": "bookmark", "href": "http://192.168.24.3:8774/flavors/48b6ea74-8aeb-4086-99ac-c4a4d18398f6"}]}, "created": "2021-09-13T18:27:35Z", "updated": "2021-09-13T18:30:11Z", "addresses": {"tempest-TrunkTest-398369782": [{"version": 4, "addr": "10.100.0.10", "OS-EXT-IPS:type": "fixed", "OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:4f:c4:11"}, {"version": 4, "addr": "192.168.24.162", "OS-EXT-IPS:type": "floating", "OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:4f:c4:11"}]}, "accessIPv4": "", "accessIPv6": "", "links": [{"rel": "self", "href": "http://192.168.24.3:8774/v2.1/servers/9da29ae5-2784-4cb2-8cb6-cd2ec19e7fbf"}, {"rel": "bookmark", "href": "http://192.168.24.3:8774/servers/9da29ae5-2784-4cb2-8cb6-cd2ec19e7fbf"}], "OS-DCF:diskConfig": "MANUAL", "progress": 0, "OS-EXT-AZ:availability_zone": "nova", "config_drive": "True", "key_name": "tempest-TrunkTest-398369782", "OS-SRV-USG:launched_at": "2021-09-13T18:27:41.000000", "OS-SRV-USG:terminated_at": null, "security_groups": [{"name": "tempest-TrunkTest-398369782"}], "OS-EXT-STS:task_state": "deleting", "OS-EXT-STS:vm_state": "active", "OS-EXT-STS:power_state": 1, "os-extended-volumes:volumes_attached": []}}'
2021-09-13 18:30:14,588 234586 INFO [tempest.lib.common.rest_client] Request (TrunkTest:_run_cleanups): 404 GET http://192.168.24.3:8774/v2.1/servers/9da29ae5-2784-4cb2-8cb6-cd2ec19e7fbf 0.046s
2021-09-13 18:30:14,589 234586 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
        Body: None
    Response - Headers: {'date': 'Mon, 13 Sep 2021 18:30:14 GMT', 'server': 'Apache', 'content-length': '111', 'openstack-api-version': 'compute 2.1', 'x-openstack-nova-api-version': '2.1', 'vary': 'OpenStack-API-Version,X-OpenStack-Nova-API-Version', 'x-openstack-request-id': 'req-28741ac7-529e-4f49-a12f-3e75e14e2a0e', 'x-compute-request-id': 'req-28741ac7-529e-4f49-a12f-3e75e14e2a0e', 'connection': 'close', 'content-type': 'application/json; charset=UTF-8', 'status': '404', 'content-location': 'http://192.168.24.3:8774/v2.1/servers/9da29ae5-2784-4cb2-8cb6-cd2ec19e7fbf'}
        Body: b'{"itemNotFound": {"code": 404, "message": "Instance 9da29ae5-2784-4cb2-8cb6-cd2ec19e7fbf could not be found."}}'
}}}

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/neutron_tempest_plugin/scenario/test_trunk.py", line 266, in test_trunk_subport_lifecycle
    self.client.add_subports(vm2.trunk['id'], subports)
  File "/usr/lib/python3.6/site-packages/neutron_tempest_plugin/services/network/json/network_client.py", line 848, in add_subports
    return self._subports_action('add_subports', trunk_id, subports)
  File "/usr/lib/python3.6/site-packages/neutron_tempest_plugin/services/network/json/network_client.py", line 842, in _subports_action
    resp, body = self.put(uri, jsonutils.dumps({'sub_ports': subports}))
  File "/usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 363, in put
    return self.request('PUT', url, extra_headers, headers, body, chunked)
  File "/usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 703, in request
    self._error_checker(resp, resp_body)
  File "/usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 824, in _error_checker
    raise exceptions.Conflict(resp_body, resp=resp)
tempest.lib.exceptions.Conflict: Conflict with state of target resource
Details: {'type': 'PortInUse', 'message': 'Unable to complete operation on port 3dd247b0-6c4e-4f02-85d5-62230d8ccaff for network b8ad6344-a87c-4373-88b9-f6f13175e3bd. Port already has an attached device 41aa6ab0-23b1-418e-9678-6c2fd0da0d46.', 'detail': ''}

https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-full-tempest-scenario-master/c8c0f3f/logs/undercloud/var/log/tempest/stestr_results.html.gz

https://logserver.rdoproject.org/21/35121/12/check/periodic-tripleo-ci-centos-8-standalone-full-tempest-scenario-master/23c0427/logs/undercloud/var/log/tempest/stestr_results.html.gz

[Ronelle Landy (rlandy)]

Failure trace from:

https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-full-tempest-scenario-master/ae5ef20/logs/undercloud/var/log/tempest/stestr_results.html.gz

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/neutron_tempest_plugin/scenario/test_trunk.py", line 266, in test_trunk_subport_lifecycle
    self.client.add_subports(vm2.trunk['id'], subports)
  File "/usr/lib/python3.6/site-packages/neutron_tempest_plugin/services/network/json/network_client.py", line 848, in add_subports
    return self._subports_action('add_subports', trunk_id, subports)
  File "/usr/lib/python3.6/site-packages/neutron_tempest_plugin/services/network/json/network_client.py", line 842, in _subports_action
    resp, body = self.put(uri, jsonutils.dumps({'sub_ports': subports}))
  File "/usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 363, in put
    return self.request('PUT', url, extra_headers, headers, body, chunked)
  File "/usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 703, in request
    self._error_checker(resp, resp_body)
  File "/usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 824, in _error_checker
    raise exceptions.Conflict(resp_body, resp=resp)
tempest.lib.exceptions.Conflict: Conflict with state of target resource
Details: {'type': 'PortInUse', 'message': 'Unable to complete operation on port 3dd247b0-6c4e-4f02-85d5-62230d8ccaff for network b8ad6344-a87c-4373-88b9-f6f13175e3bd. Port already has an attached device 41aa6ab0-23b1-418e-9678-6c2fd0da0d46.', 'detail': ''}

[Ronelle Landy (rlandy)]

https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-standalone-full-tempest-scenario-master shows that we start to hit this error at around 09/13. We have two passes in that list.

[Slawek Kaplonski (slaweq)]

I think that this issue may be caused by https://review.opendev.org/#/q/101ccbaeb3c48d9757c276716098ee6e17e12fb8
I proposed revert of that patch and I'm testing it now in https://review.rdoproject.org/r/c/testproject/+/35593

If that will be the culprit I will try to fix it in the Neutron.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/809550
Committed: https://opendev.org/openstack/neutron/commit/b0e01c7df3a8f73f0997fc6125de2b2c78a45862
Submitter: "Zuul (22348)"
Branch: master

commit b0e01c7df3a8f73f0997fc6125de2b2c78a45862
Author: Slawek Kaplonski <email address hidden>
Date: Fri Sep 17 07:54:38 2021 +0000

    Revert "update subport status when trunk/subport create/delete is triggerred"

    This reverts commit 101ccbaeb3c48d9757c276716098ee6e17e12fb8.

    Reason for revert: This commit is probably reason of some race condition
    which causes random failures in the test_trunk_subport_lifecycle scenario
    test.

    Change-Id: Ia042384e0ac333f30235b76e50fdc8748fc2b29a
    Closes-Bug: #1943708

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/neutron/+/810975

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/810975
Committed: https://opendev.org/openstack/neutron/commit/e227b0e97644cc5cbe9840392ee812f805ea4014
Submitter: "Zuul (22348)"
Branch: stable/xena

commit e227b0e97644cc5cbe9840392ee812f805ea4014
Author: Slawek Kaplonski <email address hidden>
Date: Fri Sep 17 07:54:38 2021 +0000

    Revert "update subport status when trunk/subport create/delete is triggerred"

    This reverts commit 101ccbaeb3c48d9757c276716098ee6e17e12fb8.

    Reason for revert: This commit is probably reason of some race condition
    which causes random failures in the test_trunk_subport_lifecycle scenario
    test.

    Change-Id: Ia042384e0ac333f30235b76e50fdc8748fc2b29a
    Closes-Bug: #1943708
    (cherry picked from commit b0e01c7df3a8f73f0997fc6125de2b2c78a45862)

[Bhagyashri Shewale (bhagyashri-shewale)]

We are again started hitting this issue in master periodic integration pipeline.
Added test in the tempest skiplist: https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/811701
Reported bug: https://bugs.launchpad.net/tripleo/+bug/1945443 but marked as duplicate of this bug.
log link: https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-standalone-full-tempest-scenario-master

[Slawek Kaplonski (slaweq)]

It's different issue for sure, first of all, patch which was the root cause of the original issue wasn't in victoria and second, the error in the failed test is different.
I think that it will be better if You could open new bug for that other issue.

[Bhagyashri Shewale (bhagyashri-shewale)]

Hi Slawek i have reported bug separately https://bugs.launchpad.net/tripleo/+bug/1945443

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 19.0.0.0rc2 release candidate.

[Ronelle Landy (rlandy)]

entry no longer in the skiplist - closing out tripleo bug

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 20.0.0.0rc1 release candidate.