Tempest test "test_reassign_port_between_servers " fails with server fault

[Konstantinos Kaskavelis]

Solutions QA team has a failed run where a tempest test failed with the message: "tempest.lib.exceptions.ServerFault: Got server fault"

The failure happened after we deployed Yoga on Focal.

From the tempest report:

Traceback (most recent call last):
  File "/home/ubuntu/snap/fcbtest/29/.rally/verification/verifier-485e4b80-bd8b-4d42-81ed-8b6690a64532/repo/tempest/api/compute/servers/test_attach_interfaces.py", line 337, in test_reassign_port_between_servers
    server['id'], port_id=port_id)['interfaceAttachment']
  File "/home/ubuntu/snap/fcbtest/29/.rally/verification/verifier-485e4b80-bd8b-4d42-81ed-8b6690a64532/repo/tempest/lib/services/compute/interfaces_client.py", line 48, in create_interface
    body=post_body)
  File "/home/ubuntu/snap/fcbtest/29/.rally/verification/verifier-485e4b80-bd8b-4d42-81ed-8b6690a64532/repo/tempest/lib/common/rest_client.py", line 299, in post
    return self.request('POST', url, extra_headers, headers, body, chunked)
  File "/home/ubuntu/snap/fcbtest/29/.rally/verification/verifier-485e4b80-bd8b-4d42-81ed-8b6690a64532/repo/tempest/lib/services/compute/base_compute_client.py", line 48, in request
    method, url, extra_headers, headers, body, chunked)
  File "/home/ubuntu/snap/fcbtest/29/.rally/verification/verifier-485e4b80-bd8b-4d42-81ed-8b6690a64532/repo/tempest/lib/common/rest_client.py", line 703, in request
    self._error_checker(resp, resp_body)
  File "/home/ubuntu/snap/fcbtest/29/.rally/verification/verifier-485e4b80-bd8b-4d42-81ed-8b6690a64532/repo/tempest/lib/common/rest_client.py", line 880, in _error_checker
    message=message)
tempest.lib.exceptions.ServerFault: Got server fault
Details: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'nova.exception_Remote.PortBindingFailed_Remote'>

From the logs:

2022-10-25 12:12:37.665 305601 ERROR nova.api.openstack.wsgi [req-d7f47a9a-bcbc-4852-b16d-21676f2d252d 5b37d935cf704131a7e0ffc4ca909598 6383a5bda46c49e68522db7da47854df - 541f14c98a464950bf423d15c973b6de 541f14c98a464950bf423d15c973b6de] Unexpected exception in API method: nova.exception_Remote.PortBindingFailed_Remote: Binding failed for port ab1a4b97-b704-462f-a767-f4319ab1f278, please check neutron logs for more information.
Traceback (most recent call last):

  File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/server.py", line 241, in inner
    return func(*args, **kwargs)

  File "/usr/lib/python3/dist-packages/nova/exception_wrapper.py", line 71, in wrapped
    _emit_versioned_exception_notification(

  File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
    self.force_reraise()

  File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
    raise self.value

  File "/usr/lib/python3/dist-packages/nova/exception_wrapper.py", line 63, in wrapped
    return f(self, context, *args, **kw)

  File "/usr/lib/python3/dist-packages/nova/compute/utils.py", line 1439, in decorated_function
    return function(self, context, *args, **kwargs)

  File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 212, in decorated_function
    compute_utils.add_instance_fault_from_exc(context,

  File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
    self.force_reraise()

  File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
    raise self.value

  File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 201, in decorated_function
    return function(self, context, *args, **kwargs)

  File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 7846, in attach_interface
    return do_attach_interface(context, instance, network_id, port_id,

  File "/usr/lib/python3/dist-packages/oslo_concurrency/lockutils.py", line 391, in inner
    return f(*args, **kwargs)

  File "/usr/lib/python3/dist-packages/nova/exception_wrapper.py", line 71, in wrapped
    _emit_versioned_exception_notification(

  File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
    self.force_reraise()

  File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
    raise self.value

  File "/usr/lib/python3/dist-packages/nova/exception_wrapper.py", line 63, in wrapped
    return f(self, context, *args, **kw)

  File "/usr/lib/python3/dist-packages/nova/compute/utils.py", line 1439, in decorated_function
    return function(self, context, *args, **kwargs)

  File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 212, in decorated_function
    compute_utils.add_instance_fault_from_exc(context,

  File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
    self.force_reraise()

  File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
    raise self.value

  File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 201, in decorated_function
    return function(self, context, *args, **kwargs)

  File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 7846, in attach_interface
    return do_attach_interface(context, instance, network_id, port_id,

  File "/usr/lib/python3/dist-packages/oslo_concurrency/lockutils.py", line 391, in inner
    return f(*args, **kwargs)

  File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 7843, in do_attach_interface
    return self._attach_interface(context, instance, network_id,

  File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 7914, in _attach_interface
    network_info = self.network_api.allocate_for_instance(

  File "/usr/lib/python3/dist-packages/nova/network/neutron.py", line 1213, in allocate_for_instance
    created_port_ids = self._update_ports_for_instance(

  File "/usr/lib/python3/dist-packages/nova/network/neutron.py", line 1351, in _update_ports_for_instance
    vif.destroy()

  File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
    self.force_reraise()

  File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
    raise self.value

  File "/usr/lib/python3/dist-packages/nova/network/neutron.py", line 1320, in _update_ports_for_instance
    updated_port = self._update_port(

  File "/usr/lib/python3/dist-packages/nova/network/neutron.py", line 584, in _update_port
    _ensure_no_port_binding_failure(port)

  File "/usr/lib/python3/dist-packages/nova/network/neutron.py", line 293, in _ensure_no_port_binding_failure
    raise exception.PortBindingFailed(port_id=port['id'])

nova.exception.PortBindingFailed: Binding failed for port ab1a4b97-b704-462f-a767-f4319ab1f278, please check neutron logs for more information.
2022-10-25 12:12:37.665 305601 ERROR nova.api.openstack.wsgi Traceback (most recent call last):
2022-10-25 12:12:37.665 305601 ERROR nova.api.openstack.wsgi File "/usr/lib/python3/dist-packages/nova/api/openstack/wsgi.py", line 658, in wrapped
2022-10-25 12:12:37.665 305601 ERROR nova.api.openstack.wsgi return f(*args, **kwargs)
2022-10-25 12:12:37.665 305601 ERROR nova.api.openstack.wsgi File "/usr/lib/python3/dist-packages/nova/api/validation/__init__.py", line 110, in wrapper
2022-10-25 12:12:37.665 305601 ERROR nova.api.openstack.wsgi return func(*args, **kwargs)
2022-10-25 12:12:37.665 305601 ERROR nova.api.openstack.wsgi File "/usr/lib/python3/dist-packages/nova/api/validation/__init__.py", line 110, in wrapper
2022-10-25 12:12:37.665 305601 ERROR nova.api.openstack.wsgi return func(*args, **kwargs)
2022-10-25 12:12:37.665 305601 ERROR nova.api.openstack.wsgi File "/usr/lib/python3/dist-packages/nova/api/openstack/compute/attach_interfaces.py", line 168, in create
2022-10-25 12:12:37.665 305601 ERROR nova.api.openstack.wsgi vif = self.compute_api.attach_interface(context,
2022-10-25 12:12:37.665 305601 ERROR nova.api.openstack.wsgi File "/usr/lib/python3/dist-packages/nova/compute/api.py", line 238, in inner
2022-10-25 12:12:37.665 305601 ERROR nova.api.openstack.wsgi return function(self, context, instance, *args, **kwargs)
2022-10-25 12:12:37.665 305601 ERROR nova.api.openstack.wsgi File "/usr/lib/python3/dist-packages/nova/compute/api.py", line 164, in inner
2022-10-25 12:12:37.665 305601 ERROR nova.api.openstack.wsgi return f(self, context, instance, *args, **kw)
2022-10-25 12:12:37.665 305601 ERROR nova.api.openstack.wsgi File "/usr/lib/python3/dist-packages/nova/compute/api.py", line 5258, in attach_interface
2022-10-25 12:12:37.665 305601 ERROR nova.api.openstack.wsgi return self.compute_rpcapi.attach_interface(context,
2022-10-25 12:12:37.665 305601 ERROR nova.api.openstack.wsgi File "/usr/lib/python3/dist-packages/nova/compute/rpcapi.py", line 572, in attach_interface
2022-10-25 12:12:37.665 305601 ERROR nova.api.openstack.wsgi return cctxt.call(ctxt, 'attach_interface', **kw)
2022-10-25 12:12:37.665 305601 ERROR nova.api.openstack.wsgi File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/client.py", line 189, in call
2022-10-25 12:12:37.665 305601 ERROR nova.api.openstack.wsgi result = self.transport._send(
2022-10-25 12:12:37.665 305601 ERROR nova.api.openstack.wsgi File "/usr/lib/python3/dist-packages/oslo_messaging/transport.py", line 123, in _send
2022-10-25 12:12:37.665 305601 ERROR nova.api.openstack.wsgi return self._driver.send(target, ctxt, message,
2022-10-25 12:12:37.665 305601 ERROR nova.api.openstack.wsgi File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 689, in send
2022-10-25 12:12:37.665 305601 ERROR nova.api.openstack.wsgi return self._send(target, ctxt, message, wait_for_reply, timeout,
2022-10-25 12:12:37.665 305601 ERROR nova.api.openstack.wsgi File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 681, in _send
2022-10-25 12:12:37.665 305601 ERROR nova.api.openstack.wsgi raise result
2022-10-25 12:12:37.665 305601 ERROR nova.api.openstack.wsgi nova.exception_Remote.PortBindingFailed_Remote: Binding failed for port ab1a4b97-b704-462f-a767-f4319ab1f278, please check neutron logs for more information.

Test run:

https://solutions.qa.canonical.com/testruns/testRun/0c561619-ee15-46c1-8aa6-1cf9c86883dc

Rally report:

https://oil-jenkins.canonical.com/artifacts/0c561619-ee15-46c1-8aa6-1cf9c86883dc/generated/generated/openstack/tempest_result.html

Logs:

https://oil-jenkins.canonical.com/artifacts/0c561619-ee15-46c1-8aa6-1cf9c86883dc/index.html

[Billy Olsen (billy-olsen)]

This looks like the result of a change in the 20.2.0 version for Yoga. Specifically, bug #1958501 was fixed to cause Neutron to not bind the port to a dead agent in the OVN case. It was fixed in Zed release and backported to yoga/stable in https://review.opendev.org/c/openstack/neutron/+/844494.

In this case, the agent is not determined to be alive for solqa-lab1-server-35.nosilo.lab1.solutionsqa

neutron-api-plugin-ovn_1/var/log/neutron/neutron-server.log
--------------------
2022-10-25 12:12:36.952 192922 ERROR neutron.plugins.ml2.managers [req-ef882b1c-f715-41fa-89d9-fd029f38576f a12a96846bce4d16af50a3389e87a7e9 a0c0be4aa6a3407d9b695dae6ee1861f - 58f5e7f1a82c452e8affaa6b14c7b
d00 58f5e7f1a82c452e8affaa6b14c7bd00] Failed to bind port ab1a4b97-b704-462f-a767-f4319ab1f278 on host solqa-lab1-server-35.nosilo.lab1.solutionsqa for vnic_type normal using segments [{'id': '898f8692-fdb
0-46b3-8388-8a007efbac46', 'network_type': 'geneve', 'physical_network': None, 'segmentation_id': 1376, 'network_id': 'de98a060-d23a-4520-bba1-3b6092008b6f'}]
2022-10-25 12:12:36.953 192922 INFO neutron.plugins.ml2.plugin [req-ef882b1c-f715-41fa-89d9-fd029f38576f a12a96846bce4d16af50a3389e87a7e9 a0c0be4aa6a3407d9b695dae6ee1861f - 58f5e7f1a82c452e8affaa6b14c7bd00
 58f5e7f1a82c452e8affaa6b14c7bd00] Attempt 6 to bind port ab1a4b97-b704-462f-a767-f4319ab1f278
2022-10-25 12:12:36.970 192922 WARNING neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver [req-ef882b1c-f715-41fa-89d9-fd029f38576f a12a96846bce4d16af50a3389e87a7e9 a0c0be4aa6a3407d9b695dae6ee1861f - 58f5e7f1a82c452e8affaa6b14c7bd00 58f5e7f1a82c452e8affaa6b14c7bd00] Refusing to bind port ab1a4b97-b704-462f-a767-f4319ab1f278 to dead agent: <neutron.plugins.ml2.drivers.ovn.agent.neutron_agent.ControllerAgent object at 0x7fdff23a26d0>