Comment 10 for bug 1849802

Revision history for this message
Alexey Stupnikov (astupnikov) wrote :

I have used my lab to understand original report and what is actually going on. Shortly speaking, Nova is unable to live migrate an instance because Neutron fails to create port binding on destination host (it already exists) when Nova Conductor tries to create port binding on destination host.

There could be multiple causes of such behavior and this bug is not about solving a root cause. Instead, the problem is that it is hard to isolate the issue using Neutron Server logs.

When debug is enabled Neutron generates logs [1] for such kinds of requests. It looks like for some reason the following exception raised by Neutron is not logged properly on Neutron side: we can see it in Nova logs, but not in Neutron Server logs. https://github.com/openstack/neutron/blob/master/neutron/plugins/ml2/plugin.py#L2463-L2466

As a result, it looks like this problem should be solved by improving logging for Neutron extensions in general or this particular function.

With that being said, I am changing affected product to Neutron and unassigning this bug.

[1]
  2021-10-18 14:15:58.081 18 DEBUG neutron.api.v2.base [req-c237e714-0085-47c6-abab-e7e15cec7ea1 d79d0393fbb04564bc8fd3c62d290087 b8715d57125f4787a6701319d38f61e3 - default default] Request body: {'binding': {'host': 'compute-0.redhat.local', 'vnic_type': 'normal', 'profile': {}}} prepare_request_body /usr/lib/python3.6/site-packages/neutron/api/v2/base.py:719
  2021-10-18 14:15:58.081 18 DEBUG neutron.api.v2.base [req-c237e714-0085-47c6-abab-e7e15cec7ea1 d79d0393fbb04564bc8fd3c62d290087 b8715d57125f4787a6701319d38f61e3 - default default] Unknown quota resources ['binding']. _create /usr/lib/python3.6/site-packages/neutron/api/v2/base.py:490
  2021-10-18 14:15:58.135 18 INFO neutron.api.v2.resource [req-c237e714-0085-47c6-abab-e7e15cec7ea1 d79d0393fbb04564bc8fd3c62d290087 b8715d57125f4787a6701319d38f61e3 - default default] create failed (client error): There was a conflict when trying to complete your request.
  2021-10-18 14:15:58.137 18 INFO neutron.wsgi [req-c237e714-0085-47c6-abab-e7e15cec7ea1 d79d0393fbb04564bc8fd3c62d290087 b8715d57125f4787a6701319d38f61e3 - default default] 172.17.1.17 "POST /v2.0/ports/7542a977-0586-423a-ae35-86e3ff791060/bindings HTTP/1.1" status: 409 len: 364 time: 0.3800023
  2021-10-18 14:15:58.316 21 DEBUG neutron_lib.callbacks.manager [req-334ebddd-4e81-4c12-829c-64f3b0a278ff - - - - -] Notify callbacks ['neutron.services.segments.db._update_segment_host_mapping_for_agent-8793714910767', 'neutron.plugins.ml2.plugin.Ml2Plugin._retry_binding_revived_agents-16758855'] for agent, after_update _notify_loop /usr/lib/python3.6/site-packages/neutron_lib/callbacks/manager.py:193