OpenStack Compute (nova)

Bug #1978065
Comment #5

Comment 5 for bug 1978065

Revision history for this message

YG Kumar (ygk-kmr) wrote on 2022-06-14:

Jun 14 09:05:41 test039-01-nova-api-container-c7d6cb54 nova-conductor[211058]: 2022-06-14 09:05:41.874 211058 ERROR nova.scheduler.utils [req-63606504-ed6a-4f01-bce2-f0f827f61431 6786d6082bdd4e7b81b9a49d50feccba f5446845ac2349c7ac7bd85aa8b50304 - default default] [instance: 2f3e777c-c5a9-4dd9-95ce-91fc9d904be7] Error from last host: test039-16 (node test039-16.prod.test.com): ['Traceback (most recent call last):\n', ' File "/openstack/venvs/nova-23.2.0/lib/python3.8/site-packages/nova/compute/manager.py", line 2409, in _build_and_run_instance\n self.driver.spawn(context, instance, image_meta,\n', ' File "/openstack/venvs/nova-23.2.0/lib/python3.8/site-packages/nova/virt/libvirt/driver.py", line 4189, in spawn\n xml = self._get_guest_xml(context, instance, network_info,\n', ' File "/openstack/venvs/nova-23.2.0/lib/python3.8/site-packages/nova/virt/libvirt/driver.py", line 7020, in _get_guest_xml\n network_info_str = str(network_info)\n', ' File "/openstack/venvs/nova-23.2.0/lib/python3.8/site-packages/nova/network/model.py", line 617, in __str__\n return self._sync_wrapper(fn, *args, **kwargs)\n', ' File "/openstack/venvs/nova-23.2.0/lib/python3.8/site-packages/nova/network/model.py", line 600, in _sync_wrapper\n self.wait()\n', ' File "/openstack/venvs/nova-23.2.0/lib/python3.8/site-packages/nova/network/model.py", line 632, in wait\n self[:] = self._gt.wait()\n', ' File "/openstack/venvs/nova-23.2.0/lib/python3.8/site-packages/eventlet/greenthread.py", line 181, in wait\n return self._exit_event.wait()\n', ' File "/openstack/venvs/nova-23.2.0/lib/python3.8/site-packages/eventlet/event.py", line 125, in wait\n result = hub.switch()\n', ' File "/openstack/venvs/nova-23.2.0/lib/python3.8/site-packages/eventlet/hubs/hub.py", line 313, in switch\n return self.greenlet.switch()\n', ' File "/openstack/venvs/nova-23.2.0/lib/python3.8/site-packages/eventlet/greenthread.py", line 221, in main\n result = function(*args, **kwargs)\n', ' File "/openstack/venvs/nova-23.2.0/lib/python3.8/site-packages/nova/utils.py", line 660, in context_wrapper\n return func(*args, **kwargs)\n', ' File "/openstack/venvs/nova-23.2.0/lib/python3.8/site-packages/nova/compute/manager.py", line 1780, in _allocate_network_async\n raise e\n', ' File "/openstack/venvs/nova-23.2.0/lib/python3.8/site-packages/nova/compute/manager.py", line 1759, in _allocate_network_async\n nwinfo = self.network_api.allocate_for_instance(\n', ' File "/openstack/venvs/nova-23.2.0/lib/python3.8/site-packages/nova/network/neutron.py", line 1092, in allocate_for_instance\n created_port_ids = self._update_ports_for_instance(\n', ' File "/openstack/venvs/nova-23.2.0/lib/python3.8/site-packages/nova/network/neutron.py", line 1224, in _update_ports_for_instance\n vif.destroy()\n', ' File "/openstack/venvs/nova-23.2.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/openstack/venvs/nova-23.2.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/openstack/venvs/nova-23.2.0/lib/python3.8/site-packages/nova/network/neutron.py", line 1193, in _update_ports_for_instance\n updated_port = self._update_port(\n', ' File "/openstack/venvs/nova-23.2.0/lib/python3.8/site-packages/nova/network/neutron.py", line 584, in _update_port\n _ensure_no_port_binding_failure(port)\n', ' File "/openstack/venvs/nova-23.2.0/lib/python3.8/site-packages/nova/network/neutron.py", line 298, in _ensure_no_port_binding_failure\n raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port b70f8b86-0df3-4d94-a19c-ee45a88063c7, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/openstack/venvs/nova-23.2.0/lib/python3.8/site-packages/nova/compute/manager.py", line 2232, in _do_build_and_run_instance\n self._build_and_run_instance(context, instance, image,\n', ' File "/openstack/venvs/nova-23.2.0/lib/python3.8/site-packages/nova/compute/manager.py", line 2505, in _build_and_run_instance\n raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 2f3e777c-c5a9-4dd9-95ce-91fc9d904be7 was re-scheduled: Binding failed for port b70f8b86-0df3-4d94-a19c-ee45a88063c7, please check neutron logs for more information.\n']

Jun 14 09:06:26 test039-01-nova-api-container-c7d6cb54 nova-conductor[211059]: 2022-06-14 09:06:26.540 211059 WARNING oslo_messaging._drivers.amqpdriver [req-63606504-ed6a-4f01-bce2-f0f827f61431 6786d6082bdd4e7b81b9a49d50feccba f5446845ac2349c7ac7bd85aa8b50304 - default default] reply_224f3b0323bb41209f07a0a8f372f103 doesn't exist, drop reply to 5473b0de85b14193a62216d1e65b324a: oslo_messaging.exceptions.MessageUndeliverable
Jun 14 09:06:26 test039-01-nova-api-container-c7d6cb54 nova-conductor[211059]: 2022-06-14 09:06:26.543 211059 ERROR oslo_messaging._drivers.amqpdriver [req-63606504-ed6a-4f01-bce2-f0f827f61431 6786d6082bdd4e7b81b9a49d50feccba f5446845ac2349c7ac7bd85aa8b50304 - default default] The reply 5473b0de85b14193a62216d1e65b324a failed to send after 60 seconds due to a missing queue (reply_224f3b0323bb41209f07a0a8f372f103). Abandoning...: oslo_messaging.exceptions.MessageUndeliverable
Jun 14 09:26:28 test039-01-nova-api-container-c7d6cb54 nova-api-wsgi[175873]: 2022-06-14 09:26:28.021 175873 INFO nova.api.openstack.requestlog [req-ab30cb5f-05a4-4e20-91c8-8e1e44ae163e 6786d6082bdd4e7b81b9a49d50feccba f5446845ac2349c7ac7bd85aa8b50304 - default default] 10.167.37.198 "GET /v2.1/servers/3b21ada9-9770-4772-8be7-fbef53f4896a/os-instance-actions/req-63606504-ed6a-4f01-bce2-f0f827f61431" status: 200 len: 503 microversion: 2.1 time: 0.105379

Jun 14 09:05:41 test039-01-nova-api-container-c7d6cb54 nova-conductor[211058]: 2022-06-14 09:05:41.874 211058 ERROR nova.scheduler.utils [req-63606504-ed6a-4f01-bce2-f0f827f61431 6786d6082bdd4e7b81b9a49d50feccba f5446845ac2349c7ac7bd85aa8b50304 - default default] [instance: 2f3e777c-c5a9-4dd9-95ce-91fc9d904be7] Error from last host: test039-16 (node test039-16.prod.test.com): ['Traceback (most recent call last):\n', '  File "/openstack/venvs/nova-23.2.0/lib/python3.8/site-packages/nova/compute/manager.py", line 2409, in _build_and_run_instance\n    self.driver.spawn(context, instance, image_meta,\n', '  File "/openstack/venvs/nova-23.2.0/lib/python3.8/site-packages/nova/virt/libvirt/driver.py", line 4189, in spawn\n    xml = self._get_guest_xml(context, instance, network_info,\n', '  File "/openstack/venvs/nova-23.2.0/lib/python3.8/site-packages/nova/virt/libvirt/driver.py", line 7020, in _get_guest_xml\n    network_info_str = str(network_info)\n', '  File "/openstack/venvs/nova-23.2.0/lib/python3.8/site-packages/nova/network/model.py", line 617, in __str__\n    return self._sync_wrapper(fn, *args, **kwargs)\n', '  File "/openstack/venvs/nova-23.2.0/lib/python3.8/site-packages/nova/network/model.py", line 600, in _sync_wrapper\n    self.wait()\n', '  File "/openstack/venvs/nova-23.2.0/lib/python3.8/site-packages/nova/network/model.py", line 632, in wait\n    self[:] = self._gt.wait()\n', '  File "/openstack/venvs/nova-23.2.0/lib/python3.8/site-packages/eventlet/greenthread.py", line 181, in wait\n    return self._exit_event.wait()\n', '  File "/openstack/venvs/nova-23.2.0/lib/python3.8/site-packages/eventlet/event.py", line 125, in wait\n    result = hub.switch()\n', '  File "/openstack/venvs/nova-23.2.0/lib/python3.8/site-packages/eventlet/hubs/hub.py", line 313, in switch\n    return self.greenlet.switch()\n', '  File "/openstack/venvs/nova-23.2.0/lib/python3.8/site-packages/eventlet/greenthread.py", line 221, in main\n    result = function(*args, **kwargs)\n', '  File "/openstack/venvs/nova-23.2.0/lib/python3.8/site-packages/nova/utils.py", line 660, in context_wrapper\n    return func(*args, **kwargs)\n', '  File "/openstack/venvs/nova-23.2.0/lib/python3.8/site-packages/nova/compute/manager.py", line 1780, in _allocate_network_async\n    raise e\n', '  File "/openstack/venvs/nova-23.2.0/lib/python3.8/site-packages/nova/compute/manager.py", line 1759, in _allocate_network_async\n    nwinfo = self.network_api.allocate_for_instance(\n', '  File "/openstack/venvs/nova-23.2.0/lib/python3.8/site-packages/nova/network/neutron.py", line 1092, in allocate_for_instance\n    created_port_ids = self._update_ports_for_instance(\n', '  File "/openstack/venvs/nova-23.2.0/lib/python3.8/site-packages/nova/network/neutron.py", line 1224, in _update_ports_for_instance\n    vif.destroy()\n', '  File "/openstack/venvs/nova-23.2.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n    self.force_reraise()\n', '  File "/openstack/venvs/nova-23.2.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n    raise self.value\n', '  File "/openstack/venvs/nova-23.2.0/lib/python3.8/site-packages/nova/network/neutron.py", line 1193, in _update_ports_for_instance\n    updated_port = self._update_port(\n', '  File "/openstack/venvs/nova-23.2.0/lib/python3.8/site-packages/nova/network/neutron.py", line 584, in _update_port\n    _ensure_no_port_binding_failure(port)\n', '  File "/openstack/venvs/nova-23.2.0/lib/python3.8/site-packages/nova/network/neutron.py", line 298, in _ensure_no_port_binding_failure\n    raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port b70f8b86-0df3-4d94-a19c-ee45a88063c7, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', '  File "/openstack/venvs/nova-23.2.0/lib/python3.8/site-packages/nova/compute/manager.py", line 2232, in _do_build_and_run_instance\n    self._build_and_run_instance(context, instance, image,\n', '  File "/openstack/venvs/nova-23.2.0/lib/python3.8/site-packages/nova/compute/manager.py", line 2505, in _build_and_run_instance\n    raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 2f3e777c-c5a9-4dd9-95ce-91fc9d904be7 was re-scheduled: Binding failed for port b70f8b86-0df3-4d94-a19c-ee45a88063c7, please check neutron logs for more information.\n']