Hi, Following ticket has been reported in https://bugzilla.redhat.com/show_bug.cgi?id=1964893
Based on their answer, verified the neutron command manually and it works fine.
The issue is occurring randomly and failing the instance creation usecase.
Recently had failed 2 cases,
1) it failed once out of 72 runs
2021-06-09 10:00:49.391 3194 INFO heat.engine.resource [req-6d38438b-be89-4619-8517-9edc49bc7a40 - instTermProject-1 - default default] CREATE: Net "vm_network_3" [5776a857-4316-47f9-b3fa-3a1fbff39132] Stack "vapp-Service-060821210256-1" [c7fa51bb-2012-4f5c-aa6f-1720dd47a5ed]
2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource Traceback (most recent call last):
2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 920, in _action_recorder
2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource yield
2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 1033, in _do_action
2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource yield self.action_handler_task(action, args=handler_args)
2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/scheduler.py", line 346, in wrapper
2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource step = next(subtask)
2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 982, in action_handler_task
2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource done = check(handler_data)
2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resources/openstack/neutron/net.py", line 208, in check_create_complete
2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource attributes = self._show_resource()
2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resources/openstack/neutron/neutron.py", line 139, in _show_resource
2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource res_info = client_method(*args)
2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 822, in show_network
2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource return self.get(self.network_path % (network), params=_params)
2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 354, in get
2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource headers=headers, params=params)
2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 331, in retry_request
2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource headers=headers, params=params)
2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 294, in do_request
2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource self._handle_fault_response(status_code, replybody, resp)
2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 269, in _handle_fault_response
2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource exception_handler_v20(status_code, error_body)
2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 93, in exception_handler_v20
2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource request_ids=request_ids)
2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource Unauthorized: 401-{u'error': {u'message': u'The request you have made requires authentication.', u'code': 401, u'title': u'Unauthorized'}}
2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource Neutron server returns request_ids: ['req-8eb8d67b-1b44-4caf-8c5a-37a442494b31']
2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource
2021-06-09 10:00:49.429 3194 INFO heat.engine.stack [req-6d38438b-be89-4619-8517-9edc49bc7a40 - instTermProject-1 - default default] Stack CREATE FAILED (vapp-Service-060821210256-1): Resource CREATE failed: Unauthorized: resources.vm_network_3: 401-{u'error': {u'message': u'The request you have made requires authentication.', u'code': 401, u'title': u'Unauthorized'}}
Neutron server returns request_ids: ['req-8eb8d67b-1b44-4caf-8c5a-37a442494b31']
2) it failed once out of 72 runs
2021-06-10 10:01:27.888 13501 INFO nova.compute.manager [req-c531e9c8-49b7-4e16-a06c-813cf32067d6 289cd17405e846c390fbb62a92a8adb9 53e47db492134bf4b58f3d2aadafb639 - default default] [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] Took 8.84 seconds to spawn the instance on the hypervisor.
2021-06-10 10:01:28.034 13501 ERROR nova.network.neutronv2.api [req-c531e9c8-49b7-4e16-a06c-813cf32067d6 289cd17405e846c390fbb62a92a8adb9 53e47db492134bf4b58f3d2aadafb639 - default default] Neutron client was not able to generate a valid admin token, please verify Neutron admin credential located in nova.conf: Unauthorized: 401-{u'error': {u'message': u'The request you have made requires authentication.', u'code': 401, u'title': u'Unauthorized'}}
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [req-c531e9c8-49b7-4e16-a06c-813cf32067d6 289cd17405e846c390fbb62a92a8adb9 53e47db492134bf4b58f3d2aadafb639 - default default] [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] Failed to deallocate networks: NeutronAdminCredentialConfigurationInvalid: Networking client is experiencing an unauthorized exception.
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] Traceback (most recent call last):
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2746, in _cleanup_allocated_networks
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] self._deallocate_network(context, instance, requested_networks)
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2002, in _deallocate_network
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] context, instance, requested_networks=requested_networks)
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1570, in deallocate_for_instance
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] data = neutron.list_ports(**search_opts)
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 121, in wrapper
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] ret = obj(*args, **kwargs)
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 795, in list_ports
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] **_params)
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 121, in wrapper
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] ret = obj(*args, **kwargs)
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 369, in list
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] for r in self._pagination(collection, path, **params):
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 384, in _pagination
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] res = self.get(path, params=params)
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 121, in wrapper
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] ret = obj(*args, **kwargs)
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 354, in get
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] headers=headers, params=params)
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 121, in wrapper
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] ret = obj(*args, **kwargs)
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 331, in retry_request
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] headers=headers, params=params)
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 121, in wrapper
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] ret = obj(*args, **kwargs)
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 294, in do_request
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] self._handle_fault_response(status_code, replybody, resp)
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 137, in wrapper
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] raise exception.NeutronAdminCredentialConfigurationInvalid()
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] NeutronAdminCredentialConfigurationInvalid: Networking client is experiencing an unauthorized exception.
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71]
2021-06-10 10:01:28.154 13501 INFO nova.scheduler.client.report [req-c531e9c8-49b7-4e16-a06c-813cf32067d6 289cd17405e846c390fbb62a92a8adb9 53e47db492134bf4b58f3d2aadafb639 - default default] Deleted allocation for instance f0dc327b-1416-400f-9846-86b9ef0dfc71
I did tried with neutron auth config from nova.conf manually and it works fine.
No similar issue observed with Queens openstack version.
Also many errors related to rabbitmq
2021-06-11 03:28:26.028 26179 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer
2021-06-11 03:28:26.034 26158 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer
2021-06-11 03:28:26.056 26179 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer
2021-06-11 03:28:26.057 26158 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer
openstack-nova-novncproxy-20.6.0-1.el7.noarch
openstack-nova-api-20.6.0-1.el7.noarch
openstack-nova-common-20.6.0-1.el7.noarch
puppet-nova-15.7.0-1.el7.noarch
python2-nova-20.6.0-1.el7.noarch
openstack-nova-scheduler-20.6.0-1.el7.noarch
openstack-nova-conductor-20.6.0-1.el7.noarch
python2-novaclient-15.1.1-1.el7.noarch
openstack-neutron-common-15.3.2-1.el7.noarch
puppet-neutron-15.5.0-1.el7.noarch
python2-neutronclient-6.14.1-1.el7.noarch
openstack-neutron-metering-agent-15.3.2-1.el7.noarch
openstack-neutron-15.3.2-1.el7.noarch
openstack-neutron-openvswitch-15.3.2-1.el7.noarch
python2-neutron-lib-1.29.1-1.el7.noarch
openstack-neutron-ml2-15.3.2-1.el7.noarch
python2-neutron-15.3.2-1.el7.noarch
openstack-keystone-16.0.1-1.el7.noarch
python2-keystonemiddleware-7.0.1-2.el7.noarch
puppet-keystone-15.4.0-1.el7.noarch
python2-keystoneclient-3.21.0-2.el7.noarch
python2-keystoneauth1-3.17.3-1.el7.noarch
python2-keystone-16.0.1-1.el7.noarch
Could you please check and let know if any config needs to be changed?
Best Regards
Hi,
Could you please check and suggest the way forward please? as the failures happening more occurrences recently.
Best Regards