nova cannot create instances, because it doesn't find the network

[Marcella]

Hello, just upgraded to Wallaby.
When I try to create a new instance, I get an error on the impossibility of findings the network

NetworkNotFoundClient: Network f44c2b7f-e226-427c-9b83-7d6e7f2ca31d could not be found

openstack server create --nic net-id=abc --nic net-id=def --flavor c8r60e20 --image image2 --availability-zone pool mm-test-vm
Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'neutronclient.common.exceptions.NetworkNotFoundClient'> (HTTP 500) (Request-ID: req-78bde3da-360f-4a8c-9ef1-131885157d13)

openstack network list
+--------------------------------------+--------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ID | Name | Subnets |
+--------------------------------------+--------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 630073d3-7b08-4a76-88e0-a784bc742f8f | abc | 1fa11b31-5a12-492a-88b3-ec7a36cc801f, 34d9605e-9378-4607-b36c-b369e6e29fa5, 58cb9e59-ea8c-4bce-a2d4-848635a48d60, e1760060-12de-4a7b-930e-db0547017fd0, e355070b-d4c6-4c1b-8724-fa8616961108 |
| f44c2b7f-e226-427c-9b83-7d6e7f2ca31d | def | 1635b4c2-2eb2-44d3-bed5-3aefa8a701b6, ec571acf-08be-4d94-a8c5-fac628c64a24 |
+--------------------------------------+--------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

nova-api logs:

2021-06-30 11:31:21.558 2176 ERROR nova.api.openstack.wsgi request_ids=request_ids)
2021-06-30 11:31:21.558 2176 ERROR nova.api.openstack.wsgi neutronclient.common.exceptions.Network
NotFoundClient: Network f44c2b7f-e226-427c-9b83-7d6e7f2ca31d could not be found.
2021-06-30 11:31:21.558 2176 ERROR nova.api.openstack.wsgi Neutron server returns request_ids: ['r
eq-ce5d20a5-5037-4189-9859-029d8d59f6d3']
2021-06-30 11:31:21.558 2176 ERROR nova.api.openstack.wsgi
2021-06-30 11:31:21.559 2176 INFO nova.api.openstack.wsgi [req-06a40f8a-e5bc-4619-b771-13f1bbc3b0a
d 730fb4c240ef4097928e9b1eb2751f2d 03f47207096a45b5a08d12839d7639e5 - 069eebeda32849d784734df79232
8c91 069eebeda32849d784734df792328c91] HTTP exception thrown: Unexpected API Error. Please report
this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'neutronclient.common.exceptions.NetworkNotFoundClient'>
2021-06-30 11:31:21.560 2176 INFO nova.osapi_compute.wsgi.server [req-06a40f8a-e5bc-4619-b771-13f1
bbc3b0ad 730fb4c240ef4097928e9b1eb2751f2d 03f47207096a45b5a08d12839d7639e5 - 069eebeda32849d784734
df792328c91 069eebeda32849d784734df792328c91] 10.255.24.99 "POST /v2.1/servers HTTP/1.0" status: 5
00 len: 655 time: 3.9306366
2021-06-30 11:39:40.541 2178 INFO nova.osapi_compute.wsgi.server [-] 10.255.24.99 "GET /v2.1 HTTP/
1.0" status: 200 len: 767 time: 0.0010121
2021-06-30 11:39:40.633 2179 INFO nova.osapi_compute.wsgi.server [req-c9bce6f8-b1fb-4348-b7f8-2758
552eda7e 730fb4c240ef4097928e9b1eb2751f2d 03f47207096a45b5a08d12839d7639e5 - 069eebeda32849d784734
df792328c91 069eebeda32849d784734df792328c91] 10.255.24.99 "GET /v2.1/flavors/detail?is_public=Non
e HTTP/1.0" status: 200 len: 3260 time: 0.0196841
2021-06-30 11:39:52.430 2177 INFO nova.osapi_compute.wsgi.server [req-1a887ba7-3cb0-408a-9e3f-f81f
911e647a 730fb4c240ef4097928e9b1eb2751f2d 03f47207096a45b5a08d12839d7639e5 - 069eebeda32849d784734
df792328c91 069eebeda32849d784734df792328c91] 10.255.24.99 "GET /v2.1/flavors/detail?is_public=Tru
e HTTP/1.0" status: 200 len: 3260 time: 0.3669748
2021-06-30 11:40:28.447 2176 INFO nova.osapi_compute.wsgi.server [req-2d575437-ecb2-4080-859d-75e9
8d8c01fe 730fb4c240ef4097928e9b1eb2751f2d 03f47207096a45b5a08d12839d7639e5 - 069eebeda32849d784734
df792328c91 069eebeda32849d784734df792328c91] 10.255.24.99 "GET /v2.1/flavors HTTP/1.0" status: 20
0 len: 1449 time: 0.0173318
2021-06-30 11:40:30.511 2177 ERROR nova.api.openstack.wsgi [req-cca55e27-90ba-44e8-aae7-b85f9c88da
33 730fb4c240ef4097928e9b1eb2751f2d 03f47207096a45b5a08d12839d7639e5 - 069eebeda32849d784734df7923
28c91 069eebeda32849d784734df792328c91] Unexpected exception in API method: neutronclient.common.e
xceptions.NetworkNotFoundClient: Network f44c2b7f-e226-427c-9b83-7d6e7f2ca31d could not be found.
Neutron server returns request_ids: ['req-272816ee-1601-4ed2-95e0-a9119c6af1a1']
2021-06-30 11:40:30.511 2177 ERROR nova.api.openstack.wsgi Traceback (most recent call last):
2021-06-30 11:40:30.511 2177 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.6/site-package
s/nova/api/openstack/wsgi.py", line 658, in wrapped
2021-06-30 11:40:30.511 2177 ERROR nova.api.openstack.wsgi return f(*args, **kwargs)
2021-06-30 11:40:30.511 2177 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.6/site-packages/nova/api/validation/__init__.py", line 110, in wrapper
2021-06-30 11:40:30.511 2177 ERROR nova.api.openstack.wsgi return func(*args, **kwargs)
2021-06-30 11:40:30.511 2177 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.6/site-packages/nova/api/validation/__init__.py", line 110, in wrapper
2021-06-30 11:40:30.511 2177 ERROR nova.api.openstack.wsgi return func(*args, **kwargs)
2021-06-30 11:40:30.511 2177 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.6/site-packages/nova/api/validation/__init__.py", line 110, in wrapper
2021-06-30 11:40:30.511 2177 ERROR nova.api.openstack.wsgi return func(*args, **kwargs)
2021-06-30 11:40:30.511 2177 ERROR nova.api.openstack.wsgi [Previous line repeated 9 more times]
2021-06-30 11:40:30.511 2177 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.6/site-packages/nova/api/openstack/compute/servers.py", line 701, in create
2021-06-30 11:40:30.511 2177 ERROR nova.api.openstack.wsgi **create_kwargs)
2021-06-30 11:40:30.511 2177 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.6/site-packages/nova/compute/api.py", line 2076, in create
2021-06-30 11:40:30.511 2177 ERROR nova.api.openstack.wsgi requested_hypervisor_hostname=requested_hypervisor_hostname)
2021-06-30 11:40:30.511 2177 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.6/site-packages/nova/compute/api.py", line 1516, in _create_instance
2021-06-30 11:40:30.511 2177 ERROR nova.api.openstack.wsgi reservation_id, max_count, supports_port_resource_request)
2021-06-30 11:40:30.511 2177 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.6/site-packages/nova/compute/api.py", line 1038, in _validate_and_build_base_options
2021-06-30 11:40:30.511 2177 ERROR nova.api.openstack.wsgi affinity_policy=pci_numa_affinity_policy)
2021-06-30 11:40:30.511 2177 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.6/site-packages/nova/network/neutron.py", line 2159, in create_resource_requests
2021-06-30 11:40:30.511 2177 ERROR nova.api.openstack.wsgi context, neutron, network_id)
2021-06-30 11:40:30.511 2177 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.6/site-packages/nova/network/neutron.py", line 2022, in _get_physnet_tunneled_info

We have also checked the neutron server logs and we see:
ERROR stevedore.extension [req-12a806a5-ceb3-4c0c-960b-0844cda807ed 730fb4c240ef4097928e9b1eb2751f2d 03f47207096a45b5a08d12839d7639e5 - 069eebeda32849d784734df792328c91 069eebeda32849d784734df792328c91] Could not load 'oslo_cache.etcd3gw': No module named 'etcd3gw': ModuleNotFoundError: No module named 'etcd3gw'

On the machine python3-oslo-cache-2.7.0-1.el8.noarch is installed and it contains etcd3gw.py

Do you please have any help or suggestions?
Thank you in advance

[Marcella (m-arcella)]

In neutron server logs there is also such INFO, not sure if it is related.

2021-06-30 11:40:53.771 192477 INFO neutron.pecan_wsgi.hooks.translation [req-9d7ef738-94d4-4bb8-98d6-b7bae1d1f8de 730fb4c240ef4097928e9b1eb2751f2d 03f47207096a45b5a08d12839d7639e5 - 069eebeda32849d784734df792328c91 069eebeda32849d784734df792328c91] GET failed (client error): The resource could not be found.

[Lee Yarwood (lyarwood)]

Can you look for req-272816ee-1601-4ed2-95e0-a9119c6af1a1 in your neutron logs?

That's the request from Nova to Neutron that failed for you, tracing that request-id through the logs should show you why the request failed.

[Bence Romsics (bence-romsics)]

Beyond the logs Lee suggested, at what time do you see the etcd3gw related error message? At neutron-server startup or later during every try to create an instance?

What relevant python packages does your system have?

pip3 freeze | egrep 'oslo\.cache|etcd3'

Also do you have a [cache] section in your neutron configuration? If yes, how does that look like?

[Marcella (m-arcella)]

Hello, good day. We managed to fix it in the meantime.
It seems that the olso cache was not installed, but something else.

Thanks and regards.

[Launchpad Janitor (janitor)]

[Expired for OpenStack Compute (nova) because there has been no activity for 60 days.]

[Launchpad Janitor (janitor)]

[Expired for neutron because there has been no activity for 60 days.]