2023-04-10 15:05:41 |
Felipe Reyes |
description |
[Impact]
In a fresh jammy-antelope environment the metadata service is not available to the overcloud deployed instances, this is an environment using neutron-openvswitch environment (not OVN), when looking into the logs the stacktrace below is suspicious. The metadata service becomes available when restarting the services (e.g. juju config neutron-openvswitch debug=True)
[Test Case]
```
git clone https://opendev.org/openstack/charm-neutron-openvswitch
cd charm-neutron-openvswitch
git review -d https://review.opendev.org/c/openstack/charm-neutron-openvswitch/+/873819
tox -e build # make sure charmcraft-2.1 is installed before
tox -e func-target -- jammy-antelope
```
[Stacktrace]
2023-03-31 19:35:06.093 58625 DEBUG neutron.agent.dhcp.agent [-] Calling driver for network: 6d246d86-11b5-4d5f-aa9c-c2bcbcc28b62/seg=None action: get_metadata_bind_interface _call_driver /usr/lib/python3/dist-packages/neutron/agent/dhcp/agent.py:242
2023-03-31 19:35:06.095 58625 ERROR neutron.agent.dhcp.agent [-] 'bool' object is not subscriptable: TypeError: 'bool' object is not subscriptable
2023-03-31 19:35:06.095 58625 ERROR neutron.agent.dhcp.agent Traceback (most recent call last):
2023-03-31 19:35:06.095 58625 ERROR neutron.agent.dhcp.agent File "/usr/lib/python3/dist-packages/neutron/common/utils.py", line 182, in call
2023-03-31 19:35:06.095 58625 ERROR neutron.agent.dhcp.agent return func(*args, **kwargs)
2023-03-31 19:35:06.095 58625 ERROR neutron.agent.dhcp.agent File "/usr/lib/python3/dist-packages/neutron/agent/dhcp/agent.py", line 434, in safe_configure_dhcp_for_network
2023-03-31 19:35:06.095 58625 ERROR neutron.agent.dhcp.agent self.configure_dhcp_for_network(network)
2023-03-31 19:35:06.095 58625 ERROR neutron.agent.dhcp.agent File "/usr/lib/python3/dist-packages/osprofiler/profiler.py", line 159, in wrapper
2023-03-31 19:35:06.095 58625 ERROR neutron.agent.dhcp.agent result = f(*args, **kwargs)
2023-03-31 19:35:06.095 58625 ERROR neutron.agent.dhcp.agent File "/usr/lib/python3/dist-packages/neutron/agent/dhcp/agent.py", line 447, in configure_dhcp_for_network
2023-03-31 19:35:06.095 58625 ERROR neutron.agent.dhcp.agent self.update_isolated_metadata_proxy(network)
2023-03-31 19:35:06.095 58625 ERROR neutron.agent.dhcp.agent File "/usr/lib/python3/dist-packages/osprofiler/profiler.py", line 159, in wrapper
2023-03-31 19:35:06.095 58625 ERROR neutron.agent.dhcp.agent result = f(*args, **kwargs)
2023-03-31 19:35:06.095 58625 ERROR neutron.agent.dhcp.agent File "/usr/lib/python3/dist-packages/neutron/agent/dhcp/agent.py", line 763, in update_isolated_metadata_proxy
2023-03-31 19:35:06.095 58625 ERROR neutron.agent.dhcp.agent self.enable_isolated_metadata_proxy(network)
2023-03-31 19:35:06.095 58625 ERROR neutron.agent.dhcp.agent File "/usr/lib/python3/dist-packages/osprofiler/profiler.py", line 159, in wrapper
2023-03-31 19:35:06.095 58625 ERROR neutron.agent.dhcp.agent result = f(*args, **kwargs)
2023-03-31 19:35:06.095 58625 ERROR neutron.agent.dhcp.agent File "/usr/lib/python3/dist-packages/neutron/agent/dhcp/agent.py", line 819, in enable_isolated_metadata_proxy
2023-03-31 19:35:06.095 58625 ERROR neutron.agent.dhcp.agent metadata_driver.MetadataDriver.spawn_monitored_metadata_proxy(
2023-03-31 19:35:06.095 58625 ERROR neutron.agent.dhcp.agent File "/usr/lib/python3/dist-packages/neutron/agent/metadata/driver.py", line 244, in spawn_monitored_metadata_proxy
2023-03-31 19:35:06.095 58625 ERROR neutron.agent.dhcp.agent ip_lib.IpAddrCommand(
2023-03-31 19:35:06.095 58625 ERROR neutron.agent.dhcp.agent File "/usr/lib/python3/dist-packages/neutron/agent/linux/ip_lib.py", line 609, in wait_until_address_ready
2023-03-31 19:35:06.095 58625 ERROR neutron.agent.dhcp.agent common_utils.wait_until_true(
2023-03-31 19:35:06.095 58625 ERROR neutron.agent.dhcp.agent File "/usr/lib/python3/dist-packages/neutron/common/utils.py", line 744, in wait_until_true
2023-03-31 19:35:06.095 58625 ERROR neutron.agent.dhcp.agent while not predicate():
2023-03-31 19:35:06.095 58625 ERROR neutron.agent.dhcp.agent File "/usr/lib/python3/dist-packages/neutron/agent/linux/ip_lib.py", line 594, in is_address_ready
2023-03-31 19:35:06.095 58625 ERROR neutron.agent.dhcp.agent addr_info = self.list(to=address)[0]
2023-03-31 19:35:06.095 58625 ERROR neutron.agent.dhcp.agent File "/usr/lib/python3/dist-packages/neutron/agent/linux/ip_lib.py", line 574, in list
2023-03-31 19:35:06.095 58625 ERROR neutron.agent.dhcp.agent devices = get_devices_with_ip(self._parent.namespace, name=self.name,
2023-03-31 19:35:06.095 58625 ERROR neutron.agent.dhcp.agent File "/usr/lib/python3/dist-packages/neutron/agent/linux/ip_lib.py", line 431, in name
2023-03-31 19:35:06.095 58625 ERROR neutron.agent.dhcp.agent return self._parent.name
2023-03-31 19:35:06.095 58625 ERROR neutron.agent.dhcp.agent File "/usr/lib/python3/dist-packages/neutron/agent/linux/ip_lib.py", line 416, in name
2023-03-31 19:35:06.095 58625 ERROR neutron.agent.dhcp.agent return self._name[:constants.DEVICE_NAME_MAX_LEN]
2023-03-31 19:35:06.095 58625 ERROR neutron.agent.dhcp.agent TypeError: 'bool' object is not subscriptable
2023-03-31 19:35:06.095 58625 ERROR neutron.agent.dhcp.agent
2023-03-31 19:35:06.099 58625 INFO neutron.agent.dhcp.agent [None req-4298bc12-95ea-476c-8503-7258e4454721 - - - - - -] Synchronizing state complete |
In a fresh environment running Antelope (on top of Ubuntu 22.04), with a DVR configuration (neutron-dhcp-agent and neutron-metadata-agent running on compute nodes) the metadata service is not available instances, this is an environment using neutron-openvswitch environment (not OVN), when looking into the logs the stacktrace below indicates an unexpected data type.
[Stacktrace]
2023-03-31 19:35:06.093 58625 DEBUG neutron.agent.dhcp.agent [-] Calling driver for network: 6d246d86-11b5-4d5f-aa9c-c2bcbcc28b62/seg=None action: get_metadata_bind_interface _call_driver /usr/lib/python3/dist-packages/neutron/agent/dhcp/agent.py:242
2023-03-31 19:35:06.095 58625 ERROR neutron.agent.dhcp.agent [-] 'bool' object is not subscriptable: TypeError: 'bool' object is not subscriptable
2023-03-31 19:35:06.095 58625 ERROR neutron.agent.dhcp.agent Traceback (most recent call last):
2023-03-31 19:35:06.095 58625 ERROR neutron.agent.dhcp.agent File "/usr/lib/python3/dist-packages/neutron/common/utils.py", line 182, in call
2023-03-31 19:35:06.095 58625 ERROR neutron.agent.dhcp.agent return func(*args, **kwargs)
2023-03-31 19:35:06.095 58625 ERROR neutron.agent.dhcp.agent File "/usr/lib/python3/dist-packages/neutron/agent/dhcp/agent.py", line 434, in safe_configure_dhcp_for_network
2023-03-31 19:35:06.095 58625 ERROR neutron.agent.dhcp.agent self.configure_dhcp_for_network(network)
2023-03-31 19:35:06.095 58625 ERROR neutron.agent.dhcp.agent File "/usr/lib/python3/dist-packages/osprofiler/profiler.py", line 159, in wrapper
2023-03-31 19:35:06.095 58625 ERROR neutron.agent.dhcp.agent result = f(*args, **kwargs)
2023-03-31 19:35:06.095 58625 ERROR neutron.agent.dhcp.agent File "/usr/lib/python3/dist-packages/neutron/agent/dhcp/agent.py", line 447, in configure_dhcp_for_network
2023-03-31 19:35:06.095 58625 ERROR neutron.agent.dhcp.agent self.update_isolated_metadata_proxy(network)
2023-03-31 19:35:06.095 58625 ERROR neutron.agent.dhcp.agent File "/usr/lib/python3/dist-packages/osprofiler/profiler.py", line 159, in wrapper
2023-03-31 19:35:06.095 58625 ERROR neutron.agent.dhcp.agent result = f(*args, **kwargs)
2023-03-31 19:35:06.095 58625 ERROR neutron.agent.dhcp.agent File "/usr/lib/python3/dist-packages/neutron/agent/dhcp/agent.py", line 763, in update_isolated_metadata_proxy
2023-03-31 19:35:06.095 58625 ERROR neutron.agent.dhcp.agent self.enable_isolated_metadata_proxy(network)
2023-03-31 19:35:06.095 58625 ERROR neutron.agent.dhcp.agent File "/usr/lib/python3/dist-packages/osprofiler/profiler.py", line 159, in wrapper
2023-03-31 19:35:06.095 58625 ERROR neutron.agent.dhcp.agent result = f(*args, **kwargs)
2023-03-31 19:35:06.095 58625 ERROR neutron.agent.dhcp.agent File "/usr/lib/python3/dist-packages/neutron/agent/dhcp/agent.py", line 819, in enable_isolated_metadata_proxy
2023-03-31 19:35:06.095 58625 ERROR neutron.agent.dhcp.agent metadata_driver.MetadataDriver.spawn_monitored_metadata_proxy(
2023-03-31 19:35:06.095 58625 ERROR neutron.agent.dhcp.agent File "/usr/lib/python3/dist-packages/neutron/agent/metadata/driver.py", line 244, in spawn_monitored_metadata_proxy
2023-03-31 19:35:06.095 58625 ERROR neutron.agent.dhcp.agent ip_lib.IpAddrCommand(
2023-03-31 19:35:06.095 58625 ERROR neutron.agent.dhcp.agent File "/usr/lib/python3/dist-packages/neutron/agent/linux/ip_lib.py", line 609, in wait_until_address_ready
2023-03-31 19:35:06.095 58625 ERROR neutron.agent.dhcp.agent common_utils.wait_until_true(
2023-03-31 19:35:06.095 58625 ERROR neutron.agent.dhcp.agent File "/usr/lib/python3/dist-packages/neutron/common/utils.py", line 744, in wait_until_true
2023-03-31 19:35:06.095 58625 ERROR neutron.agent.dhcp.agent while not predicate():
2023-03-31 19:35:06.095 58625 ERROR neutron.agent.dhcp.agent File "/usr/lib/python3/dist-packages/neutron/agent/linux/ip_lib.py", line 594, in is_address_ready
2023-03-31 19:35:06.095 58625 ERROR neutron.agent.dhcp.agent addr_info = self.list(to=address)[0]
2023-03-31 19:35:06.095 58625 ERROR neutron.agent.dhcp.agent File "/usr/lib/python3/dist-packages/neutron/agent/linux/ip_lib.py", line 574, in list
2023-03-31 19:35:06.095 58625 ERROR neutron.agent.dhcp.agent devices = get_devices_with_ip(self._parent.namespace, name=self.name,
2023-03-31 19:35:06.095 58625 ERROR neutron.agent.dhcp.agent File "/usr/lib/python3/dist-packages/neutron/agent/linux/ip_lib.py", line 431, in name
2023-03-31 19:35:06.095 58625 ERROR neutron.agent.dhcp.agent return self._parent.name
2023-03-31 19:35:06.095 58625 ERROR neutron.agent.dhcp.agent File "/usr/lib/python3/dist-packages/neutron/agent/linux/ip_lib.py", line 416, in name
2023-03-31 19:35:06.095 58625 ERROR neutron.agent.dhcp.agent return self._name[:constants.DEVICE_NAME_MAX_LEN]
2023-03-31 19:35:06.095 58625 ERROR neutron.agent.dhcp.agent TypeError: 'bool' object is not subscriptable
2023-03-31 19:35:06.095 58625 ERROR neutron.agent.dhcp.agent
2023-03-31 19:35:06.099 58625 INFO neutron.agent.dhcp.agent [None req-4298bc12-95ea-476c-8503-7258e4454721 - - - - - -] Synchronizing state complete |
|