| 2023-11-17 15:03:26 |
Jan Gutter |
description |
Since tooz 4.3.0 was released, grenade-skip-level-always is broken[1].
[1] https://zuul.opendev.org/t/openstack/builds?job_name=grenade-skip-level-always&project=openstack/nova
The root cause is that cinder fails to connect the etcd server because of the following error.
2023-11-17 06:29:24.781 | Nov 17 06:29:20 np0035791906 devstack@c-api.service[169832]: ERROR cinder File "/usr/local/lib/python3.10/dist-packages/tooz/drivers/etcd3gw.py", line 235, in _start
2023-11-17 06:29:24.781 | Nov 17 06:29:20 np0035791906 devstack@c-api.service[169832]: ERROR cinder self._membership_lease = self.client.lease(self.membership_timeout)
2023-11-17 06:29:24.781 | Nov 17 06:29:20 np0035791906 devstack@c-api.service[169832]: ERROR cinder File "/usr/local/lib/python3.10/dist-packages/etcd3gw/client.py", line 123, in lease
2023-11-17 06:29:24.781 | Nov 17 06:29:20 np0035791906 devstack@c-api.service[169832]: ERROR cinder result = self.post(self.get_url("/lease/grant"),
2023-11-17 06:29:24.781 | Nov 17 06:29:20 np0035791906 devstack@c-api.service[169832]: ERROR cinder File "/usr/local/lib/python3.10/dist-packages/etcd3gw/client.py", line 93, in post
2023-11-17 06:29:24.781 | Nov 17 06:29:20 np0035791906 devstack@c-api.service[169832]: ERROR cinder raise exceptions.Etcd3Exception(resp.text, resp.reason)
2023-11-17 06:29:24.781 | Nov 17 06:29:20 np0035791906 devstack@c-api.service[169832]: ERROR cinder etcd3gw.exceptions.Etcd3Exception: Not Found
2023-11-17 06:29:24.781 | Nov 17 06:29:20 np0035791906 devstack@c-api.service[169832]: ERROR cinder
2023-11-17 06:29:24.781 | Nov 17 06:29:20 np0035791906 devstack@c-api.service[169832]: unable to load app 0 (mountpoint='') (callable not found or import error)
When the node is deployed initially it is deplyoed with etcd v3.3.12.
The coordination_url parameter in cinder.conf does not contain api_version but tooz 3.2.0 used v1alpha by default which is available in etcd v3.3.12.
The problem arises when all components, including cinder are upgraded with etcd is kept to v3.3.12.
The coordination_url in cinder does not contain api_version at this point.
However the tooz 4.3.0 uses v1 API by default but this is not available in etcd v3.3.12 and eventually results in the above failure.
Although we added a release note to explain this requirement during upgrade, it's much better if we can implement some logic to avoid needing operators to update the config file. |
Since tooz 4.3.0 was released, grenade-skip-level-always is broken[1].
[1] https://zuul.opendev.org/t/openstack/builds?job_name=grenade-skip-level-always&project=openstack/nova
The root cause is that cinder fails to connect the etcd server because of the following error.
2023-11-17 06:29:24.781 | Nov 17 06:29:20 np0035791906 devstack@c-api.service[169832]: ERROR cinder File "/usr/local/lib/python3.10/dist-packages/tooz/drivers/etcd3gw.py", line 235, in _start
2023-11-17 06:29:24.781 | Nov 17 06:29:20 np0035791906 devstack@c-api.service[169832]: ERROR cinder self._membership_lease = self.client.lease(self.membership_timeout)
2023-11-17 06:29:24.781 | Nov 17 06:29:20 np0035791906 devstack@c-api.service[169832]: ERROR cinder File "/usr/local/lib/python3.10/dist-packages/etcd3gw/client.py", line 123, in lease
2023-11-17 06:29:24.781 | Nov 17 06:29:20 np0035791906 devstack@c-api.service[169832]: ERROR cinder result = self.post(self.get_url("/lease/grant"),
2023-11-17 06:29:24.781 | Nov 17 06:29:20 np0035791906 devstack@c-api.service[169832]: ERROR cinder File "/usr/local/lib/python3.10/dist-packages/etcd3gw/client.py", line 93, in post
2023-11-17 06:29:24.781 | Nov 17 06:29:20 np0035791906 devstack@c-api.service[169832]: ERROR cinder raise exceptions.Etcd3Exception(resp.text, resp.reason)
2023-11-17 06:29:24.781 | Nov 17 06:29:20 np0035791906 devstack@c-api.service[169832]: ERROR cinder etcd3gw.exceptions.Etcd3Exception: Not Found
2023-11-17 06:29:24.781 | Nov 17 06:29:20 np0035791906 devstack@c-api.service[169832]: ERROR cinder
2023-11-17 06:29:24.781 | Nov 17 06:29:20 np0035791906 devstack@c-api.service[169832]: unable to load app 0 (mountpoint='') (callable not found or import error)
When the node is deployed initially it is deployed with etcd v3.3.12.
The coordination_url parameter in cinder.conf does not contain api_version but tooz 3.2.0 used v1alpha by default which is available in etcd v3.3.12.
The problem arises when all components, including cinder are upgraded while etcd is kept at v3.3.12.
The coordination_url in cinder does not contain api_version at this point.
However the tooz 4.3.0 uses v3 API by default but this is not available in etcd v3.3.12 and eventually results in the above failure.
Although we added a release note to explain this requirement during upgrade, it's much better if we can implement some logic to avoid needing operators to update the config file. |
|
