grenade-skip-level-always because of different api versions used by default

[Takashi Kajinami]

Since tooz 4.3.0 was released, grenade-skip-level-always is broken[1].

[1] https://zuul.opendev.org/t/openstack/builds?job_name=grenade-skip-level-always&project=openstack/nova

The root cause is that cinder fails to connect the etcd server because of the following error.

2023-11-17 06:29:24.781 | Nov 17 06:29:20 np0035791906 <email address hidden>[169832]: ERROR cinder File "/usr/local/lib/python3.10/dist-packages/tooz/drivers/etcd3gw.py", line 235, in _start
2023-11-17 06:29:24.781 | Nov 17 06:29:20 np0035791906 <email address hidden>[169832]: ERROR cinder self._membership_lease = self.client.lease(self.membership_timeout)
2023-11-17 06:29:24.781 | Nov 17 06:29:20 np0035791906 <email address hidden>[169832]: ERROR cinder File "/usr/local/lib/python3.10/dist-packages/etcd3gw/client.py", line 123, in lease
2023-11-17 06:29:24.781 | Nov 17 06:29:20 np0035791906 <email address hidden>[169832]: ERROR cinder result = self.post(self.get_url("/lease/grant"),
2023-11-17 06:29:24.781 | Nov 17 06:29:20 np0035791906 <email address hidden>[169832]: ERROR cinder File "/usr/local/lib/python3.10/dist-packages/etcd3gw/client.py", line 93, in post
2023-11-17 06:29:24.781 | Nov 17 06:29:20 np0035791906 <email address hidden>[169832]: ERROR cinder raise exceptions.Etcd3Exception(resp.text, resp.reason)
2023-11-17 06:29:24.781 | Nov 17 06:29:20 np0035791906 <email address hidden>[169832]: ERROR cinder etcd3gw.exceptions.Etcd3Exception: Not Found
2023-11-17 06:29:24.781 | Nov 17 06:29:20 np0035791906 <email address hidden>[169832]: ERROR cinder
2023-11-17 06:29:24.781 | Nov 17 06:29:20 np0035791906 <email address hidden>[169832]: unable to load app 0 (mountpoint='') (callable not found or import error)

When the node is deployed initially it is deployed with etcd v3.3.12.
The coordination_url parameter in cinder.conf does not contain api_version but tooz 3.2.0 used v1alpha by default which is available in etcd v3.3.12.

The problem arises when all components, including cinder are upgraded while etcd is kept at v3.3.12.
The coordination_url in cinder does not contain api_version at this point.
However the tooz 4.3.0 uses v3 API by default but this is not available in etcd v3.3.12 and eventually results in the above failure.

Although we added a release note to explain this requirement during upgrade, it's much better if we can implement some logic to avoid needing operators to update the config file.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/etcd3gw/+/901244
Committed: https://opendev.org/openstack/etcd3gw/commit/e5faa53eada8b17795b2e68fdffb226aaceb0f92
Submitter: "Zuul (22348)"
Branch: master

commit e5faa53eada8b17795b2e68fdffb226aaceb0f92
Author: Takashi Kajinami <email address hidden>
Date: Fri Nov 17 22:53:59 2023 +0900

    Discover API version automatically

    etcd changed API version from v3alpha to v3beta, then v3, and because
    of transition available api versions are different according to
    the etcd server available.

    This change implements the mechanism to detect the current api version
    according to the etcd version obtained via the version API. Discovery
    is executed at the first request and the result is stored.

    Closes-Bug: #2043810
    Change-Id: I08bc429dd8c59e5d2b6e62f8f48afd78a116fbea

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/etcd3gw 2.3.0 release.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tooz/+/901247
Committed: https://opendev.org/openstack/tooz/commit/f180411f42b77f85123e44983f0799e688a21044
Submitter: "Zuul (22348)"
Branch: master

commit f180411f42b77f85123e44983f0799e688a21044
Author: Takashi Kajinami <email address hidden>
Date: Fri Nov 17 23:38:08 2023 +0900

    etcd3gw: Discover API version automatically

    etcd changed API version from v3beta to v3alpha, then v3, and because
    of transition available api versions are different according to
    the etcd server available.

    This change implements the mechanism to detect the current api version
    according to the etcd version obtained via the version API. So that
    the url without api version works regardless of the etcd server used.

    Closes-Bug: #2043810
    Depends-on: https://review.opendev.org/c/openstack/etcd3gw/+/901244
    Change-Id: Ib81390eb73feafe08d007db1ced4d0ceb3bde212

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tooz 5.0.0 release.