Etcd Charm

anbox-cloud-core: etcd/jammy: hook failed: "install" due to "error in Tempita setup command: use_2to3 is invalid."

Bug #2002020 reported by Mauricio Faria de Oliveira
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Anbox Cloud
Fix Released
High
Simon Fels
Anbox Cloud 1.16.2
Etcd Charm
Invalid
Undecided
Unassigned

Bug Description

The anbox-cloud/-core/ charm uses an older revision of
the etcd charm that does not support the jammy release
(install hook fails).

juju proceeds with the etcd deployment on jammy anyway
as it seems to ignore that for _bundles_, but it fails
w/ that specific reason if you deploy just etcd (below).

The (full/non-core) anbox-cloud charm uses a revision
that actually supports jammy, so it should be good.

Updating the anbox-cloud-core bundle for an etcd charm
revision as anbox-cloud (non-core) should address that
(see comment #3).

Workarounds:
- to use an overlay to unpin the revision (comment #3)
- manually address the failure on the unit (description)

...

$ juju download --quiet anbox-cloud-core && unzip -q anbox-cloud-core_*.bundle -d anbox-cloud-core && cat anbox-cloud-core/bundle.yaml | yq '.series,.applications.etcd.revision'
jammy
691

$ juju download --quiet anbox-cloud && unzip -q anbox-cloud_*.bundle -d anbox-cloud && cat anbox-cloud/bundle.yaml | yq '.series,.applications.etcd.revision'
jammy
718

...

$ juju deploy etcd --series jammy --revision 691 --channel stable etcd-r691
ERROR series "jammy" not supported by charm, supported series are: focal, bionic, xenial. Use --force to deploy the charm anyway.

$ cat bundle.yaml
series: jammy
applications:
  etcd-r691:
    charm: etcd
    revision: 691
    channel: stable

$ juju deploy ./bundle.yaml
Located charm "etcd" in charm-hub, channel stable
Executing changes:
- upload charm etcd from charm-hub for series jammy with revision 691 with architecture=amd64
...
Deploy of bundle completed.

$ juju deploy etcd --series jammy --revision 718 --channel stable etcd-r718
Located charm "etcd" in charm-hub, revision 718
Deploying "etcd-r718" from charm-hub charm "etcd", revision 718 in channel stable on jammy

$ juju status
...
etcd-r691/0* error idle 1 10.5.3.114 hook failed: "install"
etcd-r718/0* blocked idle 2 10.5.3.207 Missing relation to certificate authority.
...

...

Original Description:

`juju status` during deployment shows:

```
etcd/0* error idle 0 10.5.1.196 hook failed: "install"
```

Debug log shows:

```
$ juju debug-log -i etcd/0 --replay --no-tail
...
unit-etcd-0: 16:59:58 INFO juju.worker.uniter found queued "install" hook
unit-etcd-0: 17:02:48 WARNING unit.etcd/0.install error: subprocess-exited-with-error
unit-etcd-0: 17:02:48 WARNING unit.etcd/0.install
unit-etcd-0: 17:02:48 WARNING unit.etcd/0.install × python setup.py egg_info did not run successfully.
unit-etcd-0: 17:02:48 WARNING unit.etcd/0.install │ exit code: 1
unit-etcd-0: 17:02:48 WARNING unit.etcd/0.install ╰─> [1 lines of output]
unit-etcd-0: 17:02:48 WARNING unit.etcd/0.install error in Tempita setup command: use_2to3 is invalid.
...
unit-etcd-0: 17:02:48 WARNING unit.etcd/0.install Traceback (most recent call last):
unit-etcd-0: 17:02:48 WARNING unit.etcd/0.install File "/var/lib/juju/agents/unit-etcd-0/charm/hooks/install", line 8, in <module>
unit-etcd-0: 17:02:48 WARNING unit.etcd/0.install basic.bootstrap_charm_deps()
unit-etcd-0: 17:02:48 WARNING unit.etcd/0.install File "/var/lib/juju/agents/unit-etcd-0/charm/lib/charms/layer/basic.py", line 219, in bootstrap_charm_deps
unit-etcd-0: 17:02:48 WARNING unit.etcd/0.install check_call([pip, 'install', '-U', reinstall_flag, '--no-index',
unit-etcd-0: 17:02:48 WARNING unit.etcd/0.install File "/usr/lib/python3.10/subprocess.py", line 369, in check_call
unit-etcd-0: 17:02:48 WARNING unit.etcd/0.install raise CalledProcessError(retcode, cmd)
unit-etcd-0: 17:02:48 WARNING unit.etcd/0.install subprocess.CalledProcessError: Command '['/var/lib/juju/agents/unit-etcd-0/.venv/bin/pip', 'install', '-U', '--force-reinstall', '--no-inde
x', '--no-cache-dir', '-f', 'wheelhouse', 'charmhelpers==0.20.24', 'charms.reactive==1.5.0', 'charms.templating.jinja2==1.0.2', 'netaddr==0.7.19', 'six==1.16.0', 'Tempita==0.5.2', 'tenacity
==5.0.3', 'Jinja2==2.11.3', 'pyaml==21.10.1', 'PyYAML==5.3.1', 'pbr==5.8.1']' returned non-zero exit status 1.
unit-etcd-0: 17:02:49 ERROR juju.worker.uniter.operation hook "install" (via explicit, bespoke hook script) failed: exit status 1
...
```

The issue seems to be the deprecation of `use_2to3` in setuptools 58, per web searches.

Reproducer:

$ juju ssh etcd/0
$ . /var/lib/juju/agents/unit-etcd-0/.venv/bin/activate
$ pip install -U --force-reinstall --no-index -f /var/lib/juju/agents/unit-etcd-0/charm/wheelhouse Tempita==0.5.2

Workaround:

$ juju ssh etcd/0
$ sudo -s
# . /var/lib/juju/agents/unit-etcd-0/.venv/bin/activate
# pip install 'setuptools<58'
...
Successfully installed setuptools-57.5.0
# exit
$ exit
$ juju resolved etcd/0

Within some time:
```
etcd/0* active idle 0 10.5.1.196 2379/tcp Healthy with 1 known peer
```

Seen on Ubuntu 22.04 / Jammy:

```
$ juju ssh etcd/0 'lsb_release -cs; python3 -V'
jammy
Python 3.10.6
```

See original description
Revision history for this message
George Kraft (cynerva) wrote (last edit ):

I'm fairly confident that this has been fixed already since we added Jammy support to the charm[1], which went out with the CK 1.25 release. Specifically, removing charms.templating.jinja2 from the wheelhouse was necessary to prevent Tempita from being pulled in as a dependency.

What revision of the etcd charm did you encounter this with? The charm revision should appear in the output of `juju status`.

[1]: https://github.com/charmed-kubernetes/layer-etcd/pull/200

Changed in charm-etcd:
status: New → Incomplete
Revision history for this message
Mauricio Faria de Oliveira (mfo) wrote :

Thanks for the clarification.

This is etcd charm revision 691, pulled by `juju deploy anbox-cloud-core`.

I'm not sure how to download the bundle from charmhub to check a hardcode.
Apparently `charm pull-source` doesn't work anymore; is that still doable?

$ juju status --format yaml | yq '.applications.etcd' | grep -e charm -e series
charm: etcd
series: jammy
charm-origin: charmhub
charm-name: etcd
charm-rev: 691
charm-channel: stable
charm-version: 23cdd4f

Revision history for this message
George Kraft (cynerva) wrote :

Thanks. Revision 691 does predate the Jammy support PR, so you will need a newer revision.

You can use `juju download` to pull a charm or bundle from charmhub, and `unzip` to inspect it. For example:

$ juju download anbox-cloud-core
Fetching bundle "anbox-cloud-core" using "stable" channel and base "amd64/ubuntu/20.04"
Install the "anbox-cloud-core" bundle with:
    juju deploy ./anbox-cloud-core_3836b1b.bundle
$ unzip anbox-cloud-core_3836b1b.bundle -d anbox-cloud-core
$ cat anbox-cloud-core/bundle.yaml

I do see the etcd revision hardcoded in that bundle, so I would raise this as a bug against anbox-cloud-core.

To work around this, you should be able to deploy anbox-cloud-core with an overlay to unpin the revision:

$ juju deploy anbox-cloud-core --overlay overlay.yaml

where overlay.yaml contains:

applications:
  etcd:
    revision: null

Or you can pin it to a newer revision, if you prefer. See `juju info etcd` for the latest revisions from 1.25/stable or 1.26/stable, either of which should work.

Changed in charm-etcd:
status: Incomplete → Invalid
Revision history for this message
Mauricio Faria de Oliveira (mfo) wrote :

Thanks for the steps and workaround, George.

It turns out the anbox-cloud-core bundle is
outdated wrt the anbox-cloud bundle on etcd.

Reassigning this bug to anbox cloud project.

summary: - hook failed: "install" due to "error in Tempita setup command: use_2to3
- is invalid."
+ anbox-cloud-core: etcd/jammy: hook failed: "install" due to "error in
+ Tempita setup command: use_2to3 is invalid."
Revision history for this message
Mauricio Faria de Oliveira (mfo) wrote :

For documentation purposes, the etcd charm channels/revision at the moment.

$ juju info etcd
...
channels: |
  latest/stable: 7d28c19 2022-12-15 (722) 5MB
  latest/candidate: 8ca2bdb 2022-09-28 (718) 5MB
  latest/beta: 7d28c19 2022-12-09 (722) 5MB
  latest/edge: 7d28c19 2022-12-09 (723) 5MB
  1.27/stable: –
  1.27/candidate: –
  1.27/beta: –
  1.27/edge: 7d28c19 2022-12-09 (723) 5MB
  1.26/stable: 7d28c19 2022-12-15 (722) 5MB
  1.26/candidate: ↑
  1.26/beta: 7d28c19 2022-12-09 (722) 5MB
  1.26/edge: 7d28c19 2022-11-19 (720) 5MB
  1.25/stable: 8ca2bdb 2022-09-30 (718) 5MB
  1.25/candidate: 8ca2bdb 2022-09-28 (718) 5MB
  1.25/beta: 7d28c19 2022-12-01 (721) 5MB
  1.25/edge: 8ca2bdb 2022-09-09 (708) 5MB
  1.24/stable: 1.24+ck1 2022-08-04 (701) 5MB
  1.24/candidate: 1.24+ck1 2022-08-01 (701) 5MB
  1.24/beta: 23cdd4f 2022-05-03 (691) 5MB
  1.24/edge: 8ca2bdb 2022-07-22 (700) 5MB
  1.23/stable: –
  1.23/candidate: –
  1.23/beta: 77eef0c 2022-03-22 (682) 5MB
  1.23/edge: 77eef0c 2022-02-24 (680) 5MB

description: updated
description: updated
Revision history for this message
Simon Fels (morphis) wrote :

Hm, yes this is a bug in the anbox-cloud-core bundle. The anbox-cloud bundle pins etcd 718 which has support for jammy. We will fix for the upcoming 1.16.2 next week.

Changed in anbox-cloud:
status: New → In Progress
assignee: nobody → Simon Fels (morphis)
importance: Undecided → High
milestone: none → 1.16.2
Revision history for this message
Mauricio Faria de Oliveira (mfo) wrote :

Thanks! Marking as Fix Released per `Bugs` listed in [1].

[1] https://discourse.ubuntu.com/t/anbox-cloud-1-16-2-has-been-released/33161

Changed in anbox-cloud:
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.