octavia charm-upgrade failed on pip install wheelhouse setuptools-scm

[Marian Gasparovic]

stein-bionic cloud

Cloud was deployed on Aug 12th, all went ok. Charms are specified in the bundle with cs:charmname. Today (Aug 14th) around 8:20 UTC I removed trilio applications and used juju deploy with original bundle.yaml to redeploy it.

This re-deploy triggered charm-upgrade in octavia (I assume also others, saw that also in designate). But octavia failed the upgrade with

2020-08-14 08:26:46 DEBUG upgrade-charm Traceback (most recent call last):
2020-08-14 08:26:46 DEBUG upgrade-charm File "/tmp/pip-build-env-cfh7oyz_/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2451, in resolve
2020-08-14 08:26:46 DEBUG upgrade-charm return functools.reduce(getattr, self.attrs, module)
2020-08-14 08:26:46 DEBUG upgrade-charm AttributeError: module 'setuptools_scm.hg' has no attribute 'FILES_COMMAND'
2020-08-14 08:26:46 DEBUG upgrade-charm
2020-08-14 08:26:46 DEBUG upgrade-charm During handling of the above exception, another exception occurred:
2020-08-14 08:26:46 DEBUG upgrade-charm
2020-08-14 08:26:46 DEBUG upgrade-charm Traceback (most recent call last):
2020-08-14 08:26:46 DEBUG upgrade-charm File "<string>", line 1, in <module>
2020-08-14 08:26:46 DEBUG upgrade-charm File "/tmp/pip-install-6viilk9s/setuptools-scm/setup.py", line 53, in <module>
2020-08-14 08:26:46 DEBUG upgrade-charm setuptools.setup(**scm_config())
2020-08-14 08:26:46 DEBUG upgrade-charm File "/tmp/pip-build-env-cfh7oyz_/lib/python3.6/site-packages/setuptools/__init__.py", line 145, in setup
2020-08-14 08:26:46 DEBUG upgrade-charm return distutils.core.setup(**attrs)
2020-08-14 08:26:46 DEBUG upgrade-charm File "/usr/lib/python3.6/distutils/core.py", line 148, in setup
2020-08-14 08:26:46 DEBUG upgrade-charm dist.run_commands()
2020-08-14 08:26:46 DEBUG upgrade-charm File "/usr/lib/python3.6/distutils/dist.py", line 955, in run_commands
2020-08-14 08:26:46 DEBUG upgrade-charm self.run_command(cmd)
2020-08-14 08:26:46 DEBUG upgrade-charm File "/usr/lib/python3.6/distutils/dist.py", line 974, in run_command
2020-08-14 08:26:46 DEBUG upgrade-charm cmd_obj.run()
2020-08-14 08:26:46 DEBUG upgrade-charm File "/tmp/pip-build-env-cfh7oyz_/lib/python3.6/site-packages/setuptools/command/egg_info.py", line 296, in run
2020-08-14 08:26:46 DEBUG upgrade-charm self.find_sources()
2020-08-14 08:26:46 DEBUG upgrade-charm File "/tmp/pip-build-env-cfh7oyz_/lib/python3.6/site-packages/setuptools/command/egg_info.py", line 303, in find_sources
2020-08-14 08:26:46 DEBUG upgrade-charm mm.run()
2020-08-14 08:26:46 DEBUG upgrade-charm File "/tmp/pip-build-env-cfh7oyz_/lib/python3.6/site-packages/setuptools/command/egg_info.py", line 534, in run
2020-08-14 08:26:46 DEBUG upgrade-charm self.add_defaults()
2020-08-14 08:26:46 DEBUG upgrade-charm File "/tmp/pip-build-env-cfh7oyz_/lib/python3.6/site-packages/setuptools/command/egg_info.py", line 574, in add_defaults
2020-08-14 08:26:46 DEBUG upgrade-charm rcfiles = list(walk_revctrl())
2020-08-14 08:26:46 DEBUG upgrade-charm File "/tmp/pip-build-env-cfh7oyz_/lib/python3.6/site-packages/setuptools/command/sdist.py", line 20, in walk_revctrl
2020-08-14 08:26:46 DEBUG upgrade-charm for item in ep.load()(dirname):
2020-08-14 08:26:46 DEBUG upgrade-charm File "/tmp/pip-install-6viilk9s/setuptools-scm/src/setuptools_scm/integration.py", line 22, in find_files
2020-08-14 08:26:46 DEBUG upgrade-charm command = ep.load()
2020-08-14 08:26:46 DEBUG upgrade-charm File "/tmp/pip-build-env-cfh7oyz_/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2443, in load
2020-08-14 08:26:46 DEBUG upgrade-charm return self.resolve()
2020-08-14 08:26:46 DEBUG upgrade-charm File "/tmp/pip-build-env-cfh7oyz_/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2453, in resolve
2020-08-14 08:26:46 DEBUG upgrade-charm raise ImportError(str(exc))
2020-08-14 08:26:46 DEBUG upgrade-charm ImportError: module 'setuptools_scm.hg' has no attribute 'FILES_COMMAND'
2020-08-14 08:26:46 DEBUG upgrade-charm
2020-08-14 08:26:46 DEBUG upgrade-charm ----------------------------------------
2020-08-14 08:26:47 DEBUG upgrade-charm Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-install-6viilk9s/setuptools-scm/
2020-08-14 08:26:48 DEBUG upgrade-charm Traceback (most recent call last):
2020-08-14 08:26:48 DEBUG upgrade-charm File "/var/lib/juju/agents/unit-octavia-0/charm/hooks/upgrade-charm", line 8, in <module>
2020-08-14 08:26:48 DEBUG upgrade-charm basic.bootstrap_charm_deps()
2020-08-14 08:26:48 DEBUG upgrade-charm File "lib/charms/layer/basic.py", line 197, in bootstrap_charm_deps
2020-08-14 08:26:48 DEBUG upgrade-charm _update_if_newer(pip, pre_install_pkgs)
2020-08-14 08:26:48 DEBUG upgrade-charm File "lib/charms/layer/basic.py", line 284, in _update_if_newer
2020-08-14 08:26:48 DEBUG upgrade-charm pkg], env=_get_subprocess_env())
2020-08-14 08:26:48 DEBUG upgrade-charm File "/usr/lib/python3.6/subprocess.py", line 311, in check_call
2020-08-14 08:26:48 DEBUG upgrade-charm raise CalledProcessError(retcode, cmd)
2020-08-14 08:26:48 DEBUG upgrade-charm subprocess.CalledProcessError: Command '['/var/lib/juju/agents/unit-octavia-0/.venv/bin/pip', 'install', '-U', '--no-index', '-f', 'wheelhouse', 'setuptools-scm']' returned non-zero exit status 1.
2020-08-14 08:26:48 ERROR juju.worker.uniter.operation runhook.go:132 hook "upgrade-charm" failed: exit status 1
2020-08-14 08:26:48 DEBUG juju.machinelock machinelock.go:186 machine lock released for uniter (run upgrade-charm hook)

[Aurelien Lourot (aurelien-lourot)]

Good to mention: the OpenStack charms 20.08 got released on August 13th.

This bug is probably related to:
https://bugs.launchpad.net/charm-mysql-router/+bug/1884460
https://github.com/juju-solutions/layer-basic/issues/181
https://bugs.launchpad.net/charm-ceph-fs/+bug/1885935

[Aurelien Lourot (aurelien-lourot)]

Can easily be reproduced with:
juju deploy cs:octavia-21 --config openstack-origin=cloud:bionic-stein
juju upgrade octavia

[David Ames (thedac)]

TRIAGE:

We may be running into a setup tools SCM bug [0]. The workaround is to remove the .venv directory and retry.

This may be a layer-basic bug to fix the problem in charms.

As Aurelien has pointed out the '-' vs '_' in setuptools-scm may be at play.

Another oddity which may be unrelated is we are getting setuptools_scm-4.1.2.tar.gz when the wheelouse.txt says:
setuptools-scm>=3.0<=3.4.1

-----------
Workaround:

rm -rf /var/lib/juju/agents/unit-octavia-<UNIT>/.venv
juju resolved octavia/<UNIT>

[0] https://github.com/fink/fink-distributions/issues/219

[Aurelien Lourot (aurelien-lourot)]

We end up with setuptools 4 because of a comma missing in "setuptools-scm>=3.0<=3.4.1". It should be "setuptools-scm>=3.0,<=3.4.1". However this doesn't solve the original problem, which is most likely [0] as David mentioned. Maybe we can put our workaround in the upgrade hook somehow?

0: https://github.com/fink/fink-distributions/issues/219

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/746740

[David Ames (thedac)]

Layer basic fix to remove the venv on charm upgrade: https://github.com/juju-solutions/layer-basic/pull/188

Tested with the following:
juju deploy cs:octavia-21 --config openstack-origin=cloud:bionic-stein --contstraints mem=3040
# Built charm with layer-basic change.
juju upgrade-charm octavia --switch .

The bug is resolved by the change.

[Cory Johns (johnsca)]

Layer basic PR merged

[David Ames (thedac)]

With [0] landed, [1] will rebuild Octavia with the update. Once [1] lands, we can back-port to stable.

[0] https://github.com/juju-solutions/layer-basic/pull/188
[1] https://review.opendev.org/#/c/746740/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/746740
Committed: https://git.openstack.org/cgit/openstack/charm-octavia/commit/?id=009dd33942938cc29acfbd1ad267352b9d0c7fba
Submitter: Zuul
Branch: master

commit 009dd33942938cc29acfbd1ad267352b9d0c7fba
Author: Aurelien Lourot <email address hidden>
Date: Tue Aug 18 16:39:19 2020 +0200

    Fix setuptools-scm's version range

    Because of this typo we ended up installing version 4 instead of 3

    Change-Id: Ic2d884b8c589d00531b372ba4f742cbe651c7e48
    Related-Bug: #1891639

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/20.08
Review: https://review.opendev.org/746908

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/746908
Committed: https://git.openstack.org/cgit/openstack/charm-octavia/commit/?id=4cb3409e8929b5a5d90d4f24e1e3a23194c7cad6
Submitter: Zuul
Branch: stable/20.08

commit 4cb3409e8929b5a5d90d4f24e1e3a23194c7cad6
Author: Aurelien Lourot <email address hidden>
Date: Tue Aug 18 16:39:19 2020 +0200

    Fix setuptools-scm's version range

    Because of this typo we ended up installing version 4 instead of 3

    (cherry picked from commit 009dd33942938cc29acfbd1ad267352b9d0c7fba)

    This triggers a rebuild and picks
    https://github.com/juju-solutions/layer-basic/pull/188 which resolves
    the linked bug.

    Change-Id: I86e94f5a9fb4823f612718d5ef16fa9d263f3749
    Closes-Bug: #1891639

[David Ames (thedac)]

The fix to layer basic [0] has now landed in master [1] and 20.08 stable [2].

[0] https://github.com/juju-solutions/layer-basic/pull/188
[1] https://review.opendev.org/#/c/746740/
[2] https://review.opendev.org/#/c/746908/