hook error on Bionic with "ERROR: You must give at least one requirement to install"

[Moises Emilio Benzan Mora]

While running the install hook for the kubernetes-control-plane units, the hook enters an error state due to a failure in pip originating from a misconstructed command:

unit-kubernetes-control-plane-0: 17:51:23 DEBUG unit.kubernetes-control-plane/0.install Looking in links: wheelhouse
unit-kubernetes-control-plane-0: 17:51:23 DEBUG unit.kubernetes-control-plane/0.install Collecting flit_core<4,>=3.2
unit-kubernetes-control-plane-0: 17:51:23 DEBUG unit.kubernetes-control-plane/0.install Installing build dependencies: started
unit-kubernetes-control-plane-0: 17:51:23 DEBUG unit.kubernetes-control-plane/0.install Installing build dependencies: finished with status 'error'
unit-kubernetes-control-plane-0: 17:51:23 DEBUG unit.kubernetes-control-plane/0.install Complete output from command /var/lib/juju/agents/unit-kubernetes-control-plane-0/.venv/bin/python3 -m pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-tij_jhq8 --no-warn-script-location --no-binary :none: --only-binary :none: --no-index --find-links wheelhouse --:
unit-kubernetes-control-plane-0: 17:51:23 DEBUG unit.kubernetes-control-plane/0.install ERROR: You must give at least one requirement to install (maybe you meant "pip install wheelhouse"?)
unit-kubernetes-control-plane-0: 17:51:23 DEBUG unit.kubernetes-control-plane/0.install
unit-kubernetes-control-plane-0: 17:51:23 DEBUG unit.kubernetes-control-plane/0.install ----------------------------------------
unit-kubernetes-control-plane-0: 17:51:23 DEBUG unit.kubernetes-control-plane/0.install Command "/var/lib/juju/agents/unit-kubernetes-control-plane-0/.venv/bin/python3 -m pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-tij_jhq8 --no-warn-script-location --no-binary :none: --only-binary :none: --no-index --find-links wheelhouse --" failed with error code 1 in None
unit-kubernetes-control-plane-0: 17:51:23 DEBUG unit.kubernetes-control-plane/0.install
unit-kubernetes-control-plane-0: 17:51:23 DEBUG unit.kubernetes-control-plane/0.install ----------------------------------------
unit-kubernetes-control-plane-0: 17:51:24 WARNING unit.kubernetes-control-plane/0.install Command "/var/lib/juju/agents/unit-kubernetes-control-plane-0/.venv/bin/python3 -m pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-aq1_3uf8 --no-warn-script-location --no-binary :none: --only-binary :none: --no-index --find-links wheelhouse -- "flit_core >=3.2,<4"" failed with error code 1 in None
unit-kubernetes-control-plane-0: 17:51:25 WARNING unit.kubernetes-control-plane/0.install Traceback (most recent call last):
unit-kubernetes-control-plane-0: 17:51:25 WARNING unit.kubernetes-control-plane/0.install File "/var/lib/juju/agents/unit-kubernetes-control-plane-0/charm/hooks/install", line 8, in <module>
unit-kubernetes-control-plane-0: 17:51:25 WARNING unit.kubernetes-control-plane/0.install basic.bootstrap_charm_deps()
unit-kubernetes-control-plane-0: 17:51:25 WARNING unit.kubernetes-control-plane/0.install File "lib/charms/layer/basic.py", line 226, in bootstrap_charm_deps
unit-kubernetes-control-plane-0: 17:51:25 WARNING unit.kubernetes-control-plane/0.install env=_get_subprocess_env())
unit-kubernetes-control-plane-0: 17:51:25 WARNING unit.kubernetes-control-plane/0.install File "/usr/lib/python3.6/subprocess.py", line 311, in check_call
unit-kubernetes-control-plane-0: 17:51:25 WARNING unit.kubernetes-control-plane/0.install raise CalledProcessError(retcode, cmd)
unit-kubernetes-control-plane-0: 17:51:25 WARNING unit.kubernetes-control-plane/0.install subprocess.CalledProcessError: Command '['/var/lib/juju/agents/unit-kubernetes-control-plane-0/.venv/bin/pip', 'install', '-U', '--force-reinstall', '--no-index', '--no-cache-dir', '-f', 'wheelhouse', 'multidict==5.2.0', 'yarl==1.7.2', 'psutil==5.9.2', 'netaddr==0.7.19', 'six==1.16.0', 'charms.reactive==1.5.1', 'aiohttp==3.7.4.post0', 'ops-reactive-interface==1.0.1', 'charset-normalizer==2.0.12', 'attrs==22.1.0', 'requests==2.27.1', 'urllib3==1.26.12', 'gunicorn==20.1.0', 'pbr==5.10.0', 'marshmallow==3.14.1', 'idna-ssl==1.1.0', 'loadbalancer-interface==1.1.1', 'Jinja2==3.0.3', 'certifi==2022.9.24', 'idna==3.4', 'PyYAML==5.3.1', 'marshmallow-enum==1.5.1', 'pyaml==21.10.1', 'typing-extensions==3.10.0.2', 'async-timeout==3.0.1', 'ops==1.5.2', 'cached-property==1.5.2', 'tenacity==5.0.3', 'chardet==4.0.0', 'hvac==0.11.2', 'netifaces==0.11.0', 'flit-core==3.7.1', 'charmhelpers==1.2.1']' returned non-zero exit status 1.
unit-kubernetes-control-plane-0: 17:51:25 ERROR juju.worker.uniter.operation hook "install" (via explicit, bespoke hook script) failed: exit status 1
unit-kubernetes-control-plane-0: 17:51:25 DEBUG juju.machinelock created rotating log file "/var/log/juju/machine-lock.log" with max size 10 MB and max backups 5
unit-kubernetes-control-plane-0: 17:51:25 DEBUG juju.machinelock machine lock released for kubernetes-control-plane/0 uniter (run install hook)
unit-kubernetes-control-plane-0: 17:51:25 DEBUG juju.worker.uniter.operation lock released for kubernetes-control-plane/0
unit-kubernetes-control-plane-0: 17:51:25 INFO juju.worker.uniter awaiting error resolution for "install" hook
unit-kubernetes-control-plane-0: 17:51:25 DEBUG juju.worker.uniter [AGENT-STATUS] error: hook failed: "install"

Logs: https://oil-jenkins.canonical.com/artifacts/c39c5776-96f0-4ba7-841b-5bc3089a7a1b/generated/generated/kubernetes-maas/juju-crashdump-kubernetes-maas-2022-10-05-17.52.14.tar.gz

Run artifacts: https://oil-jenkins.canonical.com/artifacts/c39c5776-96f0-4ba7-841b-5bc3089a7a1b/index.html

[George Kraft (cynerva)]

I can't quite make sense of this yet, but it's definitely an issue with the kubernetes-control-plane charm. I'm trivially able to reproduce with

juju deploy kubernetes-core --overlay overlay.yaml

where overlay.yaml contains:

series: bionic

[George Kraft (cynerva)]

It's failing during the install of idna and flit_core, so it may be related to this change: https://github.com/charmed-kubernetes/charm-kubernetes-control-plane/pull/246

[Adam Dyess (addyess)]

idna is a package required by hvac (used in layer-vault-kv) and also by aiohttp
idna >= 3.4 and beyond requires flit-core>3.2,<4 to build the idna wheel
flit-core (3.7.1) is encountering an issue pip installing itself into this environment.

A detailed view of the installation shows this error:
https://paste.ubuntu.com/p/Q9nvjPSWDv/

where pip seems to be generating a malformed subprocess call to itself for installing the flit-core package.

flit-core doesn't have any requirements (https://github.com/pypa/flit/blob/3.7.1/flit_core/pyproject.toml) -- but pip think that it should run the install command for those empty requirements which manifests the failure. This could be due a bug that's be since fixed in pip

[Adam Dyess (addyess)]

i was able to work around the install issue by using pip==20.0.2

this workaround can be applied with:

$ juju ssh kubernetes-control-plane/<unit>
$ sudo su -
$ rm /var/lib/juju/agents/unit-kubernetes-control-plane-*/charm/wheelhouse/pip-18.1.tar.gz
$ PIP_PKG=https://files.pythonhosted.org/packages/8e/76/66066b7bc71817238924c7e4b448abdb17eb0c92d645769c223f9ace478f/pip-20.0.2.tar.gz
$ wget $PIP_PKG -P /var/lib/juju/agents/unit-kubernetes-control-plane-*/charm/wheelhouse/

I tried all the versions of pip between 18.1 and 20.0.2 all of which encountered errors. 20.0.2 is the same version of pip installed by default in focal. It should be ok to use this version of pip in a virtualenv for kubernetes-control-plane since layer-basic performs an upgrade on pip in the event it doesn't downgrade the current series installation of pip.

[Adam Dyess (addyess)]

Official Fix PR[1]

[1] (https://github.com/charmed-kubernetes/charm-kubernetes-control-plane/pull/252)