hook failed: "install" - sriov-netplan-shim/setup.py

[Andrew McLeod]

focal ussuri s390x (network restricted)

Fails during install hook when trying to run sriov-netplan-shim/setup.py

partial log:

unit-ovn-chassis-0: 14:35:54 DEBUG unit.ovn-chassis/0.install command: /var/lib/juju/agents/unit-ovn-chassis-0/.venv/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-q43hizbl/sriov-netplan-shim
/setup.py'"'"'; __file__='"'"'/tmp/pip-install-q43hizbl/sriov-netplan-shim/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"
'"'exec'"'"'))' egg_info --egg-base /tmp/pip-install-q43hizbl/sriov-netplan-shim/pip-egg-info unit-ovn-chassis-0: 14:35:54 DEBUG unit.ovn-chassis/0.install cwd: /tmp/pip-install-q43hizbl/sriov-netplan-shim/ unit-ovn-chassis-0: 14:35:54 DEBUG unit.ovn-chassis/0.install Complete output (37 lines):
unit-ovn-chassis-0: 14:35:54 DEBUG unit.ovn-chassis/0.install WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.VerifiedHTTP
SConnection object at 0x3ff95fc91c0>: Failed to establish a new connection: [Errno 101] Network is unreachable')': /simple/pbr/

complete log:

https://pastebin.ubuntu.com/p/V7yYzJX627/

[Frode Nordahl (fnordahl)]

Added charm-layer-ovn as that is the origin of the bug, keeping the charm tasks for tracking. Triaging as critical as this will make the charms unusable in offline deployments and must be fixed before release.

[Frode Nordahl (fnordahl)]

The package mentioned in the trace, pbr, is included as a wheel, so there must be missing build dependencies, a version inter dependency mismatch or something like that.

[James Page (james-page)]

I'm not sure why you need sriov-netplan-shim in the wheelhouse? its provided by the networking tools PPA.

[Frode Nordahl (fnordahl)]

The reactive charm imports the PCI library from the package, and layer-basic is configured to not include system packages. So the charm consumes the package both as a wheel and as a deb.

Wheel for direct charm code dependencies and deb for run-/boot- time system configuration.

[Frode Nordahl (fnordahl)]

First attempt at an instrumented reproducer did not reproduce the issue.

Steps taken:

  - deployed a charm on LXD provider with modified layer-basic with increased verbosity
  - changed the firewall rules to only allow communication between juju controller and unit
  - issued a charm upgrade

All wheels were sourced from the local whelhouse with file:// links and I saw now network related errors.

It may be the initial install has brought something to the system that make it not display the behavior on charm upgrade, so I'll do another attempt where the initial deploy is also isolated.

[Frode Nordahl (fnordahl)]

s/now/no

[Frode Nordahl (fnordahl)]

Instrumented test with no networking available on initial install reproduces the issue.

The charm attempts to reach out to PyPI while processing the wheelhouse.

It does not happen for all Python packages, only some, so let's see what we can figure out as to what the difference is.

[Frode Nordahl (fnordahl)]

https://github.com/juju-solutions/layer-basic/issues/171

[Frode Nordahl (fnordahl)]

https://github.com/juju-solutions/layer-basic/pull/172

[Frode Nordahl (fnordahl)]

Setting invalid for layer as the break and fix is in a upstream layer. Keeping tasks for charms for tracking of rebuild.

[Dmitrii Shcherbakov (dmitriis)]

Added charm-ovn-dedicated-chassis since it also depends on pbr which uses easy_install => also affected by the layer-basic regression.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Dmitrii Shcherbakov (<email address hidden>) on branch: master
Review: https://review.opendev.org/728764

[OpenStack Infra (hudson-openstack)]

Change abandoned by Dmitrii Shcherbakov (<email address hidden>) on branch: master
Review: https://review.opendev.org/728792