Charm fails to install on focal {main development branch}

[Tilman Baumann]

I am unable to build this charm from source. (on focal and for focal)

I initially tried the stable/21.10 branch with the build.lock. No matter what I try, it either has missing dependencies that not packed in the wheelhouse directory or if I try pulling them in I get a circular conflict with flit_core being at the same time required to be below version 3 and between version 3 and 4.

I spend two days adding a bunch of dependencies in the wheelhouse.txt file with no luck.

I took the downloaded charm from charmhub and created a wheelhosue.txt based on what packages are distributed there with no luck.

dnspython==1.16.0
Jinja2==2.10.1
MarkupSafe==1.1.1
netaddr==0.7.19
netifaces==0.11.0
pbr==5.6.0
pip==18.1
psutil==5.8.0
pyaml==21.10.1
PyYAML==5.2
setuptools==41.6.0
setuptools_scm==1.17.0
six==1.16.0
sriov-netplan-shim==0.1.4
Tempita==0.4
wheel==0.33.6
pkgutil-resolve-name==1.0.0

The same is basically true with master (after I fixed charmcraft.yaml)

I seem unable to build a permutation that is ready to run on focal (python3.8)
Which is odd, because the charm was obviously build for release by Canonical a while ago. Python hasn't changed tremendously since then and for that reason the lockfile and venv is used.

May I ask for some help? I can't see the wood for all the trees at this point.

Cheers
 Tilman

[Tilman Baumann (tilmanbaumann)]

# Fixed charmcraft

type: charm

parts:
  charm:
    build-packages:
      - tox
      - git
      - python3-dev
    source: ./src/
    plugin: reactive

bases:
  - build-on:
      - name: ubuntu
        channel: "20.04"
        architectures:
          - amd64
    run-on:
      - name: ubuntu
        channel: "20.04"
        architectures: [amd64, s390x, ppc64el, arm64]
      - name: ubuntu
        channel: "22.04"
        architectures: [amd64, s390x, ppc64el, arm64]

[Alex Kavanagh (ajkavanagh)]

I'm not sure what's going on, but I've just successfully built charm-ovn-chassis master using:

$ lsb_release -c
Codename: focal
$ charmcraft version
1.5.0
$ pwd
../charm-ovn-chassis
$ tox -e build

...

2022-08-26 11:43:25.432 Charm packed ok
build run-test: commands[2] | /home/alex/Projects/Canonical/git/opendev.org/x/charm-ovn-chassis/rename.sh
renaming ovn-chassis_*.charm to ovn-chassis.charm
pwd: /home/alex/Projects/Canonical/git/opendev.org/x/charm-ovn-chassis
total 29328
drwxr-xr-x 8 alex alex 4096 Aug 26 11:43 .
drwxr-xr-x 7 alex alex 4096 Jan 29 2020 ..
drwxrwxr-x 3 alex alex 4096 Apr 16 18:49 build
-rw-rw-r-- 1 alex alex 359 Apr 16 18:51 build-requirements.txt
-rw-rw-r-- 1 alex alex 876 Apr 16 18:51 charmcraft.yaml
drwxr-xr-x 8 alex alex 4096 Aug 26 11:34 .git
-rw-rw-r-- 1 alex alex 71 Apr 16 18:51 .gitignore
-rw-rw-r-- 1 alex alex 76 Aug 26 11:34 .gitreview
drwxr-xr-x 2 alex alex 4096 Jan 30 2020 layers
lrwxrwxrwx 1 alex alex 17 Apr 16 18:51 metadata.yaml -> src/metadata.yaml
-rw-rw-r-- 1 alex alex 260 Apr 16 18:51 osci.yaml
-rw-r--r-- 1 alex alex 14610349 Apr 16 20:22 ovn-chassis.charm
-rw-r--r-- 1 alex alex 15326143 Aug 26 11:43 ovn-chassis_ubuntu-20.04-amd64-s390x-ppc64el-arm64_ubuntu-22.04-amd64-s390x-ppc64el-arm64.charm
-rwxrwxr-x 1 alex alex 787 Jan 27 2022 pip.sh
lrwxrwxrwx 1 alex alex 13 Jan 29 2020 README.md -> src/README.md
-rw-rw-r-- 1 alex alex 197 Apr 16 18:51 rebuild
-rwxrwxr-x 1 alex alex 321 Apr 16 18:51 rename.sh
-rw-rw-r-- 1 alex alex 1062 Apr 16 18:50 requirements.txt
drwxr-xr-x 7 alex alex 4096 Aug 26 11:34 src
-rw-r--r-- 1 alex alex 44 May 12 2021 .stestr.conf
-rw-rw-r-- 1 alex alex 2010 Aug 26 11:34 test-requirements.txt
drwxrwxr-x 6 alex alex 4096 Aug 26 11:35 .tox
-rw-rw-r-- 1 alex alex 4193 Apr 16 18:51 tox.ini
-rw-r--r-- 1 alex alex 1140 May 12 2021 .travis.yml
drwxr-xr-x 2 alex alex 4096 May 5 2021 unit_tests
-rw-r--r-- 1 alex alex 92 May 12 2021 .zuul.yaml
Removing bad downloaded charm maybe?
Renaming charm here.
________________________________________________________________ summary ________________________________________________________________
  build: commands succeeded
  congratulations :)

$ ls -al *.charm
-rw-r--r-- 1 alex alex 15326143 Aug 26 11:43 ovn-chassis.charm

So what's different about your setup please?

[Tilman Baumann (tilmanbaumann)]

Oh it builds fine. But it will not install. I'm quite convinced it will be the same for you...

I tried charm build command as snap and as pip. On focal.

charmcraft version 2.0.0

tilli@zuse:~$ snap info charm
name: charm
summary: charm and charm-tools
publisher: Canonical✓
store-url: https://snapcraft.io/charm
contact: https://discourse.juju.is/c/charming
license: unset
description: |
  charmstore-client and charm-tools
commands:
  - charm
snap-id: 2Rryoc2ylScfbFl4eQtpntHD9iuZuMvt
tracking: latest/stable
refresh-date: 2022-02-17
channels:
  latest/stable: 2.8.2 2021-02-01 (609) 119MB classic
  latest/candidate: 2.8.2 2021-02-01 (609) 119MB classic
  latest/beta: 2.8.2 2021-02-01 (609) 119MB classic
  latest/edge: 2.8.3+git-1-736b1ad 2021-02-23 (620) 119MB classic
installed: 2.8.2 (609) 119MB classic

[Tilman Baumann (tilmanbaumann)]

PS: Sorry I see I said in the title of the bug that it fails to build. It actually fails to build a charm that is installable.

[Alex Kavanagh (ajkavanagh)]

> PS: Sorry I see I said in the title of the bug that it fails to build. It actually fails to build a charm that is installable.

Ah, that's a different beast :) Will try that now too.

[Tilman Baumann (tilmanbaumann)]

Also stable/21.10 branch with the build.lock which is the real curious one.

[Alex Kavanagh (ajkavanagh)]

Confirmed:

unit-ovn-chassis-0: 11:28:01 DEBUG unit.ovn-chassis/0.install Processing ./wheelhouse/jsonschema-4.14.0.tar.gz
unit-ovn-chassis-0: 11:28:01 DEBUG unit.ovn-chassis/0.install Installing build dependencies: started
unit-ovn-chassis-0: 11:28:03 DEBUG unit.ovn-chassis/0.install Installing build dependencies: finished with status 'error'
unit-ovn-chassis-0: 11:28:03 WARNING unit.ovn-chassis/0.install error: subprocess-exited-with-error
unit-ovn-chassis-0: 11:28:03 WARNING unit.ovn-chassis/0.install
unit-ovn-chassis-0: 11:28:03 WARNING unit.ovn-chassis/0.install × pip subprocess to install build dependencies did not run successfully.
unit-ovn-chassis-0: 11:28:03 WARNING unit.ovn-chassis/0.install │ exit code: 1
unit-ovn-chassis-0: 11:28:03 WARNING unit.ovn-chassis/0.install ╰─> [3 lines of output]
unit-ovn-chassis-0: 11:28:03 WARNING unit.ovn-chassis/0.install Looking in links: wheelhouse
unit-ovn-chassis-0: 11:28:03 WARNING unit.ovn-chassis/0.install ERROR: Could not find a version that satisfies the requirement hatchling (from versions: none)
unit-ovn-chassis-0: 11:28:03 WARNING unit.ovn-chassis/0.install ERROR: No matching distribution found for hatchling
unit-ovn-chassis-0: 11:28:03 WARNING unit.ovn-chassis/0.install [end of output]
unit-ovn-chassis-0: 11:28:03 WARNING unit.ovn-chassis/0.install
unit-ovn-chassis-0: 11:28:03 WARNING unit.ovn-chassis/0.install note: This error originates from a subprocess, and is likely not a problem with pip.
unit-ovn-chassis-0: 11:28:03 WARNING unit.ovn-chassis/0.install error: subprocess-exited-with-error
unit-ovn-chassis-0: 11:28:03 WARNING unit.ovn-chassis/0.install
unit-ovn-chassis-0: 11:28:03 WARNING unit.ovn-chassis/0.install × pip subprocess to install build dependencies did not run successfully.
unit-ovn-chassis-0: 11:28:03 WARNING unit.ovn-chassis/0.install │ exit code: 1
unit-ovn-chassis-0: 11:28:03 WARNING unit.ovn-chassis/0.install ╰─> See above for output.
unit-ovn-chassis-0: 11:28:03 WARNING unit.ovn-chassis/0.install
unit-ovn-chassis-0: 11:28:03 WARNING unit.ovn-chassis/0.install note: This error originates from a subprocess, and is likely not a problem with pip.
unit-ovn-chassis-0: 11:28:04 WARNING unit.ovn-chassis/0.install Traceback (most recent call last):
unit-ovn-chassis-0: 11:28:04 WARNING unit.ovn-chassis/0.install File "/var/lib/juju/agents/unit-ovn-chassis-0/charm/hooks/install", line 8, in <module>
unit-ovn-chassis-0: 11:28:04 WARNING unit.ovn-chassis/0.install basic.bootstrap_charm_deps()
unit-ovn-chassis-0: 11:28:04 WARNING unit.ovn-chassis/0.install File "/var/lib/juju/agents/unit-ovn-chassis-0/charm/lib/charms/layer/basic.py", line 224, in bootstrap_charm_deps
unit-ovn-chassis-0: 11:28:04 WARNING unit.ovn-chassis/0.install check_call([pip, 'install', '-U', reinstall_flag, '--no-index',
unit-ovn-chassis-0: 11:28:04 WARNING unit.ovn-chassis/0.install File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
unit-ovn-chassis-0: 11:28:04 WARNING unit.ovn-chassis/0.install raise CalledProcessError(retcode, cmd)
unit-ovn-chassis-0: 11:28:04 WARNING unit.ovn-chassis/0.install subprocess.CalledProcessError: C...

[Alex Kavanagh (ajkavanagh)]

The jsonschema package is pulled in by layer-ovn; pinning jsonschema < 4.6 stops hatchling from being required. This would be a temporary fix until everything is unpinned.

[Alex Kavanagh (ajkavanagh)]

PR on layer-ovn: https://github.com/openstack-charmers/charm-layer-ovn/pull/64

[Tilman Baumann (tilmanbaumann)]

I think you are onto something there.

But it still fails for me with jsonschema-4.5.1
unit-ovn-chassis-0: 17:41:28 INFO juju.worker.uniter awaiting error resolution for "upgrade-charm" hook
unit-ovn-chassis-0: 17:41:39 WARNING unit.ovn-chassis/0.upgrade-charm ERROR: Command errored out with exit status 1:
unit-ovn-chassis-0: 17:41:39 WARNING unit.ovn-chassis/0.upgrade-charm command: /var/lib/juju/agents/unit-ovn-chassis-0/.venv/bin/python /var/lib/juju/agents/unit-ovn-chassis-0/.venv/lib/python3.8/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-hbce81gi/overlay --no-warn-script-location --no-binary :none: --only-binary :none: --no-index --find-links wheelhouse -- 'setuptools>=40.6.0' 'setuptools_scm[toml]>=3.4' wheel
unit-ovn-chassis-0: 17:41:39 WARNING unit.ovn-chassis/0.upgrade-charm cwd: None
unit-ovn-chassis-0: 17:41:39 WARNING unit.ovn-chassis/0.upgrade-charm Complete output (4 lines):
unit-ovn-chassis-0: 17:41:39 WARNING unit.ovn-chassis/0.upgrade-charm Looking in links: wheelhouse
unit-ovn-chassis-0: 17:41:39 WARNING unit.ovn-chassis/0.upgrade-charm Processing /root/.cache/pip/wheels/18/0c/77/f4a092c136db163656355c7b9960d0d20bfd197e7d63d59dad/setuptools-41.6.0-py2.py3-none-any.whl
unit-ovn-chassis-0: 17:41:39 WARNING unit.ovn-chassis/0.upgrade-charm ERROR: Could not find a version that satisfies the requirement setuptools_scm[toml]>=3.4 (from versions: 1.17.0)
unit-ovn-chassis-0: 17:41:39 WARNING unit.ovn-chassis/0.upgrade-charm ERROR: No matching distribution found for setuptools_scm[toml]>=3.4
unit-ovn-chassis-0: 17:41:39 WARNING unit.ovn-chassis/0.upgrade-charm ----------------------------------------
unit-ovn-chassis-0: 17:41:39 WARNING unit.ovn-chassis/0.upgrade-charm ERROR: Command errored out with exit status 1: /var/lib/juju/agents/unit-ovn-chassis-0/.venv/bin/python /var/lib/juju/agents/unit-ovn-chassis-0/.venv/lib/python3.8/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-hbce81gi/overlay --no-warn-script-location --no-binary :none: --only-binary :none: --no-index --find-links wheelhouse -- 'setuptools>=40.6.0' 'setuptools_scm[toml]>=3.4' wheel Check the logs for full command output.
unit-ovn-chassis-0: 17:41:39 WARNING unit.ovn-chassis/0.upgrade-charm Traceback (most recent call last):
unit-ovn-chassis-0: 17:41:39 WARNING unit.ovn-chassis/0.upgrade-charm File "/var/lib/juju/agents/unit-ovn-chassis-0/charm/hooks/upgrade-charm", line 8, in <module>
unit-ovn-chassis-0: 17:41:39 WARNING unit.ovn-chassis/0.upgrade-charm basic.bootstrap_charm_deps()
unit-ovn-chassis-0: 17:41:39 WARNING unit.ovn-chassis/0.upgrade-charm File "/var/lib/juju/agents/unit-ovn-chassis-0/charm/lib/charms/layer/basic.py", line 224, in bootstrap_charm_deps
unit-ovn-chassis-0: 17:41:39 WARNING unit.ovn-chassis/0.upgrade-charm check_call([pip, 'install', '-U', reinstall_flag, '--no-index',
unit-ovn-chassis-0: 17:41:39 WARNING unit.ovn-chassis/0.upgrade-charm File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
unit-ovn-chassis-0: 17:41:39 WARNING unit.ovn-chassis/0.upgrade-charm raise CalledProcessError(retcode, cmd)
unit-ovn-chassis-0: ...

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/x/charm-ovn-dedicated-chassis/+/854819

[Alex Kavanagh (ajkavanagh)]

So, I got successful builds and installs using charmcraft 1.5/stable. At some point in the near future we'll get charmcraft 2.0/stable working with the charm(s).

[Tilman Baumann (tilmanbaumann)]

It would be a surprise if the build tools make a difference.

I tried stable and master branch.

unit-ovn-chassis-0: 18:39:46 WARNING unit.ovn-chassis/0.upgrade-charm ERROR: Command errored out with exit status 1:
unit-ovn-chassis-0: 18:39:46 WARNING unit.ovn-chassis/0.upgrade-charm command: /var/lib/juju/agents/unit-ovn-chassis-0/.venv/bin/python /var/lib/juju/agents/unit-ovn-chassis-0/.venv/lib/python3.8/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-9t364s9x/overlay --no-warn-script-location --no-binary :none: --only-binary :none: --no-index --find-links wheelhouse -- 'setuptools>=56' 'setuptools_scm[toml]>=3.4.1'
unit-ovn-chassis-0: 18:39:46 WARNING unit.ovn-chassis/0.upgrade-charm cwd: None
unit-ovn-chassis-0: 18:39:46 WARNING unit.ovn-chassis/0.upgrade-charm Complete output (3 lines):
unit-ovn-chassis-0: 18:39:46 WARNING unit.ovn-chassis/0.upgrade-charm Looking in links: wheelhouse
unit-ovn-chassis-0: 18:39:46 WARNING unit.ovn-chassis/0.upgrade-charm ERROR: Could not find a version that satisfies the requirement setuptools>=56 (from versions: 41.6.0)
unit-ovn-chassis-0: 18:39:46 WARNING unit.ovn-chassis/0.upgrade-charm ERROR: No matching distribution found for setuptools>=56
unit-ovn-chassis-0: 18:39:46 WARNING unit.ovn-chassis/0.upgrade-charm ----------------------------------------
unit-ovn-chassis-0: 18:39:46 WARNING unit.ovn-chassis/0.upgrade-charm ERROR: Command errored out with exit status 1: /var/lib/juju/agents/unit-ovn-chassis-0/.venv/bin/python /var/lib/juju/agents/unit-ovn-chassis-0/.venv/lib/python3.8/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-9t364s9x/overlay --no-warn-script-location --no-binary :none: --only-binary :none: --no-index --find-links wheelhouse -- 'setuptools>=56' 'setuptools_scm[toml]>=3.4.1' Check the logs for full command output.
unit-ovn-chassis-0: 18:39:46 WARNING unit.ovn-chassis/0.upgrade-charm Traceback (most recent call last):
unit-ovn-chassis-0: 18:39:46 WARNING unit.ovn-chassis/0.upgrade-charm File "/var/lib/juju/agents/unit-ovn-chassis-0/charm/hooks/upgrade-charm", line 8, in <module>
unit-ovn-chassis-0: 18:39:46 WARNING unit.ovn-chassis/0.upgrade-charm basic.bootstrap_charm_deps()
unit-ovn-chassis-0: 18:39:46 WARNING unit.ovn-chassis/0.upgrade-charm File "/var/lib/juju/agents/unit-ovn-chassis-0/charm/lib/charms/layer/basic.py", line 224, in bootstrap_charm_deps
unit-ovn-chassis-0: 18:39:46 WARNING unit.ovn-chassis/0.upgrade-charm check_call([pip, 'install', '-U', reinstall_flag, '--no-index',
unit-ovn-chassis-0: 18:39:46 WARNING unit.ovn-chassis/0.upgrade-charm File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
unit-ovn-chassis-0: 18:39:46 WARNING unit.ovn-chassis/0.upgrade-charm raise CalledProcessError(retcode, cmd)
unit-ovn-chassis-0: 18:39:46 WARNING unit.ovn-chassis/0.upgrade-charm subprocess.CalledProcessError: Command '['/var/lib/juju/agents/unit-ovn-chassis-0/.venv/bin/pip', 'install', '-U', '--force-reinstall', '--no-index', '--no-cache-dir', '-f', 'wheelhouse', 'dnspython==1.16.0', 'importlib-metadata==4.8.3', 'netaddr==0.7.19', 'attrs==22.1.0', 'netifaces==0....

[Tilman Baumann (tilmanbaumann)]

I see the automated tests for https://review.opendev.org/c/x/charm-ovn-dedicated-chassis/+/854819 failed too. The error looks a bit different. I will try to make sense of it on Monday...

[Tilman Baumann (tilmanbaumann)]

Alex can you have another look at this? It's definitely still broken. (charmcraft 1.5, charmcraft 2.0 or charm-build makes no difference)

[Frode Nordahl (fnordahl)]

Note that the OVN charm stable branches follow the charmhub tracks. I saw you mentioned stable/21.10 earlier, that branch is not in use, please use stable/21.09 instead.

[Tilman Baumann (tilmanbaumann)]

I was actually working with master recently.

We currently have all the openstack charms on stable/21.10 as that was the last openstack release when we did the upgrade.
I'm surprised about 21.09. Are there other charms where 21.10 is not good?

[Alex Kavanagh (ajkavanagh)]

> We currently have all the openstack charms on stable/21.10 as that was the last openstack release when we did the upgrade.
I'm surprised about 21.09. Are there other charms where 21.10 is not good?

It's a bit confusing for sure. So the "21.10" charms was the last pre-charmhub track release of the whole set of charms that the OpenStack team provided. This was pushed to the latest/stable track (ultimately).

Since then, the charms have split into the following groups:

- OpenStack charms
- OVN charms (ovn-chassis, ovn-dedicated-chassis, ovn-central)
- Ceph charms (ceph-osd, -mon, etc.)
- Support charms (rabbitmq-server, mysql-innodb-cluster, mysql-router, hacluster, vault)

The different sets of charms have different tracks on charmhub that mostly map to stable branches on the git repositories.

e.g. if you do a "juju info" for keystone, NO 21.10 track exists, and that charm is only available (currently) on latest/stable. However, yoga, xena, etc. tracks do exist.

For the ceph charms, the track names are the upstream codenames: quincy, pacific, nautilus, etc.

And for the OVN charms, the tracks also follow the upstream codenames which are (unfortunately for our previous naming convention), yy.mm and a recent one was 21.09. But juju info ovn-chassis also shows 22.03, 20.12 and 20.03 as releases.

In order to see how all this maps together, we have some docs at: https://docs.openstack.org/charm-guide/latest/project/charm-delivery.html

So the stable/21.10 branch is no longer a target for development as, for charmhub, there are other active and stable branches: 20.03, 20.12, 21.09, 22.03 and master.

Hope that this helps clarify the situation.

[Tilman Baumann (tilmanbaumann)]

Oh that's very interesting. I wasn't aware of those changes.
I always assumed that latest stable will be backwards compatible to older openstack releases as it always was.

Thanks a lot for the hint. I didn't see those changes.

Oh the complexity...

PS: I suppose the stable/releasename branches follow what's released on charmhub? So for example stable/ussuri will be the right version to build for me while we use ussuri?
I suppose you will backport fixes for it during the lifetime of the release?

PPS: Are you still looking into fixing this charm? Or should it work and I'm doing it wrong?

[Tilman Baumann (tilmanbaumann)]

There is still no branch of this charm that succeeds in building a compatible version via charmcraft.

[Frode Nordahl (fnordahl)]

That is simply not true, we have recently merged several changes to these charms with passing builds and functional tests:

https://review.opendev.org/q/project:x/charm-ovn-chassis+status:merged
https://review.opendev.org/q/project:x/charm-ovn-dedicated-chassis+status:merged
https://review.opendev.org/q/project:x/charm-ovn-central+status:merged

The state of building reactive charms is currently in a bit of a flux and the approach in the charms may unfortunately be a bit convoluted.

We are fortunately working to address this in the tooling:
https://github.com/canonical/charmcraft/pulls?q=is%3Aclosed+is%3Apr+author%3Afnordahl
https://github.com/juju/charm-tools/pull/634/commits/d17d173d8dc653b446c5b7a63f678a64cfdb4a27

[Tilman Baumann (tilmanbaumann)]

Apologies, you are right it works on master branch.

[Tilman Baumann (tilmanbaumann)]

stable/21.09 with charmcraft 1.5 is a working combination. Thanks for helping to point out that I'm using the wrong versions.

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Alex Kavanagh <email address hidden>" on branch: master
Review: https://review.opendev.org/c/x/charm-ovn-chassis/+/854817
Reason: OBE; charm has been rebuilt since, so this is no longer needed.

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Alex Kavanagh <email address hidden>" on branch: master
Review: https://review.opendev.org/c/x/charm-ovn-dedicated-chassis/+/854819
Reason: OBE; charm has been rebuilt since and thus this is no longer needed.