charm build fails due to duplicate wheelhouse dependency

[Frode Nordahl]

charmtools.build.tactics: Processing wheelhouse:
charmtools.build.tactics: # layer:basic
charmtools.build.tactics: # pip is pinned to <19.0 to avoid https://github.com/pypa/pip/issues/6164
charmtools.build.tactics: # even with installing setuptools before upgrading pip ends up with pip seeing
charmtools.build.tactics: # the older setuptools at the system level if include_system_packages is true
charmtools.build.tactics: pip>=18.1,<19.0
charmtools.build.tactics: # pin Jinja2 and PyYAML to the last versions supporting python 3.4 for trusty
charmtools.build.tactics: Jinja2<=2.10.1
charmtools.build.tactics: PyYAML<=5.2
charmtools.build.tactics: setuptools<42
charmtools.build.tactics: setuptools-scm<=1.17.0
charmtools.build.tactics: charmhelpers>=0.4.0,<1.0.0
charmtools.build.tactics: charms.reactive>=0.1.0,<2.0.0
charmtools.build.tactics: wheel<0.34
charmtools.build.tactics: # pin netaddr to avoid pulling importlib-resources
charmtools.build.tactics: netaddr<=0.7.19
charmtools.build.tactics:
charmtools.build.tactics: # layer:openstack
charmtools.build.tactics: # charmhelpers.contrib.openstack.cert_utils charms with certificates relation
charmtools.build.tactics: dnspython
charmtools.build.tactics: # charmhelpers.contrib.network all
charmtools.build.tactics: # netifaces>=0.10.4 # overridden by ceph-fs
charmtools.build.tactics: # charmhelpers.contrib.openstack.context all
charmtools.build.tactics: # psutil # overridden by ceph-fs
charmtools.build.tactics: # charms_openstack - pull from github until first release
charmtools.build.tactics: git+https://github.com/openstack/charms.openstack.git#egg=charms.openstack
charmtools.build.tactics:
charmtools.build.tactics: # layer:ceph
charmtools.build.tactics: git+https://github.com/openstack/charms.ceph.git#egg=charms.ceph
charmtools.build.tactics:
charmtools.build.tactics: # ceph-fs
charmtools.build.tactics: netifaces
charmtools.build.tactics: dnspython3
charmtools.build.tactics: ceph_api
charmtools.build.tactics: pyxattr
charmtools.build.tactics: jinja2
charmtools.build.tactics: psutil
charmtools.build.tactics:
utils: b"ERROR: Double requirement given: jinja2 (from -r /tmp/tmpaw45nofw/wheelhouse.txt (line 35)) (already in Jinja2<=2.10.1 (from -r /tmp/tmpaw45nofw/wheelhouse.txt (line 7)), name='jinja2')\n"
utils: process: bash -c . /tmp/tmpm3_6dn_0/bin/activate ; pip3 download --no-binary :all: -d /tmp/tmpgntmhu5b -r /tmp/tmpaw45nofw/wheelhouse.txt (1)
build: Traceback (most recent call last):
  File "/var/lib/jenkins/workspace/charm_pusher_ceph-fs_master/ceph-fs/.tox/build/lib/python3.5/site-packages/charmtools/build/builder.py", line 952, in main
    build()
  File "/var/lib/jenkins/workspace/charm_pusher_ceph-fs_master/ceph-fs/.tox/build/lib/python3.5/site-packages/charmtools/build/builder.py", line 652, in __call__
    self.generate()
  File "/var/lib/jenkins/workspace/charm_pusher_ceph-fs_master/ceph-fs/.tox/build/lib/python3.5/site-packages/charmtools/build/builder.py", line 597, in generate
    self.exec_plan(self.plan, self.layers)
  File "/var/lib/jenkins/workspace/charm_pusher_ceph-fs_master/ceph-fs/.tox/build/lib/python3.5/site-packages/charmtools/build/builder.py", line 570, in exec_plan
    tactic()
  File "/var/lib/jenkins/workspace/charm_pusher_ceph-fs_master/ceph-fs/.tox/build/lib/python3.5/site-packages/charmtools/build/tactics.py", line 1138, in __call__
    self._process_combined(wheelhouse)
  File "/var/lib/jenkins/workspace/charm_pusher_ceph-fs_master/ceph-fs/.tox/build/lib/python3.5/site-packages/charmtools/build/tactics.py", line 1160, in _process_combined
    self._add(wheelhouse, '-r', wh_file)
  File "/var/lib/jenkins/workspace/charm_pusher_ceph-fs_master/ceph-fs/.tox/build/lib/python3.5/site-packages/charmtools/build/tactics.py", line 1095, in _add
    *reqs)
  File "/var/lib/jenkins/workspace/charm_pusher_ceph-fs_master/ceph-fs/.tox/build/lib/python3.5/site-packages/charmtools/build/tactics.py", line 1124, in _pip
    return self._run_in_venv('pip3', *args)
  File "/var/lib/jenkins/workspace/charm_pusher_ceph-fs_master/ceph-fs/.tox/build/lib/python3.5/site-packages/charmtools/build/tactics.py", line 1120, in _run_in_venv
    raise BuildError(res.output)
charmtools.build.errors.BuildError: ERROR: Double requirement given: jinja2 (from -r /tmp/tmpaw45nofw/wheelhouse.txt (line 35)) (already in Jinja2<=2.10.1 (from -r /tmp/tmpaw45nofw/wheelhouse.txt (line 7)), name='jinja2')

build: ERROR: Double requirement given: jinja2 (from -r /tmp/tmpaw45nofw/wheelhouse.txt (line 35)) (already in Jinja2<=2.10.1 (from -r /tmp/tmpaw45nofw/wheelhouse.txt (line 7)), name='jinja2')
ERROR: InvocationError: '/var/lib/jenkins/workspace/charm_pusher_ceph-fs_master/ceph-fs/.tox/build/bin/charm-build --log-level DEBUG -o /var/lib/jenkins/workspace/charm_pusher_ceph-fs_master/ceph-fs/build src'
___________________________________ summary ____________________________________
ERROR: build: commands failed
 . /var/lib/jenkins/workspace/charm_pusher_ceph-fs_master/ceph-fs is a multi-series charm based on its metadata.yaml
 ! Built charm will not pass later proof tests: /var/lib/jenkins/workspace/charm_pusher_ceph-fs_master/ceph-fs/build/builds/ceph-fs
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording fingerprints
No emails were triggered.
Finished: FAILURE

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/738875

[Aurelien Lourot (aurelien-lourot)]

Also hitting http://osci:8080/job/charm_pusher_ceph-rbd-mirror_master/32/consoleFull

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/738892

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/738892
Committed: https://git.openstack.org/cgit/openstack/charm-ceph-rbd-mirror/commit/?id=43a209d09b7796af2db1bfafd23a15024c4ad9a6
Submitter: Zuul
Branch: master

commit 43a209d09b7796af2db1bfafd23a15024c4ad9a6
Author: Frode Nordahl <email address hidden>
Date: Wed Jul 1 18:52:23 2020 +0200

    Remove duplicate requirement

    Change-Id: I6ecc780ce05ad7e9d392888139f80678e2b343f5
    Closes-Bug: #1885935

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/738875
Committed: https://git.openstack.org/cgit/openstack/charm-ceph-fs/commit/?id=4c02fa9c420e99ceb3bc0f2e667cbf611c3fe290
Submitter: Zuul
Branch: master

commit 4c02fa9c420e99ceb3bc0f2e667cbf611c3fe290
Author: Frode Nordahl <email address hidden>
Date: Wed Jul 1 18:00:57 2020 +0200

    Remove duplicate requirement

    Func-Test-Pr: https://github.com/openstack-charmers/zaza-openstack-tests/pull/342
    Change-Id: I2137edb851fbb7926f372783fa140d89190c813b
    Closes-Bug: #1885935

[Aurelien Lourot (aurelien-lourot)]

Similarly hitting neutron-api-plugin-ovn

http://osci:8080/job/charm_pusher_neutron-api-plugin-ovn_stable/1/consoleFull

build: ERROR: Double requirement given: charmhelpers from git+https://github.com/juju/charm-helpers.git@stable/20.05#egg=charmhelpers (from -r /tmp/tmpn7y57j7s/wheelhouse.txt (line 32)) (already in charmhelpers<1.0.0,>=0.4.0 (from -r /tmp/tmpn7y57j7s/wheelhouse.txt (line 11)), name='charmhelpers')

[Aurelien Lourot (aurelien-lourot)]

Also hits mysql-innodb-cluster: https://review.opendev.org/#/c/739736/

[Aurelien Lourot (aurelien-lourot)]

Hits charm-barbican-softhsm as well: https://review.opendev.org/#/c/739932/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/739932
Committed: https://git.openstack.org/cgit/openstack/charm-barbican-softhsm/commit/?id=d9b05879b322ee60dc7ec9b8a17a01bc89e730dd
Submitter: Zuul
Branch: master

commit d9b05879b322ee60dc7ec9b8a17a01bc89e730dd
Author: Aurelien Lourot <email address hidden>
Date: Wed Jul 8 11:15:25 2020 +0200

    Add Ussuri and Victoria tests

    Closes-Bug: #1885935
    Change-Id: Ia4ab72a73ab44c3263d627bc7f18c1dbafaae2ad

[Aurelien Lourot (aurelien-lourot)]

hitting manila-ganesha as well: http://osci:8080/job/charm_pusher_manila-ganesha_master/15/consoleFull

[Aurelien Lourot (aurelien-lourot)]

hitting neutron-dynamic-routing as well: http://osci:8080/job/charm_pusher_neutron-dynamic-routing_master/57/consoleFull

[Aurelien Lourot (aurelien-lourot)]

hitting masakari-monitors as well: http://osci:8080/job/charm_pusher_masakari-monitors_master/69/consoleFull

[Aurelien Lourot (aurelien-lourot)]

hitting keystone-saml-mellon as well: http://osci:8080/job/charm_pusher_keystone-saml-mellon_master/16/consoleFull

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/739736
Committed: https://git.openstack.org/cgit/openstack/charm-mysql-innodb-cluster/commit/?id=5fa577e818420e0f559b87bf9ac1d41df4de4119
Submitter: Zuul
Branch: master

commit 5fa577e818420e0f559b87bf9ac1d41df4de4119
Author: Aurelien Lourot <email address hidden>
Date: Tue Jul 7 14:58:59 2020 +0200

    Add Groovy test bundles

    Closes-Bug: #1885935
    Change-Id: Iac4ec3a98c66164d3d6e254ebe41e46e4e8ba5df

[David Ames (thedac)]

Note: *All* stable branch charms will fail due to the charm helpers override until [0] is fixed.

i.e. neutron-api-pluing-ovn passes at master but fails at stable.

[0] https://github.com/juju/charm-tools/issues/571

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/740807

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/740808

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/740735
Committed: https://git.openstack.org/cgit/openstack/charm-keystone-saml-mellon/commit/?id=979f2969dd7695ff978f9534b2acd80c5df9b513
Submitter: Zuul
Branch: master

commit 979f2969dd7695ff978f9534b2acd80c5df9b513
Author: David Ames <email address hidden>
Date: Mon Jul 13 10:26:01 2020 -0700

    Remove duplicate dependency: Jinja2

    Closes-Bug: #1885935
    Change-Id: I9911a3a43956f49d8fad76aa872cd96ce809aae4

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/740734
Committed: https://git.openstack.org/cgit/openstack/charm-neutron-dynamic-routing/commit/?id=c94e0f3067c2f544268f6ef4cc1b49f3762d057d
Submitter: Zuul
Branch: master

commit c94e0f3067c2f544268f6ef4cc1b49f3762d057d
Author: David Ames <email address hidden>
Date: Mon Jul 13 10:18:47 2020 -0700

    Remove duplicate dependency: Jinja2

    Closes-Bug: #1885935
    Change-Id: If970fdc135c78c7a5ad1ae14daba3e96b9363c39

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/740808
Committed: https://git.openstack.org/cgit/openstack/charm-manila-ganesha/commit/?id=dc4199a94522d953affaf5983e428236b690c8fc
Submitter: Zuul
Branch: master

commit dc4199a94522d953affaf5983e428236b690c8fc
Author: David Ames <email address hidden>
Date: Mon Jul 13 13:45:47 2020 -0700

    Remove duplicate dependency: Jinja2

    Change-Id: I309c84081cfd18ac4fac1c4cc488a6e7da2be27d
    Closes-Bug: #1885935

[David Ames (thedac)]

With the landing of [0] stable branches' overrides should now work. Any stable branches that still fail have a duplicate (non-override) dependency that will need to be corrected.

[0] https://github.com/juju/charm-tools/pull/572

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/740807
Committed: https://git.openstack.org/cgit/openstack/charm-masakari-monitors/commit/?id=0f7e7851657e8c9f7e1ec1e0fec8a40c42d0a940
Submitter: Zuul
Branch: master

commit 0f7e7851657e8c9f7e1ec1e0fec8a40c42d0a940
Author: David Ames <email address hidden>
Date: Mon Jul 13 13:43:08 2020 -0700

    Remove duplicate dependency: Jinja2

    Change-Id: I78bbc765a67e8f92a618a7be1aed86b37f41051d
    Closes-Bug: #1885935