f28 master standalone job fails container-image-prepare - No module named 'pkg_resources'

[Ronelle Landy]

periodic-tripleo-ci-fedora-28-standalone-master is failing the tripleo-container-image-prepare step with the following error:

Traceback (most recent call last):
  File "/usr/bin/tripleo-container-image-prepare", line 23, in <module>
    from tripleo_common.image import image_uploader
  File "/usr/lib/python3.6/site-packages/tripleo_common/image/image_uploader.py", line 33, in <module>
    from oslo_concurrency import processutils
  File "/usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py", line 34, in <module>
    from oslo_utils import timeutils
  File "/usr/lib/python3.6/site-packages/oslo_utils/timeutils.py", line 25, in <module>
    from debtcollector import removals
  File "/usr/lib/python3.6/site-packages/debtcollector/__init__.py", line 20, in <module>
    'debtcollector').version_string()
  File "/usr/lib/python3.6/site-packages/pbr/version.py", line 467, in version_string
    return self.semantic_version().brief_string()
  File "/usr/lib/python3.6/site-packages/pbr/version.py", line 462, in semantic_version
    self._semantic = self._get_version_from_pkg_resources()
  File "/usr/lib/python3.6/site-packages/pbr/version.py", line 439, in _get_version_from_pkg_resources
    import pkg_resources
ModuleNotFoundError: No module named 'pkg_resources'

Full log is linked below:

https://logs.rdoproject.org/openstack-periodic-master/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-fedora-28-standalone-master/cdbb808/logs/undercloud/var/log/tripleo-container-image-prepare.log.txt.gz

https://logs.rdoproject.org/openstack-periodic-master/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-fedora-28-standalone-master/cdbb808/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz#_2019-04-17_13_50_55

[Ronelle Landy (rlandy)]

Watching to see if this error repeats in next promotion

[Ronelle Landy (rlandy)]

This fails in stein as well ...
https://logs.rdoproject.org/openstack-periodic-latest-released/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-fedora-28-standalone-stein/37309be/logs/undercloud/var/log/tripleo-container-image-prepare.log.txt.gz

[wes hayutin (weshayutin)]

https://github.com/rdo-packages/tripleo-common-distgit/blob/rpm-master/openstack-tripleo-common.spec#L63

https://github.com/rdo-infra/fedora-stable-config/blob/master/fedora_constraints#L1682

Hrm.. odd the module is not installed.

[wes hayutin (weshayutin)]

python2 and python3 setuptools is installed on the box
https://logs.rdoproject.org/openstack-periodic-latest-released/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-fedora-28-standalone-stein/37309be/logs/undercloud/var/log/extra/rpm-list.txt.gz

[Quique Llorente (quiquell)]

Let's try to reproduce this

[Quique Llorente (quiquell)]

Reproducer is able to run the image prepare

Maybe this is caused by https://review.opendev.org/#/c/652066/

[Quique Llorente (quiquell)]

dib elements at upstream for the image
https://opendev.org/openstack/project-config/src/branch/master/nodepool/nodepool.yaml#L163-L184

dib elements at RDO for the image
https://softwarefactory-project.io/r/gitweb?p=config.git;a=blob;f=nodepool/rdo-cloud.yaml;h=0bf89db118e00aaf3dac5baaef6323ad5ebd7d1d;hb=refs/heads/master#l149

They look the same :-/

[Quique Llorente (quiquell)]

Testing with revert https://review.rdoproject.org/r/#/c/20419/
Testing without revert https://review.rdoproject.org/r/#/c/20420/

[Quique Llorente (quiquell)]

Looks like revert works, but we have to discover why is not needed upstream.

[Quique Llorente (quiquell)]

After holding nodes with and without the revert, looks like f28 rdo dib has the python3-setuptools RPM installed but the content deleted

[zuul@upstream-fedora-28-rdo-cloud-tripleo-0001921565 ~]$ ls -la /usr/lib/python3.6/site-packages/pkg_resources/
ls: cannot access '/usr/lib/python3.6/site-packages/pkg_resources/': No such file or directory

while the one with pytho3-setuptool reinstalled have it right
[zuul@standalone ~]$ ls -la /usr/lib/python3.6/site-packages/pkg_resources/
total 148
drwxr-xr-x. 5 root root 4096 Apr 23 11:02 .
drwxr-xr-x. 387 root root 20480 Apr 23 11:30 ..
-rw-r--r--. 1 root root 106677 Mar 1 12:06 __init__.py
drwxr-xr-x. 2 root root 4096 Apr 23 11:02 __pycache__
drwxr-xr-x. 4 root root 4096 Apr 23 11:02 _vendor
drwxr-xr-x. 3 root root 4096 Apr 23 11:02 extern
-rw-r--r--. 1 root root 558 Mar 1 12:06 py31compat.py

[Quique Llorente (quiquell)]

This is the point were exclude is added to the fedora-28 dib

https://github.com/openstack/diskimage-builder/blob/37dff9738afd245772cf029995476f29e68fb0f0/diskimage_builder/elements/pip-and-virtualenv/install.d/pip-and-virtualenv-source-install/04-install-pip#L188

[Quique Llorente (quiquell)]

Could be that RDO nodepool dib is using old version of dib and does not do the following
https://github.com/openstack/diskimage-builder/commit/ea1735b6a259c5424d2a0ca52eca371bf9213ac9

[Quique Llorente (quiquell)]

diskimage-builder versin at RDO 2.18.0

https://softwarefactory-project.io/nodepool-log/upstream-fedora-28-0000000297.log

[Quique Llorente (quiquell)]

So 2.18.0 is on 4 Nov 2018 and does not have the fix this is the issue.

[Quique Llorente (quiquell)]

Fix https://softwarefactory-project.io/r/#/c/15427/

[Quique Llorente (quiquell)]

Looks like also image are now wrongly build as fedora-29

fix: https://softwarefactory-project.io/r/#/c/15430/