tripleo

openstack: 'undercloud install' is not an openstack command. See 'openstack --help'.

Bug #1921679 reported by chandan kumar
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
Critical
Unassigned
tripleo xena-2

Bug Description

ussuri promotion pipeline is broken for standalone, undercloud and ovb jobs at openstack cli invocations.
https://logserver.rdoproject.org/openstack-periodic-integration-stable2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-ussuri/f00fcb5/logs/undercloud/home/zuul/undercloud_install.log.txt.gz

```
2021-03-29 05:26:01 | + openstack undercloud install
2021-03-29 05:26:03 | WARNING: Failed to import plugin identity.
2021-03-29 05:26:03 | WARNING: Failed to import plugin tripleoclient.
2021-03-29 05:26:03 | openstack: 'undercloud install' is not an openstack command. See 'openstack --help'.
2021-03-29 05:26:03 | Did you mean one of these?
2021-03-29 05:26:03 | hypervisor list
2021-03-29 05:26:03 | hypervisor show
2021-03-29 05:26:03 | hypervisor stats show
2021-03-29 05:26:03 | versions show

```

and

https://logserver.rdoproject.org/openstack-periodic-integration-stable2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-ussuri/b59637b/job-output.txt

```
2021-03-29 05:12:35.107534 | primary | TASK [tripleo.operator.tripleo_container_image_prepare_default : Container image prepare default] ***
2021-03-29 05:12:35.107600 | primary | Monday 29 March 2021 05:12:35 +0000 (0:00:00.043) 0:05:14.404 **********
2021-03-29 05:12:36.792646 | primary | fatal: [undercloud]: FAILED! => {
2021-03-29 05:12:36.792696 | primary | "changed": true,
2021-03-29 05:12:36.792704 | primary | "cmd": "openstack tripleo container image prepare default --output-env-file /home/zuul/containers-prepare-parameters.yaml ",
2021-03-29 05:12:36.792711 | primary | "delta": "0:00:01.367517",
2021-03-29 05:12:36.792729 | primary | "end": "2021-03-29 05:12:36.766959",
2021-03-29 05:12:36.792736 | primary | "rc": 2,
2021-03-29 05:12:36.792742 | primary | "start": "2021-03-29 05:12:35.399442"
2021-03-29 05:12:36.792747 | primary | }
2021-03-29 05:12:36.792754 | primary |
2021-03-29 05:12:36.792759 | primary | STDOUT:
2021-03-29 05:12:36.792765 | primary |
2021-03-29 05:12:36.792771 | primary | openstack: 'tripleo container image prepare default --output-env-file /home/zuul/containers-prepare-parameters.yaml' is not an openstack command. See 'openstack --help'.
2021-03-29 05:12:36.792777 | primary | Did you mean one of these?
2021-03-29 05:12:36.792783 | primary | tsigkey create
2021-03-29 05:12:36.792789 | primary | tsigkey delete
2021-03-29 05:12:36.792795 | primary | tsigkey list
2021-03-29 05:12:36.792801 | primary | tsigkey set
2021-03-29 05:12:36.792824 | primary | tsigkey show
2021-03-29 05:12:36.792829 | primary |
2021-03-29 05:12:36.792835 | primary |
2021-03-29 05:12:36.792841 | primary | STDERR:
2021-03-29 05:12:36.792847 | primary |
2021-03-29 05:12:36.792852 | primary | WARNING: Failed to import plugin identity.
2021-03-29 05:12:36.792858 | primary | WARNING: Failed to import plugin tripleoclient.
```

While looking at

https://logserver.rdoproject.org/openstack-periodic-integration-stable2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-ussuri/f00fcb5/logs/undercloud/home/zuul/install_packages.sh.log.txt.gz

```
2021-03-29 05:15:45 | python3-tripleoclient noarch 13.4.3-0.20210327084048.71693e3.el8 delorean-current 535 k

2021-03-29 05:15:45 | (56/518): python3-openstackclient-5.2.0-0.20200 40 MB/s | 1.1 MB 00:00
```

There might be some issue with openstackclient and python-tripleoclient package.

Revision history for this message
chandan kumar (chkumar246) wrote :

We are seeing similar failure on train main line: https://logserver.rdoproject.org/openstack-periodic-integration-stable3/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-full-tempest-scenario-train/50a9cc2/logs/undercloud/home/zuul/standalone_container_prep.log.txt.gz

```
2021-03-29 05:00:56 | WARNING: Failed to import plugin identity.
2021-03-29 05:00:56 | WARNING: Failed to import plugin tripleoclient.
2021-03-29 05:00:56 | openstack: 'tripleo container image prepare default --output-env-file /home/zuul/containers-prepare-parameters.yaml' is not an openstack command. See 'openstack --help'.
2021-03-29 05:00:56 | Did you mean one of these?
2021-03-29 05:00:56 | tsigkey create
2021-03-29 05:00:56 | tsigkey delete
2021-03-29 05:00:56 | tsigkey list
2021-03-29 05:00:56 | tsigkey set
2021-03-29 05:00:56 | tsigkey show
```

Revision history for this message
Rabi Mishra (rabi) wrote :
Download full text (3.6 KiB)

So I had a look at the node provided by Chandan. While debugging it seems to fail loading the identity plugin with the below traceback. Looks like some 'keyring.backends' entrypoints are incorrect. Very difficult to know what has caused this.

WARNING: Failed to import plugin identity.
Traceback (most recent call last):
  File "/usr/bin/openstack", line 6, in <module>
    from openstackclient.shell import main
  File "/usr/lib/python3.6/site-packages/openstackclient/shell.py", line 28, in <module>
    from openstackclient.common import clientmanager
  File "/usr/lib/python3.6/site-packages/openstackclient/common/clientmanager.py", line 183, in <module>
    'openstack.cli.base',
  File "/usr/lib/python3.6/site-packages/openstackclient/common/clientmanager.py", line 149, in get_plugin_modules
    __import__(ep.module_name)
  File "/usr/lib/python3.6/site-packages/openstackclient/identity/client.py", line 18, in <module>
    from keystoneclient.v2_0 import client as identity_client_v2
  File "/usr/lib/python3.6/site-packages/keystoneclient/v2_0/__init__.py", line 1, in <module>
    from keystoneclient.v2_0.client import Client # noqa
  File "/usr/lib/python3.6/site-packages/keystoneclient/v2_0/client.py", line 21, in <module>
    from keystoneclient import httpclient
  File "<frozen importlib._bootstrap>", line 1020, in _handle_fromlist
  File "/usr/lib/python3.6/site-packages/keystoneclient/__init__.py", line 72, in __getattr__
    return importlib.import_module('keystoneclient.%s' % name)
  File "/usr/lib64/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/usr/lib/python3.6/site-packages/keystoneclient/httpclient.py", line 43, in <module>
    import keyring
  File "/usr/lib/python3.6/site-packages/keyring/__init__.py", line 1, in <module>
    from .core import (
  File "/usr/lib/python3.6/site-packages/keyring/core.py", line 192, in <module>
    init_backend()
  File "/usr/lib/python3.6/site-packages/keyring/core.py", line 96, in init_backend
    filter(limit, backend.get_all_keyring()),
  File "/usr/lib/python3.6/site-packages/keyring/util/__init__.py", line 22, in wrapper
    func.always_returns = func(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/keyring/backend.py", line 216, in get_all_keyring
    _load_plugins()
  File "/usr/lib/python3.6/site-packages/keyring/backend.py", line 199, in _load_plugins
    entry_points = metadata.entry_points()['keyring.backends']
  File "/usr/local/lib/python3.6/site-packages/importlib_metadata/__init__.py", line 893, in entry_points
    return SelectableGroups.load(eps).select(**params)
  File "/usr/local/lib/python3.6/site-packages/importlib_metadata/__init__.py", line 331, in load
    ordered = sorted(eps, key=by_group)
  File "/usr/local/lib/python3.6/site-packages/importlib_metadata/__init__.py", line 891, in <genexpr>
    dist.entry_points for dist in unique(distributions())
  File "/usr/local/lib/python3.6/site-packages/importlib_metadata/__init__.py", line 517, in entry_points
    return EntryPoints._from_text_for(self.read_text('entry_points.txt'), self)
  File "/usr/local/lib/python3.6/site-packages...

Read more...

Revision history for this message
chandan kumar (chkumar246) wrote :

https://logserver.rdoproject.org/69/781069/2/openstack-check/tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039/368a301/logs/undercloud/home/zuul/undercloud_install.log.txt.gz

It might be related to the same issue
```
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/stevedore/_cache.py", line 159, in _get_data_for_path
    with open(filename, 'r') as f:
FileNotFoundError: [Errno 2] No such file or directory: '/home/zuul/.cache/python-entrypoints/c9ec26dbebd9eb84e915ddfbe88b919d10a8377885ca0efefcf347dfad5449dd'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/openstack", line 6, in <module>
    from openstackclient.shell import main
  File "/usr/lib/python3.6/site-packages/openstackclient/shell.py", line 26, in <module>
    from openstackclient.common import clientmanager
  File "/usr/lib/python3.6/site-packages/openstackclient/common/clientmanager.py", line 196, in <module>
    'openstack.cli.base',
  File "/usr/lib/python3.6/site-packages/openstackclient/common/clientmanager.py", line 147, in get_plugin_modules
    mgr = stevedore.ExtensionManager(group)
  File "/usr/lib/python3.6/site-packages/stevedore/extension.py", line 136, in __init__
    verify_requirements)
  File "/usr/lib/python3.6/site-packages/stevedore/extension.py", line 218, in _load_plugins
    for ep in self.list_entry_points():
  File "/usr/lib/python3.6/site-packages/stevedore/extension.py", line 207, in list_entry_points
    eps = list(_cache.get_group_all(self.namespace))
  File "/usr/lib/python3.6/site-packages/stevedore/_cache.py", line 179, in get_group_all
    data = self._get_data_for_path(path)
  File "/usr/lib/python3.6/site-packages/stevedore/_cache.py", line 162, in _get_data_for_path
    data = _build_cacheable_data(path)
  File "/usr/lib/python3.6/site-packages/stevedore/_cache.py", line 107, in _build_cacheable_data
    real_groups = importlib_metadata.entry_points()
  File "/usr/local/lib/python3.6/site-packages/importlib_metadata/__init__.py", line 893, in entry_points
    return SelectableGroups.load(eps).select(**params)
  File "/usr/local/lib/python3.6/site-packages/importlib_metadata/__init__.py", line 331, in load
    ordered = sorted(eps, key=by_group)
  File "/usr/local/lib/python3.6/site-packages/importlib_metadata/__init__.py", line 891, in <genexpr>
    dist.entry_points for dist in unique(distributions())
  File "/usr/local/lib/python3.6/site-packages/importlib_metadata/__init__.py", line 517, in entry_points
    return EntryPoints._from_text_for(self.read_text('entry_points.txt'), self)
  File "/usr/local/lib/python3.6/site-packages/importlib_metadata/__init__.py", line 244, in _from_text_for
    return cls(ep._for(dist) for ep in cls._from_text(text))
  File "/usr/local/lib/python3.6/site-packages/importlib_metadata/__init__.py", line 244, in <genexpr>
    return cls(ep._for(dist) for ep in cls._from_text(text))
  File "/usr/local/lib/python3.6/site-packages/importlib_metadata/__init__.py", line 255, in <genexpr>
    for name, value in values
ValueError: not enough values to unpack (expected 2, got 1)

```

Revision history for this message
Rabi Mishra (rabi) wrote :

This seems to be due to importlib_metadata 3.9.0.

This does not seem to parse the comments in entry_points.txt properly after[1]. Removing the comment in paste[2] fixes it/

[1] https://github.com/python/importlib_metadata/commit/01cf7b4591671b5a44c28cb054e07074ce0ee760

[2] https://github.com/cdent/paste/blob/3.2.4/setup.py#L91

Revision history for this message
Rabi Mishra (rabi) wrote :

I don't know why ussuri and train are using latest importlib_metadata.

Revision history for this message
chandan kumar (chkumar246) wrote :

Based on rdoinfo https://github.com/redhat-openstack/rdoinfo/search?q=importlib

for ussuri and train - python-importlib-metadata-0.23-1.el8 and for victoria onwards python-importlib-metadata-1.7.0-1.el8 is used

Revision history for this message
Rabi Mishra (rabi) wrote :

[zuul@standalone ~]$ sudo pip freeze | grep importlib-metadata
importlib-metadata==3.9.0
[zuul@standalone ~]$ rpm -qa | grep importlib-metadata
python3-importlib-metadata-0.23-1.el8.noarch
[zuul@standalone ~]$

Revision history for this message
chandan kumar (chkumar246) wrote :

By looking at https://logserver.rdoproject.org/openstack-periodic-integration-stable2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-ussuri/f00fcb5/logs/undercloud/var/log/extra/pip.txt.gz

importlib-metadata (3.9.0) latest is getting installed via pip somewhere, which might be causing issue due to mixing of pip and rpm packages.

Revision history for this message
Rabi Mishra (rabi) wrote :

We seem to install bunch of stuff from source including tripleo-ansible with tripleo-quickstart.

https://github.com/openstack/tripleo-quickstart/blob/master/quickstart-extras-requirements.txt#L15

I've tried to blacklist importlib-metadata with https://review.opendev.org/c/openstack/tripleo-ansible/+/783636 to see if it helps.

Revision history for this message
chandan kumar (chkumar246) wrote :

testing the above patch here: https://review.rdoproject.org/r/c/testproject/+/32855

Revision history for this message
Sorin Sbarnea (ssbarnea) wrote :

Based on the error received I have strong reasons to believe is caused by use of an old version of stevedore library, as I found a bug fixed in November 2020 which seems to have addressed a random error related to access to /tmp, one particularly affecting ansible users. That merged change was included in 3.3.0 release but we are using a much older version than this.

To be clear, I am referring to this exception in particular:

    with open(filename, 'r') as f:
2021-03-29 11:40:03.930165 | primary | FileNotFoundError: [Errno 2] No such file or directory: '/home/zuul/.cache/python-entrypoints/894a0d78993ecb74fe95b38bd9286ed9413fc78f0bed7b0e8625f68b791f05eb'
2021-03-29 11:40:03.930174 | primary |

I believe the other ones are side effects of this one.

wes hayutin (weshayutin)
Changed in tripleo:
importance: High → Critical
Revision history for this message
Sorin Sbarnea (ssbarnea) wrote :

An hour ago a hotfix for https://pypi.org/project/importlib-metadata/3.9.1/ was released which is supposed to address this issue.

Once we get the confirmation that this is true, we can close this as fixed.

Revision history for this message
Rabi Mishra (rabi) wrote :

Though latest importlib_metadata fixes the regression of parsing comments in entrypoints, we should understand and fix why we're pip installing the latest which can cause these kind of issues from time to time.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on tripleo-ansible (master)

Change abandoned by "Rabi Mishra <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/tripleo-ansible/+/783636

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-image-elements (stable/victoria)

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/tripleo-image-elements/+/788003

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-image-elements (stable/ussuri)

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/tripleo-image-elements/+/788004

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-image-elements (stable/train)

Fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/tripleo-image-elements/+/788005

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-image-elements (stable/train)

Reviewed: https://review.opendev.org/c/openstack/tripleo-image-elements/+/788005
Committed: https://opendev.org/openstack/tripleo-image-elements/commit/44c79f626b3a66d07b71fb12cf416286b55d614b
Submitter: "Zuul (22348)"
Branch: stable/train

commit 44c79f626b3a66d07b71fb12cf416286b55d614b
Author: Sorin Sbarnea <email address hidden>
Date: Mon Mar 29 13:13:16 2021 +0100

    Drop pbr version ceiling

    - Ceiling of pbr version can create installation conflicts with pip.
    - Fixes tox default env list to be python version agnostic
    - Includes hacking version bump to fix dependencies

    NOTE: Partial backport of the master fix. This patch excludes the py39
    job updates.

    Bug: #1921679
    Change-Id: Ifd0d2f810799992faa1db41b80bf93bc97d5d19d
    (cherry picked from commit c69ead8b45d28befc693a5d0583868cef501ff25)
    (cherry picked from commit 55b2437394abddc23be5781aa495ffabf24f2f31)
    (cherry picked from commit 2b5e3aaefdeb3e1de5d47efdba5e5d82952b89b3)

tags: added: in-stable-train
tags: added: in-stable-ussuri
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-image-elements (stable/ussuri)

Reviewed: https://review.opendev.org/c/openstack/tripleo-image-elements/+/788004
Committed: https://opendev.org/openstack/tripleo-image-elements/commit/2b5e3aaefdeb3e1de5d47efdba5e5d82952b89b3
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit 2b5e3aaefdeb3e1de5d47efdba5e5d82952b89b3
Author: Sorin Sbarnea <email address hidden>
Date: Mon Mar 29 13:13:16 2021 +0100

    Drop pbr version ceiling

    - Ceiling of pbr version can create installation conflicts with pip.
    - Fixes tox default env list to be python version agnostic
    - Includes hacking version bump to fix dependencies

    NOTE: Partial backport of the master fix. This patch excludes the py39
    job updates.

    Bug: #1921679
    Change-Id: Ifd0d2f810799992faa1db41b80bf93bc97d5d19d
    (cherry picked from commit c69ead8b45d28befc693a5d0583868cef501ff25)
    (cherry picked from commit 55b2437394abddc23be5781aa495ffabf24f2f31)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-image-elements (stable/victoria)

Reviewed: https://review.opendev.org/c/openstack/tripleo-image-elements/+/788003
Committed: https://opendev.org/openstack/tripleo-image-elements/commit/b73f4154cab14ed635f2ab25fad528360b509a21
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit b73f4154cab14ed635f2ab25fad528360b509a21
Author: Sorin Sbarnea <email address hidden>
Date: Mon Mar 29 13:13:16 2021 +0100

    Drop pbr version ceiling

    - Ceiling of pbr version can create installation conflicts with pip.
    - Fixes tox default env list to be python version agnostic
    - Includes hacking version bump to fix dependencies

    NOTE: Partial backport of the master fix. This patch excludes the py39
    job updates.

    Bug: #1921679
    Change-Id: Ifd0d2f810799992faa1db41b80bf93bc97d5d19d
    (cherry picked from commit c69ead8b45d28befc693a5d0583868cef501ff25)

tags: added: in-stable-victoria
Marios Andreou (marios-b)
Changed in tripleo:
milestone: wallaby-rc1 → xena-1
Revision history for this message
Marios Andreou (marios-b) wrote :

i am seeing something very similar on undercloud upgrade master job (working on re-adding that) and filed https://bugs.launchpad.net/tripleo/+bug/1930416 but i think it may be duplicate? not sure yet but similar error

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-image-elements (stable/wallaby)

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/tripleo-image-elements/+/794632

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on tripleo-image-elements (stable/wallaby)

Change abandoned by "Sorin Sbârnea <email address hidden>" on branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/tripleo-image-elements/+/794632

Marios Andreou (marios-b)
Changed in tripleo:
milestone: xena-1 → xena-2
Revision history for this message
Marios Andreou (marios-b) wrote :

moving fix released - we are no longer seeing the issue (master undercloud upgrade is now failing on something else :/ but gets passed this point)

Changed in tripleo:
status: Triaged → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.