openstack command failed in kolla_toolbox container

[hanyi-tsai]

My kolla-ansible was failed at the Task: [nova : Waiting for nova-compute service up]

Error msg:

FAILED - RETRYING: Waiting for nova-compute service up (20 retries left).
FAILED - RETRYING: Waiting for nova-compute service up (19 retries left).
FAILED - RETRYING: Waiting for nova-compute service up (18 retries left).
FAILED - RETRYING: Waiting for nova-compute service up (17 retries left).
FAILED - RETRYING: Waiting for nova-compute service up (16 retries left).
FAILED - RETRYING: Waiting for nova-compute service up (15 retries left).
FAILED - RETRYING: Waiting for nova-compute service up (14 retries left).
FAILED - RETRYING: Waiting for nova-compute service up (13 retries left).
FAILED - RETRYING: Waiting for nova-compute service up (12 retries left).
FAILED - RETRYING: Waiting for nova-compute service up (11 retries left).
FAILED - RETRYING: Waiting for nova-compute service up (10 retries left).
FAILED - RETRYING: Waiting for nova-compute service up (9 retries left).
FAILED - RETRYING: Waiting for nova-compute service up (8 retries left).
FAILED - RETRYING: Waiting for nova-compute service up (7 retries left).
FAILED - RETRYING: Waiting for nova-compute service up (6 retries left).
FAILED - RETRYING: Waiting for nova-compute service up (5 retries left).
FAILED - RETRYING: Waiting for nova-compute service up (4 retries left).
FAILED - RETRYING: Waiting for nova-compute service up (3 retries left).
FAILED - RETRYING: Waiting for nova-compute service up (2 retries left).
FAILED - RETRYING: Waiting for nova-compute service up (1 retries left).
fatal: [control01 -> 10.60.7.34]: FAILED! => {"attempts": 20, "changed": false, "cmd": ["docker", "exec", "kolla_toolbox", "openstack", "--os-interface", "internal", "--os-auth-url", "http://10.60.6.32:35357", "--os-identity-api-version", "3", "--os-project-domain-name", "default", "--os-tenant-name", "admin", "--os-username", "admin", "--os-password", "openstack", "--os-user-domain-name", "default", "compute", "service", "list", "-f", "json", "--service", "nova-compute"], "delta": "0:00:00.426035", "end": "2018-10-22 05:03:40.743733", "msg": "non-zero return code", "rc": 1, "start": "2018-10-22 05:03:40.317698", "stderr": "Traceback (most recent call last):\n File \"/opt/ansible/bin/openstack\", line 7, in <module>\n from openstackclient.shell import main\n File \"/opt/ansible/local/lib/python2.7/site-packages/openstackclient/shell.py\", line 29, in <module>\n from openstackclient.common import clientmanager\n File \"/opt/ansible/local/lib/python2.7/site-packages/openstackclient/common/clientmanager.py\", line 166, in <module>\n 'openstack.cli.base',\n File \"/opt/ansible/local/lib/python2.7/site-packages/openstackclient/common/clientmanager.py\", line 137, in get_plugin_modules\n __import__(ep.module_name)\n File \"/opt/ansible/local/lib/python2.7/site-packages/openstackclient/network/client.py\", line 17, in <module>\n from openstack import profile\nImportError: cannot import name profile", "stderr_lines": ["Traceback (most recent call last):", " File \"/opt/ansible/bin/openstack\", line 7, in <module>", " from openstackclient.shell import main", " File \"/opt/ansible/local/lib/python2.7/site-packages/openstackclient/shell.py\", line 29, in <module>", " from openstackclient.common import clientmanager", " File \"/opt/ansible/local/lib/python2.7/site-packages/openstackclient/common/clientmanager.py\", line 166, in <module>", " 'openstack.cli.base',", " File \"/opt/ansible/local/lib/python2.7/site-packages/openstackclient/common/clientmanager.py\", line 137, in get_plugin_modules", " __import__(ep.module_name)", " File \"/opt/ansible/local/lib/python2.7/site-packages/openstackclient/network/client.py\", line 17, in <module>", " from openstack import profile", "ImportError: cannot import name profile"], "stdout": "", "stdout_lines": []}

The root cause is:
  the profile class was removed from the openstacksdk v0.18.0, however the Dockerfile of kolla_toolbox didn't assign the openstacksdk under v0.18.0

[Takashi Sogabe (sogabe)]

I encountered this issue by using ubuntu/xenial queens configurations.

As a tentative solution, upgrading python-openstackclient to 3.14.0 worked for my environment.

diff --git a/docker/kolla-toolbox/Dockerfile.j2 b/docker/kolla-toolbox/Dockerfile.j2
index 606a2b5..4c3753c 100644
--- a/docker/kolla-toolbox/Dockerfile.j2
+++ b/docker/kolla-toolbox/Dockerfile.j2
@@ -86,7 +86,7 @@ ENV PATH {{ virtualenv_path }}/bin:$PATH
         'os-client-config==1.28.0',
         'pbr==2.0.0',
         'pymongo',
- 'python-openstackclient==3.12.0',
+ 'python-openstackclient==3.14.0',
         'pytz',
         'pyudev',
         'shade==1.16.0'

Are there any blockers for resolving the issue?

[Mark Goddard (mgoddard)]

Perhaps a safer option for stable branchs would be to pin the openstacksdk library? We could bump the version of python-openstackclient in master, or just rely on upper-constraints.

[Mark Goddard (mgoddard)]

Takashi Sogabe, is this something you are able to propose a fix for?

[Mark Goddard (mgoddard)]

Fixed by https://review.openstack.org/#/c/612684/2.

[Takashi Sogabe (sogabe)]

Mark, Thank you for fixing the issue.
Pinning ofr openstacksdk<0.18 would be enough for my environments.