kolla: zun-ui or cloud-shel not work

[PrinzElvis]

hi Stackers,
Am running Kolla und Kolla-ansible Release Rocky with Ubuntu_18TLS
zun-ui or cloud-shel not work

2019-01-15 10:17:45.789 21 DEBUG oslo_db.api [req-74d384f1-cac8-417c-abbc-a365483d0d80 17558defae314a18b1761d44f1e6bedd ff45243cd8384d00a77b8d47be7aa45c default - -] Loading backend 'sqlalchemy' from 'zun.db.sqlalchemy.api' _load_backend /var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_db/api.py:261
2019-01-15 10:17:45.951 21 DEBUG oslo_db.sqlalchemy.engines [req-74d384f1-cac8-417c-abbc-a365483d0d80 17558defae314a18b1761d44f1e6bedd ff45243cd8384d00a77b8d47be7aa45c default - -] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION _check_effective_sql_mode /var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py:308
2019-01-15 10:17:45.989 21 DEBUG zun.common.exception [req-74d384f1-cac8-417c-abbc-a365483d0d80 17558defae314a18b1761d44f1e6bedd ff45243cd8384d00a77b8d47be7aa45c default - -] Container cloud-shell-admin-admin-Default-RegionOne could not be found. wrapped /var/lib/kolla/venv/local/lib/python2.7/site-packages/zun/common/exception.py:128
2019-01-15 10:17:47.760 18 DEBUG zun.common.exception [req-9333168f-c02b-4caf-a531-07628380e7d8 17558defae314a18b1761d44f1e6bedd ff45243cd8384d00a77b8d47be7aa45c default - -] Container cloud-shell-admin-admin-Default-RegionOne could not be found. wrapped /var/lib/kolla/venv/local/lib/python2.7/site-packages/zun/common/exception.py:128
2019-01-15 10:17:56.905 20 DEBUG zun.common.exception [req-dde55ec6-5e43-4eb5-9ac7-43bcf4130a64 17558defae314a18b1761d44f1e6bedd ff45243cd8384d00a77b8d47be7aa45c default - -] Container cloud-shell-admin-admin-Default-RegionOne could not be found. wrapped /var/lib/kolla/venv/local/lib/python2.7/site-packages/zun/common/exception.py:128
2019-01-15 10:39:59.167 18 DEBUG zun.api.controllers.v1 [req-9333168f-c02b-4caf-a531-07628380e7d8 17558defae314a18b1761d44f1e6bedd ff45243cd8384d00a77b8d47be7aa45c default - -] Processing request: url: http://api.enostack.net:9517/v1/containers?run=true, POST, body: {"name": "cloud-shell-admin-admin-Default-RegionOne", "image": "gbraad/openstack-client:alpine", "labels": {"cloud-shell": "cloud-shell-admin-admin-Default-RegionOne"}, "environment": {"OS_CLOUD": "openstack"}, "command": "/bin/bash", "interactive": true} _route /var/lib/kolla/venv/local/lib/python2.7/site-packages/zun/api/controllers/v1/__init__.py:197
2019-01-15 10:39:59.175 18 DEBUG zun.common.exception [req-a33a4bdb-631b-4056-bbea-d70eb18b019a 17558defae314a18b1761d44f1e6bedd ff45243cd8384d00a77b8d47be7aa45c default - -] Invalid input for field 'command'.Value: '/bin/bash'. u'/bin/bash' is not of type 'array', 'null'

Failed validating 'type' in schema['properties']['command']:
    {'type': ['array', 'null']}

On instance:
    u'/bin/bash' wrapped /var/lib/kolla/venv/local/lib/python2.7/site-packages/zun/common/exception.py:128

[hongbin (hongbin034)]

Hi @PrinzElvis,

This error only happened in the UI/cloud shell? Doesn't the same error occur if you are using the CLI?

[hongbin (hongbin034)]

s/Doesn't/Does/

[PrinzElvis (eno237)]

This error only happened in the UI/cloud shell.
CLI is working fine.

[PrinzElvis (eno237)]

Pull Image with UI is not possible and I can not also select Host "Select host that stores the image"
But on CLI is possible:

+--------------------------------------+------------+-----------+------+------------+
| uuid | hostname | mem_total | cpus | disk_total |
+--------------------------------------+------------+-----------+------+------------+
| b3addba2-f5bf-433e-8978-f5d3616536f2 | eprinz | 32047 | 8 | 181

UI error for Cloud Shell:

2019-01-16 22:21:58.040 19 DEBUG zun.common.exception [req-0492d668-cdbc-4ac4-863c-a4a1dc803b41 a85c7cda39144319a4d3cfa4f0b0ff69 3b61c2c78da24fb183a4ac5154ba041c default - -] Invalid input for field 'command'.Value: '/bin/bash'. u'/bin/bash' is not of type 'array', 'null'

Failed validating 'type' in schema['properties']['command']:
    {'type': ['array', 'null']}

On instance:
    u'/bin/bash' wrapped /var/lib/kolla/venv/local/lib/python2.7/site-packages/zun/common/exception.py:128

[hongbin (hongbin034)]

@PrinzElvis, thanks for providing the detailed information. It looks there are two issues there. One is the cloud shell issue, the other is the pull image issue. This bug is confirmed.

[PrinzElvis (eno237)]

I found this too in # tail -f /var/log/kolla/horizon/horizon.log

[Thu Jan 17 02:20:57.705242 2019] [wsgi:error] [pid 20:tid 140667541022464] [remote 10.37.4.1:54784] error invoking apiclient
[Thu Jan 17 02:20:57.705264 2019] [wsgi:error] [pid 20:tid 140667541022464] [remote 10.37.4.1:54784] Traceback (most recent call last):
[Thu Jan 17 02:20:57.705270 2019] [wsgi:error] [pid 20:tid 140667541022464] [remote 10.37.4.1:54784] File "/var/lib/kolla/venv/lib/python2.7/site-packages/openstack_dashboard/api/rest/utils.py", line 127, in _wrapped
[Thu Jan 17 02:20:57.705275 2019] [wsgi:error] [pid 20:tid 140667541022464] [remote 10.37.4.1:54784] data = function(self, request, *args, **kw)
[Thu Jan 17 02:20:57.705280 2019] [wsgi:error] [pid 20:tid 140667541022464] [remote 10.37.4.1:54784] File "/var/lib/kolla/venv/lib/python2.7/site-packages/zun_ui/api/rest_api.py", line 39, in get
[Thu Jan 17 02:20:57.705285 2019] [wsgi:error] [pid 20:tid 140667541022464] [remote 10.37.4.1:54784] return change_to_id(client.container_show(request, id).to_dict())
[Thu Jan 17 02:20:57.705290 2019] [wsgi:error] [pid 20:tid 140667541022464] [remote 10.37.4.1:54784] File "/var/lib/kolla/venv/lib/python2.7/site-packages/zun_ui/api/client.py", line 195, in container_show
[Thu Jan 17 02:20:57.705294 2019] [wsgi:error] [pid 20:tid 140667541022464] [remote 10.37.4.1:54784] return zunclient(request).containers.get(id)
[Thu Jan 17 02:20:57.705299 2019] [wsgi:error] [pid 20:tid 140667541022464] [remote 10.37.4.1:54784] File "/var/lib/kolla/venv/lib/python2.7/site-packages/zunclient/v1/containers.py", line 93, in get
[Thu Jan 17 02:20:57.705329 2019] [wsgi:error] [pid 20:tid 140667541022464] [remote 10.37.4.1:54784] qparams=kwargs)[0]
[Thu Jan 17 02:20:57.705334 2019] [wsgi:error] [pid 20:tid 140667541022464] [remote 10.37.4.1:54784] File "/var/lib/kolla/venv/lib/python2.7/site-packages/zunclient/common/base.py", line 128, in _list
[Thu Jan 17 02:20:57.705339 2019] [wsgi:error] [pid 20:tid 140667541022464] [remote 10.37.4.1:54784] resp, body = self.api.json_request('GET', url)
[Thu Jan 17 02:20:57.705345 2019] [wsgi:error] [pid 20:tid 140667541022464] [remote 10.37.4.1:54784] File "/var/lib/kolla/venv/lib/python2.7/site-packages/zunclient/common/httpclient.py", line 367, in json_request
[Thu Jan 17 02:20:57.705361 2019] [wsgi:error] [pid 20:tid 140667541022464] [remote 10.37.4.1:54784] resp = self._http_request(url, method, **kwargs)
[Thu Jan 17 02:20:57.705366 2019] [wsgi:error] [pid 20:tid 140667541022464] [remote 10.37.4.1:54784] File "/var/lib/kolla/venv/lib/python2.7/site-packages/zunclient/common/httpclient.py", line 350, in _http_request
[Thu Jan 17 02:20:57.705370 2019] [wsgi:error] [pid 20:tid 140667541022464] [remote 10.37.4.1:54784] error_json.get('debuginfo'), method, url)
[Thu Jan 17 02:20:57.705374 2019] [wsgi:error] [pid 20:tid 140667541022464] [remote 10.37.4.1:54784] NotFound: Container cloud-shell-admin-admin-Default-RegionOne could not be found. (HTTP 404) (Request-ID: req-336ee267-971d-4618-b716-4806dbe11bf6)
[Thu Jan 17 02:20:58.852169 2019] [wsgi:error] [pid 23:tid 140667541022464] [remote 10.37.4.1:54890] error invoki...

[hongbin (hongbin034)]

Note: according to the feedback from IRC, https://review.openstack.org/#/c/631556/ doesn't fix the problem. I will find another time to look into it.

PrinzElvis Hi all, 11:49
PrinzElvis I try this review https://review.openstack.org/#/c/631556/ for cloudshell bugs https://bugs.launchpad.net/zun/+bug/1811791 but doesn’t work. I’m using Master und Rocky kolla & Kolla-Ansible. It will be very nice if this new feature work. Thanks for your help. 11:55
openstack Launchpad bug 1811791 in Zun "kolla: zun-ui or cloud-shel not work" [Medium,Confirmed] - Assigned to <email address hidden> (feng-shengqin) 11:55
PrinzElvis I mean I’ve two kolla Openstack environments (Master and rocky release)

[hongbin (hongbin034)]

Hi PrinzElvis,

I tried to reproduce the error by spinning up a VM (Ubuntu 16.04, 2 cpus, 8G ram) and install kolla and kolla-ansible. However, I couldn't reproduce the error. The cloudshell was working fine.

Here is how I deployed Zun using Kolla-ansible: http://paste.openstack.org/show/743926/

Could you share how you deployed Zun so that I can reproduce the error?

[PrinzElvis (eno237)]

Hi Hongbin,
I have used Ubuntu18.04 und Rocky:
https://docs.openstack.org/project-deploy-guide/kolla-ansible/rocky/quickstart.html
And
https://docs.openstack.org/kolla-ansible/rocky/reference/zun-guide.html

What you did it look like I did, but without the master Release. Did you use the devsatck?
Anyway I will try it again, with Ubuntu16.04 and Master Release.

Cheers
Prinz

[PrinzElvis (eno237)]

Hi Hongbin,
with Ubuntu 16.04 und master Release Zun_ui is working,
but Cloud Shell on the UI or Browser is don't working fine,
still waiting the terminal prompt. Look the screenshot attached.

I have use your Setup. What I'm doing wrong ?

+--------------------------------------+-------------------------------------------+--------------------------------+---------+------------+------------+-------+
| uuid | name | image | status | task_state | addresses | ports |
+--------------------------------------+-------------------------------------------+--------------------------------+---------+------------+------------+-------+
| 0ca4f2ed-8880-4275-a6f7-f8a993527b06 | cloud-shell-admin-admin-Default-RegionOne | gbraad/openstack-client:alpine | Running | None | 10.0.2.159 | [] |
+--------------------------------------+-------------------------------------------+--------------------------------+---------+------------+------------+-------+

root@prinzdev:~# docker exec -it -u root zun-0ca4f2ed-8880-4275-a6f7-f8a993527b06 bash
bash-4.3# openstack server list
Password:
+--------------------------------------+-------+--------+---------------------+--------+---------+
| ID | Name | Status | Networks | Image | Flavor |
+--------------------------------------+-------+--------+---------------------+--------+---------+
| a555cdf2-8fba-483b-8f66-74beb04025df | demo1 | ACTIVE | demo-net=10.0.0.154 | cirros | m1.tiny |
+--------------------------------------+-------+--------+---------------------+--------+---------+
bash-4.3#

Cheers,
Prinz.

[hongbin (hongbin034)]

The best guess I can make is that some ports are not open. In particular, the port 6784 is the zun websocket proxy port. The browser will use that port to connect to the container's shell.

[Shu Muto (shu-mutou)]

Here is my zun settings to use websocket for cloud shell.

* /lib/systemd/system/docker.service

[Service]
.....
ExecStart=/usr/bin/dockerd -H unix:///var/run/docker.sock -H tcp://0.0.0.0:2375 --group stack
.....

After editing this, reload and restart docker.

* /etc/zun/zun.conf

......
[websocket_proxy]
base_url = ws://[ip address of your zun-wsproxy]:6784/
allowed_origins = '[ip address of your horizon]'
wsproxy_host = [ip address of your zun-wsproxy]
wsproxy_port = 6784
......

After editing this, restart services zun-api, zun-compute and zun-wsproxy.

I hope this helps you. Thanks.