Error: Unable to retrieve the Zun Availability Zones.

Bug #1829516 reported by Ajith Rajan
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Zun UI
Fix Committed
High
hongbin

Bug Description

Openstack rocky with horizon installed. Installed Zun and zun-ui and able to create containers using zun commands like openstack appcontainer run , but while using horizon with zun-ui it is giving error as "Unable to retrieve the Zun Availability Zones." when I browse to Container in horizon. The zun list is giving proper output though.
[(keystone_user)]# zun list
+--------------------------------------+-----------+--------+---------+------------+--------------+-------+
| uuid | name | image | status | task_state | addresses | ports |
+--------------------------------------+-----------+--------+---------+------------+--------------+-------+
| 7ba65fcb-127c-4fd1-88b3-8806c57f9e12 | container | cirros | Running | None | xx.xx.xx.xx | [] |
+--------------------------------------+-----------+--------+---------+------------+--------------+-------+

Masked the IP in above output. If I select the option to create a container from UI , it displays "Error: Unable to retrieve the Zun Availability Zones." As admin user if I run cli command it shows availability zone as nova.
[(keystone_admin)]# openstack appcontainer service list
+----+------------+-------------+-------+----------+-----------------+----------------------------+-------------------+
| Id | Host | Binary | State | Disabled | Disabled Reason | Updated At | Availability Zone |
+----+------------+-------------+-------+----------+-----------------+----------------------------+-------------------+
| 1 | ostk-node2 | zun-compute | up | False | None | 2019-05-17T12:39:02.000000 | nova |
+----+------------+-------------+-------+----------+-----------------+----------------------------+-------------------+

Seems like the javascripts under /usr/share/openstack-dashboard/static/dashboard/container/containers are not giving correct return which maybe causing problem. Attaching screenshot . If there are more logs needed please let me know.

zun --version
3.3.0
openstack --version
openstack 3.16.2

zun-compute --version
3.1.0.dev55

zun-api --version
3.1.0.dev55

Did all the steps mentioned in installation doc https://docs.openstack.org/zun-ui/latest/install/index.html

Revision history for this message
Ajith Rajan (ajithrajan) wrote :
Revision history for this message
hongbin (hongbin034) wrote :

@Ajith,

According to your attached screenshot, your horizon was not able to access Zun's API at all.

It cannot access both:

* The container API: /containers
* The availability zone API: /availability-zones

However, if you was able to access using CLI (but couldn't access it in browser), then you might want to check the horizon logs to see the real error. The logs are under /var/log/apache2/horizon* on Ubuntu.

My best guess is that your Zun endpoint in keystone (i.e. http://controller:9517/v1) might not be accessible from browser. You might want to double check that.

Revision history for this message
hongbin (hongbin034) wrote :

The command to check the endpoints:

$ openstack endpoint list

Revision history for this message
Ajith Rajan (ajithrajan) wrote :
Download full text (9.0 KiB)

Thank you hongbin for looking into this. I suspected the same but couldn't find what is wrong. I could see the endpoints are listed properly.

[~(keystone_admin)]# openstack endpoint list|grep -i zun
| cbc4e64f03784e378bc9500dad94c973 | RegionOne | zun | container | True | admin | http://xx.xx.xx.xx:9517/v1 |
| d76eb9125bf64b3fb2214a5a7f88ea8b | RegionOne | zun | container | True | public | http://xx.xx.xx.xx:9517/v1 |
| fcf5094c449c4bc1947be5ac9fa849ac | RegionOne | zun | container | True | internal | http://xx.xx.xx.xx:9517/v1 |

Http logs are given below. Please check if you can suggest something from this.

error.log:

[Mon May 20 06:23:12.227734 2019] [:error] [pid 32611] ERROR openstack_dashboard.api.rest.utils error invoking apiclient
[Mon May 20 06:23:12.227802 2019] [:error] [pid 32611] Traceback (most recent call last):
[Mon May 20 06:23:12.227811 2019] [:error] [pid 32611] File "/usr/share/openstack-dashboard/openstack_dashboard/api/rest/utils.py", line 127, in _wrapped
[Mon May 20 06:23:12.227819 2019] [:error] [pid 32611] data = function(self, request, *args, **kw)
[Mon May 20 06:23:12.227853 2019] [:error] [pid 32611] File "/usr/lib/python2.7/site-packages/zun_ui/api/rest_api.py", line 124, in get
[Mon May 20 06:23:12.227887 2019] [:error] [pid 32611] result = client.container_list(request)
[Mon May 20 06:23:12.227905 2019] [:error] [pid 32611] File "/usr/lib/python2.7/site-packages/zun_ui/api/client.py", line 198, in container_list
[Mon May 20 06:23:12.227924 2019] [:error] [pid 32611] sort_dir)
[Mon May 20 06:23:12.227953 2019] [:error] [pid 32611] File "/usr/lib/python2.7/site-packages/zunclient/v1/containers.py", line 84, in list
[Mon May 20 06:23:12.227960 2019] [:error] [pid 32611] "containers", qparams=kwargs)
[Mon May 20 06:23:12.227967 2019] [:error] [pid 32611] File "/usr/lib/python2.7/site-packages/zunclient/common/base.py", line 125, in _list
[Mon May 20 06:23:12.227978 2019] [:error] [pid 32611] resp, body = self.api.json_request('GET', url)
[Mon May 20 06:23:12.228002 2019] [:error] [pid 32611] File "/usr/lib/python2.7/site-packages/zunclient/common/httpclient.py", line 367, in json_request
[Mon May 20 06:23:12.228010 2019] [:error] [pid 32611] resp = self._http_request(url, method, **kwargs)
[Mon May 20 06:23:12.228017 2019] [:error] [pid 32611] File "/usr/lib/python2.7/site-packages/zunclient/common/httpclient.py", line 350, in _http_request
[Mon May 20 06:23:12.228024 2019] [:error] [pid 32611] error_json.get('debuginfo'), method, url)
[Mon May 20 06:23:12.228032 2019] [:error] [pid 32611] NotAcceptable: Not Acceptable (HTTP 406) (Request-ID: req-4000b0ca-b6d7-4585-b664-3ac232cf82aa)

keystone_access.log:

xx.xx.xx.xx - - [20/May/2019:11:55:54 +0530] "GET /v3/users/f10726130dcf42a3a02f455a022d3d48/projects HTTP/1.1" 200 259 "-" "python-keystoneclient"
xx.xx.xx.xx - - [20/May/2019:11:55:54 +0530] "GET /v3/ HTTP/1.1" 200 194 "-" "keystoneauth1/3.10.0 python-requests/2.19.1 CPython/2.7.5"
xx.xx.xx.xx - - [20/May/2019:11:55:54 +0530] "GET /v3/users/f10726130dcf42a3a02f455a...

Read more...

Revision history for this message
Ajith Rajan (ajithrajan) wrote :

@hongbin

Can you tell me any way to test if the web authentication is happening as expected w.r.t zun-ui . I am not sure about the syntax which helps me test the rest api calls are authenticating and providing result as expected.

Revision history for this message
Ajith Rajan (ajithrajan) wrote :

I tried REST API calls using the same token I generated using the user credentials I used for creating container and it is giving proper result(screenshot). It seems the keystone authentication is working proper and API calls are getting proper results but the GUI portion in horizon is somehow not getting the proper returns from it seems from the javascripts/python codes. Not sure though. Please have a check.

Revision history for this message
hongbin (hongbin034) wrote :

Hi @Ajith,

Yes, it looks your endpoints were setup properly and there is no problem on keystone authentication.

In your access log, this error catches my eye:

xx.xx.xx.xx - - [20/May/2019:11:57:35 +0530] "GET /dashboard/api/zun/containers/ HTTP/1.1" 500 82 "http://xx.xx.xx.xx/dashboard/project/container/containers" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.157 Safari/537.36"

I saw the request issued by browser received a 500 response from zun-api. Therefore, you might want to check your zun-api and zun-compute logs to locate the real error:

  $ sudo journalctl -u zun-api > zun-api.log
  $ sudo journalctl -u zun-compute > zun-compute.log

Revision history for this message
hongbin (hongbin034) wrote :

You probably don't need this, but this is the answer for your previous question.

This is the way I tested the endpoint:

source /opt/stack/devstack/openrc demo demo
TOKEN=$(openstack token issue -f value -c id)
curl -X GET http://xxx.xx.x.xxx/container/v1/containers -H "Accept: application/json" -H "Content-Type: application/json" -H "OpenStack-API-Version: container 1.32" -H "X-Auth-Token: $TOKEN"

The way you tested it in the tool also works.

An easy way to figure out how the HTTP works is adding the '--debug' option in the corresponding CLI command. For example:

  $ zun --debug list
  $ zun --debug run ...

Revision history for this message
Ajith Rajan (ajithrajan) wrote :
Download full text (3.9 KiB)

Thank you for the details and commands.

I couldn't see much from the api or compute logs. Below is the output.

tail -50 zun-compute.log

May 20 17:11:11 ostk-node2 systemd[1]: Stopping OpenStack Container Service Compute Agent...
May 20 17:11:17 ostk-node2 systemd[1]: Stopped OpenStack Container Service Compute Agent.
May 20 17:11:17 ostk-node2 systemd[1]: Started OpenStack Container Service Compute Agent.
May 22 11:07:19 ostk-node2 systemd[1]: Stopping OpenStack Container Service Compute Agent...
May 22 11:07:26 ostk-node2 systemd[1]: Stopped OpenStack Container Service Compute Agent.
May 22 11:07:26 ostk-node2 systemd[1]: Started OpenStack Container Service Compute Agent.
May 22 12:03:18 ostk-node2 systemd[1]: Stopping OpenStack Container Service Compute Agent...
May 22 12:03:45 ostk-node2 systemd[1]: Stopped OpenStack Container Service Compute Agent.
May 22 12:03:45 ostk-node2 systemd[1]: Started OpenStack Container Service Compute Agent.
May 22 12:08:31 ostk-node2 systemd[1]: Stopping OpenStack Container Service Compute Agent...
May 22 12:08:32 ostk-node2 systemd[1]: Stopped OpenStack Container Service Compute Agent.
May 22 12:08:32 ostk-node2 systemd[1]: Started OpenStack Container Service Compute Agent.
May 22 12:34:03 ostk-node2 systemd[1]: Stopping OpenStack Container Service Compute Agent...
May 22 12:34:06 ostk-node2 systemd[1]: Stopped OpenStack Container Service Compute Agent.
May 22 12:34:06 ostk-node2 systemd[1]: Started OpenStack Container Service Compute Agent.
[root@ostk-node2 ~]# date
Wed May 22 12:51:23 IST 2019

tail -50 zun-api.log

May 20 14:14:26 ostk-node1 systemd[1]: Stopped OpenStack Container Service API.
May 20 14:14:26 ostk-node1 systemd[1]: Started OpenStack Container Service API.
May 20 17:10:46 ostk-node1 systemd[1]: Stopping OpenStack Container Service API...
May 20 17:10:46 ostk-node1 systemd[1]: Stopped OpenStack Container Service API.
May 20 17:10:46 ostk-node1 systemd[1]: Started OpenStack Container Service API.
May 22 11:07:01 ostk-node1 systemd[1]: Stopping OpenStack Container Service API...
May 22 11:07:02 ostk-node1 systemd[1]: Stopped OpenStack Container Service API.
May 22 11:07:02 ostk-node1 systemd[1]: Started OpenStack Container Service API.
May 22 11:11:50 ostk-node1 systemd[1]: Stopping OpenStack Container Service API...
May 22 11:11:50 ostk-node1 systemd[1]: Stopped OpenStack Container Service API.
May 22 11:11:50 ostk-node1 systemd[1]: Started OpenStack Container Service API.
May 22 12:03:36 ostk-node1 systemd[1]: Stopping OpenStack Container Service API...
May 22 12:03:36 ostk-node1 systemd[1]: Stopped OpenStack Container Service API.
May 22 12:03:36 ostk-node1 systemd[1]: Started OpenStack Container Service API.
May 22 12:08:39 ostk-node1 systemd[1]: Stopping OpenStack Container Service API...
May 22 12:08:39 ostk-node1 systemd[1]: Stopped OpenStack Container Service API.
May 22 12:08:39 ostk-node1 systemd[1]: Started OpenStack Container Service API.
[root@ostk-node1 dashboard(keystone_user)]# date
Wed May 22 12:52:26 IST 2019

One thing I doubt when I did a search for the path dashboard/project/container/containers is that this path doesn't exists as such. I am not sure ...

Read more...

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to zun-ui (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/660871

Revision history for this message
hongbin (hongbin034) wrote :

@Ajith Rajan,

I think I know what is wrong and here might be the right fix: https://review.opendev.org/#/c/660871/ .

You can apply the patch to your local installation if you want:

* go to your zun-ui repo (/opt/stack/zun-ui)
* apply the patch (you can cherry-pick or manually change the code)
* restart the httpd/apache server.

Revision history for this message
hongbin (hongbin034) wrote :

"One thing I doubt when I did a search for the path dashboard/project/container/containers is that this path doesn't exists as such."

Yes, it seems there are some url translation happened in somewhere. I am not sure the details since the code was written by another developer.

I saw your zun-ui traffic was hitting the zun-api server, so that part seems to be working properly.

Revision history for this message
Ajith Rajan (ajithrajan) wrote : Re: [Bug 1829516] Re: Error: Unable to retrieve the Zun Availability Zones.
Download full text (3.8 KiB)

Thanks a lot hongbin. After I changed the code as you described in path it
is working as expected.

I have one more question which is related to kuryr network. I am running 2
node openstack with a flat network configured and there is no tenant
network. Is it possible to run containers with a single flat network ?

When I try creating a docker network it is complaining that there is no
tenant network available.

[root@ostk-node2 kuryr]# docker network create --driver kuryr --ipam-driver
kuryr \
> --subnet 10.10.0.0/16 --gateway=10.10.0.1 test_net
Error response from daemon: NetworkDriver.CreateNetwork: Unable to create
the network. No tenant network is available for allocation.

On Thu, May 23, 2019 at 9:15 AM hongbin <email address hidden> wrote:

> "One thing I doubt when I did a search for the path
> dashboard/project/container/containers is that this path doesn't exists
> as such."
>
> Yes, it seems there are some url translation happened in somewhere. I am
> not sure the details since the code was written by another developer.
>
> I saw your zun-ui traffic was hitting the zun-api server, so that part
> seems to be working properly.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1829516
>
> Title:
> Error: Unable to retrieve the Zun Availability Zones.
>
> Status in Zun UI:
> New
>
> Bug description:
> Openstack rocky with horizon installed. Installed Zun and zun-ui and
> able to create containers using zun commands like openstack appcontainer
> run , but while using horizon with zun-ui it is giving error as "Unable to
> retrieve the Zun Availability Zones." when I browse to Container in
> horizon. The zun list is giving proper output though.
> [(keystone_user)]# zun list
>
> +--------------------------------------+-----------+--------+---------+------------+--------------+-------+
> | uuid | name | image | status |
> task_state | addresses | ports |
>
> +--------------------------------------+-----------+--------+---------+------------+--------------+-------+
> | 7ba65fcb-127c-4fd1-88b3-8806c57f9e12 | container | cirros | Running |
> None | xx.xx.xx.xx | [] |
>
> +--------------------------------------+-----------+--------+---------+------------+--------------+-------+
>
> Masked the IP in above output. If I select the option to create a
> container from UI , it displays "Error: Unable to retrieve the Zun
> Availability Zones." As admin user if I run cli command it shows
> availability zone as nova.
> [(keystone_admin)]# openstack appcontainer service list
>
> +----+------------+-------------+-------+----------+-----------------+----------------------------+-------------------+
> | Id | Host | Binary | State | Disabled | Disabled Reason |
> Updated At | Availability Zone |
>
> +----+------------+-------------+-------+----------+-----------------+----------------------------+-------------------+
> | 1 | ostk-node2 | zun-compute | up | False | None |
> 2019-05-17T12:39:02.000000 | nova |
>
> +----+------------+-------------+----...

Read more...

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to zun-ui (stable/rocky)

Reviewed: https://review.opendev.org/660871
Committed: https://git.openstack.org/cgit/openstack/zun-ui/commit/?id=9af84c569e2924de2dfb74d667c15f5da1417dd0
Submitter: Zuul
Branch: stable/rocky

commit 9af84c569e2924de2dfb74d667c15f5da1417dd0
Author: Hongbin Lu <email address hidden>
Date: Thu May 23 03:21:24 2019 +0000

    Hard code api version to 1.20

    Change-Id: Icf013378f636cf2bf266efc7f91dd0ba42600373
    Closes-Bug: #1829516

tags: added: in-stable-rocky
Revision history for this message
hongbin (hongbin034) wrote :

@Ajith Rajan,

I believe it is possible, because Zun specifies the uuid of the neutron network and neutron subnet on creating the docker network. It is roughly equivalent to:

  docker network create --driver kuryr --ipam-driver kuryr \
    --subnet 10.10.0.0/16 --gateway=10.10.0.1 test_net \
    -o neutron.net.uuid=xxx \
    -o neutron.subnet.uuid=xxx \
    ...

It should be able to search the flat network regardless of its tenant.

Revision history for this message
hongbin (hongbin034) wrote :

It looks the issue reported by this report is resolved. I am going to close this ticket but please feel free to re-open it if I am wrong. Furthermore, feel free to open another ticket if you encounter another issues.

Changed in zun-ui:
assignee: nobody → hongbin (hongbin034)
importance: Undecided → High
status: New → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/zun-ui rocky-eol

This issue was fixed in the openstack/zun-ui rocky-eol release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.