Zun

ConnectionRefusedError: [Errno 111] ECONNREFUSED for Cloud Shell or Console

Bug #1902453 reported by PeterS
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Zun
New
Undecided
Unassigned

Bug Description

I have installed Openstack Ussuri with 1 controller (for now), 4 Nova Nodes and one Docker Node with Zun on top. I installed the Zun WSProxy and Zun-UI from the Ussuri branch on the controller and the Zun with Kyrur on the Docker Node. i can launch containers and exec the shell using the command line 'docker exec -it <id> bash', but i am unable to load the Cloud Shell or Console. The screen stays black and a blinking prompt appears. The connections is attempted and closed (bottom of console screen)

browser console tells me:
The connection to ws://my-controller:6784/?token=20f9a80e-afce-4cc2-908c-efdaa59efa01&uuid=fddd688a-cc72-49f7-ac72-8b0240e83889 was interrupted while the page was loading.

trying to run a command through the Openstack CLI fails:
openstack --debug appcontainer run --name container --interactive --net network=internal-net cirros ping 8.8.8.8
...
putting tty into raw mode
starting main loop in client
                            restoring tty configuration
                                                       Remote host closed connection:Connection is already closed.:58c76857-fd5e-469f-82c5-c4c104191c6c

zun-wsproxy.log after trying the browser:
2020-11-01 21:50:49.318 141687 INFO zun.websocket.websocketproxy [req-65180ff2-79e6-46fb-b36a-b23ba49536a1 - - - - -] handler exception: unsupported operand type(s) for %: 'ConnectionRefused>
2020-11-01 21:50:49.319 141687 DEBUG zun.websocket.websocketproxy [req-65180ff2-79e6-46fb-b36a-b23ba49536a1 - - - - -] exception vmsg /usr/lib/python3/dist-packages/websockify/websockifyserv>
2020-11-01 21:50:49.319 141687 ERROR zun.websocket.websocketproxy Traceback (most recent call last):
2020-11-01 21:50:49.319 141687 ERROR zun.websocket.websocketproxy File "/usr/local/lib/python3.8/dist-packages/zun/websocket/websocketclient.py", line 33, in connect
2020-11-01 21:50:49.319 141687 ERROR zun.websocket.websocketproxy self.ws = websocket.create_connection(url,
2020-11-01 21:50:49.319 141687 ERROR zun.websocket.websocketproxy File "/usr/local/lib/python3.8/dist-packages/websocket/_core.py", line 515, in create_connection
2020-11-01 21:50:49.319 141687 ERROR zun.websocket.websocketproxy websock.connect(url, **options)
2020-11-01 21:50:49.319 141687 ERROR zun.websocket.websocketproxy File "/usr/local/lib/python3.8/dist-packages/websocket/_core.py", line 222, in connect
2020-11-01 21:50:49.319 141687 ERROR zun.websocket.websocketproxy self.sock, addrs = connect(url, self.sock_opt, proxy_info(**options),
2020-11-01 21:50:49.319 141687 ERROR zun.websocket.websocketproxy File "/usr/local/lib/python3.8/dist-packages/websocket/_http.py", line 121, in connect
2020-11-01 21:50:49.319 141687 ERROR zun.websocket.websocketproxy sock = _open_socket(addrinfo_list, options.sockopt, options.timeout)
2020-11-01 21:50:49.319 141687 ERROR zun.websocket.websocketproxy File "/usr/local/lib/python3.8/dist-packages/websocket/_http.py", line 201, in _open_socket
2020-11-01 21:50:49.319 141687 ERROR zun.websocket.websocketproxy raise err
2020-11-01 21:50:49.319 141687 ERROR zun.websocket.websocketproxy File "/usr/local/lib/python3.8/dist-packages/websocket/_http.py", line 176, in _open_socket
2020-11-01 21:50:49.319 141687 ERROR zun.websocket.websocketproxy sock.connect(address)
2020-11-01 21:50:49.319 141687 ERROR zun.websocket.websocketproxy File "/usr/lib/python3/dist-packages/eventlet/greenio/base.py", line 253, in connect
2020-11-01 21:50:49.319 141687 ERROR zun.websocket.websocketproxy socket_checkerr(fd)
2020-11-01 21:50:49.319 141687 ERROR zun.websocket.websocketproxy File "/usr/lib/python3/dist-packages/eventlet/greenio/base.py", line 51, in socket_checkerr
2020-11-01 21:50:49.319 141687 ERROR zun.websocket.websocketproxy raise socket.error(err, errno.errorcode[err])

2020-11-01 21:50:49.319 141687 ERROR zun.websocket.websocketproxy ConnectionRefusedError: [Errno 111] ECONNREFUSED

2020-11-01 21:50:49.319 141687 ERROR zun.websocket.websocketproxy
2020-11-01 21:50:49.319 141687 ERROR zun.websocket.websocketproxy During handling of the above exception, another exception occurred:
2020-11-01 21:50:49.319 141687 ERROR zun.websocket.websocketproxy
2020-11-01 21:50:49.319 141687 ERROR zun.websocket.websocketproxy Traceback (most recent call last):
2020-11-01 21:50:49.319 141687 ERROR zun.websocket.websocketproxy File "/usr/lib/python3/dist-packages/websockify/websockifyserver.py", line 691, in top_new_client
2020-11-01 21:50:49.319 141687 ERROR zun.websocket.websocketproxy client = self.do_handshake(startsock, address)
2020-11-01 21:50:49.319 141687 ERROR zun.websocket.websocketproxy File "/usr/lib/python3/dist-packages/websockify/websockifyserver.py", line 619, in do_handshake
2020-11-01 21:50:49.319 141687 ERROR zun.websocket.websocketproxy self.RequestHandlerClass(retsock, address, self)
2020-11-01 21:50:49.319 141687 ERROR zun.websocket.websocketproxy File "/usr/local/lib/python3.8/dist-packages/zun/websocket/websocketproxy.py", line 301, in __init__
2020-11-01 21:50:49.319 141687 ERROR zun.websocket.websocketproxy websockify.ProxyRequestHandler.__init__(self, *args, **kwargs)
2020-11-01 21:50:49.319 141687 ERROR zun.websocket.websocketproxy File "/usr/lib/python3/dist-packages/websockify/websockifyserver.py", line 99, in __init__
2020-11-01 21:50:49.319 141687 ERROR zun.websocket.websocketproxy SimpleHTTPRequestHandler.__init__(self, req, addr, server)
2020-11-01 21:50:49.319 141687 ERROR zun.websocket.websocketproxy File "/usr/lib/python3.8/http/server.py", line 647, in __init__
2020-11-01 21:50:49.319 141687 ERROR zun.websocket.websocketproxy super().__init__(*args, **kwargs)
2020-11-01 21:50:49.319 141687 ERROR zun.websocket.websocketproxy File "/usr/lib/python3.8/socketserver.py", line 720, in __init__
2020-11-01 21:50:49.319 141687 ERROR zun.websocket.websocketproxy self.handle()
2020-11-01 21:50:49.319 141687 ERROR zun.websocket.websocketproxy File "/usr/lib/python3/dist-packages/websockify/websockifyserver.py", line 315, in handle
2020-11-01 21:50:49.319 141687 ERROR zun.websocket.websocketproxy SimpleHTTPRequestHandler.handle(self)
2020-11-01 21:50:49.319 141687 ERROR zun.websocket.websocketproxy File "/usr/lib/python3.8/http/server.py", line 427, in handle
2020-11-01 21:50:49.319 141687 ERROR zun.websocket.websocketproxy self.handle_one_request()
2020-11-01 21:50:49.319 141687 ERROR zun.websocket.websocketproxy File "/usr/lib/python3/dist-packages/websockify/websocketserver.py", line 47, in handle_one_request
2020-11-01 21:50:49.319 141687 ERROR zun.websocket.websocketproxy super(WebSocketRequestHandlerMixIn, self).handle_one_request()
2020-11-01 21:50:49.319 141687 ERROR zun.websocket.websocketproxy File "/usr/lib/python3.8/http/server.py", line 415, in handle_one_request
2020-11-01 21:50:49.319 141687 ERROR zun.websocket.websocketproxy method()
2020-11-01 21:50:49.319 141687 ERROR zun.websocket.websocketproxy File "/usr/lib/python3/dist-packages/websockify/websocketserver.py", line 60, in _websocket_do_GET
2020-11-01 21:50:49.319 141687 ERROR zun.websocket.websocketproxy self.handle_upgrade()
2020-11-01 21:50:49.319 141687 ERROR zun.websocket.websocketproxy File "/usr/lib/python3/dist-packages/websockify/websockifyserver.py", line 221, in handle_upgrade
2020-11-01 21:50:49.319 141687 ERROR zun.websocket.websocketproxy WebSocketRequestHandlerMixIn.handle_upgrade(self)
2020-11-01 21:50:49.319 141687 ERROR zun.websocket.websocketproxy File "/usr/lib/python3/dist-packages/websockify/websocketserver.py", line 87, in handle_upgrade
2020-11-01 21:50:49.319 141687 ERROR zun.websocket.websocketproxy self.handle_websocket()
2020-11-01 21:50:49.319 141687 ERROR zun.websocket.websocketproxy File "/usr/lib/python3/dist-packages/websockify/websockifyserver.py", line 259, in handle_websocket
2020-11-01 21:50:49.319 141687 ERROR zun.websocket.websocketproxy self.new_websocket_client()
2020-11-01 21:50:49.319 141687 ERROR zun.websocket.websocketproxy File "/usr/local/lib/python3.8/dist-packages/zun/websocket/websocketproxy.py", line 208, in new_websocket_client
2020-11-01 21:50:49.319 141687 ERROR zun.websocket.websocketproxy self._new_websocket_client(container, token, uuid)
2020-11-01 21:50:49.319 141687 ERROR zun.websocket.websocketproxy File "/usr/local/lib/python3.8/dist-packages/zun/websocket/websocketproxy.py", line 225, in _new_websocket_client
2020-11-01 21:50:49.319 141687 ERROR zun.websocket.websocketproxy wscls.connect()
2020-11-01 21:50:49.319 141687 ERROR zun.websocket.websocketproxy File "/usr/local/lib/python3.8/dist-packages/zun/websocket/websocketclient.py", line 36, in connect
2020-11-01 21:50:49.319 141687 ERROR zun.websocket.websocketproxy raise exception.ConnectionFailed(e)
2020-11-01 21:50:49.319 141687 ERROR zun.websocket.websocketproxy File "/usr/local/lib/python3.8/dist-packages/zun/common/exception.py", line 204, in __init__
2020-11-01 21:50:49.319 141687 ERROR zun.websocket.websocketproxy self.message = self.message % kwargs
2020-11-01 21:50:49.319 141687 ERROR zun.websocket.websocketproxy TypeError: unsupported operand type(s) for %: 'ConnectionRefusedError' and 'dict'

I tried the websocket setting on the controller node with all names and all IP:
[websocket_proxy]
wsproxy_host = 10.0.0.1
webproxy_port = 6784
base_url = ws://10.0.0.1:6784/
allowed_origins = my-controller,10.0.0.1,my-workstation,10.0.0.100

and added the docker API settings on the Docker node:
[docker]
docker_remote_api_url = tcp://my-docker:2375
docker_remote_api_host = my-docker

Did i miss something that i should have configured?

Regards,
Peter

Revision history for this message
PeterS (beholder101) wrote :
Download full text (8.2 KiB)

I completely reinstalled the controller and the docker node. The zun-wsproxy service on the controller gives me the same error:

handler exception: unsupported operand type(s) for %: 'ConnectionRefusedError' and 'dict'

/var/log/zun-wsproxy.log gives me again:
2020-11-03 20:06:29.985 22383 INFO zun.websocket.websocketproxy [req-0090fd14-0b3b-48ab-9f82-44fe994fa09b - - - - -] handler exception: unsupported operand type(s) for %: 'ConnectionRefusedError' and 'dict'
2020-11-03 20:06:29.985 22383 DEBUG zun.websocket.websocketproxy [req-0090fd14-0b3b-48ab-9f82-44fe994fa09b - - - - -] exception vmsg /usr/lib/python3/dist-packages/websockify/websockifyserver.py:634
2020-11-03 20:06:29.985 22383 ERROR zun.websocket.websocketproxy Traceback (most recent call last):
2020-11-03 20:06:29.985 22383 ERROR zun.websocket.websocketproxy File "/usr/local/lib/python3.8/dist-packages/zun/websocket/websocketclient.py", line 33, in connect
2020-11-03 20:06:29.985 22383 ERROR zun.websocket.websocketproxy self.ws = websocket.create_connection(url,
2020-11-03 20:06:29.985 22383 ERROR zun.websocket.websocketproxy File "/usr/local/lib/python3.8/dist-packages/websocket/_core.py", line 515, in create_connection
2020-11-03 20:06:29.985 22383 ERROR zun.websocket.websocketproxy websock.connect(url, **options)
2020-11-03 20:06:29.985 22383 ERROR zun.websocket.websocketproxy File "/usr/local/lib/python3.8/dist-packages/websocket/_core.py", line 222, in connect
2020-11-03 20:06:29.985 22383 ERROR zun.websocket.websocketproxy self.sock, addrs = connect(url, self.sock_opt, proxy_info(**options),
2020-11-03 20:06:29.985 22383 ERROR zun.websocket.websocketproxy File "/usr/local/lib/python3.8/dist-packages/websocket/_http.py", line 121, in connect
2020-11-03 20:06:29.985 22383 ERROR zun.websocket.websocketproxy sock = _open_socket(addrinfo_list, options.sockopt, options.timeout)
2020-11-03 20:06:29.985 22383 ERROR zun.websocket.websocketproxy File "/usr/local/lib/python3.8/dist-packages/websocket/_http.py", line 201, in _open_socket
2020-11-03 20:06:29.985 22383 ERROR zun.websocket.websocketproxy raise err
2020-11-03 20:06:29.985 22383 ERROR zun.websocket.websocketproxy File "/usr/local/lib/python3.8/dist-packages/websocket/_http.py", line 176, in _open_socket
2020-11-03 20:06:29.985 22383 ERROR zun.websocket.websocketproxy sock.connect(address)
2020-11-03 20:06:29.985 22383 ERROR zun.websocket.websocketproxy File "/usr/lib/python3/dist-packages/eventlet/greenio/base.py", line 253, in connect
2020-11-03 20:06:29.985 22383 ERROR zun.websocket.websocketproxy socket_checkerr(fd)
2020-11-03 20:06:29.985 22383 ERROR zun.websocket.websocketproxy File "/usr/lib/python3/dist-packages/eventlet/greenio/base.py", line 51, in socket_checkerr
2020-11-03 20:06:29.985 22383 ERROR zun.websocket.websocketproxy raise socket.error(err, errno.errorcode[err])

2020-11-03 20:06:29.985 22383 ERROR zun.websocket.websocketproxy ConnectionRefusedError: [Errno 111] ECONNREFUSED

2020-11-03 20:06:29.985 22383 ERROR zun.websocket.websocketproxy
2020-11-03 20:06:29.985 22383 ERROR zun.websocket.websocketproxy During handling of the above exception, another excepti...

Read more...

Revision history for this message
Beholder101 (vandensteen) wrote :

the proxy service on the controller is up:
netstat -ntulp | grep 6784
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp 0 0 10.0.0.1:6784 0.0.0.0:* LISTEN -

and connections from the controller and the docker node are unrestricted (no firewalls):

telnet 10.0.0.1 6784
Trying 10.0.0.1...
Connected to 10.0.0.1.
Escape character is '^]'.
Connection closed by foreign host.

Revision history for this message
Beholder101 (vandensteen) wrote :

And this is resolved. The documentation in the Openstack guide does not mention the configuration of the API connection with the Docker service (2375).

You should also add on your compute/docker node in /etc/zun/zun.conf

[docker]
docker_remote_api_url = tcp://<docker ip>:2375
docker_remote_api_host = <docker ip>

After that change it did not work because i had used the hostname and not DNS name or IP in the service for docker:

in /etc/systemd/system/docker.service.d/docker.conf

[Service]
ExecStart=
ExecStart=/usr/bin/dockerd --group zun -H tcp://<docker name>:2375 -H unix:///var/run/docker.sock --cluster-store etcd://<controller name>:2379

I had to change tcp://<docker name>:2375 to tcp://<docker ip>:2375

On the controller node i already added in /etc/zun/zun.conf

[websocket_proxy]
wsproxy_host = <controller ip>
wsproxy_port = 6784
base_url = ws://controller ip>:6784/
allowed_origins = <controller name>,<controller ip>

Restarting the zun-api en zun-wspproxy service on the controller and zun-compute with docker service on the docker node finished it.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.