Cannot access container's console via wss

[hongbin]

@fishbone_ reported in IRC that he cannot connect to container's console via horizon. Below are the steps provided by @fishbone_ to reproduce the error:

- install zun as per documentation instructions here: https://docs.openstack.org/zun/queens/install/
- install horizon on a separate node as per horizon as per documentation instructions here: https://docs.openstack.org/horizon/queens/install/install-ubuntu.html
- setup https access to dashboard via instructions here: https://docs.openstack.org/horizon/latest/admin/customize-configure.html
- add configuration for 'allowed_origins' = HOSTNAME_OF_HORIZON_SERVER to zun conf files on compute nodes and cloud controller
- install zun-ui on horizon node via git repository at branch queens
-- run setup.py install and copy all files from enabled folder to openstack-dashboard/local/enabled, excluding console.py
-- collectstatic and compress
- restart horizon service
- connect to horizon dashboard and create a container
- connect to the console tab
- console of container will not be visible, check web browser console
- you will be informed that there are contradicting protocols on the same connection; 'ws is not secure and cannot be used with https, please connect using wss'

[David Ivey (yevi)]

@hongbin, I was getting ready to open another bug for wss based on our discussion last week, but it looks like this is the same issue.

We are also using queens specifically version 1.0.1 with a multi-node setup using ubuntu 16.04.
  - controller: zun-api and zun-wsproxy
  - compute: zun-compute, kuryr-libnetwork and docker
  - zun-ui is on a separate node with horizon.

The above error in regards to trying to connect to a ws:/ socket while in a https:// session happens if they are setup for HTTPS but the conf file for zun has the base_url set to ws:// instead of wss://
and should produce this error "Error: Failed to construct 'WebSocket': An insecure WebSocket connection may not be initiated from a page loaded over HTTPS". I believe this is and should be expected behavior. So the problem is when you set the base_url as wss:// to support a secure connection you still do not have access to the container for an interactive session with zun-ui or zun cli. The connection is refused at zun-wsproxy and never makes it to the compute node.

If everything is configured for https/wss you can interactively connect to any running container with "openstack appcontainer exec --interactive mycontainer /bin/sh", but can not connect through zun-ui.

If you try to spin up a container with an interactive session with "openstack appcontainer run -i --name mycontainer --net network=netuuid cirros /bin/sh" you get a connection refused just like you do in zun-ui. To my understanding zun-ui uses this same method to connect to the console session.

I bypassed zun-ui and the requirement for https to test ws:// and everything worked as expected. I can get any logs/configs you need, but a good chunk of it is in irc logs here: http://eavesdrop.openstack.org/irclogs/%23openstack-zun/%23openstack-zun.2018-11-08.log.html if needed for reference. The only thing that is not there is the successful one after switching back to ws://

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/619934

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/619942

[hongbin (hongbin034)]

After a couple of fixes [1][2][3], I managed to get the "wss" connection work in master branch. I will backport the fixes to stable branches after they are merged in master.

To setup secure websocket correctly, it requires additional steps to configure the zun websocket proxy as following:

    [websocket_proxy]
    ...
    ssl_only = True
    cert = /opt/stack/data/devstack-cert.pem

In which, the "cert" is the certificates used in the OpenStack deployment.

[1] https://review.openstack.org/#/c/619907/
[2] https://review.openstack.org/#/c/619934/
[3] https://review.openstack.org/#/c/619942/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/619934
Committed: https://git.openstack.org/cgit/openstack/zun/commit/?id=1bb994962be5822fdf992adf2ead07a72f6064f4
Submitter: Zuul
Branch: master

commit 1bb994962be5822fdf992adf2ead07a72f6064f4
Author: Hongbin Lu <email address hidden>
Date: Sun Nov 25 16:21:50 2018 +0000

    Add support for secure websocket connection

    Add a few SSL config for operator to turn on secure connection
    on zun websocket proxy.

    Change-Id: Ica077eda90e0d782c206eea7c446b6af646668f8
    Closes-Bug: #1762511

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/619907
Committed: https://git.openstack.org/cgit/openstack/python-zunclient/commit/?id=ff3b380680c109d0819c19b5530b483082c6935b
Submitter: Zuul
Branch: master

commit ff3b380680c109d0819c19b5530b483082c6935b
Author: Hongbin Lu <email address hidden>
Date: Sat Nov 24 19:35:42 2018 +0000

    Allow wss as websocket protocol

    Allow users to establish "wss" (secure web socket) connection to
    the container. This is a client side fix but we also need to fix
    the server to fully support "wss"

    Change-Id: I1dd4f4872be03d1d80e3decbab18b41376230aba
    Partial-Bug: #1762511

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/619942
Committed: https://git.openstack.org/cgit/openstack/python-zunclient/commit/?id=67660449617b2a884735f000f922faafe69659b0
Submitter: Zuul
Branch: master

commit 67660449617b2a884735f000f922faafe69659b0
Author: Hongbin Lu <email address hidden>
Date: Sun Nov 25 21:46:44 2018 +0000

    Set HTTP header "Origin" when using websocket

    This is because if this header is missing, the websocket client
    will populate this header as "http://...", which is incorrect
    if we are using secure connection (in which this header should
    be "https://...").

    Change-Id: I441946dc168db645744da093e215f65fa1ca3637
    Related-Bug: #1762511

[David Ivey (yevi)]

@hongbin I patched my deployment with your fix and I can now use wss with openstack cli via "openstack appcontainer exec --interactive mycontainer --net network=netuuid cirros /bin/sh". I am still having issues with zun-ui console connections. zun-ui for me sits on a separate node than my controller and compute. So I am wondering if I might have missed something there. I patched websocketclient.py with the two patches you have for zunclient on the nodes with zun-ui, but I still can not get a connection. Does zun-ui request the websocket console connections in the same way as openstack cli? Or does it have another dependency I might have missed.

[David Ivey (yevi)]

That was my fault, I made a mistake. I have wss working throughout my environment now. Thanks for the patch. I had to modify it slightly to work in queens specifically in the websocketclient.py for the do_exec function.

[hongbin (hongbin034)]

@David,

Good to hear that you get everything working. I am about to backport the fixes to stable brance. If possible, could you show me what you modified?

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/620441

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/620442

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/rocky
Review: https://review.openstack.org/620443

[David Ivey (yevi)]

I really just used your patch, but queens did not have the conditional statement in the function do_exec(387) like master so I added it, but left the variables from queens in zunclient/common/websocketclient/websocketclient.py.

def do_exec(zunclient, url, container_id, exec_id, escape, close_wait):
    if url.startswith("ws://") or url.startswith("wss://"):
        try:
            httpcls = HTTPClient(zunclient=zunclient, url=url,
                               exec_id=exec_id,
                               id=container_id, escape=escape,
                               close_wait=close_wait)
            httpcls.connect()
            httpcls.handle_resize()
            httpcls.start_loop()
        except exceptions.ContainerWebSocketException as e:
            print("%(e)s:%(container)s" %
                  {'e': e, 'container': container_id})
    else:
        raise exceptions.InvalidWebSocketLink(container_id)

[Hongbin Lu (hongbin.lu)]

Got it. Thanks for the information, David.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/620620

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/queens
Review: https://review.openstack.org/620622

[David Ivey (yevi)]

No problem, Glad I could help

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/620637

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/620443
Committed: https://git.openstack.org/cgit/openstack/python-zunclient/commit/?id=3adcd73ac7024ae084532d0d8fb264dad245b694
Submitter: Zuul
Branch: stable/rocky

commit 3adcd73ac7024ae084532d0d8fb264dad245b694
Author: Hongbin Lu <email address hidden>
Date: Sun Nov 25 21:46:44 2018 +0000

    Set HTTP header "Origin" when using websocket

    This is because if this header is missing, the websocket client
    will populate this header as "http://...", which is incorrect
    if we are using secure connection (in which this header should
    be "https://...").

    Change-Id: I441946dc168db645744da093e215f65fa1ca3637
    Related-Bug: #1762511
    (cherry picked from commit 67660449617b2a884735f000f922faafe69659b0)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/620620
Committed: https://git.openstack.org/cgit/openstack/zun/commit/?id=b8d8595fa376d56c87600510b8aba8fdcc013390
Submitter: Zuul
Branch: stable/queens

commit b8d8595fa376d56c87600510b8aba8fdcc013390
Author: Hongbin Lu <email address hidden>
Date: Sun Nov 25 16:21:50 2018 +0000

    Add support for secure websocket connection

    Add a few SSL config for operator to turn on secure connection
    on zun websocket proxy.

    Change-Id: Ica077eda90e0d782c206eea7c446b6af646668f8
    Closes-Bug: #1762511
    (cherry picked from commit 1bb994962be5822fdf992adf2ead07a72f6064f4)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/620441
Committed: https://git.openstack.org/cgit/openstack/zun/commit/?id=e5c588acfd7070b309b5285ac55dd3f0f6bfbfbe
Submitter: Zuul
Branch: stable/rocky

commit e5c588acfd7070b309b5285ac55dd3f0f6bfbfbe
Author: Hongbin Lu <email address hidden>
Date: Sun Nov 25 16:21:50 2018 +0000

    Add support for secure websocket connection

    Add a few SSL config for operator to turn on secure connection
    on zun websocket proxy.

    Change-Id: Ica077eda90e0d782c206eea7c446b6af646668f8
    Closes-Bug: #1762511
    (cherry picked from commit 1bb994962be5822fdf992adf2ead07a72f6064f4)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/620622
Committed: https://git.openstack.org/cgit/openstack/python-zunclient/commit/?id=216ba98b4e285c3d23ab68a876afb38b3cc2482e
Submitter: Zuul
Branch: stable/queens

commit 216ba98b4e285c3d23ab68a876afb38b3cc2482e
Author: Hongbin Lu <email address hidden>
Date: Sun Nov 25 21:46:44 2018 +0000

    Set HTTP header "Origin" when using websocket

    This is because if this header is missing, the websocket client
    will populate this header as "http://...", which is incorrect
    if we are using secure connection (in which this header should
    be "https://...").

    Change-Id: I441946dc168db645744da093e215f65fa1ca3637
    Related-Bug: #1762511
    (cherry picked from commit 67660449617b2a884735f000f922faafe69659b0)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/620637
Committed: https://git.openstack.org/cgit/openstack/python-zunclient/commit/?id=5a0a13283020d59005a8fcd43f84c1a02fd48abd
Submitter: Zuul
Branch: stable/queens

commit 5a0a13283020d59005a8fcd43f84c1a02fd48abd
Author: Hongbin Lu <email address hidden>
Date: Sat Nov 24 19:35:42 2018 +0000

    Allow wss as websocket protocol

    Allow users to establish "wss" (secure web socket) connection to
    the container. This is a client side fix but we also need to fix
    the server to fully support "wss"

    Change-Id: I1dd4f4872be03d1d80e3decbab18b41376230aba
    Partial-Bug: #1762511
    (cherry picked from commit ff3b380680c109d0819c19b5530b483082c6935b)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/620442
Committed: https://git.openstack.org/cgit/openstack/python-zunclient/commit/?id=617f1126ac21119708050a64a18b7fba339a82f4
Submitter: Zuul
Branch: stable/rocky

commit 617f1126ac21119708050a64a18b7fba339a82f4
Author: Hongbin Lu <email address hidden>
Date: Sat Nov 24 19:35:42 2018 +0000

    Allow wss as websocket protocol

    Allow users to establish "wss" (secure web socket) connection to
    the container. This is a client side fix but we also need to fix
    the server to fully support "wss"

    Change-Id: I1dd4f4872be03d1d80e3decbab18b41376230aba
    Partial-Bug: #1762511
    (cherry picked from commit ff3b380680c109d0819c19b5530b483082c6935b)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/zun 3.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/zun queens-eol release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/zun rocky-eol release.