Access to VM console not working as Horizon redirects to novncproxy.openstack.svc.cluster.local

[Chris Winnicki]

Brief Description
-----------------
Access to VM console not working as Horizon redirects to novncproxy.openstack.svc.cluster.local

Severity
--------
Major: Vbox installation cannot access VMs consoles

Steps to Reproduce
------------------
Install Vbox controller-0, controller-1 as All-in-one
Launch VMs
Attempt to access VM console from Horizon

Expected Behavior
------------------
Access to VM console from Horizon should work

Actual Behavior
----------------
Access to VM console from Horizon does not work
As Horizon redirects to: http://novncproxy.openstack.svc.cluster.local
(see attached screenshots)

Reproducibility
---------------
100% reproducible on build: 20190423T135120Z

System Configuration
--------------------
1+1 system (Vbox AIO-DX)

Branch/Pull Time/Commit
-----------------------
BUILD_ID="20190423T135120Z"
JOB="STX_build_master_master"
<email address hidden>"

Last Pass
---------
n/a

Timestamp/Logs
--------------
not available / not required

Test Activity
-------------
Vbox Installation (All-in-one)

[Frank Miller (sensfan22)]

Marking stx.2.0 gating as access to VM consoles is a basic requirement for using VMs.

Assigning to Cindy and request assistance to identify a prime to investigate this issue.

[yong hu (yhu6)]

Hi @chriswinnicki have you done these settings? I got this recipe from mailing list previously.

type these two commands on active controller:
sudo iptables -A INPUT -p tcp -m multiport --dports 80 -m comment --comment "500 horizon ingress dashboard" -j ACCEPT
sudo iptables -A INPUT -p tcp -m multiport --dports 443 -m comment --comment "500 horizon ingress dashboard" -j ACCEPT

As well if you are testing with VMs (instead of bare metal) for StarlingX, you might also have to add this setting in /etc/hosts in test host (on which StarlingX nodes are running as VMs) . This is one time setting.
10.10.10.2 novncproxy.openstack.svc.cluster.loca

[Elio Martinez (elio1979)]

Hello Yong Hu, i tried to follow your WA but i'm not able to see the console after applying that values to the IP tables plus modifying /etc/hosts, any other suggestion?

[Erich Cordoba (ericho)]

We tested the mentioned steps today as Elio mentioned. The steps worked before but know is not possible to reach the console. However, we tried a ssh tunnel with :

ssh -L 6080:localhost:6080 wrsroot@10.10.10.3

and then changing the URL to 127.0.0.1:6080 instead of novncproxy.openstack.svc.cluster.local and the console appeared in the browser.

In some point recently, the configuration change making unable to reach the novnc service.

[yong hu (yhu6)]

The root cause was 6080 on OAM was blocked by GlobalNetworkPolicy controller-oam-if-gnp

You can check it by cmd: "kubectl describe GlobalNetworkPolicy controller-oam-if-gnp" and 6080 was NOT in TCP Allow list.

  Ingress:
    Action: Allow
    Destination:
      Ports:
        22
        18002
        4545
        15491
        6385
        7777
        6443
        7480
        9311
        5000
        8080
    Ip Version: 4
    Protocol: TCP

I will apply a patch to add this port.

[Fernando Hernandez Gonzalez (fhernan2)]

I tried WA from Erich,
I opened a tunnel with ssh command from my jumper to the controller:
   ssh -L 6080:localhost:6080 wrsroot@10.10.10.3
   Remark: the tunnel need to keep alive in order to keep the VM consoles opened.
I enabled "AllowTcpForwarding yes" flag in "/etc/ssh/sshd_config" file.
Once edited I restart the sshd config by "sudo systemctl restart sshd"

and invoked the graphic console successfully.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/658288

[yong hu (yhu6)]

after this fix is accepted and with the original settings (https://wiki.openstack.org/wiki/StarlingX/Containers/Installation#Instance_Console_Access), the VM VNC console in Horizon (:31000) should be seen.

[Matt Peters (mpeters-wrs)]

OpenStack is not a platform service and should not be exposed via the OAM firewall by default. When OpenStack is installed, the either with http or https enabled, then the appropriate GlobalNetworkPolicy should be configured to enable access to that port via the OAM network.

Example Policy:

apiVersion: crd.projectcalico.org/v1
kind: GlobalNetworkPolicy
metadata:
  name: allow-openstack-endpoints
spec:
  ingress:
  - action: Allow
    destination:
      ports:
      - 80
    protocol: TCP
  order: 500
  selector: has(iftype) && iftype == 'oam'
  types:
  - Ingress

[OpenStack Infra (hudson-openstack)]

Change abandoned by yong hu (<email address hidden>) on branch: master
Review: https://review.opendev.org/658288

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/662748

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/662748
Committed: https://git.openstack.org/cgit/starlingx/config/commit/?id=9b0820d1d65dca4c2d647f6e6aff435c58e84592
Submitter: Zuul
Branch: master

commit 9b0820d1d65dca4c2d647f6e6aff435c58e84592
Author: yhu6 <email address hidden>
Date: Tue May 28 03:59:11 2019 +0000

    add helm chart for nginx ports control

    this chart is added as a part of "stx-openstack" application,
    in the same chart group as openstack-ingress chart, so that
    when "nginx-ingress-controller" starts working, http and https
    ports are allowed for nginx which accepts http/https requests
    and forwards to internal services accordingly.
    In the following LP#1827246, the http request of opening console
    of VM instance is sent to nginx 80 first, and then nginx forwards
    the request to "nova-novncproxy" at port 6080 internally.

    Closes-Bug: 1827246

    Change-Id: I183f7edc92f1a9e0bdedad0afe35e3d03e20e7d5
    Signed-off-by: yhu6 <email address hidden>

[Peng Peng (ppeng)]

Issue was reproduced on
WCP_105
20190718T013000Z

[yong hu (yhu6)]

I just now tested on 0724 build and this feature did work well. see the attachment.

controller-1:~$ cat /etc/build.info
###
### StarlingX
### Built from master
###

OS="centos"
SW_VERSION="19.01"
BUILD_TARGET="Host Installer"
BUILD_TYPE="Formal"
BUILD_ID="20190724T013000Z"

JOB="STX_build_master_master"
<email address hidden>"
BUILD_NUMBER="191"
BUILD_HOST="starlingx_mirror"
BUILD_DATE="2019-07-24 01:30:00 +0000"

[yong hu (yhu6)]

did you set this as this wiki says "https://wiki.openstack.org/wiki/StarlingX/Containers/Installation#Instance_Console_Access"??

Add an alias to the novncproxy default local domain in /etc/hosts file on your machine or in other words, from the Horizon Graphical interfaces is launched.

<your Controller IP address> novncproxy.openstack.svc.cluster.local

[Yang Liu (yliu12)]

This issue reappeared on baremetal system as Peng commented above.
The wiki link for instance console access provided by Yong seems no longer valid, but I would expect no extra manual configuration needed on baremetal system. With the same environment, I was able to access the vm console from Horizon about a month ago.

[Chris Winnicki (chriswinnicki)]

The issue is still present - as of: 2019-08-12_20-59-00

[yong hu (yhu6)]

@Frank, please assign some one to resolve this issue.
It was not about the nova-vnc port for which I made the patch anymore.
it requires a solution, mentioned by Matt in the attached mail.

[yong hu (yhu6)]

@Zhipeng is to implement a node-port for novnc-proxy so that novnc console can be accessed by users on its explicit port, similar to Horizon (:31000).

Consider this change is a part of openstack helm, instead of OpenStack itself, tag this LP with stx.containers.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/677126

[zhipeng liu (zhipengs)]

Hi all,

The root cause of why we need to enable 6080 in OAM firewall is novncproxy using hostNetwork.
in deployment-novncproxy.yaml, it set hostNetwork to true.
In iptables, we can see oam:30680 go to 192.168.204.3:6080 without forward. then it will be filtered if we do not enable 6080 port in oam firewall.

So, there are 2 solutions.
1) No need to expose 30680 node port, just need enable 6080 in oam firewall and connect to oam:6080 directly.
2) Submit a patch for openstack-helm to set hostNetwork to false. Then it will forward to cluster network instead of host network, then it can work as we expected before.

I already verified both solutions

Any comment?

Thanks!
Zhipeng

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/679913

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/680908

[OpenStack Infra (hudson-openstack)]

Change abandoned by zhipeng liu (<email address hidden>) on branch: master
Review: https://review.opendev.org/679913
Reason: Abandon as repo changed.
Resubmit with below patch
https://review.opendev.org/#/c/680908/

[ANIRUDH GUPTA (anyrude10)]

Hi Team,

I have downloaded Starlingx R2.0 and still facing the same issue in opening of Console.

The URL is
"http://novncproxy.openstack.svc.cluster.local/vnc_auto.html?token=6cb9811a-4102-4313-99b4-16df5fbceec7&title=vm1(343d2e13-d8a8-4b69-88ec-db12edaa06d1)"

I have also made an entry in /etc/hosts file for "novncproxy.openstack.svc.cluster.local", but still unable to open it.

I have also opened the port mentioned in the comments, but of no use

sudo iptables -A INPUT -p tcp -m multiport --dports 80 -m comment --comment "500 horizon ingress dashboard" -j ACCEPT
sudo iptables -A INPUT -p tcp -m multiport --dports 443 -m comment --comment "500 horizon ingress dashboard" -j ACCEPT

As I can see port 6080 is still not open

"kubectl describe GlobalNetworkPolicy controller-oam-if-gnp"

Destination:
      Ports:
        22
        18002
        4545
        15491
        6385
        7777
        6443
        7480
        9311
        5000
        8080
Can someone please update on it.

Thanks
Anirudh

[zhipeng liu (zhipengs)]

Hi Anirudh,

Two patches is under review.
https://review.opendev.org/#/c/680908/
https://review.opendev.org/#/c/677126/

You can try to access VM with workaround method.

Zhipeng

[ANIRUDH GUPTA (anyrude10)]

Hi Zhipeng

As per the link mentioned

https://review.opendev.org/#/c/677126/

"The novncproxy would be available at the OAMIP:30680"

But in my case, Port 30680 is not open in my setup.
I ran the command
netstat -apn | grep 30680

But, there was no connection being made with the port.

And as per the 2nd workaround

https://review.opendev.org/#/c/680908/

I can't find any "deployment-novncproxy.yaml" file.

Can you please elaborate the steps that I need to follow in order to resolve the console issue?
Do I need to create a new setup or these changes can be implemented in my already created Simplex Virtual Setup?

[zhipeng liu (zhipengs)]

My patch is ready for merge and I'm pushing now, hope it can be merged this week.
You can use new version with this patch after some days.

Zhipeng

[ANIRUDH GUPTA (anyrude10)]

Hi Zhipeng

Please let me know once your patched is merged and suggest the way how can we use the new version?
Do I need to download the ISO once it's merged or need to do the changes in the existing setup?

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/680908
Committed: https://git.openstack.org/cgit/starlingx/openstack-armada-app/commit/?id=2e63d41e5c429b616454fed0ac2073a73ad5d5b5
Submitter: Zuul
Branch: master

commit 2e63d41e5c429b616454fed0ac2073a73ad5d5b5
Author: zhipengl <email address hidden>
Date: Mon Sep 9 21:42:06 2019 +0800

    Add a network config item for novncproxy

    In deployment-novncproxy.yaml, it set hostNetwork = true.
    We want to let it use cluster network instead of hostNetwork.
    This patch will add a config item, so that we can override it
    to use cluster network. Then no need to enable 6080 port in
    local network firewall for novncproxy access.

    Upstream patch submitted as below.
    https://review.opendev.org/#/c/679891

    Below test pass!
    Access to VM console through horizon works!

    Closes-bug: 1827246

    Change-Id: Icb0cfa39839e151d5869c64bc8f0151d0d9faf49
    Signed-off-by: zhipengl <email address hidden>

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/677126
Committed: https://git.openstack.org/cgit/starlingx/config/commit/?id=f8ec30d0c69ec0c843b8077bd892b8e932279c6c
Submitter: Zuul
Branch: master

commit f8ec30d0c69ec0c843b8077bd892b8e932279c6c
Author: zhipengl <email address hidden>
Date: Mon Sep 9 22:33:44 2019 +0800

    Expose the novncproxy port

    The novncproxy would be available at the OAMIP:30680.

    Test pass on simplex setup.
    VM console can be accessed through horizon.

    Closes-bug: 1827246

    Depends-on: https://review.opendev.org/#/c/680908/
    Change-Id: I6c450d391a8842d8336a0085dc31f939cb8789d9
    Signed-off-by: zhipengl <email address hidden>

[Ghada Khalil (gkhalil)]

@Zhipeng, the fix will need to be cherry-picked to the r/stx.2.0 branch so that it's included in the next maintenance release. Please do this asap.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: r/stx.2.0
Review: https://review.opendev.org/684440

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: r/stx.2.0
Review: https://review.opendev.org/684443

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/684443
Committed: https://git.openstack.org/cgit/starlingx/upstream/commit/?id=104189e1618a5c4f2cebe538cbf2ce6ce3e2dfec
Submitter: Zuul
Branch: r/stx.2.0

commit 104189e1618a5c4f2cebe538cbf2ce6ce3e2dfec
Author: zhipengl <email address hidden>
Date: Mon Sep 9 21:42:06 2019 +0800

    Add a network config item for novncproxy

    In deployment-novncproxy.yaml, it set hostNetwork = true.
    We want to let it use cluster network instead of hostNetwork.
    This patch will add a config item, so that we can override it
    to use cluster network. Then no need to enable 6080 port in
    local network firewall for novncproxy access.

    Upstream patch submitted as below.
    https://review.opendev.org/#/c/679891

    Below test pass!
    Access to VM console through horizon works!

    Closes-bug: 1827246

    Change-Id: Icb0cfa39839e151d5869c64bc8f0151d0d9faf49
    Signed-off-by: zhipengl <email address hidden>

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/684440
Committed: https://git.openstack.org/cgit/starlingx/config/commit/?id=707cdd9d68f223c84cd4a5d9ffb8fad0434629d2
Submitter: Zuul
Branch: r/stx.2.0

commit 707cdd9d68f223c84cd4a5d9ffb8fad0434629d2
Author: zhipengl <email address hidden>
Date: Mon Sep 9 22:33:44 2019 +0800

    Expose the novncproxy port

    The novncproxy would be available at the OAMIP:30680.

    Test pass on simplex setup.
    VM console can be accessed through horizon.

    Closes-bug: 1827246

    Depends-on: https://review.opendev.org/#/c/684443/
    Change-Id: I6c450d391a8842d8336a0085dc31f939cb8789d9
    Signed-off-by: zhipengl <email address hidden>
    (cherry picked from commit f8ec30d0c69ec0c843b8077bd892b8e932279c6c)

[ANIRUDH GUPTA (anyrude10)]

Hi,

The fix would be available in next official release or it would be in latest_green_builds available?

If it would be available in latest_green_build, on which date's green build we can have the solution?

[Ghada Khalil (gkhalil)]

@Anirudh Gupta, The fix will be available in the next stx.2.0.1 (maintenance) release. Please monitor the stx-discuss mailing list for availability. A re-install of the new ISO will be required.

[ANIRUDH GUPTA (anyrude10)]

Hi Ghada,

As per the latest discussion on stx-discuss, the build 2.0.1 was released on 7th October

http://lists.starlingx.io/pipermail/starlingx-discuss/2019-October/006422.html

But I can't find it anywhere on the Cengn Mirror.

http://mirror.starlingx.cengn.ca/mirror/starlingx/

Can you please update by when it would be available for the users and what all bugs are being fixed in that release?

[Volker (skydiablo)]

hi,
this is my first try at STX 2.0 and i also run in this issue. after add "10.10.10.2 novncproxy.openstack.svc.cluster.local" to my host-file, the console-GUI is shown but my browser-debug-console tells me "cant connect to ws://localhost:17523/".

so my questions are:
- is this a new/other bug?
- @gkhalil mention to reinstall the complete cluster with version 2.0.1 RC, to fix this bug
-- can i leave my current version (STX 2.0) and get all the stuff by auto-updates to new release 2.0.1? sure, until this time my console keep being broken.

volker.

[Volker (skydiablo)]

*
-- can i KEEP my current version...

sry for my bad english ;)

[Wendy Mitchell (wmitchellwr)]

verified on load 20200115T023003Z (HWlab IP 20-27)
successfully accessed instance console (as admin/tenant)
successful access also when logged into floating IP.