VNC Console in Dashboard fails to connect ot server (code: 1006)

[steve A]

This is a multi server installation of openstack with 1 'Cloud controller ' runnign all nova services + 2 compute nodes
I have installed both nova-novncproxy and novnc packages and it appears novnc itself is working ok.
All hosts are 12.04 using the Ubuntu cloud repo for folsom, all updated in the last week.

my nova.conf is as follows:
novncproxy_base_url=http://172.28.137.65:6080/vnc_auto.html
vncserver_listen=0.0.0.0
vncserver_proxyclient_address=172.28.137.65

my host IP is 172.28.137.65 for Cloud controller

summary -
I can request a noVNC url as follows:

| ID | Name | Status | Networks |
+--------------------------------------+--------------+--------+-------------------------+
| 0c6ce9b3-2064-4189-8d86-29df9b4ddaff | steveasles02 | ACTIVE | lab-test=10.0.10.10 |
| ff8c5c9d-fb77-4451-b08c-4eee610f3b47 | steveaubu02 | ACTIVE | lab-test=10.0.10.11 |
| db9952e3-ced8-4a70-8892-473b258731d6 | steveawin02 | ACTIVE lab-test=10.0.10.12 |
| 17323ecc-1f93-4f4e-8139-e629a02a97e1 | steveawin04 | ACTIVE | lab-test=10.0.10.14 |
+--------------------------------------+--------------+--------+-------------------------+
root@iswubu33:/etc/nova# nova get-vnc-console ff8c5c9d-fb77-4451-b08c-4eee610f3b47 novnc
+-------+------------------------------------------------------------------------------------+
| Type | Url |
+-------+------------------------------------------------------------------------------------+
| novnc | http://172.28.137.65:6080/vnc_auto.html?token=0599f402-9150-462d-9c14-c2ed10c69492 |
+-------+------------------------------------------------------------------------------------+

i can connect to each VM succesfully from Chrome or firefox via VNC from this URL.
I DO have to refresh the window 2-3 times for the connection to be established, this may be an issue in itself ???.

So far so good, i now want to access this via the Folsom dashboard so i login to the dashboard as a project user, select a VM instance of which I as a user have launched , from the drop down - select VNC console.
VNC Console then appears but is unable to connect to the same Vms that i could access before directly from the nova get-vnc-console command.

so in summary, i cant access the VNC console in the dashboard
error reported back in the browser is a 'failed to connect to server (code: 1006)
I CAN connect via nova get-vnc-console <serverid> novnc

Steps to reproduce -
login to the dashboard as a project user, select a VM instances of which I as a user have launched , from the drop down - select VNC console. An empty VNC window appears but cannot connect to the VM
100% repeatable

current Firefox browser (V16.0.1), shockwave flash version 11.4.402.287
current Chrome (Version 22.0.1229.94 m)

any guidance on how to workaround or resolve would be really appreciated

[Gabriel Hurley (gabriel-hurley)]

The fact that you're having to try to connect multiple times is suspicious. If your setup were completely working correctly it would connect every time.

The Nova team is much better equipped to debug NoVNC issues than the Horizon team. All Horizon does is get that URL and connect to it in an iFrame.

That said, one reason I *have* seen the NoVNC service fail intermittently like that is in multi-node/proxied systems where you're running multiple NoVNC services (such as on each node) and so when the dashboard tries to connect to the URL that the API returned it ends up hitting a service that didn't originate the URL/token so it gets rejected. That may or may not be the case here, but is one avenue worth investigating.

All in all I'm pretty sure this isn't a Horizon bug, at least without further details.

[[e-mail] (bullardza)]

I can reproduce this on my setup with Folsom. 4th refresh connects me every time.

[Gabriel Hurley (gabriel-hurley)]

If it's consistent like that I strongly suspect that it has to do with running multiple nova-consoleauth services and the tokens ending up in the wrong places as I said in my last comment...

I could always be wrong, though. ;-)

[Vish Ishaya (vishvananda)]

agreed this looks like multiple copies of consoleauth or novncproxy. There should only be one of these running.

[Justin Palmer (s7p4-julkon-kkws)]

Gabriel is exactly right. I was experiencing this problem and stopped nova-consoleauth on all but one node and the problem disappeared.

Thanks!

[steve A (dafridgie)]

thanks for the detailed feedback, I did have multiple nova-consoleauth instances running on my compute nodes as well as the Cloud controller system. I disabled nova-consoleauth on the compute nodes i can confirm as well that issue is now resolved.

I can see now in the documentation that only 1 consoleauth process is required. thanks for clarifying this !

Steve a

[Davanum Srinivas (DIMS) (dims-v)]

Marking bug as invalid per last comment from @dafridgie

[YunQiang Su (wzssyqa)]

I install all of these components on one node, and it will happen once I reset the physical machine.

IT seems that there is only one nova-consoleauth instance running.
root@eayun-manager:/var/log/nova# ps aux |grep nova-con
nova 1275 0.0 0.0 37952 1316 ? Ss 09:01 0:00 su -s /bin/sh -c exec nova-consoleauth --config-file=/etc/nova/nova.conf nova
nova 1276 0.3 2.3 187580 47220 ? S 09:01 0:01 /usr/bin/python /usr/bin/nova-consoleauth --config-file=/etc/nova/nova.conf
root 6748 0.0 0.0 10024 932 pts/2 S+ 09:06 0:00 grep --color=auto nova-con

[YunQiang Su (wzssyqa)]

ohh, my fault, after adding

resume_guests_state_on_host_boot=true

to nova.conf, it resolved

[pippo (claudio-enjoy)]

I m follow the multiple node(https://github.com/mseknibilel/OpenStack-Grizzly-Install-Guide/blob/OVS_MultiNode/OpenStack_Grizzly_Install_Guide.rst)
also I noticed the Glizzly I have less debugs then Folsom and quantum 's working :D
Now the only thing doesnt work is the vnc on the browser I get error : "
Failed to connect to server (code: 1006)
"
I have tried different browser and safari too.

Now my controller ip is 192.168.2.231 and ext 192.168.221.231

both controller and compute /etc/nova/nova.conf have :

novnc_enabled=true
novncproxy_base_url=http://192.168.221.231:6080/vnc_auto.html
novncproxy_port=6080
vncserver_proxyclient_address=192.168.2.231
vncserver_listen=0.0.0.0

all services are fine , nova and quantum ...

only get problem on nova-novncproxy that is running but :

 cat /var/log/upstart/nova-novncproxy.log
WebSocket server settings:
  - Listen on 0.0.0.0:6080
  - Flash security policy server
  - Web server. Web root: /usr/share/novnc
  - No SSL/TLS support (no cert file)
  - proxying from 0.0.0.0:6080 to ignore:ignore

1: 192.168.221.1: new handler Process
  2: 192.168.221.1: new handler Process
  3: 192.168.221.1: new handler Process
  1: 192.168.221.1: "GET /vnc_auto.html?token=f70fbd58-695b-4ddf-8e76-417aa958ee79&title=vm234(f944ad21-54c0-43aa-b218-44c8800a7a53) HTTP/1.1" 200 -
  4: 192.168.221.1: new handler Process
  4: 192.168.221.1: Plain non-SSL (ws://) WebSocket connection
  4: 192.168.221.1: Version hybi-13, base64: 'True'
  4: 192.168.221.1: Path: '/websockify'
  4: connecting to: 192.168.2.231:5900
  4: handler exception: [Errno 111] Connection refused
  4: Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/websockify/websocket.py", line 797, in top_new_client
    self.new_client()
  File "/usr/lib/python2.7/dist-packages/nova/console/websocketproxy.py", line 62, in new_client
    tsock = self.socket(host, port, connect=True)
  File "/usr/lib/python2.7/dist-packages/websockify/websocket.py", line 193, in socket
    sock.connect(addrs[0][4])
  File "/usr/lib/python2.7/socket.py", line 224, in meth
    return getattr(self._sock,name)(*args)
error: [Errno 111] Connection refused

  2: 192.168.221.1: ignoring socket not ready
  3: 192.168.221.1: ignoring socket not ready

in controller the port 5900 is not open but is open the 6080
sudo lsof -i:6080
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
nova-novn 29447 nova 3u IPv4 446639 0t0 TCP *:6080 (LISTEN)

and in compute is open 5900 but not 6080
 lsof -i:5900
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
kvm 2016 libvirt-qemu 16u IPv4 792378 0t0 TCP *:5900 (LISTEN)

if you can help to resolve that thanks :)

[pippo (claudio-enjoy)]

nova get-vnc-console <id-instance > novnc

get the Url , but then doesn't work either.

Also I can see before became red label with error message "Failed to connect to server (code: 1006)" on the top of the console
the label is yellow and said "Starting VNC handshake" .

[xciter327 (xciter327)]

I am following the single node OVS guide from the same source and I encounter this issue as well.

[pippo (claudio-enjoy)]

ok fixed !
set in compute machine :
in nova.conf
 vncserverproxyclientaddress = compute ip

in case of the grizzly guide is eth0 so 10.10.10.53

[najeh nafti (nouray)]

@Claudio (claudio-enjoy)
#13

vncserverproxyclientaddress = compute ip

compute ip is the management or provider ip