novncproxy: code 404, message File not found

[Paul Mathews]

2015-03-19 06:09:49.295 24982 INFO nova.console.websocketproxy [-] WebSocket server settings:
2015-03-19 06:09:49.295 24982 INFO nova.console.websocketproxy [-] WebSocket server settings:
2015-03-19 06:09:49.296 24982 INFO nova.console.websocketproxy [-] - Listen on 192.168.0.3:6080
2015-03-19 06:09:49.296 24982 INFO nova.console.websocketproxy [-] - Listen on 192.168.0.3:6080
2015-03-19 06:09:49.296 24982 INFO nova.console.websocketproxy [-] - Flash security policy server
2015-03-19 06:09:49.296 24982 INFO nova.console.websocketproxy [-] - Flash security policy server
2015-03-19 06:09:49.296 24982 INFO nova.console.websocketproxy [-] - Web server. Web root: /usr/share/novnc
2015-03-19 06:09:49.296 24982 INFO nova.console.websocketproxy [-] - Web server. Web root: /usr/share/novnc
2015-03-19 06:09:49.296 24982 INFO nova.console.websocketproxy [-] - No SSL/TLS support (no cert file)
2015-03-19 06:09:49.296 24982 INFO nova.console.websocketproxy [-] - No SSL/TLS support (no cert file)
2015-03-19 06:09:49.297 24982 INFO nova.console.websocketproxy [-] - proxying from 192.168.0.3:6080 to None:None
2015-03-19 06:09:49.297 24982 INFO nova.console.websocketproxy [-] - proxying from 192.168.0.3:6080 to None:None
192.168.0.2 - - [19/Mar/2015 06:11:44] "GET /include/util.js HTTP/1.1" 200 -
192.168.0.2 - - [19/Mar/2015 06:11:45] "GET /include/websock.js HTTP/1.1" 200 -
192.168.0.2 - - [19/Mar/2015 06:11:45] "GET /include/keyboard.js HTTP/1.1" 200 -
192.168.0.2 - - [19/Mar/2015 06:11:45] "GET /include/display.js HTTP/1.1" 200 -
192.168.0.2 - - [19/Mar/2015 06:11:45] code 404, message File not found
192.168.0.2 - - [19/Mar/2015 06:11:45] "GET /include/keysym.js HTTP/1.1" 404 -

The new MOX 6.0 deployment for IWassa (46.16.184.66) was improperly configured (novncproxy_base_url was missing http://host:port - only had /vnc_auto.html) and after properly setting this, we get a 404 back from novncproxy. Watching all novncproxy instances, we see the token come through:

192.168.0.2 - - [19/Mar/2015 06:11:44] "GET /vnc_auto.html?token=1e522ad7-2669-45c5-8fa0-8f6f82eee1ac HTTP/1.1" 200 -

But we get a 404 back. When the page is visited, the vnc client loads, but never proceeds to the guest vnc, which is listening.

[Paul Mathews (pmathews-i)]

Poking at this a bit more this morning, I set debug=true in nova.conf, hoping to get more verbose output. I started novncproxy from the cli, and it's suddenly working.

[Bogdan Dobrelya (bogdando)]

6.0.1 is in HCF, high bugs can be assigned only for 6.0.2

[Aleksandr Didenko (adidenko)]

Confirmed on 6.1 in bug: https://bugs.launchpad.net/fuel/+bug/1434562

Diagnostic snapshot:
https://drive.google.com/a/mirantis.com/file/d/0B6SjzarTGFxaNjl3NE5VTTF6WTA/view?usp=sharing

I can see the same messages in logs:
http://paste.openstack.org/show/197175/

VNC itself works fine, I'm able to connect with vncviewer.

[Aleksandr Didenko (adidenko)]

OK, it looks like our 'novnc' package is outdated. I got console working after downloading master branch from https://github.com/kanaka/noVNC to /usr/share/novnc/ on controller and restarting nova-novncproxy service. So we need to update 'novnc' package.
Forwarding this bug to OSCI team.

[Aleksander Mogylchenko (amogylchenko)]

What do you mean by "outdated"? We have the latest 0.5.1 release for 6.1, which is known to be working:
https://bugs.launchpad.net/fuel/6.0.x/+bug/1418567

If you have some specific bugs that are fixed in master but are present in 0.5.1, please provide links to them - we will consider backporting.

[Aleksandr Didenko (adidenko)]

OK. Here are some additional details. I can see 3 JS files are missing for novnc on Ubuntu envs:

# grep 404 /var/log/upstart/nova-novncproxy.log | grep 'GET /include/' | awk '{print $7}' | sort -u
/include/keyboard.js
/include/keysymdef.js
/include/keysym.js

# ll /usr/share/novnc/include/{keysymdef.js,keyboard.js,keysym.js}
ls: cannot access /usr/share/novnc/include/keysymdef.js: No such file or directory
ls: cannot access /usr/share/novnc/include/keyboard.js: No such file or directory
ls: cannot access /usr/share/novnc/include/keysym.js: No such file or directory

# dpkg -L novnc | grep /include/ | egrep 'keysymdef.js|keyboard.js|keysym.js' || echo Not found
Not found

Due to this, novnc in Horizon loads forever and I can't get console.

Checking our deb package:

# wget http://osci-mirror-msk.msk.mirantis.net/mos/ubuntu//pool/main/n/novnc/novnc_0.5.1+gitfc00821eba~mira6.1_all.deb
# dpkg -c novnc_0.5.1+gitfc00821eba~mira6.1_all.deb | grep /include/ | egrep 'keysymdef.js|keyboard.js|keysym.js' || echo Not found
Not found

CentOS rpm has those files in place so only Ubuntu is affected. As soon as I put those missing files into /usr/share/novnc/include/ on controllers, novnc console starts to work. So it looks like this problem is caused by missing files in out DEB package. Please check.

[OSCI Robot (oscirobot)]

package novnc has been built for project packages/trusty/novnc
Package version == 1:0.5.1+gitfc00821eba~mira6.1, package release == ubuntu1

Changeset: https://review.fuel-infra.org/5098
project: packages/trusty/novnc
branch: 6.1
author: Aleksandr Mogylchenko
committer: Aleksandr Mogylchenko
subject: novncproxy: code 404, message File not found
status: patchset-created

Files placed on repository:

NOTE: Changeset is not merged, created temporary package repository.
 repository URL: /trusty-fuel-6.1-stable-5098/

[Aleksander Mogylchenko (amogylchenko)]

Could you please recheck newly built package?

[Aleksandr Didenko (adidenko)]

Works fine on my local test env. Thanks.

[Aleksander Mogylchenko (amogylchenko)]

Status of 6.0.x is not clear, so no fix this time

[Alexander Nevenchannyy (anevenchannyy)]

Verified on MOS 6.1 ISO #429
Steps to Verify:
root@node-1:~# ls /usr/share/novnc/include -1 | egrep 'keysymdef.js|keyboard.js|keysym.js'
keyboard.js
keysymdef.js
keysym.js

[sbartel (samuel-bartel)]

"Status of 6.0.x is not clear, so no fix this time" << does that mean we won't have fix for this issue in 6.0.x?

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: 6.0.1
Change author: Aleksandr Mogylchenko <email address hidden>
Review: https://review.fuel-infra.org/9973

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/9973
Submitter: Michael Semenov <email address hidden>
Branch: 6.0.1

Commit: 60b8fc2d4a277cfdbfea27d1ebe0b680839f40af
Author: Aleksandr Mogylchenko <email address hidden>
Date: Thu Jul 30 09:00:09 2015

novncproxy: code 404, message File not found

Some JS files were missing in the package, resulting in non-working novnc.

Change-Id: I8b67a598b8b1493876a837f851ec7b4bd1d8fa71
Closes-Bug: #1433894

[Aleksander Mogylchenko (amogylchenko)]

6.0.1 status is not clear, but I've pushed the fix to our stable repos.

[Michael Polenchuk (mpolenchuk)]

Confirmed on Fuel 10:

# tail /var/log/nova/nova-novncproxy.log
...
2017-04-07 15:19:09.622 153296 INFO nova.console.websocketproxy [-] 192.168.4.2 - - [07/Apr/2017 15:19:09] code 404, message File not found

[Michael Polenchuk (mpolenchuk)]

Will mark this as "incomplete" while details (logs, cluster configs) are coming.

[Ivan Suzdal (isuzdal)]

novnc in 10.0 shouldn't be affected. Marked as 'invalid'.

dpkg -c novnc_0.6.1-1~u16.04%2bmos1_all.deb | grep -E 'key(.*).js$'
-rw-r--r-- root/root 20840 2017-03-27 12:39 ./usr/share/novnc/include/keyboard.js
-rw-r--r-- root/root 22527 2017-03-27 12:39 ./usr/share/novnc/include/keysym.js
-rw-r--r-- root/root 21512 2017-03-27 12:39 ./usr/share/novnc/include/keysymdef.js