Comment 0 for bug 1576218

Revision history for this message
Sergey Arkhipov (sarkhipov) wrote : Cannot connect to VNC after nova-compute restart

Detailed bug description:
Cannot connect to VNC (or connection is unstable) after restarting of nova-compute service on all computes. Got 'Failed to connect to server (code: 1006)' message in Horizon and following tracebacks in logs (nova-novncproxy):

InvalidToken: The token '598d6469-663c-4ada-8a7d-ce3acc75cdb7' is invalid or has expired

or

socket.error(last_err) error: timed out

What is interesting, I've never managed to connect from inlined console on Instance page, but connected using standalone 'vnc_auto.html' ("Click here to show only console" link). Connection was unstable though

Steps to reproduce:
1. Run rally tests to create load
2. Restart nova-compute service on all computes with `service nova-compute restart`
3. Wait 5-6 minutes
4. Check 'crm status' that all is ok
5. Check 'Host aggregates' page that Nova is up and running
6. Try to connect to some instance using VNC in Horizon (proceed to instance page and click "Console")

Expected results:
1. Connection to VNC is established without any problems

Actual result:
1. Got 'Failed to connect to server (code: 1006)' error

Workaround:
Proceed to "Click here to show only console" link. Standalone page almost always connects.

Description of the environment:
* 10 baremetal nodes:
   - CPU: 12 x 2.10 GHz
   - Disks: 2 drives (SSD - 80 GB, HDD - 931.5 GB), 1006.0 GB total
   - Memory: 2 x 16.0 GB, 32.0 GB total
   - NUMA topology: 1 NUMA node
* Node roles:
  - 1 ElasticSearch / Kibana node
  - 1 InfluxDB / Grafana node
  - 3 controllers (1 was is offline because of disk problems)
  - 5 computes
* Details:
  - OS: Mitaka on Ubuntu 14.04
  - Compute: KVM
  - Neutron with VLAN segmentation
  - Ceph RBD for volumes (Cinder)
  - Ceph RadosGW for objects (Swift API)
  - Ceph RBD for ephemeral volumes (Nova)
  - Ceph RBD for images (Glance)
* MOS 8.0, build 227

Additional information:
Logs from controller and compute running VM '19908832-c817-4e20-a80f-2853d8a2ff42' are here:
http://mos-scale-share.mirantis.com/env14/28-04-2016-novnc-problems-logs.tar.xz

I've used VM with UUID '19908832-c817-4e20-a80f-2853d8a2ff42' for all tests.