Instance VNC console can't connect on the first attempt

[Anastasia Palkina]

"build_id": "2014-05-27_05-51-41",
"mirantis": "yes",
"build_number": "26",
"ostf_sha": "a8b7660082a6f152794c610d6abe30d360fd577d",
"nailgun_sha": "bd09f89ef56176f64ad5decd4128933c96cb20f4",
"production": "docker",
"api": "1.0",
"fuelmain_sha": "505741e4f431f85a8d0252fc42754d10c0326c1a",
"astute_sha": "a7eac46348dc77fc2723c6fcc3dbc66cc1a83152",
"release": "5.0",
"fuellib_sha": "2f79c0415159651fc1978d99bd791079d1ae4a06"

1. Create new environment (Ubuntu, HA mode)
2. Choose GRE segmentation
3. Add 3 controllers, compute and cinder
4. Start deployment. It was successful
5. Login to Horizon
6. Create instance, associate floating IP
7. Move to Instance console in Horizon
8. Result: Failed to connect

Dmitry Ilyin found information: https://answers.launchpad.net/nova/+question/216534
We experimented and found that if only one nova-consoleuth is started instance console connect on the first attempt.
If two nova-consoleuth are started instance console can't connect on the first attempt

[Dmitry Borodaenko (angdraug)]

Why is this High priority?

[Mike Scherbakov (mihgen)]

This is one of the basic OpenStack features. If fails 1 of 3 times in HA mode, then I think it can be considered as High priority issue.

[Jesse Pretorius (jesse-pretorius)]

I think it's high priority - this is something that should work every time.

In order to workaround this at the moment the only option is to shut down two of the three nova-consoleauth services on the controllers.

[Dmitry Borodaenko (angdraug)]

Makes sense. Added 5.0.1 to target milestones.

[Jesse Pretorius (jesse-pretorius)]

Proper workaround:

Under the [DEFAULT] section in nova.conf, add the memcached_servers option with a list of each controller's IP address. For example:

memcached_servers=192.168.0.9:11211,192.168.0.4:11211,192.168.0.15:11211

[Jesse Pretorius (jesse-pretorius)]

It seems this is the source of the problem:

The metadata_api manifest, which used to set the memcached_servers on the controllers, is no longer included:
https://github.com/stackforge/fuel-library/blob/master/deployment/puppet/openstack/manifests/nova/controller.pp#L256-L265

The config option is still set on the compute nodes though:
https://github.com/stackforge/fuel-library/blob/master/deployment/puppet/openstack/manifests/compute.pp#L171-L174

So it would seem that the options just need to be set inside the controller manifest to resolve the issue.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/102387

[Dmitry Borodaenko (angdraug)]

The proposed fix shouldn't be merged ahead of upstream puppet module nova. Reassigning back to Stanislaw.

[Joshua Dotson (tns9)]

I'm on 5.0.1 built over the weekend with nova-network, Ceph for all, plus H/A. Consoles do not work for me.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/102387
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=a235c6a6d91a691472ba3a84fe9fe782f884dfbc
Submitter: Jenkins
Branch: master

commit a235c6a6d91a691472ba3a84fe9fe782f884dfbc
Author: Andrew Woodward <email address hidden>
Date: Tue Jun 24 16:32:30 2014 -0700

    Add memcached_servers so that nova-consoleauth can share tokens properly

    add memcached_servers into the same location as in upstream puppet-nova.
    Also updated openstack::compute to use same code path

    Change-Id: I5f94322bb74dd52fcfd3f923a6583bd742618849
    Closes-bug: #1323705

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/5.0
Review: https://review.openstack.org/104650

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/104650
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=8a7d86a033b82520abe611bc2c286a10eae42d93
Submitter: Jenkins
Branch: stable/5.0

commit 8a7d86a033b82520abe611bc2c286a10eae42d93
Author: Andrew Woodward <email address hidden>
Date: Tue Jun 24 16:32:30 2014 -0700

    Add memcached_servers so that nova-consoleauth can share tokens properly

    add memcached_servers into the same location as in upstream puppet-nova.
    Also updated openstack::compute to use same code path

    Change-Id: I5f94322bb74dd52fcfd3f923a6583bd742618849
    Closes-bug: #1323705

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/108790

[Josh Juzwik (n-josh)]

Also on 5.0.1, nova-network, Ceph for all, plus H/A. Consoles did not work for me...

Applied the workaround from post #6 on the 3 controllers, and consoles are working now.

[Bogdan Dobrelya (bogdando)]

If there would be 4.1.2 by any chances, please update the status for its milestone back to in progress/confirmed

[Aleksandr Didenko (adidenko)]

In master (5.1) it's already fixed as a part of https://blueprints.launchpad.net/fuel/+spec/merge-openstack-puppet-modules

[OpenStack Infra (hudson-openstack)]

Change abandoned by Ryan Moe (<email address hidden>) on branch: master
Review: https://review.openstack.org/108790

[Sergii Golovatiuk (sgolovatiuk)]

In 5.1 we had a regression as we merged puppet upstream manifests. Since we agreed not to change upstream manifests I simply added memcache option in openstack class which solved the problem for 5.1. Manifests for 5.0.1 seem to be fine.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/110895
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=dd4a1c0eb6bb48e7091b1fa84f189ab2ebc49171
Submitter: Jenkins
Branch: master

commit dd4a1c0eb6bb48e7091b1fa84f189ab2ebc49171
Author: Sergii Golovatiuk <email address hidden>
Date: Thu Jul 31 10:29:05 2014 +0000

    Specify memcache servers for nova::controller

    Specify all memcache servers instead of 127.0.0.1.
    Otherwise, noVNC service will be broken

    Change-Id: Id1bb3fb36aab812d8c52a16bddc27231619a684b
    Closes-Bug: 1350706
    Closes-Bug: 1323705

[7.NoJ (hejiantao5-f)]

Not only the first time
If you have n nova-consoleuth nodes, you will success connect the console every n times.
because the nova-scheduler will choose nova-consoleauth in turns, only the one generate the token can verify it, while others will fail in authorization.

[Sergii Golovatiuk (sgolovatiuk)]

nova-consoleauth on each node verifies if token is not expired. It doesn't matter which one of nova-consoleuth will be used. I verified Fuel and MOS with latest patches. Issue was indeed related to memcached issue which is already resolved and waiting for QA approval.

[Anastasia Palkina (apalkina)]

Verified on ISO #11 for HA CentOS and Ubuntu

"build_id": "2014-09-17_21-40-34", "ostf_sha": "64cb59c681658a7a55cc2c09d079072a41beb346", "build_number": "11", "auth_required": true, "api": "1.0", "nailgun_sha": "eb8f2b358ea4bb7eb0b2a0075e7ad3d3a905db0d", "production": "docker", "fuelmain_sha": "8ef433e939425eabd1034c0b70e90bdf888b69fd", "astute_sha": "f5fbd89d1e0e1f22ef9ab2af26da5ffbfbf24b13", "feature_groups": ["mirantis"], "release": "5.1", "release_versions": {"2014.1.1-5.1": {"VERSION": {"build_id": "2014-09-17_21-40-34", "ostf_sha": "64cb59c681658a7a55cc2c09d079072a41beb346", "build_number": "11", "api": "1.0", "nailgun_sha": "eb8f2b358ea4bb7eb0b2a0075e7ad3d3a905db0d", "production": "docker", "fuelmain_sha": "8ef433e939425eabd1034c0b70e90bdf888b69fd", "astute_sha": "f5fbd89d1e0e1f22ef9ab2af26da5ffbfbf24b13", "feature_groups": ["mirantis"], "release": "5.1", "fuellib_sha": "d9b16846e54f76c8ebe7764d2b5b8231d6b25079"}}}, "fuellib_sha": "d9b16846e54f76c8ebe7764d2b5b8231d6b25079"

[fullmoon4465 (fullmoon4465)]

this has been reintroduced :(. I installed Mirantis Fuel 9.0 and updated to 9.2 per their docs. Just deployed an openstack cluster and this was the issue why I couldn't console.