Instance VNC console can't connect on the first attempt

Bug #1323705 reported by Anastasia Palkina on 2014-05-27
30
This bug affects 5 people
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
High
Sergii Golovatiuk
4.1.x
High
Stanislaw Bogatkin
5.0.x
High
Andrew Woodward

Bug Description

"build_id": "2014-05-27_05-51-41",
"mirantis": "yes",
"build_number": "26",
"ostf_sha": "a8b7660082a6f152794c610d6abe30d360fd577d",
"nailgun_sha": "bd09f89ef56176f64ad5decd4128933c96cb20f4",
"production": "docker",
"api": "1.0",
"fuelmain_sha": "505741e4f431f85a8d0252fc42754d10c0326c1a",
"astute_sha": "a7eac46348dc77fc2723c6fcc3dbc66cc1a83152",
"release": "5.0",
"fuellib_sha": "2f79c0415159651fc1978d99bd791079d1ae4a06"

1. Create new environment (Ubuntu, HA mode)
2. Choose GRE segmentation
3. Add 3 controllers, compute and cinder
4. Start deployment. It was successful
5. Login to Horizon
6. Create instance, associate floating IP
7. Move to Instance console in Horizon
8. Result: Failed to connect

Dmitry Ilyin found information: https://answers.launchpad.net/nova/+question/216534
We experimented and found that if only one nova-consoleuth is started instance console connect on the first attempt.
If two nova-consoleuth are started instance console can't connect on the first attempt

Anastasia Palkina (apalkina) wrote :
Ryan Moe (rmoe) on 2014-06-04
Changed in fuel:
status: New → Confirmed
Changed in fuel:
status: Confirmed → Triaged
importance: Undecided → Low
Changed in fuel:
importance: Low → High
Dmitry Borodaenko (angdraug) wrote :

Why is this High priority?

Mike Scherbakov (mihgen) wrote :

This is one of the basic OpenStack features. If fails 1 of 3 times in HA mode, then I think it can be considered as High priority issue.

I think it's high priority - this is something that should work every time.

In order to workaround this at the moment the only option is to shut down two of the three nova-consoleauth services on the controllers.

Dmitry Borodaenko (angdraug) wrote :

Makes sense. Added 5.0.1 to target milestones.

Proper workaround:

Under the [DEFAULT] section in nova.conf, add the memcached_servers option with a list of each controller's IP address. For example:

memcached_servers=192.168.0.9:11211,192.168.0.4:11211,192.168.0.15:11211

It seems this is the source of the problem:

The metadata_api manifest, which used to set the memcached_servers on the controllers, is no longer included:
https://github.com/stackforge/fuel-library/blob/master/deployment/puppet/openstack/manifests/nova/controller.pp#L256-L265

The config option is still set on the compute nodes though:
https://github.com/stackforge/fuel-library/blob/master/deployment/puppet/openstack/manifests/compute.pp#L171-L174

So it would seem that the options just need to be set inside the controller manifest to resolve the issue.

tags: added: low-hanging-fruit
Changed in fuel:
assignee: Fuel Library Team (fuel-library) → Stanislaw Bogatkin (sbogatkin)
Mike Scherbakov (mihgen) on 2014-06-19
tags: added: customer-found

Fix proposed to branch: master
Review: https://review.openstack.org/102387

Changed in fuel:
assignee: Stanislaw Bogatkin (sbogatkin) → Andrew Woodward (xarses)
status: Triaged → In Progress

The proposed fix shouldn't be merged ahead of upstream puppet module nova. Reassigning back to Stanislaw.

Changed in fuel:
assignee: Andrew Woodward (xarses) → Stanislaw Bogatkin (sbogatkin)
Joshua Dotson (tns9) wrote :

I'm on 5.0.1 built over the weekend with nova-network, Ceph for all, plus H/A. Consoles do not work for me.

Reviewed: https://review.openstack.org/102387
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=a235c6a6d91a691472ba3a84fe9fe782f884dfbc
Submitter: Jenkins
Branch: master

commit a235c6a6d91a691472ba3a84fe9fe782f884dfbc
Author: Andrew Woodward <email address hidden>
Date: Tue Jun 24 16:32:30 2014 -0700

    Add memcached_servers so that nova-consoleauth can share tokens properly

    add memcached_servers into the same location as in upstream puppet-nova.
    Also updated openstack::compute to use same code path

    Change-Id: I5f94322bb74dd52fcfd3f923a6583bd742618849
    Closes-bug: #1323705

Changed in fuel:
status: In Progress → Fix Committed

Reviewed: https://review.openstack.org/104650
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=8a7d86a033b82520abe611bc2c286a10eae42d93
Submitter: Jenkins
Branch: stable/5.0

commit 8a7d86a033b82520abe611bc2c286a10eae42d93
Author: Andrew Woodward <email address hidden>
Date: Tue Jun 24 16:32:30 2014 -0700

    Add memcached_servers so that nova-consoleauth can share tokens properly

    add memcached_servers into the same location as in upstream puppet-nova.
    Also updated openstack::compute to use same code path

    Change-Id: I5f94322bb74dd52fcfd3f923a6583bd742618849
    Closes-bug: #1323705

Changed in fuel:
status: Fix Committed → In Progress

Also on 5.0.1, nova-network, Ceph for all, plus H/A. Consoles did not work for me...

Applied the workaround from post #6 on the 3 controllers, and consoles are working now.

Changed in fuel:
assignee: Stanislaw Bogatkin (sbogatkin) → Bogdan Dobrelya (bogdando)
Changed in fuel:
assignee: Bogdan Dobrelya (bogdando) → Andrew Woodward (xarses)
status: In Progress → Fix Committed
Bogdan Dobrelya (bogdando) wrote :

If there would be 4.1.2 by any chances, please update the status for its milestone back to in progress/confirmed

Changed in fuel:
status: Fix Committed → In Progress
Aleksandr Didenko (adidenko) wrote :
Changed in fuel:
status: In Progress → Fix Committed
assignee: Andrew Woodward (xarses) → Fuel Library Team (fuel-library)

Change abandoned by Ryan Moe (<email address hidden>) on branch: master
Review: https://review.openstack.org/108790

Changed in fuel:
assignee: Fuel Library Team (fuel-library) → Sergii Golovatiuk (sgolovatiuk)

In 5.1 we had a regression as we merged puppet upstream manifests. Since we agreed not to change upstream manifests I simply added memcache option in openstack class which solved the problem for 5.1. Manifests for 5.0.1 seem to be fine.

Reviewed: https://review.openstack.org/110895
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=dd4a1c0eb6bb48e7091b1fa84f189ab2ebc49171
Submitter: Jenkins
Branch: master

commit dd4a1c0eb6bb48e7091b1fa84f189ab2ebc49171
Author: Sergii Golovatiuk <email address hidden>
Date: Thu Jul 31 10:29:05 2014 +0000

    Specify memcache servers for nova::controller

    Specify all memcache servers instead of 127.0.0.1.
    Otherwise, noVNC service will be broken

    Change-Id: Id1bb3fb36aab812d8c52a16bddc27231619a684b
    Closes-Bug: 1350706
    Closes-Bug: 1323705

Not only the first time
If you have n nova-consoleuth nodes, you will success connect the console every n times.
because the nova-scheduler will choose nova-consoleauth in turns, only the one generate the token can verify it, while others will fail in authorization.

tags: removed: low-hanging-fruit

nova-consoleauth on each node verifies if token is not expired. It doesn't matter which one of nova-consoleuth will be used. I verified Fuel and MOS with latest patches. Issue was indeed related to memcached issue which is already resolved and waiting for QA approval.

tags: added: in progress
Anastasia Palkina (apalkina) wrote :

Verified on ISO #11 for HA CentOS and Ubuntu

"build_id": "2014-09-17_21-40-34", "ostf_sha": "64cb59c681658a7a55cc2c09d079072a41beb346", "build_number": "11", "auth_required": true, "api": "1.0", "nailgun_sha": "eb8f2b358ea4bb7eb0b2a0075e7ad3d3a905db0d", "production": "docker", "fuelmain_sha": "8ef433e939425eabd1034c0b70e90bdf888b69fd", "astute_sha": "f5fbd89d1e0e1f22ef9ab2af26da5ffbfbf24b13", "feature_groups": ["mirantis"], "release": "5.1", "release_versions": {"2014.1.1-5.1": {"VERSION": {"build_id": "2014-09-17_21-40-34", "ostf_sha": "64cb59c681658a7a55cc2c09d079072a41beb346", "build_number": "11", "api": "1.0", "nailgun_sha": "eb8f2b358ea4bb7eb0b2a0075e7ad3d3a905db0d", "production": "docker", "fuelmain_sha": "8ef433e939425eabd1034c0b70e90bdf888b69fd", "astute_sha": "f5fbd89d1e0e1f22ef9ab2af26da5ffbfbf24b13", "feature_groups": ["mirantis"], "release": "5.1", "fuellib_sha": "d9b16846e54f76c8ebe7764d2b5b8231d6b25079"}}}, "fuellib_sha": "d9b16846e54f76c8ebe7764d2b5b8231d6b25079"

Changed in fuel:
status: Fix Committed → Fix Released
tags: removed: in progress
Pavel Vaylov (pvaylov) on 2014-10-10
summary: - Instance console can't connect on the first attempt
+ Instance VNC console can't connect on the first attempt
fullmoon4465 (fullmoon4465) wrote :

this has been reintroduced :(. I installed Mirantis Fuel 9.0 and updated to 9.2 per their docs. Just deployed an openstack cluster and this was the issue why I couldn't console.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers