consoleauth cannot be run in HA configuration without external memcache

Bug #1243306 reported by Stanislaw Pitucha
26
This bug affects 4 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Opinion
Low
Unassigned

Bug Description

Running more than one consoleauth service causes silent failures where tokens simply don't get authenticated, because only one of the processes has it cached.

There are two ways to fix this:
- process sending the new token has to use the fanout queue rather than a direct message, so that all consoleauth services are updated
- token can be sent to the database, rather than consoleauth directly - this allows restarting services and adding new ones without creating new problems

Ideally both ways could be implemented at the same time.

Tags: console
Yaguang Tang (heut2008)
Changed in nova:
assignee: nobody → Yaguang Tang (heut2008)
Revision history for this message
Michael H Wilson (geekinutah) wrote :

I don't this think this is a bug at all, you can already use a shared memcache instance so that tokens are shared between consoleauth services. You just need to define memcached_servers for each consoleauth process ie.

memcached_server=sharedhost1:11211, sharedhost2:11211

The memcache client will take care of hashing keys to the different memcache instances.

Revision history for this message
Stanislaw Pitucha (stanislaw-pitucha) wrote :

This is a solution for sharding the tokens. That's something completely different from HA.

By setting up two memcache servers, you have 2 SPOFs - one is each of the memcache servers (because tokens are not shared between them), another one is the consoleauth (if that server goes down, you can't access any server). If you setup another consoleauth server, you're just ending up with 1 SPOF again (token stored in one memcache).

This bug is about removing that SPOF - make sure that the information itself is replicated in case of multiple servers. There are also reasons not to rely on external memcache (as mentioned in the bug title).

Revision history for this message
Michael H Wilson (geekinutah) wrote :

And I should read the title of the bug better..... disregard my previous comment.

Matt Riedemann (mriedem)
tags: added: consoleauth
Changed in nova:
assignee: Yaguang Tang (heut2008) → nobody
Changed in nova:
assignee: nobody → Stanislaw Pitucha (stanislaw-pitucha)
Changed in nova:
importance: Undecided → Wishlist
status: New → Confirmed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/75140

Changed in nova:
status: Confirmed → In Progress
Allison Randal (allison)
tags: added: console
removed: consoleauth
Revision history for this message
Joe Gordon (jogo) wrote :

Is this still valid, patch was abandoned.

Changed in nova:
status: In Progress → Incomplete
Revision history for this message
Stanislaw Pitucha (stanislaw-pitucha) wrote :

I'm not working with Nova an daily basis anymore, handed this patch to someone else. I'll make sure the bug is reassigned.

Changed in nova:
assignee: Stanislaw Pitucha (stanislaw-pitucha) → David McNally (dave-mcnally)
Changed in nova:
status: Incomplete → In Progress
Revision history for this message
Bjørnar Ness (bjornar-ness) wrote :

What is progress on this.

I would like to see fanout + memory cache (local/memcached). Dont see the reason to stress database
with this, as worst case scenario is "open console again"

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on nova (master)

Change abandoned by Sean Dague (<email address hidden>) on branch: master
Review: https://review.openstack.org/75140
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

Revision history for this message
Davanum Srinivas (DIMS) (dims-v) wrote :

Removing "In Progress" status and assignee as change is abandoned.

Changed in nova:
status: In Progress → Confirmed
assignee: David McNally (dave-mcnally) → nobody
Alex Xu (xuhj)
Changed in nova:
assignee: nobody → Alex Xu (xuhj)
Eli Qiao (taget-9)
Changed in nova:
assignee: Alex Xu (xuhj) → Eli Qiao (taget-9)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/177078

Changed in nova:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on nova (master)

Change abandoned by Eli Qiao (<email address hidden>) on branch: master
Review: https://review.openstack.org/177078

Revision history for this message
Eli Qiao (taget-9) wrote :

there was a nova sepc to address this bug. https://review.openstack.org/#/c/165838

but it has been postpone to M release.

Changed in nova:
status: In Progress → Incomplete
Revision history for this message
Markus Zoeller (markus_z) (mzoeller) wrote :

wrt to comment #12

The bug report looks still valid. A blueprint which also intents to
solve this bug shouldn't be a reason to put it to "incomplete" IMO.
That's why I change it back to "confirmed".
As there is no open patch available, I remove the assignee too.

Changed in nova:
status: Incomplete → Confirmed
assignee: Eli Qiao (taget-9) → nobody
Revision history for this message
Markus Zoeller (markus_z) (mzoeller) wrote :

In addition to comment #13:

This bug report is pretty old. I'm closing it.
Please re-test this issue when [1] has merged. If it hasn't then reopen this report.

References:
[1] https://review.openstack.org/#/c/301158/

Changed in nova:
importance: Wishlist → Low
status: Confirmed → Opinion
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.