nova-consoleauth doesn't play well with memcached

[Dr. Jens Harbott]

When running with the Ubuntu Xenial Mitaka packages, I'm seeing the following behaviour:

Following the release notes at http://docs.openstack.org/releasenotes/nova/mitaka.html I remove the old option "memcached_servers" and set up a cache section with

[cache]
enabled = true
memcache_servers = host1:11211,host2:11211,host3:11211

The result is that there are errors logged when creating a token:

2016-04-19 10:17:33.501 15952 WARNING nova.consoleauth.manager [req-ad043de9-616d-4cbb-a93f-fc5e345f1f53 1b0106d4f339406bb7012a83162ba5f2 e3c253d3e8344a8796e70bc4f96b6166 - - -] Token: dd0e7bbc-c593-421f-a56b-52578aaec20e failed to save into memcached.
2016-04-19 10:17:33.503 15952 WARNING nova.consoleauth.manager [req-ad043de9-616d-4cbb-a93f-fc5e345f1f53 1b0106d4f339406bb7012a83162ba5f2 e3c253d3e8344a8796e70bc4f96b6166 - - -] Instance: d3267abb-258d-4b5d-a22e-8e3b0a39905c failed to save into memcached

Only after a lot of debugging it turns out that the default backend for oslo_cache is dogpile.cache.null, implying that no values get cached and token validation always fails. Only after adding

[cache]
backend = oslo_cache.memcache_pool

does console authentication start working. Strangely though, the above warning messages are still being logged in the working setup, which made debugging this even more difficult.

So I suggest the following fixes:

1. Change the text of the warnings from "failed to save into memcached" to "failed to save into cache", as with the change to using oslo_cache, there may be other backends in use instead of memcached.
2. Either override the default of using the null backend or refuse to run with it or at the very least give a big fat warning that the configuration can not work.
3. Stop generating the warning messages when the data got in fact saved into cache properly.

Package versions for reference:
# dpkg -l | grep nova
ii nova-api-metadata 2:13.0.0-0ubuntu2 all OpenStack Compute - metadata API frontend
ii nova-api-os-compute 2:13.0.0-0ubuntu2 all OpenStack Compute - OpenStack Compute API frontend
ii nova-cert 2:13.0.0-0ubuntu2 all OpenStack Compute - certificate management
ii nova-common 2:13.0.0-0ubuntu2 all OpenStack Compute - common files
ii nova-conductor 2:13.0.0-0ubuntu2 all OpenStack Compute - conductor service
ii nova-consoleauth 2:13.0.0-0ubuntu2 all OpenStack Compute - Console Authenticator
ii nova-scheduler 2:13.0.0-0ubuntu2 all OpenStack Compute - virtual machine scheduler
ii nova-spiceproxy 2:13.0.0-0ubuntu2 all OpenStack Compute - spice html5 proxy
ii python-nova 2:13.0.0-0ubuntu2 all OpenStack Compute Python libraries
ii python-novaclient 2:3.3.1-2 all client library for OpenStack Compute API - Python 2.7
# dpkg -l | grep oslo
ii python-oslo.cache 1.6.0-2 all cache storage for Openstack projects - Python 2.7
ii python-oslo.concurrency 3.7.0-2 all concurrency and locks for OpenStack projects - Python 2.x
ii python-oslo.config 1:3.9.0-3 all Common code for Openstack Projects (configuration API) - Python 2.x
ii python-oslo.context 2.2.0-2 all WSGI context helpers for OpenStack - Python 2.x
ii python-oslo.db 4.7.0-2ubuntu1 all database connectivity to the different backends and helper utils - Python 2.x
ii python-oslo.i18n 3.5.0-2 all Oslo Internationalization Utilities - Python 2.x
ii python-oslo.log 3.2.0-2 all OpenStack logging configuration library - Python 2.x
ii python-oslo.messaging 4.6.1-2ubuntu1 all oslo messaging library - Python 2.x
ii python-oslo.middleware 3.8.0-2 all various WSGI middleware components for OpenStack - Python 2.x
ii python-oslo.policy 1.6.0-2 all RBAC policy enforcement library for OpenStack - Python 2.x
ii python-oslo.reports 1.7.0-2 all reports serialized in various data types - Python 2.7
ii python-oslo.rootwrap 4.1.0-2 all allows fine filtering of shell commands to run as root - Python 2.x
ii python-oslo.serialization 2.4.0-2 all utilities for serialization , especially JSON - Python 2.x
ii python-oslo.service 1.8.0-1ubuntu1 all library for running OpenStack services - Python 2.x
ii python-oslo.utils 3.8.0-2 all set of utility functions for OpenStack - Python 2.x
ii python-oslo.versionedobjects 1.8.0-1 all deals with DB schema versions and code expectations - Python 2.x
ii python-oslo.vmware 2.5.0-2 all VMware library for OpenStack projects - Python 2.7

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/307698

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/307718

[Dr. Jens Harbott (j-harbott)]

Hmm, even after setting everything up properly, I am seeing warnings like this:

2016-04-26 19:49:01.731 30404 DEBUG oslo.cache.core [req-e1c84b09-256e-4a50-b6b2-2961626ab954 013fadbb4e0c42669ac9e34643b09b5f 4a0a30294e954953a41a9791d4e7f437 - - -] CACHE_SET: Key: "'eee2cd7c98a7757eb08b8c1c9be32dcf15e4b4fc'" Value: "('["c2e6cb39-439c-48ed-840e-2c9609325717", "f6dfd635-534b-4ee4-95dc-38d33defcd6f"]', {'v': 1, 'ct': 1461700141.731355})" set /usr/lib/python2.7/dist-packages/oslo_cache/core.py:94
2016-04-26 19:49:01.731 30404 WARNING nova.consoleauth.manager [req-e1c84b09-256e-4a50-b6b2-2961626ab954 013fadbb4e0c42669ac9e34643b09b5f 4a0a30294e954953a41a9791d4e7f437 - - -] Instance: cce6cbe3-e6c7-43d4-b2a6-f4bc2d1e7dfb failed to save into memcached

Though the CACHE_SET has succeeded, as I can read that value from memcached:

# nc 10.250.1.11 11211
get eee2cd7c98a7757eb08b8c1c9be32dcf15e4b4fc
VALUE eee2cd7c98a7757eb08b8c1c9be32dcf15e4b4fc 1 177
cdogpile.cache.api
CachedValue
...
.
END

If I interpret the dogpile.cache docs properly, the set function will never have a return value, so checking for that in nova/consoleauth/manager.py L94+L106 seems outdated.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/307698
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=dcd14adb61b018501f3c0525ecfd7c13535cf386
Submitter: Jenkins
Branch: master

commit dcd14adb61b018501f3c0525ecfd7c13535cf386
Author: Jens Rosenboom <email address hidden>
Date: Tue Apr 19 12:35:17 2016 +0200

    Avoid unconditional warnings in nova-consoleauth

    With the introduction of oslo_cache, there is no feedback anymore about
    whether set()ting a cache item was successful, so we stop generating
    warnings about this all the time.

    Change-Id: I2ca0cdd3dd30498a23da2ef4f352afd199496862
    Partial-Bug: 1572062

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/307718
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=59192cfbf9482fe3798cdcd4c2044d8c85f66e95
Submitter: Jenkins
Branch: master

commit 59192cfbf9482fe3798cdcd4c2044d8c85f66e95
Author: Jens Rosenboom <email address hidden>
Date: Tue Apr 19 13:18:52 2016 +0200

    Warn when using null cache backend

    The default backend for oslo_cache is dogpile.cache.null, leading to a
    broken setup when following the current release notes or the deprecation
    message verbatim. So we should at least emit a warning when the config
    does not specify a different backend.

    Note: I'm not sure whether it is possible to amend the release note like
    this or whether there needs a new note to be added, please advise.

    Change-Id: I16647e424d8382dae98f13cb1f73a7e0c0aebaf5
    Closes-Bug: 1572062

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 14.0.0.0b3 development milestone.