Cinder Volume and API fail to start if backend isn't set for [key_manager] in cinder.conf

[Jay Bryant]

There was some code that was in Cinder in Ocata that set a default API class for Castellan to use if it wasn't defined in cinder.conf. This code was removed with the assumption that it would always be set because devstack sets it. This, however, isn't guaranteed to be set in non-devstack installs. The code in Castellan expects that either the api_class or backend are sent into set_defaults. If this isn't the case then you see the following error:

Oct 19 18:42:35 jaysdevvm cinder-volume[17260]: DEBUG oslo_db.sqlalchemy.engines [None req-fac34938-da7f-46e9-b8aa-cb8caa4e190a None None] MySQL server mode set to STRICT_
Oct 19 18:42:35 jaysdevvm cinder-volume[17260]: ERROR cinder.cmd.volume [None req-fac34938-da7f-46e9-b8aa-cb8caa4e190a None None] Volume service jaysdevvm@v3700 failed to
Oct 19 18:42:35 jaysdevvm cinder-volume[17260]: ERROR cinder.cmd.volume Traceback (most recent call last):
Oct 19 18:42:35 jaysdevvm cinder-volume[17260]: ERROR cinder.cmd.volume File "/opt/stack/cinder/cinder/cmd/volume.py", line 98, in main
Oct 19 18:42:35 jaysdevvm cinder-volume[17260]: ERROR cinder.cmd.volume cluster=cluster)
Oct 19 18:42:35 jaysdevvm cinder-volume[17260]: ERROR cinder.cmd.volume File "/opt/stack/cinder/cinder/service.py", line 395, in create
Oct 19 18:42:35 jaysdevvm cinder-volume[17260]: ERROR cinder.cmd.volume cluster=cluster)
Oct 19 18:42:35 jaysdevvm cinder-volume[17260]: ERROR cinder.cmd.volume File "/opt/stack/cinder/cinder/service.py", line 148, in __init__
Oct 19 18:42:35 jaysdevvm cinder-volume[17260]: ERROR cinder.cmd.volume *args, **kwargs)
Oct 19 18:42:35 jaysdevvm cinder-volume[17260]: ERROR cinder.cmd.volume File "/opt/stack/cinder/cinder/volume/manager.py", line 242, in __init__
Oct 19 18:42:35 jaysdevvm cinder-volume[17260]: ERROR cinder.cmd.volume self.key_manager = key_manager.API(CONF)
Oct 19 18:42:35 jaysdevvm cinder-volume[17260]: ERROR cinder.cmd.volume File "/opt/stack/cinder/cinder/keymgr/__init__.py", line 29, in API
Oct 19 18:42:35 jaysdevvm cinder-volume[17260]: ERROR cinder.cmd.volume cls = importutils.import_class(conf.key_manager.backend)
Oct 19 18:42:35 jaysdevvm cinder-volume[17260]: ERROR cinder.cmd.volume File "/usr/local/lib/python2.7/dist-packages/oslo_utils/importutils.py", line 30, in import_class
Oct 19 18:42:35 jaysdevvm cinder-volume[17260]: ERROR cinder.cmd.volume __import__(mod_str)
Oct 19 18:42:35 jaysdevvm cinder-volume[17260]: ERROR cinder.cmd.volume ValueError: Empty module name

So, this is kind of a bug in Castellan but it seems like it is a problem we can catch and correct in Cinder. I am going to propose a fix to Cinder and see what everyone thinks.

[Jay Bryant (jsbryant)]

My initial debug wasn't quite right. It appears that 'backend' is being set but it is set to 'barbican'. I am not totally sure where that is happening or why it is causing a failure. Going to add Kaitlin to see if she might be able to help me understand the problem here.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/514456

[Kaitlin Farr (kaitlin-farr)]

Hey Jay, the bug is in Cinder, and I've put up a patch to fix it. Hopefully the commit message provides enough details/explanation.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/514456
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=1a27bcab802431ff408154a6e246a8ff9697338c
Submitter: Zuul
Branch: master

commit 1a27bcab802431ff408154a6e246a8ff9697338c
Author: Kaitlin Farr <email address hidden>
Date: Mon Oct 23 16:39:43 2017 -0400

    Fix key_manager API call

    If you pass importutils.import_class a string that is not actually a
    reference to a python class, you get an "empty module" error.
    Cinder's keymgr.API call used to try to import the castellan backend
    directly, but this was not necessary as the castellan key_manager.API
    call will return the correct backend based on the api_class or backend
    option in the config file. If no api_class or backend option is
    specified, castellan key_manager API call will return a barbican backend
    by default [1].

    Castellan used to require the full class name to specify the backend (i.e.
    api_class=castellan.key_manager.barbican_key_manager.BarbicanKeyManager),
    so the import_class call in cinder was no issue. Recently castellan
    was updated to use stevedore entry points and the config option name was changed
    to "backend" [2]. The change in castellan is backwards compatible so that if
    you use a full module name instead of the stevedore entry point name, it will
    still get the correct backend. However, the cinder code was not updated accordingly,
    so any time the shorthand stevedore entry point name for the backend is used
    instead of the full module name, the import_class call in cinder will fail.

    This change removes the unnecessary import_class call and adds an appropriate
    unit test that shows the "empty module" error does not occur anymore.

    1. https://github.com/openstack/castellan/blob/b13187b34ddcc88bee57e4e99b0115cb92540b97/castellan/key_manager/__init__.py#L25
    2. https://github.com/openstack/castellan/commit/8980bf7da55dd084ad84c84534fe937f0d43b9c0

    Closes-Bug: #1724952
    Change-Id: I3f95cfe439d65d9cf746d4e3844bc70bc705625e

[Jay Bryant (jsbryant)]

Kaitlin,

Thank you so much for fixing this!

The issue also exists in Pike so I am going to backport.

Jay

[Kaitlin Farr (kaitlin-farr)]

Hey Jay! Yes, that line of code that I removed still exists in Pike, but it only posed a problem when Castellan switched to using stevedore entry points instead of full class names. The Castellan release that includes stevedore was 0.13.0, and it looks like the upper constraints for Pike was was 0.12.0. Backporting won't hurt anything, but I don't think the bug was an issue in Pike?

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/cinder 12.0.0.0b2 development milestone.