Fix key manager service endpoints in devstack Nova ephemeral

[Max]

Using a Devstack setup to configure Nova ephemeral with Barbican key manager failed, when creating an instance.

Detailed description:

1. Version of Nova/Barbican we are using:

   - stable/Liberty

2. Relevant log files:

    - n-api.log Relevant section of the log file is attached with the bug report.

3. Reproduce steps:
    a) Install via devstack, with following nova post-configuration in local.conf (complete local.conf is attached)

       #...
       [[post-config|$NOVA_CONF]]
       [keymgr]
       api_class = nova.keymgr.barbican.BarbicanKeyManager

       [libvirt]
       images_type = lvm
       images_volume_group = stack-volumes-default

        [ephemeral_storage_encryption]
        cipher = aes-xts-plain64
        enabled = True
       key_size = 256
       #...

     b) Launch an instance from Horizon or using the following command line:
         <prompt>$ nova boot --flavor m1.tiny --image cirros-0.3.4-x86_64-uec --nic net-id=YOUR_NET_ID --security-group default demo-instance1

     c) Launch instance failed with following error:
         ERROR (ClientException): Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach...
         <class 'barbicanclient.exceptions.HTTPClientError'> (HTTP 500) (Request-ID: req-3be2a530-9df3-4e34-...)

4) Possible issue:
     -After analyzing the log files, the issue seems to be in nova key manager version discovery.

[Max (maxaa)]

complete local.conf file:

[[local|localrc]]

DEST=/opt/stack
GIT_BASE=https://git.openstack.org

LOGFILE=$DEST/logs/stack.sh.log
VERBOSE=True
LOG_COLOR=False
SCREEN_LOGDIR=$DEST/logs/screen
IP_VERSION=4

ADMIN_PASSWORD=openstack
MYSQL_PASSWORD=openstack
DATABASE_PASSWORD=openstack
RABBIT_PASSWORD=openstack
SERVICE_PASSWORD=openstack
SERVICE_TOKEN=tokentoken

GLANCE_BRANCH=stable/liberty
HORIZON_BRANCH=stable/liberty
KEYSTONE_BRANCH=stable/liberty
NOVA_BRANCH=stable/liberty
NEUTRON_BRANCH=stable/liberty
HEAT_BRANCH=stable/liberty
CELIOMETER_BRANCH=stable/liberty
BARBICAN_BRANCH=stable/liberty

DISABLED_SERVICES=n-net
ENABLED_SERVICES+=,q-svc,q-agt,q-dhcp,q-l3,q-meta,q-metering,neutron

#ENABLED_SERVICES+=,q-lbass
#ENABLED_SERVICES+=,heat,h-api,h-api-cfn,h-api-cw,h-eng
#HEAT_STANDALONE=True

#ENABLED_SERVICES+=,ceilometer-acompute,ceilometer-acentral,ceilometer-collector,ceilometer-api
#ENABLED_SERVICES+=,ceilometer-alarm-notify,ceilometer-alarm-eval

ENABLED_SERVICES+=,barbican

[[post-config|$NOVA_CONF]]
[keymgr]
api_class = nova.keymgr.barbican.BarbicanKeyManager

[libvirt]
images_type = lvm
images_volume_group = stack-volumes-default

[ephemeral_storage_encryption]
cipher = aes-xts-plain64
enabled = True
key_size = 256

[Markus Zoeller (markus_z) (mzoeller)]

@Max:

Could that be an issue with the network setup? The log file you attached
has these entries:

    Network Error (dns_unresolved_hostname)
    Your requested host "v1" could not be resolved by DNS.

[Max (maxaa)]

Hi Markus:
I have already confirmed the issue is not related to network setup or endpoint/host alive issue. We can see the successful request/response between services in the log files:

------ n-api.log (part) -------
2015-10-13 15:55:05.697 DEBUG keystoneclient.session [req-feafaf22-d0ab-4ac5-ba2c-13ee0ba52d92 demo demo] REQ: curl -g -i -X GET http://16.103.150.157:9696/v2.0/ports.json?tenant_id...

2015-10-13 15:55:05.837 DEBUG keystoneclient.session [req-feafaf22-d0ab-4ac5-ba2c-13ee0ba52d92 demo demo] RESP: [200]...

2015-10-13 15:55:06.541 DEBUG nova.compute.api [req-feafaf22-d0ab-4ac5-ba2c-13ee0ba52d92 demo demo] Going to run 1 instances...
----------------------------------

The problem seems to be in the construction of the barbican endpoint string, in nova key manger. The uri is wrong in the debug log below, hence the dns hostname error:

2015-10-13 15:55:06.543 DEBUG keystoneclient.session [req-feafaf22-d0ab-4ac5-ba2c-13ee0ba52d92 demo demo] REQ: curl -g -i -X POST http://v1/orders/ -H ...

2015-10-13 15:55:06.912 DEBUG keystoneclient.session [req-feafaf22-d0ab-4ac5-ba2c-13ee0ba52d92 demo demo] RESP: [404]

[Markus Zoeller (markus_z) (mzoeller)]

@Brianna Poulos: Could you have a look at it please? It could be connected with the blueprint [1] you implemented around 8 months ago.

[1] https://blueprints.launchpad.net/nova/+spec/encryption-with-barbican

[Dave McCowan (dave-mccowan)]

I don't think Barbican can be installed in DevStack that way.

Instead of:

ENABLED_SERVICES+=,barbican

try:

enable_plugin barbican https://git.openstack.org/openstack/barbican stable/liberty

[Brianna Poulos (brianna-poulos)]

@Markus Zoeller: Yes, I will take a look at this.

[Markus Zoeller (markus_z) (mzoeller)]

@Brianna Poulos: Thanks!

I can confirm this bug report in the current master branch (Mitaka).
Versions:
    Devstack: 1d7a4279 2015-10-20 Merge "modify stackforge/swift[...]
    Nova: fdd88d69 2015-10-19 Merge "update .gitreview for s[...]
    Barbican: 9d1dbf29 2015-10-20 Remove old gate code[...]
    Keystone: 81b3fe89 2015-10-19 Updated from global requiremen[...]

$ grep ERROR /opt/stack/logs/screen/screen-*
    http://paste.openstack.org/show/477010/

Devstack "local.conf" file:
    http://paste.openstack.org/show/477011/

[Dave McCowan (dave-mccowan)]

I noticed something else as well.
I'd expect to see encryption_auth_url and encryption_api_url also specificed nova.conf, like this:

[keymgr]
api_class = nova.keymgr.barbican.BarbicanKeyManager
encryption_auth_url = http://10.0.0.32:5000
encryption_api_url = http://10.0.0.32:9311

[Dane Fichter (dane-fichter)]

I am able to recreate the bug as well. I'm seeing the following line in the Nova API log:

ConnectionRefused: Unable to establish connection to http://v1/orders/

This is consistent with what others have posted, and gives us a good hint about what the error is (Nova has no idea who it's supposed to be talking to). There's a patch up for review in Barbican that might address this. I'll try merging this patch into my local Barbican and seeing if it resolves the bug.

[Markus Zoeller (markus_z) (mzoeller)]

@Dave McCowan:

> I'd expect to see encryption_auth_url and encryption_api_url also
> specificed nova.conf

I couldn't find those config options in Nova. But they are available
in Cinder: https://github.com/openstack/cinder/blob/master/cinder/keymgr/key_mgr.py#L25

[Arun Kant (arukant)]

In Nova, barbican URL is specified as endpoint_template, https://github.com/openstack/nova/blob/master/nova/keymgr/barbican.py#L42

This config endpoint template value is used to define _base_url https://github.com/openstack/nova/blob/master/nova/keymgr/barbican.py#L99 and which is eventually used to derive barbican endpoint used in barbican client https://github.com/openstack/nova/blob/master/nova/keymgr/barbican.py#L106

The issue appears to be in nova side, specifically around usage of keystone's versioned endpoint discovery logic which uses service catalog data to identify service endpoint.

[Kaitlin Farr (kaitlin-farr)]

@Markus Zoeller, I am planningn on reworking the key manager in Nova this development cycle, which should hopefully resolve this bug

[Markus Zoeller (markus_z) (mzoeller)]

@Kaitlin in response to comment #13:

Thanks, that's great news! Maybe you ping Max too, he assigned himself to this bug (which usually means that he intends to provide a solution for this bug).

Just as a reminder, it's useful if you have your IRC handle (and timezone) added to your Launchpad profile [1]. Sometimes it's easier to discuss things in real time.

[1] https://launchpad.net/~kaitlin-farr

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/243322

[Kaitlin Farr (kaitlin-farr)]

@Markus, thanks for the helpful advice in comment #14. I followed your suggestions about the Launchpad info.

I was hoping that by commenting on this thread, it would alert Max about the possible upcoming changes. I happened upon this bug report today when I decided to check what bugs were open against the key manager while I was planning out the changes. Max, if you see this, I'll start hanging out in the nova chat room.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Arun Kant (<email address hidden>) on branch: master
Review: https://review.openstack.org/243322
Reason: Abandoning this change, as nova has moved to Castellan usage for its key-manager functionality and files in my change has been removed so its no longer needed.
Castellan change review: https://review.openstack.org/#/c/309614/

[Anusha Unnam (anusha-unnam)]

@Arun Kant,@Kaitlin Farr

Is this bug fixed?

[Arun Kant (arukant)]

As the code base is entirely changed with introduction of castellan library usage, the related bug is no longer valid and can be marked 'invalid' from newton release onwards. Someone will need to revisit with castellan to see if this issue exists there or not.