OpenStack Compute (nova)

Ephemeral storage encryption is broken with interface mismatch

  1. Series ocata
  2. Bug #1651887
Bug #1651887 reported by Guang Yee
16
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Medium
Kaitlin Farr
Newton
Fix Committed
Medium
Matt Riedemann
Ocata
Fix Committed
Medium
Kaitlin Farr

Bug Description

Description
===========
Ephemeral storage encryption is broken because of interface mismatch. The default key manager (Castellan with Barbican)'s create_key() interface required at least 4 arguments. See

https://github.com/openstack/castellan/blob/0.4.0/castellan/key_manager/barbican_key_manager.py#L200

However, Nova is only passing in 3. Looks like the 'algorithm' argument is missing. See

https://github.com/openstack/nova/blob/stable/newton/nova/compute/api.py#L1401

This will result in "TypeError: create_key() takes exactly 4 arguments (3 given)" on server create.

Steps to reproduce
==================

1. Install devstack with Barbican plugin enabled. i.e.

cat local.conf
[[local|localrc]]
enable_plugin barbican https://git.openstack.org/openstack/barbican stable/newton

2. After devstack is installed, enable ephemeral storage encryption in nova.conf. i.e.

[libvirt]
images_type = lvm
images_volume_group = vg-comp

[ephemeral_storage_encryption]
key_size = 256
cipher = aes-xts-plain64
enabled = True

3. restart nova-api

4. using the nova user account, try to create a server. i.e.

gyee@abacus:~$ env | grep OS_
OS_PROJECT_DOMAIN_ID=default
OS_USER_DOMAIN_ID=default
OS_PROJECT_NAME=service
OS_IDENTITY_API_VERSION=3
OS_PASSWORD=secrete
OS_AUTH_URL=http://localhost:5000
OS_USERNAME=nova
gyee@abacus:~$ openstack flavor list
+----+-----------+-------+------+-----------+-------+-----------+
| ID | Name | RAM | Disk | Ephemeral | VCPUs | Is Public |
+----+-----------+-------+------+-----------+-------+-----------+
| 1 | m1.tiny | 512 | 1 | 0 | 1 | True |
| 2 | m1.small | 2048 | 20 | 0 | 1 | True |
| 3 | m1.medium | 4096 | 40 | 0 | 2 | True |
| 4 | m1.large | 8192 | 80 | 0 | 4 | True |
| 42 | m1.nano | 64 | 0 | 0 | 1 | True |
| 5 | m1.xlarge | 16384 | 160 | 0 | 8 | True |
| 84 | m1.micro | 128 | 0 | 0 | 1 | True |
| c1 | cirros256 | 256 | 0 | 0 | 1 | True |
| d1 | ds512M | 512 | 5 | 0 | 1 | True |
| d2 | ds1G | 1024 | 10 | 0 | 1 | True |
| d3 | ds2G | 2048 | 10 | 0 | 2 | True |
| d4 | ds4G | 4096 | 20 | 0 | 4 | True |
+----+-----------+-------+------+-----------+-------+-----------+
gyee@abacus:~$ openstack image list
+----------------------------------+---------------------------------+--------+
| ID | Name | Status |
+----------------------------------+---------------------------------+--------+
| da447cd9-619a- | cirros-0.3.4-x86_64-uec | active |
| 41b3-9772-4a9a80fa55f9 | | |
| 718fff25-9d61-4a37-a974-fdef2f1f | cirros-0.3.4-x86_64-uec-ramdisk | active |
| 570a | | |
| 91c06518-a752-48ec-a7fd- | cirros-0.3.4-x86_64-uec-kernel | active |
| 3c0ad020d9a4 | | |
+----------------------------------+---------------------------------+--------+
gyee@abacus:~$ openstack server create --image 91c06518-a752-48ec-a7fd-3c0ad020d9a4 --flavor 1 test_eph_enc
Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<type 'exceptions.TypeError'> (HTTP 500) (Request-ID: req-6d2eb531-b239-429d-8d25-f06b4fe6309c)

5. And you'll see a traceback similiar to this.

2016-12-21 14:04:40.903 ERROR nova.api.openstack.extensions [req-6d2eb531-b239-429d-8d25-f06b4fe6309c nova service] Unexpected exception in API method
2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions Traceback (most recent call last):
2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions File "/opt/stack/nova/nova/api/openstack/extensions.py", line 338, in wrapped
2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions return f(*args, **kwargs)
2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions File "/opt/stack/nova/nova/api/validation/__init__.py", line 73, in wrapper
2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions return func(*args, **kwargs)
2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions File "/opt/stack/nova/nova/api/validation/__init__.py", line 73, in wrapper
2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions return func(*args, **kwargs)
2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions File "/opt/stack/nova/nova/api/validation/__init__.py", line 73, in wrapper
2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions return func(*args, **kwargs)
2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions File "/opt/stack/nova/nova/api/validation/__init__.py", line 73, in wrapper
2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions return func(*args, **kwargs)
2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions File "/opt/stack/nova/nova/api/validation/__init__.py", line 73, in wrapper
2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions return func(*args, **kwargs)
2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions File "/opt/stack/nova/nova/api/openstack/compute/servers.py", line 631, in create
2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions **create_kwargs)
2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions File "/opt/stack/nova/nova/hooks.py", line 154, in inner
2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions rv = f(*args, **kwargs)
2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions File "/opt/stack/nova/nova/compute/api.py", line 1528, in create
2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions check_server_group_quota=check_server_group_quota)
2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions File "/opt/stack/nova/nova/compute/api.py", line 1159, in _create_instance
2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions key_pair)
2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions File "/opt/stack/nova/nova/compute/api.py", line 1022, in _provision_instances
2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions quotas.rollback()
2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions self.force_reraise()
2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions six.reraise(self.type_, self.value, self.tb)
2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions File "/opt/stack/nova/nova/compute/api.py", line 944, in _provision_instances
2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions shutdown_terminate, create_instance=False)
2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions File "/opt/stack/nova/nova/compute/api.py", line 1451, in create_db_entry_for_new_instance
2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions num_instances, shutdown_terminate)
2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions File "/opt/stack/nova/nova/compute/api.py", line 1403, in _populate_instance_for_create
2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions length=CONF.ephemeral_storage_encryption.key_size)
2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions TypeError: create_key() takes exactly 4 arguments (3 given)
2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions
2016-12-21 14:04:40.905 INFO nova.api.openstack.wsgi [req-6d2eb531-b239-429d-8d25-f06b4fe6309c nova service] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<type 'exceptions.TypeError'>

Expected result
===============
server should be successfully created

Actual result
=============
HTTP 500, with the following traceback in the log.

2016-12-21 14:04:40.903 ERROR nova.api.openstack.extensions [req-6d2eb531-b239-429d-8d25-f06b4fe6309c nova service] Unexpected exception in API method
2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions Traceback (most recent call last):
2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions File "/opt/stack/nova/nova/api/openstack/extensions.py", line 338, in wrapped
2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions return f(*args, **kwargs)
2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions File "/opt/stack/nova/nova/api/validation/__init__.py", line 73, in wrapper
2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions return func(*args, **kwargs)
2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions File "/opt/stack/nova/nova/api/validation/__init__.py", line 73, in wrapper
2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions return func(*args, **kwargs)
2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions File "/opt/stack/nova/nova/api/validation/__init__.py", line 73, in wrapper
2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions return func(*args, **kwargs)
2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions File "/opt/stack/nova/nova/api/validation/__init__.py", line 73, in wrapper
2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions return func(*args, **kwargs)
2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions File "/opt/stack/nova/nova/api/validation/__init__.py", line 73, in wrapper
2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions return func(*args, **kwargs)
2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions File "/opt/stack/nova/nova/api/openstack/compute/servers.py", line 631, in create
2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions **create_kwargs)
2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions File "/opt/stack/nova/nova/hooks.py", line 154, in inner
2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions rv = f(*args, **kwargs)
2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions File "/opt/stack/nova/nova/compute/api.py", line 1528, in create
2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions check_server_group_quota=check_server_group_quota)
2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions File "/opt/stack/nova/nova/compute/api.py", line 1159, in _create_instance
2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions key_pair)
2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions File "/opt/stack/nova/nova/compute/api.py", line 1022, in _provision_instances
2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions quotas.rollback()
2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions self.force_reraise()
2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions six.reraise(self.type_, self.value, self.tb)
2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions File "/opt/stack/nova/nova/compute/api.py", line 944, in _provision_instances
2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions shutdown_terminate, create_instance=False)
2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions File "/opt/stack/nova/nova/compute/api.py", line 1451, in create_db_entry_for_new_instance
2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions num_instances, shutdown_terminate)
2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions File "/opt/stack/nova/nova/compute/api.py", line 1403, in _populate_instance_for_create
2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions length=CONF.ephemeral_storage_encryption.key_size)
2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions TypeError: create_key() takes exactly 4 arguments (3 given)
2016-12-21 14:04:40.903 TRACE nova.api.openstack.extensions
2016-12-21 14:04:40.905 INFO nova.api.openstack.wsgi [req-6d2eb531-b239-429d-8d25-f06b4fe6309c nova service] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<type 'exceptions.TypeError'>

Environment
===========
1. Ubuntu 16.04

cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04.1 LTS"

2. Devstack + Barbican plugin, both on stable/newton

int32bit (int32bit)
Changed in nova:
status: New → Confirmed
assignee: nobody → int32bit (int32bit)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/414233

Changed in nova:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: master
Review: https://review.openstack.org/446072

Changed in nova:
assignee: int32bit (int32bit) → Kaitlin Farr (kaitlin-farr)
Matt Riedemann (mriedem)
Changed in nova:
importance: Undecided → Medium
tags: added: newton-backport-potential ocata-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/446072
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=1d3acad111c5106592b0921628480fcf41e0fb4c
Submitter: Jenkins
Branch: master

commit 1d3acad111c5106592b0921628480fcf41e0fb4c
Author: Kaitlin Farr <email address hidden>
Date: Fri Mar 10 18:09:49 2017 -0500

    Parse algorithm from cipher for ephemeral disk encryption

    Nova's keymgr implementation used to have default values
    for the algorithm and bit length. Castellan does not have
    default values, and when Castellan replaced keymgr in
    Ib563b0ea4b8b4bc1833bf52bf49a68546c384996, the parameters
    to the create_key method were not updated. This change
    parses the algorithm from the cipher value and passes it
    to Castellan's key manager interface.

    Closes-Bug: #1651887
    Change-Id: Ib90bc7571aef59325be0efe123fcf12e86252b85

Changed in nova:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/newton)

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/466196

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/ocata)

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/466307

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on nova (stable/newton)

Change abandoned by guang-yee (<email address hidden>) on branch: stable/newton
Review: https://review.openstack.org/466196
Reason: let wait for the ocata one first

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/ocata)

Reviewed: https://review.openstack.org/466307
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=fcc931f4017ab4bb830ff645431edaa8a18bf880
Submitter: Jenkins
Branch: stable/ocata

commit fcc931f4017ab4bb830ff645431edaa8a18bf880
Author: Kaitlin Farr <email address hidden>
Date: Fri Mar 10 18:09:49 2017 -0500

    Parse algorithm from cipher for ephemeral disk encryption

    Nova's keymgr implementation used to have default values
    for the algorithm and bit length. Castellan does not have
    default values, and when Castellan replaced keymgr in
    Ib563b0ea4b8b4bc1833bf52bf49a68546c384996, the parameters
    to the create_key method were not updated. This change
    parses the algorithm from the cipher value and passes it
    to Castellan's key manager interface.

    Conflicts:
          nova/tests/unit/compute/test_compute.py

    NOTE(mriedem): The conflicts are due to import order changes
    in pike for flake8 order checking.

    Closes-Bug: #1651887
    Change-Id: Ib90bc7571aef59325be0efe123fcf12e86252b85
    (cherry picked from commit 1d3acad111c5106592b0921628480fcf41e0fb4c)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on nova (stable/newton)

Change abandoned by guang-yee (<email address hidden>) on branch: stable/newton
Review: https://review.openstack.org/466196
Reason: need to re-do

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 15.0.5

This issue was fixed in the openstack/nova 15.0.5 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 16.0.0.0b2

This issue was fixed in the openstack/nova 16.0.0.0b2 development milestone.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on nova (master)

Change abandoned by Sean Dague (<email address hidden>) on branch: master
Review: https://review.openstack.org/414233
Reason: This review is > 4 weeks without comment, and is not mergable in it's current state. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/newton)

Reviewed: https://review.openstack.org/466196
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=c589b76e13f54f8c0d753b1d2c4d4c2ef3399359
Submitter: Jenkins
Branch: stable/newton

commit c589b76e13f54f8c0d753b1d2c4d4c2ef3399359
Author: Kaitlin Farr <email address hidden>
Date: Fri Mar 10 18:09:49 2017 -0500

    Parse algorithm from cipher for ephemeral disk encryption

    Nova's keymgr implementation used to have default values
    for the algorithm and bit length. Castellan does not have
    default values, and when Castellan replaced keymgr in
    Ib563b0ea4b8b4bc1833bf52bf49a68546c384996, the parameters
    to the create_key method were not updated. This change
    parses the algorithm from the cipher value and passes it
    to Castellan's key manager interface.

    Conflicts:
          nova/conf/ephemeral_storage.py

    NOTE(mriedem): The conflicts are due to not having
    Ic8ea9b0852d5b6f5d8a468fe0a03d21e220a8913 in newton, which
    was a refactor/cleanup change in ocata. It's just preserved
    here for readability.

    Closes-Bug: #1651887
    Change-Id: Ib90bc7571aef59325be0efe123fcf12e86252b85
    (cherry picked from commit 1d3acad111c5106592b0921628480fcf41e0fb4c)
    (cherry picked from commit fcc931f4017ab4bb830ff645431edaa8a18bf880)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 14.0.8

This issue was fixed in the openstack/nova 14.0.8 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.