OpenStack Compute (nova)

Deleted keypair causes metadata failure

Bug #1592167 reported by James Dempsey on 2016-06-13

This bug affects 5 people

Affects		Status	Importance	Assigned to	Milestone
	OpenStack Compute (nova)	Fix Released	Medium	Matt Riedemann
	Mitaka	Fix Released	Medium	Matt Riedemann

Bug Description

Description
===========

If a user deletes a keypair that was used to create an instance, that instance receives HTTP 400 errors when attempting to get metadata via http://169.254.169.254/openstack/latest/meta_data.json.

This causes problems in the instance when cloud-init fails to retrieve the OpenStack datasource.

Steps to reproduce
==================

1. Create instance with SSH keypair defined.
2. Delete SSH keypair
3. Attempt 'curl http://169.254.169.254/openstack/latest/meta_data.json' from the instance

Expected result
===============

Instance receives metadata from http://169.254.169.254/openstack/latest/meta_data.json

Actual result
=============

Instance receives HTTP 400 error. Additionally, Ubuntu Cloud Image instances will fail back to the ec2 datasource and re-generate Host SSH keys.

Environment
===========

Nova: 2015.1.4.2
Hypervisor: Libvirt + KVM
Storage: Ceph
Network: Liberty Neutron ML2+OVS

Logs
====

[req-a8385839-6993-4289-96dc-1714afe82597 - - - - -] FaultWrapper error
Traceback (most recent call last):
  File "/opt/cat/openstack/nova/local/lib/python2.7/site-packages/nova/api/ec2/__init__.py", line 93, in __call__
    return req.get_response(self.application)
  File "/opt/cat/openstack/nova/local/lib/python2.7/site-packages/webob/request.py", line 1299, in send
    application, catch_exc_info=False)
  File "/opt/cat/openstack/nova/local/lib/python2.7/site-packages/webob/request.py", line 1263, in call_application
    app_iter = application(self.environ, start_response)
  File "/opt/cat/openstack/nova/local/lib/python2.7/site-packages/webob/dec.py", line 130, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/opt/cat/openstack/nova/local/lib/python2.7/site-packages/webob/dec.py", line 195, in call_func
    return self.func(req, *args, **kwargs)
  File "/opt/cat/openstack/nova/local/lib/python2.7/site-packages/nova/api/ec2/__init__.py", line 105, in __call__
    rv = req.get_response(self.application)
  File "/opt/cat/openstack/nova/local/lib/python2.7/site-packages/webob/request.py", line 1299, in send
    application, catch_exc_info=False)
  File "/opt/cat/openstack/nova/local/lib/python2.7/site-packages/webob/request.py", line 1263, in call_application
    app_iter = application(self.environ, start_response)
  File "/opt/cat/openstack/nova/local/lib/python2.7/site-packages/webob/dec.py", line 130, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/opt/cat/openstack/nova/local/lib/python2.7/site-packages/webob/dec.py", line 195, in call_func
    return self.func(req, *args, **kwargs)
  File "/opt/cat/openstack/nova/local/lib/python2.7/site-packages/nova/api/metadata/handler.py", line 137, in __call__
    data = meta_data.lookup(req.path_info)
  File "/opt/cat/openstack/nova/local/lib/python2.7/site-packages/nova/api/metadata/base.py", line 418, in lookup
    data = self.get_openstack_item(path_tokens[1:])
  File "/opt/cat/openstack/nova/local/lib/python2.7/site-packages/nova/api/metadata/base.py", line 297, in get_openstack_item
    return self._route_configuration().handle_path(path_tokens)
  File "/opt/cat/openstack/nova/local/lib/python2.7/site-packages/nova/api/metadata/base.py", line 491, in handle_path
    return path_handler(version, path)
  File "/opt/cat/openstack/nova/local/lib/python2.7/site-packages/nova/api/metadata/base.py", line 316, in _metadata_as_json
    self.instance.key_name)
  File "/opt/cat/openstack/nova/local/lib/python2.7/site-packages/nova/objects/base.py", line 163, in wrapper
    result = fn(cls, context, *args, **kwargs)
  File "/opt/cat/openstack/nova/local/lib/python2.7/site-packages/nova/objects/keypair.py", line 60, in get_by_name
    db_keypair = db.key_pair_get(context, user_id, name)
  File "/opt/cat/openstack/nova/local/lib/python2.7/site-packages/nova/db/api.py", line 937, in key_pair_get
    return IMPL.key_pair_get(context, user_id, name)
  File "/opt/cat/openstack/nova/local/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 233, in wrapper
    return f(*args, **kwargs)
  File "/opt/cat/openstack/nova/local/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 2719, in key_pair_get
    raise exception.KeypairNotFound(user_id=user_id, name=name)
KeypairNotFound: Keypair keypair_name not found for user ffffffffffffffffffffffffffffffff

Tags:

Revision history for this message

Matt Riedemann (mriedem) wrote on 2016-06-14:

You're using an end of life branch for Nova (kilo). Can you try to recreate this against at least mitaka, but preferably master and see if it's still an issue?

tags:	added: api metadata
Changed in nova:
status:	New → Incomplete

Revision history for this message

Matt Riedemann (mriedem) wrote on 2016-06-14:

From looking at the stable/kilo code I can see how this would fail.

Looking at master, the keypairs are stored with the instance (extra) and retrieved that way:

https://github.com/openstack/nova/blob/14.0.0.0b1/nova/api/metadata/base.py#L309

https://github.com/openstack/nova/blob/14.0.0.0b1/nova/objects/instance.py#L848

So the keypairs, like flavors, are stored in the instance_extra table now, so they don't get deleted unless you delete the instance.

If the keypair by name isn't found, the instance.keypairs list is just empty:

https://github.com/openstack/nova/blob/14.0.0.0b1/nova/objects/instance.py#L869

Which I guess would blow up here with an IndexError:

https://github.com/openstack/nova/blob/14.0.0.0b1/nova/api/metadata/base.py#L309

So yeah, this is still probably a bug in master, at least if the keypair wasn't stored in instance_extra so we have to perform this lookup.

Changed in nova:
status:	Incomplete → Confirmed

Revision history for this message

Matt Riedemann (mriedem) wrote on 2016-06-14:

stable/mitaka would get a KeypairNotFound in the metadata API, like in Kilo:

https://github.com/openstack/nova/blob/stable/mitaka/nova/api/metadata/base.py#L327

But Kilo is EOL, and this isn't a security bug, so we wouldn't backport to stable/liberty which is in critical/security fix only mode.

Changed in nova:
assignee:	nobody → Matt Riedemann (mriedem)
importance:	Undecided → Medium

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-06-14: Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/329661

Changed in nova:
status:	Confirmed → In Progress

Revision history for this message

James Dempsey (jamespd) wrote on 2016-06-14:

Thanks Matt; much appreciated!

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-06-14: Fix merged to nova (master)

Reviewed: https://review.openstack.org/329661
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=4317166b72bb0aadd0321acdf9f2450c1a99d0a4
Submitter: Jenkins
Branch: master

commit 4317166b72bb0aadd0321acdf9f2450c1a99d0a4
Author: Matt Riedemann <email address hidden>
Date: Tue Jun 14 16:05:35 2016 -0400

Handle keypair not found from metadata server

    With commit e83842b80b73c451f78a4bb9e7bd5dfcebdefcab we
    attempt to load keypairs for an instance from instance_extra,
    but if that hasn't been migrated yet we fall back to loading
    the keypair from the database by name.

    If the keypair was deleted, the instance object will just set
    an empty KeyPairList for instance.keypairs and we'll get an
    IndexError when using self.instance.keypairs[0] in
    _metadata_as_json.

    This adds a check that instance.keypairs actually has
    something in it. If not, we log a message and don't return
    any key values in the metadata dict - same as if instance.key_name
    wasn't set to begin with.

Change-Id: If823867d1df4bafa46978e62e05826d1f12c9269
Closes-Bug: #1592167

Changed in nova:
status:	In Progress → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-06-15: Fix proposed to nova (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/330174

Revision history for this message

Doug Hellmann (doug-hellmann) wrote on 2016-07-14: Fix included in openstack/nova 14.0.0.0b2

This issue was fixed in the openstack/nova 14.0.0.0b2 development milestone.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-08-03: Fix merged to nova (stable/mitaka)

Reviewed: https://review.openstack.org/330174
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=9228bb5b096edc132f42858c0b40562956d5793b
Submitter: Jenkins
Branch: stable/mitaka

commit 9228bb5b096edc132f42858c0b40562956d5793b
Author: Matt Riedemann <email address hidden>
Date: Tue Jun 14 16:05:35 2016 -0400

Handle keypair not found from metadata server

Conflicts:
nova/api/metadata/base.py

    NOTE(mriedem): This is basically a different patch given in mitaka
    the keypairs weren't migrated to the instance_extra table, so this
    is more or less a mitaka-specific fix for the same KeypairNotFound
    failure.

    Change-Id: If823867d1df4bafa46978e62e05826d1f12c9269
    Closes-Bug: #1592167
    (cherry picked from commit 4317166b72bb0aadd0321acdf9f2450c1a99d0a4)

Revision history for this message

Doug Hellmann (doug-hellmann) wrote on 2016-08-09: Fix included in openstack/nova 13.1.1

#10

This issue was fixed in the openstack/nova 13.1.1 release.

Revision history for this message

Luis Pigueiras (lpigueiras) wrote on 2017-06-21:

#11

This is bug is still there if you are using cells, I proposed a PR in https://review.openstack.org/#/c/476122/

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-09-05: Fix proposed to nova (stable/pike)

#12

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/500953

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-09-05: Fix proposed to nova (stable/ocata)

#13

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/500954

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-09-12: Fix merged to nova (master)

#14

Reviewed: https://review.openstack.org/476122
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=682276be6d3f965739774643b6b8b51be9d5fa54
Submitter: Jenkins
Branch: master

commit 682276be6d3f965739774643b6b8b51be9d5fa54
Author: Luis Pigueiras <email address hidden>
Date: Wed Jun 21 14:44:27 2017 +0200

Handle keypair not found from metadata server using cells

    Fixes https://bugs.launchpad.net/nova/+bug/1592167 for the
    cells case. The fix done in https://bugs.launchpad.net/nova/+bug/1592167
    only solves the problem when cells are not used at all

Closes-bug: #1592167

Change-Id: Id663b426261150a1cce310cb4a61d9572f78c016

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-09-15: Fix merged to nova (stable/pike)

#15

Reviewed: https://review.openstack.org/500953
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=b9a1ccc5fd65b59f384f075b0fb2aa35558a3e43
Submitter: Jenkins
Branch: stable/pike

commit b9a1ccc5fd65b59f384f075b0fb2aa35558a3e43
Author: Luis Pigueiras <email address hidden>
Date: Wed Jun 21 14:44:27 2017 +0200

Handle keypair not found from metadata server using cells

    Fixes https://bugs.launchpad.net/nova/+bug/1592167 for the
    cells case. The fix done in https://bugs.launchpad.net/nova/+bug/1592167
    only solves the problem when cells are not used at all

Closes-bug: #1592167

Change-Id: Id663b426261150a1cce310cb4a61d9572f78c016
(cherry picked from commit 682276be6d3f965739774643b6b8b51be9d5fa54)

tags:

added: in-stable-pike

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-09-25: Fix included in openstack/nova 16.0.1

#16

This issue was fixed in the openstack/nova 16.0.1 release.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-10-23: Fix merged to nova (stable/ocata)

#17

Reviewed: https://review.openstack.org/500954
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=2b5c5439c74881cffddae0679bdfec1163d45b95
Submitter: Zuul
Branch: stable/ocata

commit 2b5c5439c74881cffddae0679bdfec1163d45b95
Author: Luis Pigueiras <email address hidden>
Date: Wed Jun 21 14:44:27 2017 +0200

Handle keypair not found from metadata server using cells

    Fixes https://bugs.launchpad.net/nova/+bug/1592167 for the
    cells case. The fix done in https://bugs.launchpad.net/nova/+bug/1592167
    only solves the problem when cells are not used at all

Closes-bug: #1592167

    Change-Id: Id663b426261150a1cce310cb4a61d9572f78c016
    (cherry picked from commit 682276be6d3f965739774643b6b8b51be9d5fa54)
    (cherry picked from commit b9a1ccc5fd65b59f384f075b0fb2aa35558a3e43)

tags:

added: in-stable-ocata

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-10-24: Fix included in openstack/nova 17.0.0.0b1

#18

This issue was fixed in the openstack/nova 17.0.0.0b1 development milestone.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-11-15: Fix included in openstack/nova 15.0.8

#19

This issue was fixed in the openstack/nova 15.0.8 release.

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.