Detach volume fails with "Unexpected KeyError" in EC2 interface.

[Rui Chen]

Detach volume fails with "Unexpected KeyError" in EC2 interface when I detach a "attaching" status volume.

The volume with "attaching" status don't contain a property"instance_uuid", a KeyError will be raised at the following function.

    def _get_instance_from_volume(self, context, volume):
        if volume['instance_uuid']:
            ......

Attaching volume dict:

{
 'status': u'attaching',
 'volume_type_id': u'None',
 'display_name': None,
 'attach_time': '',
 'availability_zone': u'nova',
 'created_at': u'2014-02-13T16: 50: 53.620080',
 'attach_status': 'detached',
 'display_description': None,
 'volume_metadata': {

 },
 'snapshot_id': None,
 'mountpoint': '',
 'id': u'99d118ee-3666-4983-8825-f8c096bccbd1',
 'size': 1
}

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/73788

[Mark McLoughlin (markmc)]

Hmm, this looks the same as:

http://logs.openstack.org/38/84238/1/check/check-grenade-dsvm/88847c1/logs/new/screen-n-api.txt.gz?level=TRACE

i.e. "Unexpected KeyError raised" when doing a detach volume via the EC2 API

I haven't looked in detail yet, but if you search for ""Unexpected KeyError raised"" in kibana, it looks like this is occurring a lot in the past 24 hours

[Matt Riedemann (mriedem)]

Yeah, confirmed, also hitting it here:

http://logs.openstack.org/37/84037/5/check/check-grenade-dsvm-partial-ncpu/db32161/console.html

message:"InstanceRunTest" AND message:"Unexpected KeyError raised" AND tags:"screen-n-api.txt"

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiSW5zdGFuY2VSdW5UZXN0XCIgQU5EIG1lc3NhZ2U6XCJVbmV4cGVjdGVkIEtleUVycm9yIHJhaXNlZFwiIEFORCB0YWdzOlwic2NyZWVuLW4tYXBpLnR4dFwiIiwiZmllbGRzIjpbXSwib2Zmc2V0IjowLCJ0aW1lZnJhbWUiOiI2MDQ4MDAiLCJncmFwaG1vZGUiOiJjb3VudCIsInRpbWUiOnsidXNlcl9pbnRlcnZhbCI6MH0sInN0YW1wIjoxMzk2MzU4NTU0NzY0fQ==

It's not a failure every time that shows up, but the failure rate is 97.6%. It's mostly the check queue right now but it's hitting on multiple patches.

[Matt Riedemann (mriedem)]

elastic-recheck query is here: https://review.openstack.org/#/c/84449/

[Matt Riedemann (mriedem)]

Looks like the test was recently enabled again: https://review.openstack.org/81762

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/73788
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=fa72e49845179e29cf918e0742ef735ae868bb4c
Submitter: Jenkins
Branch: master

commit fa72e49845179e29cf918e0742ef735ae868bb4c
Author: ruichen <email address hidden>
Date: Sat Feb 15 15:32:47 2014 +0800

    Fix bug detach volume fails with "KeyError" in EC2

    Modified volume['instance_uuid'] to volume.get('instance_uuid'),
    to avoid "KeyError", when detach a volume with "attaching" status.

    Change-Id: Ic1aacab212c2a2026d18ef57ad9d77fdc6d86b47
    Closes-Bug: #1280532

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: milestone-proposed
Review: https://review.openstack.org/86492

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/havana
Review: https://review.openstack.org/86493

[Openstack Gerrit (openstack-gerrit)]

Fix proposed to branch: stable/icehouse
Review: https://review.openstack.org/88686

[Openstack Gerrit (openstack-gerrit)]

Reviewed: https://review.openstack.org/88686
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=2728f1e9afa7d5123ab43a8f2af6fe14be3db3c7
Submitter: Jenkins
Branch: stable/icehouse

commit 2728f1e9afa7d5123ab43a8f2af6fe14be3db3c7
Author: ruichen <email address hidden>
Date: Sat Feb 15 15:32:47 2014 +0800

    Fix bug detach volume fails with "KeyError" in EC2

    Modified volume['instance_uuid'] to volume.get('instance_uuid'),
    to avoid "KeyError", when detach a volume with "attaching" status.

    Change-Id: Ic1aacab212c2a2026d18ef57ad9d77fdc6d86b47
    Closes-Bug: #1280532
    (cherry picked from commit fa72e49845179e29cf918e0742ef735ae868bb4c)

[Joe Gordon (jogo)]

Still seeing some of these:

message:"Unexpected KeyError raised" AND module:"nova.api.ec2"

http://logs.openstack.org/86/43986/44/check/check-tempest-dsvm-postgres-full/d3c52f4/logs/screen-n-api.txt

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/86493
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=9f05659eff8a0dae1afabb3ca288a021305a8c36
Submitter: Jenkins
Branch: stable/havana

commit 9f05659eff8a0dae1afabb3ca288a021305a8c36
Author: ruichen <email address hidden>
Date: Sat Feb 15 15:32:47 2014 +0800

    Fix bug detach volume fails with "KeyError" in EC2

    Modified volume['instance_uuid'] to volume.get('instance_uuid'),
    to avoid "KeyError", when detach a volume with "attaching" status.

    Change-Id: Ic1aacab212c2a2026d18ef57ad9d77fdc6d86b47
    Closes-Bug: #1280532
    (cherry picked from commit fa72e49845179e29cf918e0742ef735ae868bb4c)