incorrect code while clearing pvc_id while spawn failing

[Terry Yao]

there will be 3 attempts for nova spawn if we hit spawn failing in powervc side:

1. the first time failure, I saw powervc driver post a REST call to PowerVC side, and PowerVC fail to spawn, then PowerVC Driver will delete the instance in PowerVC with it's metadata's pvc_id..but I saw an error code in powervc nova driver._clean_vm_and_save_fault_message:
       if constants.PVC_ID in meta.keys():
            del(meta[constants.PVC_ID])
        update_properties = {'metadata': meta}
        db.instance_update(context, instance['uuid'], update_properties)

        # destory vm in pvc side
        instance['metadata'] = {constants.PVC_ID: pvc_id}

This code is wrong, it will not remove the pvc_id from instance metadata eventually, the later set the instance['metadata'] will be saved to db as instance.save() will be called by nova framework after spawn failure.

2. then the second attempt to boot, I saw powervc driver also post the second post call to powervc side, and powervc fail to spawn, but at the powervc side, you will see, the created instance is associated with an pvc_id in it's metadata...I think the PowerVC instance should not contain such pvc_id as we never intend to save such metadata into powervc side. Here is the powervc side, nova show output for metadata:
[root@dev4powervc ~]# nova show 83bd7cce-e9cd-40bb-b439-e9872cbcc8f2 | grep metadata
| metadata | {"decision_id": "1434", "pvc_id": "63930915-6daf-4d1d-8692-393640e76443"}

And powervc driver code will save such pvc_id back to powervc side by another error code in service._update_local_instance_by_pvc_created_instance()
        created_instance = created_server.__dict__
        # get original metadata from DB and insert the pvc_id
        meta = db.instance_metadata_get(context, orig_instance['uuid'])
        meta.update(pvc_id=created_instance['id'])
        # update powervc specified metadata to hosting os vm instance
        powervc_meta = created_instance.get('metadata')
        if powervc_meta:
            meta.update(powervc_meta)

We need to call meta.update(pvc_id=created_instance['id']) in the last line, then even if the powervc instance has a 'pvc_id' metadata, then we can also override it to the right instance pvc_id..
as such code, while powervc eventually fail to spawn, the powervc driver will not delete the instance in powervc side.

3. then the 3rd attempt to boot, as driver does not remove the instance in powervc side, I saw the last attempt rest call is failing as there is not network id passed into rest call...

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/136782
Committed: https://git.openstack.org/cgit/stackforge/powervc-driver/commit/?id=e506e9a9b46950efab9a6a82e228d5d71c804d11
Submitter: Jenkins
Branch: master

commit e506e9a9b46950efab9a6a82e228d5d71c804d11
Author: terryyao <email address hidden>
Date: Mon Nov 24 23:02:33 2014 +0800

    Clear pvc_id during instance spawn failing

    1. In powevc nova driver, it's better to use instance.save() instead of
    db call

    2. clear pvc_id after destory the powervc instance.

    3. override pvc_id regardless powervc instance's metadata pvc_id.

    Closes-Bug: #1395731
    Change-Id: I2bb2494a8a2976064494101af811f6be22379472

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/149934

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/149934
Committed: https://git.openstack.org/cgit/stackforge/powervc-driver/commit/?id=4fcd0c76dfdf1dbeb9f90d368695f61a474cdc00
Submitter: Jenkins
Branch: stable/juno

commit 4fcd0c76dfdf1dbeb9f90d368695f61a474cdc00
Author: terryyao <email address hidden>
Date: Mon Nov 24 23:02:33 2014 +0800

    Clear pvc_id during instance spawn failing

    1. In powevc nova driver, it's better to use instance.save() instead of
    db call

    2. clear pvc_id after destory the powervc instance.

    3. override pvc_id regardless powervc instance's metadata pvc_id.

    Closes-Bug: #1395731
    Change-Id: I2bb2494a8a2976064494101af811f6be22379472
    (cherry picked from commit e506e9a9b46950efab9a6a82e228d5d71c804d11)