stack-update fails when an instance with a Neutron port is added to the stack

[Sergey Kolekonov]

Heat can't update stack when one more Nova server with a Neutron port added.

How to reproduce:
 - Create a stack using the following template:
-------------------------------------------------------------
{
    "AWSTemplateFormatVersion" : "2010-09-09",
    "Description" : "Sample Stack",

    "Resources" : {

        "instance-port" : {
            "Type" : "OS::Neutron::Port",
            "Properties" : {
               "network_id" : "931daa2e-2e22-44ea-8e5a-61d7cc644756"
            }
        },
        "instance" : {
            "Type" : "OS::Nova::Server",
            "Properties" : {
                "flavor" : "42",
                "image" : "988bb0ad-c526-4556-852c-e4c6eb52ffb6",
                "networks" : [{ "port" : { "Ref" : "instance-port" }}]
            }
        }
    },

    "Outputs" : {}
}
-------------------------------------------------------------
- Wait until the stack is created successfully.
- Update stack using the following template:
-------------------------------------------------------------
{
    "AWSTemplateFormatVersion" : "2010-09-09",
    "Description" : "Sample Stack",

    "Resources" : {

        "instance-port" : {
            "Type" : "OS::Neutron::Port",
            "Properties" : {
               "network_id" : "931daa2e-2e22-44ea-8e5a-61d7cc644756"
            }
        },
        "instance" : {
            "Type" : "OS::Nova::Server",
            "Properties" : {
                "flavor" : "42",
                "image" : "988bb0ad-c526-4556-852c-e4c6eb52ffb6",
                "networks" : [{ "port" : { "Ref" : "instance-port" }}]
            }
        },
        "instance2-port" : {
            "Type" : "OS::Neutron::Port",
            "Properties" : {
               "network_id" : "931daa2e-2e22-44ea-8e5a-61d7cc644756"
            }
        },
        "instance2" : {
            "Type" : "OS::Nova::Server",
            "Properties" : {
                "flavor" : "42",
                "image" : "988bb0ad-c526-4556-852c-e4c6eb52ffb6",
                "networks" : [{ "port" : { "Ref" : "instance2-port" }}]
            }
        }
    },

    "Outputs" : {}
}
-------------------------------------------------------------

Expected result:
Stack is updated successfully

Actual result:
Stack update fails with the following error from Heat Engine:
-------------------------------------------------------------
2014-02-26 09:03:41.939 INFO heat.engine.resource [-] creating Server "instance2" Stack "mystack" [7bee244c-31a7-47c5-8475-a7a79132197e]
2014-02-26 09:03:42.362 ERROR heat.engine.resource [-] CREATE : Server "instance2" Stack "mystack" [7bee244c-31a7-47c5-8475-a7a79132197e]
2014-02-26 09:03:42.362 TRACE heat.engine.resource Traceback (most recent call last):
2014-02-26 09:03:42.362 TRACE heat.engine.resource File "/opt/stack/heat/heat/engine/resource.py", line 413, in _do_action
2014-02-26 09:03:42.362 TRACE heat.engine.resource pre_func()
2014-02-26 09:03:42.362 TRACE heat.engine.resource File "/opt/stack/heat/heat/engine/properties.py", line 313, in validate
2014-02-26 09:03:42.362 TRACE heat.engine.resource raise exception.StackValidationFailed(message=msg)
2014-02-26 09:03:42.362 TRACE heat.engine.resource StackValidationFailed: Property error : instance2: networks The Parameter (instance2-port) was not provided.
2014-02-26 09:03:42.362 TRACE heat.engine.resource
-------------------------------------------------------------

[Alexander Ignatov (aignatov)]

This blocks Savanna feature to work with Heat in master from scaling hadoop clusters.

[Steve Baker (steve-stevebaker)]

Yes, there is an issue with server/port interaction on stack update.

I think what is happening is the following.
On update, if server is replaced then the new server is assigned a new port by neutron, but heat doesn't know what the new port UUID is, and assumes the old port UUID is still valid (even though it was deleted as a side-effect of deleting the nova server).

With luck, all that is required is for the server resource to detach any ports that are specified by networks.port before deleting the server.

Its possible that the fix for this will also fix bug 1196479

[Steve Baker (steve-stevebaker)]

As an aside, can savanna workaround this by not specifying a port in the template? Why do you need a port resource?

[Alexander Ignatov (aignatov)]

Steve, is it possible to use some Heat template to launch instance and assign floating IP without port assignment in Neutron?
As for example, Savanna generates the following template to launch cluster with several instances:

https://github.com/openstack/savanna/blob/master/savanna/tests/unit/resources/test_serialize_resources_use_neutron.heat

[Steve Baker (steve-stevebaker)]

You're right, that is not currently possible to have a floatingip without a port resource.. I've raised this to track that https://bugs.launchpad.net/heat/+bug/1285899. If you end up working on that then feel free to assign it to yourself and mark it in progress.

We also need to fix this port update bug too.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/77923

[Sergey Kraynev (skraynev)]

I have researched this problem and have find, that the real problem is related with incorrect resolving method for reference.
According presented in description templates we have reference on one of new resources. So for resolve should be used Resource Ref class, but if you look at error message, it tell "The Parameter (instance2-port) was not provided.". Also both template have empty parameter section.
The problem related with changes introduced in https://blueprints.launchpad.net/heat/+spec/function-plugins.
Currently Ref class makes decision about, what class for resolving will be used, based on Resources stored in template
https://github.com/openstack/heat/blob/master/heat/engine/cfn/functions.py#L118.
However when we update resource https://github.com/openstack/heat/blob/master/heat/engine/update.py#L123
we call __setitem__ method of stack and then change resource reference on stack https://github.com/openstack/heat/blob/master/heat/engine/parser.py#L261 , but old template haven't new resources.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/77923
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=7aeeab13450f8b16874ff310f4026482b9592dfa
Submitter: Jenkins
Branch: master

commit 7aeeab13450f8b16874ff310f4026482b9592dfa
Author: Sergey Kraynev <email address hidden>
Date: Tue Mar 4 10:58:36 2014 -0500

    Fix resolving for Ref function

    Fix problem with incorrect choicing of Ref class during update.
    When we call reparse function in __setitem__ method we use,
    old version of stack template. So Ref function cann't find
    new 'Resouces' in template.

    Closes-bug: #1285067
    Change-Id: I82eeeef0c3c626f0ec3df5a827a7ec049929538f

[Pavlo Shchelokovskyy (pshchelo)]

I am seeing exactly the same behavior on heat master again.

[Pavlo Shchelokovskyy (pshchelo)]

although now the stack is updated successfully, the first instance looses the prort/fixed IP

[Pavlo Shchelokovskyy (pshchelo)]

Will open a new bug as behavior is different now