Instance loses port attachment on template update

[Pavlo Shchelokovskyy]

1) Create a stack using template bug.yaml (put your own flavor, image and network_id)

heat_template_version: 2013-05-23

description: Sample Stack

resources:

  instance-port:
    type: OS::Neutron::Port
    properties:
      network_id: dca58aa6-caeb-469f-8200-c5f85b07e106

  instance:
    type: OS::Nova::Server
    properties:
      flavor: m1.nano
      image: TestVM
      networks:
        - port : { get_resource : instance-port }

2) Wait for stack to go to CREATE_COMPLETE
3) Update the stack with bug2.yaml
heat_template_version: 2013-05-23

description: Sample Stack

resources:

  instance-port:
    type: OS::Neutron::Port
    properties:
      network_id: dca58aa6-caeb-469f-8200-c5f85b07e106

  instance:
    type: OS::Nova::Server
    properties:
      flavor: m1.nano
      image: TestVM
      networks:
        - port : { get_resource : instance-port }

  instance2-port:
    type: OS::Neutron::Port
    properties:
      network_id: dca58aa6-caeb-469f-8200-c5f85b07e106

  instance2:
    type: OS::Nova::Server
    properties:
      flavor: m1.nano
      image: TestVM
      networks:
        - port : { get_resource : instance2-port }

4) Wait for stack to go to UPDATE_COMPLETE

Expected:
Stack has 2 instances with fixed IPs

Observed:
First instance lost the port attachment (and thus the fixed IP), although the port is still there.

Might be related:
I see the following in heat-engine logs:
2014-11-17 12:15:39.585 DEBUG heat.engine.scheduler [-] Task Stack "bug" [446854a3-84c8-458e-894e-ceed62784095] Update complete from (pid=8275) step /opt/stack/heat/heat
/engine/scheduler.py:224
2014-11-17 12:15:39.585 DEBUG heat.engine.stack [-] Deleting backup stack from (pid=8275) update_task /opt/stack/heat/heat/engine/stack.py:810
2014-11-17 12:15:39.602 INFO heat.engine.stack [-] Stack DELETE IN_PROGRESS (bug): Stack DELETE started
2014-11-17 12:15:39.607 DEBUG heat.engine.scheduler [-] Task DependencyTaskGroup((destroy) {Port "instance-port" [c1e4bed7-7de9-4082-8199-36efef41f9c8] Stack "bug" [3a83
b3ad-d470-4fff-aec7-fa3629b9a983]: {}, Server "instance" Stack "bug" [3a83b3ad-d470-4fff-aec7-fa3629b9a983]: {Port "instance-port" [c1e4bed7-7de9-4082-8199-36efef41f9c8]
 Stack "bug" [3a83b3ad-d470-4fff-aec7-fa3629b9a983]}}) starting from (pid=8275) start /opt/stack/heat/heat/engine/scheduler.py:190
2014-11-17 12:15:39.607 DEBUG heat.engine.scheduler [-] Task DependencyTaskGroup((destroy) {Port "instance-port" [c1e4bed7-7de9-4082-8199-36efef41f9c8] Stack "bug" [3a83
b3ad-d470-4fff-aec7-fa3629b9a983]: {}, Server "instance" Stack "bug" [3a83b3ad-d470-4fff-aec7-fa3629b9a983]: {Port "instance-port" [c1e4bed7-7de9-4082-8199-36efef41f9c8] Stack "bug" [3a83b3ad-d470-4fff-aec7-fa3629b9a983]}}) running from (pid=8275) step /opt/stack/heat/heat/engine/scheduler.py:218
2014-11-17 12:15:39.607 DEBUG heat.engine.scheduler [-] Task destroy from None starting from (pid=8275) start /opt/stack/heat/heat/engine/scheduler.py:190
2014-11-17 12:15:39.607 DEBUG heat.engine.scheduler [-] Task destroy from None running from (pid=8275) step /opt/stack/heat/heat/engine/scheduler.py:218
2014-11-17 12:15:39.607 DEBUG heat.engine.scheduler [-] Task destroy from None complete from (pid=8275) step /opt/stack/heat/heat/engine/scheduler.py:224
2014-11-17 12:15:39.607 DEBUG heat.engine.scheduler [-] Task DependencyTaskGroup((destroy) {Port "instance-port" [c1e4bed7-7de9-4082-8199-36efef41f9c8] Stack "bug" [3a83b3ad-d470-4fff-aec7-fa3629b9a983]: {}, Server "instance" Stack "bug" [3a83b3ad-d470-4fff-aec7-fa3629b9a983]: {Port "instance-port" [c1e4bed7-7de9-4082-8199-36efef41f9c8] Stack "bug" [3a83b3ad-d470-4fff-aec7-fa3629b9a983]}}) sleeping from (pid=8275) _sleep /opt/stack/heat/heat/engine/scheduler.py:163
/usr/local/lib/python2.7/dist-packages/amqp/channel.py:616: VDeprecationWarning: The auto_delete flag for exchanges has been deprecated and will be removed
from py-amqp v1.5.0.
  warn(VDeprecationWarning(EXCHANGE_AUTODELETE_DEPRECATED))
/usr/local/lib/python2.7/dist-packages/amqp/channel.py:616: VDeprecationWarning: The auto_delete flag for exchanges has been deprecated and will be removed
from py-amqp v1.5.0.
  warn(VDeprecationWarning(EXCHANGE_AUTODELETE_DEPRECATED))
2014-11-17 12:15:40.608 DEBUG heat.engine.scheduler [-] Task DependencyTaskGroup((destroy) {Port "instance-port" [c1e4bed7-7de9-4082-8199-36efef41f9c8] Stack "bug" [3a83b3ad-d470-4fff-aec7-fa3629b9a983]: {}, Server "instance" Stack "bug" [3a83b3ad-d470-4fff-aec7-fa3629b9a983]: {Port "instance-port" [c1e4bed7-7de9-4082-8199-36efef41f9c8] Stack "bug" [3a83b3ad-d470-4fff-aec7-fa3629b9a983]}}) running from (pid=8275) step /opt/stack/heat/heat/engine/scheduler.py:218
2014-11-17 12:15:40.608 DEBUG heat.engine.scheduler [-] Task destroy from None starting from (pid=8275) start /opt/stack/heat/heat/engine/scheduler.py:190
2014-11-17 12:15:40.608 DEBUG heat.engine.scheduler [-] Task destroy from None running from (pid=8275) step /opt/stack/heat/heat/engine/scheduler.py:218
2014-11-17 12:15:40.608 DEBUG heat.engine.scheduler [-] Task destroy from None complete from (pid=8275) step /opt/stack/heat/heat/engine/scheduler.py:224
2014-11-17 12:15:40.608 DEBUG heat.engine.scheduler [-] Task DependencyTaskGroup((destroy) {Port "instance-port" [c1e4bed7-7de9-4082-8199-36efef41f9c8] Stack "bug" [3a83b3ad-d470-4fff-aec7-fa3629b9a983]: {}, Server "instance" Stack "bug" [3a83b3ad-d470-4fff-aec7-fa3629b9a983]: {Port "instance-port" [c1e4bed7-7de9-4082-8199-36efef41f9c8] Stack "bug" [3a83b3ad-d470-4fff-aec7-fa3629b9a983]}}) sleeping from (pid=8275) _sleep /opt/stack/heat/heat/engine/scheduler.py:163
/usr/local/lib/python2.7/dist-packages/amqp/channel.py:616: VDeprecationWarning: The auto_delete flag for exchanges has been deprecated and will be removed
from py-amqp v1.5.0.
  warn(VDeprecationWarning(EXCHANGE_AUTODELETE_DEPRECATED))
/usr/local/lib/python2.7/dist-packages/amqp/channel.py:616: VDeprecationWarning: The auto_delete flag for exchanges has been deprecated and will be removed
from py-amqp v1.5.0.
  warn(VDeprecationWarning(EXCHANGE_AUTODELETE_DEPRECATED))
2014-11-17 12:15:41.608 DEBUG heat.engine.scheduler [-] Task DependencyTaskGroup((destroy) {Port "instance-port" [c1e4bed7-7de9-4082-8199-36efef41f9c8] Stack "bug" [3a83b3ad-d470-4fff-aec7-fa3629b9a983]: {}, Server "instance" Stack "bug" [3a83b3ad-d470-4fff-aec7-fa3629b9a983]: {Port "instance-port" [c1e4bed7-7de9-4082-8199-36efef41f9c8] Stack "bug" [3a83b3ad-d470-4fff-aec7-fa3629b9a983]}}) running from (pid=8275) step /opt/stack/heat/heat/engine/scheduler.py:218
2014-11-17 12:15:41.609 DEBUG heat.engine.scheduler [-] Task DependencyTaskGroup((destroy) {Port "instance-port" [c1e4bed7-7de9-4082-8199-36efef41f9c8] Stack "bug" [3a83b3ad-d470-4fff-aec7-fa3629b9a983]: {}, Server "instance" Stack "bug" [3a83b3ad-d470-4fff-aec7-fa3629b9a983]: {Port "instance-port" [c1e4bed7-7de9-4082-8199-36efef41f9c8] Stack "bug" [3a83b3ad-d470-4fff-aec7-fa3629b9a983]}}) complete from (pid=8275) step /opt/stack/heat/heat/engine/scheduler.py:224
2014-11-17 12:15:41.627 INFO heat.engine.stack [-] Stack DELETE COMPLETE (bug): Stack DELETE completed successfully
2014-11-17 12:15:41.717 DEBUG heat.engine.scheduler [-] Task update_task from Stack "bug" [446854a3-84c8-458e-894e-ceed62784095] complete from (pid=8275) step /opt/stack/heat/heat/engine/scheduler.py:224
2014-11-17 12:15:41.727 DEBUG heat.engine.stack_lock [-] Engine 73829574-b712-4ff9-a12f-11db2baabb7d released lock on stack 446854a3-84c8-458e-894e-ceed62784095 from (pid=8275) release /opt/stack/heat/heat/engine/stack_lock.py:123

[Stan Lagun (slagun)]

This issues happens in stable/juno but not in stable/icehouse

[Pavlo Shchelokovskyy (pshchelo)]

I see that in course of such update the the original "instance-port" resource is being replaced (its name/id changes after update)

[Pavlo Shchelokovskyy (pshchelo)]

btw, I see in on devstack master as well

[Steve Baker (steve-stevebaker)]

OS::Neutron::Port replacement_policy: REPLACE_ALWAYS means a new port will be created on every stack update, even if the server doesn't change (this is to workaround problems with modelling ports as heat resources, and nova's current behaviour of deleting ports when servers are deleted).

REPLACE_ALWAYS should in theory work since the server (and the floating association) will be switched over to the new port on each update.

The above example doesn't include a OS::Neutron::FloatingIPAssociation in the same resource, is there a reason for that?

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/135297

[Andy Edmonds (andy-edmonds)]

Have experienced the same issue on a Juno Mirantis Fuel 6.0 deployment. Whereas it's likely incorrect and/or bring a new self-induced problem, changing REPLACE_ALWAYS to AUTO in heat/engine/resources/neutron/port.py works for our simple cases. Anyone have any comments/pointers to possible ill-side-effects of this?

BTW: We also manually applied the patch above but could not get to to resolve the issue.

Andy

[Serg Melikyan (smelikyan)]

Tracked in Murano as https://bugs.launchpad.net/murano/+bug/1414008

[Martin Gerhard Loschwitz (martin-loschwitz)]

We're running into this actively. Is there any work on this being done currently, or are there any plans to fix the issue within a forseeable amount of time? Our customers tried to run the "update-stack" command and were slightly puzzled to find their VM having no NIC anymore afterwards ...

[Steve Baker (steve-stevebaker)]

Now that a fix for nova bug 1158684 has landed we can now consider making AUTO the default, which will avoid this problem.

Longer term we'll be moving away from representing neutron ports as heat resources. Our advice would be where possible to not use port resources, and use the other 'networks' options to allow nova to create the ports for you, for example where {get_param: private_net} returns the *name* of the neutron network, you can do the following:

  server:
    type: OS::Nova::Server
    properties:
      networks:
      - network: {get_param: private_net}

  server_floating_ip_assoc:
    type: OS::Neutron::FloatingIPAssociation
    properties:
      floatingip_id: {get_resource: floating_ip}
      port_id: {get_attr: [server, addresses, {get_param: private_net}, 0, port]}

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/169543

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/169543
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=2b4a4010821ebb242806ae6ff4fc1ad4db62fecb
Submitter: Jenkins
Branch: master

commit 2b4a4010821ebb242806ae6ff4fc1ad4db62fecb
Author: Steve Baker <email address hidden>
Date: Wed Apr 1 11:33:49 2015 +1300

    OS::Neutron::Port default replacement_policy=AUTO

    Now that nova bug 1158684 has been fixed heat can assume that a port
    won't be deleted by nova during a stack update.

    REPLACE_ALWAYS as a default caused its share of issues, including
    servers not handling the port churn on stack updates.

    REPLACE_ALWAYS is left as an option for standalone heat orchestrating
    older OpenStack releases.

    Change-Id: Ie9b2ebe8b29bd5ed6006dfb59c6dccd4a595832f
    Closes-Bug: #1393376
    Related-Bug: #1158684

[OpenStack Infra (hudson-openstack)]

Change abandoned by Sergey Kraynev (<email address hidden>) on branch: master
Review: https://review.openstack.org/135297
Reason: old and not actual patch