OpenStack Heat

stack update fail when server has a port

Bug #1439708 reported by Ethan Lynn on 2015-04-02

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	OpenStack Heat	Fix Released	High	Ethan Lynn	OpenStack Heat 5.0.0 "liberty"
	Kilo	Fix Released	High	Angus Salkeld	OpenStack Heat 2015.1.0 "kilo"

Bug Description

If update image property of a server and this server has a port, stack update will fail with following error message.

2015-04-02 21:55:54.245 INFO heat.engine.resource [-] CREATE: Server "server" Stack "tt" [49c52e13-96ae-4da1-ac06-edb250bac653]
2015-04-02 21:55:54.245 TRACE heat.engine.resource Traceback (most recent call last):
2015-04-02 21:55:54.245 TRACE heat.engine.resource File "/opt/stack/heat/heat/engine/resource.py", line 489, in _action_recorder
2015-04-02 21:55:54.245 TRACE heat.engine.resource yield
2015-04-02 21:55:54.245 TRACE heat.engine.resource File "/opt/stack/heat/heat/engine/resource.py", line 559, in _do_action
2015-04-02 21:55:54.245 TRACE heat.engine.resource yield self.action_handler_task(action, args=handler_args)
2015-04-02 21:55:54.245 TRACE heat.engine.resource File "/opt/stack/heat/heat/engine/scheduler.py", line 295, in wrapper
2015-04-02 21:55:54.245 TRACE heat.engine.resource step = next(subtask)
2015-04-02 21:55:54.245 TRACE heat.engine.resource File "/opt/stack/heat/heat/engine/resource.py", line 530, in action_handler_task
2015-04-02 21:55:54.245 TRACE heat.engine.resource handler_data = handler(*args)
2015-04-02 21:55:54.245 TRACE heat.engine.resource File "/opt/stack/heat/heat/engine/resources/openstack/nova/server.py", line 693, in handle_create
2015-04-02 21:55:54.245 TRACE heat.engine.resource admin_pass=admin_pass)
2015-04-02 21:55:54.245 TRACE heat.engine.resource File "/usr/lib/python2.7/site-packages/novaclient/v2/servers.py", line 929, in create
2015-04-02 21:55:54.245 TRACE heat.engine.resource **boot_kwargs)
2015-04-02 21:55:54.245 TRACE heat.engine.resource File "/usr/lib/python2.7/site-packages/novaclient/v2/servers.py", line 557, in _boot
2015-04-02 21:55:54.245 TRACE heat.engine.resource return_raw=return_raw, **kwargs)
2015-04-02 21:55:54.245 TRACE heat.engine.resource File "/usr/lib/python2.7/site-packages/novaclient/base.py", line 152, in _create
2015-04-02 21:55:54.245 TRACE heat.engine.resource _resp, body = self.api.client.post(url, body=body)
2015-04-02 21:55:54.245 TRACE heat.engine.resource File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 443, in post
2015-04-02 21:55:54.245 TRACE heat.engine.resource return self._cs_request(url, 'POST', **kwargs)
2015-04-02 21:55:54.245 TRACE heat.engine.resource File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 418, in _cs_request
2015-04-02 21:55:54.245 TRACE heat.engine.resource resp, body = self._time_request(url, method, **kwargs)
2015-04-02 21:55:54.245 TRACE heat.engine.resource File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 392, in _time_request
2015-04-02 21:55:54.245 TRACE heat.engine.resource resp, body = self.request(url, method, **kwargs)
2015-04-02 21:55:54.245 TRACE heat.engine.resource File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 386, in request
2015-04-02 21:55:54.245 TRACE heat.engine.resource raise exceptions.from_response(resp, body, url, method)
2015-04-02 21:55:54.245 TRACE heat.engine.resource Conflict: Port d493ae33-369b-43ce-b65b-1b471deb2f94 is still in use. (HTTP 409) (Request-ID: req-b2d0ac59-c04d-463f-bfa1-dcfd94a353dc)

Revision history for this message

Ethan Lynn (ethanlynn) wrote on 2015-04-02:

As a temporary workaround, set IMAGE_UPDATE_POLICY default to REBUILD will avoid this problem.

Changed in heat:
assignee:	nobody → Ethan Lynn (ethanlynn)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-04-02: Fix proposed to heat (master)

Fix proposed to branch: master
Review: https://review.openstack.org/170145

Changed in heat:
status:	New → In Progress

Steve Baker (steve-stevebaker) on 2015-04-07

tags:

added: kilo-rc-potential

Steve Baker (steve-stevebaker) on 2015-04-07

Changed in heat:
importance:	Undecided → High
milestone:	none → next

Revision history for this message

Steve Baker (steve-stevebaker) wrote on 2015-04-07:

Actually https://review.openstack.org/170145 is fine on its own but lets call it a partial fix.

A port is an exclusive resource but during a stack-update server replacement there are 2 servers needing the same port until the old server is deleted.

Now that nova doesn't delete ports on detach, it may now be possible to call nova().servers.interface_detach on any port that is already attached before calling nova().servers.create.

The tricky bit will be doing an interface_attach on the old server during rollback, but maybe we'll need to wait for convergence to do the truly correct thing.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-04-08: Fix merged to heat (master)

Reviewed: https://review.openstack.org/170145
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=9062453aa5aa0bb197623e80128f4ada63cd5fac
Submitter: Jenkins
Branch: master

commit 9062453aa5aa0bb197623e80128f4ada63cd5fac
Author: Ethan Lynn <email address hidden>
Date: Thu Apr 2 18:34:37 2015 +0800

Set IMAGE_UPDATE_POLICY default to REBUILD

    If a server network using port id, update image property will fail
    with error 'port is still in use'. Set IMAGE_UPDATE_POLICY to
    REBUILD will preserve the port information when update the image.

Closes-Bug: #1439708
Change-Id: I023ee076682bd5febb960dc8754d61986fd9bedf

Changed in heat:
status:	In Progress → Fix Committed

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-04-21: Fix proposed to heat (stable/kilo)

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/176125

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-04-22: Fix merged to heat (stable/kilo)

Reviewed: https://review.openstack.org/176125
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=7dbaa72f591d420e1c979f27be3020b69815e461
Submitter: Jenkins
Branch: stable/kilo

commit 7dbaa72f591d420e1c979f27be3020b69815e461
Author: Ethan Lynn <email address hidden>
Date: Thu Apr 2 18:34:37 2015 +0800

Set IMAGE_UPDATE_POLICY default to REBUILD

    Closes-Bug: #1439708
    Change-Id: I023ee076682bd5febb960dc8754d61986fd9bedf
    (cherry picked from commit 9062453aa5aa0bb197623e80128f4ada63cd5fac)

Thierry Carrez (ttx) on 2015-04-23

tags:

removed: kilo-rc-potential

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-04-30: Fix proposed to heat (master)

Fix proposed to branch: master
Review: https://review.openstack.org/179291

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-06-10: Fix merged to heat (master)

Download full text (7.7 KiB)

Reviewed: https://review.openstack.org/179291
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=5d725ecd09a35e3d5fb042956c4ecebb309c0a41
Submitter: Jenkins
Branch: master

commit b5c32027e5f207cca51796ee2aba346413ad48da
Author: Steven Hardy <email address hidden>
Date: Thu Apr 23 14:14:26 2015 +1000

Fix ResourceGroup validate when there are removal_policies

    We can't call child_template() from a validate in ResourceGroup
    as it accesses nested(). So move the validation to validate_nested_stack()
    so we can control the test template used for validation.

Unfortunately coverage for this got removed during recent test rework
and it looks like the interface has got broken.

    Co-Authored-by: Angus Salkeld <email address hidden>
    Change-Id: Ibea738a343847736b041cc49a2c486fa71e562d9
    Closes-Bug: #1446861

commit c958ecb5b4b4fc5a0236b70c21d7d62864cd162b
Author: Ethan Lynn <email address hidden>
Date: Wed Apr 22 13:46:11 2015 +0800

Fix unicodeerror when heat-engine start

If change local language to other languages(like ja), heat-engine
will failed to start.

    Closes-Bug: #1446958
    Change-Id: Ia3cbcebb86257bc1c52ea758c1583191e5a52a2d
    (cherry picked from commit 0b3dd7be3d3cd9842616a16843d03e6c13cdcb08)

commit 94c4ae56e0fece995659d718ed1ab6e56b2deb9f
Author: Ethan Lynn <email address hidden>
Date: Tue Apr 21 17:09:42 2015 +0800

Add v6-fixed-ip support for nova server

Now nova server fixed-ip only support ipv4 address,
this patch add IPv6 fixed ip support for nova server.

    Closes-Bug: #1446532
    Change-Id: I647abeedd36352f053a043d00ab87a84ee2470f3
    (cherry picked from commit 3e1a6f3b826bbc2a638ed7ea239221d13678024c)

commit d07f91615a159663261091e672ce62f90e6ad607
Author: Angus Salkeld <email address hidden>
Date: Tue Apr 7 09:25:50 2015 +1000

Persist parent_resource_name and make sure it's available

    We are persisting for a number of reasons:
    - so we don't have to pass this through ever rpc call
    - the API exposes parent_resource (currently always None as
      it is not persisted)

    Closes-bug: #1438978
    Change-Id: Id2db36c0234a085ec4f0ce2ab114ec483ea29d81
    (cherry picked from commit edf86aeac2f7b68243b0eccc3c49fa1a579e664b)

commit f944c86d4c26877f4d8dab56790815af8e83ba45
Author: Michal Rostecki <email address hidden>
Date: Wed Mar 25 06:47:33 2015 +0100

Novaclient v2 instead of v1_1

Nova API v1.1 is now deprecated and causes warnings.

    Change-Id: Ib4b57a308b7637f4015a07b4e888ccd6347cb947
    Closes-Bug: #1437158
    (cherry picked from commit d9d68cf52bd7027e7d1cafd023a399c2a342b2c5)

commit 370f3c98c5eccf8bce1be425acc31af5e7224171
Author: Zane Bitter <email address hidden>
Date: Thu Apr 16 17:20:05 2015 -0400

Don't re-bind environment for get_file during stack update

    While we're in the process of updating a stack, we set the stack's
    environment to the new, updated values. However, we don't want to change
    existing resources' idea of their own values until we have explivitly done
    an update of them to bring them into line with th...

Reviewed:  https://review.openstack.org/179291
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=5d725ecd09a35e3d5fb042956c4ecebb309c0a41
Submitter: Jenkins
Branch:    master

commit b5c32027e5f207cca51796ee2aba346413ad48da
Author: Steven Hardy <shardy@redhat.com>
Date:   Thu Apr 23 14:14:26 2015 +1000

Fix ResourceGroup validate when there are removal_policies
    
    We can't call child_template() from a validate in ResourceGroup
    as it accesses nested(). So move the validation to validate_nested_stack()
    so we can control the test template used for validation.
    
    Unfortunately coverage for this got removed during recent test rework
    and it looks like the interface has got broken.
    
    Co-Authored-by: Angus Salkeld <asalkeld@mirantis.com>
    Change-Id: Ibea738a343847736b041cc49a2c486fa71e562d9
    Closes-Bug: #1446861

commit c958ecb5b4b4fc5a0236b70c21d7d62864cd162b
Author: Ethan Lynn <xjunlin@cn.ibm.com>
Date:   Wed Apr 22 13:46:11 2015 +0800

Fix unicodeerror when heat-engine start
    
    If change local language to other languages(like ja), heat-engine
    will failed to start.
    
    Closes-Bug: #1446958
    Change-Id: Ia3cbcebb86257bc1c52ea758c1583191e5a52a2d
    (cherry picked from commit 0b3dd7be3d3cd9842616a16843d03e6c13cdcb08)

commit 94c4ae56e0fece995659d718ed1ab6e56b2deb9f
Author: Ethan Lynn <xjunlin@cn.ibm.com>
Date:   Tue Apr 21 17:09:42 2015 +0800

Add v6-fixed-ip support for nova server
    
    Now nova server fixed-ip only support ipv4 address,
    this patch add IPv6 fixed ip support for nova server.
    
    Closes-Bug: #1446532
    Change-Id: I647abeedd36352f053a043d00ab87a84ee2470f3
    (cherry picked from commit 3e1a6f3b826bbc2a638ed7ea239221d13678024c)

commit d07f91615a159663261091e672ce62f90e6ad607
Author: Angus Salkeld <asalkeld@mirantis.com>
Date:   Tue Apr 7 09:25:50 2015 +1000

Persist parent_resource_name and make sure it's available
    
    We are persisting for a number of reasons:
    - so we don't have to pass this through ever rpc call
    - the API exposes parent_resource (currently always None as
      it is not persisted)
    
    Closes-bug: #1438978
    Change-Id: Id2db36c0234a085ec4f0ce2ab114ec483ea29d81
    (cherry picked from commit edf86aeac2f7b68243b0eccc3c49fa1a579e664b)

commit f944c86d4c26877f4d8dab56790815af8e83ba45
Author: Michal Rostecki <michal.rostecki@allegrogroup.com>
Date:   Wed Mar 25 06:47:33 2015 +0100

Novaclient v2 instead of v1_1
    
    Nova API v1.1 is now deprecated and causes warnings.
    
    Change-Id: Ib4b57a308b7637f4015a07b4e888ccd6347cb947
    Closes-Bug: #1437158
    (cherry picked from commit d9d68cf52bd7027e7d1cafd023a399c2a342b2c5)

commit 370f3c98c5eccf8bce1be425acc31af5e7224171
Author: Zane Bitter <zbitter@redhat.com>
Date:   Thu Apr 16 17:20:05 2015 -0400

Don't re-bind environment for get_file during stack update
    
    While we're in the process of updating a stack, we set the stack's
    environment to the new, updated values. However, we don't want to change
    existing resources' idea of their own values until we have explivitly done
    an update of them to bring them into line with the new
    template/environment. So when parsing a template snippet containing the
    get_file intrinsic, store the files dict to which it will refer. get_file
    will resolve to the new file only when the template has been updated.
    
    Closes-Bug: #1445170
    Change-Id: I8ec32b0e38389cd314e4b2b2d8c65f61f8a9c41e
    (cherry picked from commit e4447e7b66cf4691218b067643b5e25de72e31ba)

commit d61ba3ba5222dd6338528cbb4232df6ed959f540
Author: Kanagaraj Manickam <kanagaraj.manickam@hp.com>
Date:   Mon Apr 6 10:52:57 2015 +0530

Cleans nullable=True in db model column declaration
    
    By default sqlalchemy considers the Column's nullable=True and
    this patch cleans those column declaration which is having
    default value for the nullable field as it is not required
    explicitly
    
    Change-Id: I22b0d62238298fa49da09c53019408bc2049b893
    Closes-bug: #1439959
    (cherry picked from commit 1dbb1ed98f98a8b474f4d54df6554e4e1d93a296)

commit 7dbaa72f591d420e1c979f27be3020b69815e461
Author: Ethan Lynn <xjunlin@cn.ibm.com>
Date:   Thu Apr 2 18:34:37 2015 +0800

Set IMAGE_UPDATE_POLICY default to REBUILD
    
    If a server network using port id, update image property will fail
    with error 'port is still in use'. Set IMAGE_UPDATE_POLICY to
    REBUILD will preserve the port information when update the image.
    
    Closes-Bug: #1439708
    Change-Id: I023ee076682bd5febb960dc8754d61986fd9bedf
    (cherry picked from commit 9062453aa5aa0bb197623e80128f4ada63cd5fac)

commit 49c3c2008b95ac63ddc4ec524eda656bfe3ce440
Author: huangtianhua <huangtianhua@huawei.com>
Date:   Thu Apr 2 14:22:35 2015 +0800

To attach interface for server when using 'uuid'
    
    If user using 'uuid' to specify the network id, to make
    sure to attach interface for server when updating.
    
    Change-Id: Ie7ceb9c54ab4da4ad9a5d19803791edd41e5604c
    Closes-Bug: #1439497
    (cherry picked from commit 6523afe8c9739a098955d462551f6da829045ce0)

commit 1f977aa5fa28ed1e351f337191291198384abe02
Author: Eric Brown <browne@vmware.com>
Date:   Mon Apr 20 15:44:37 2015 -0700

Set auth_encryption_key option to be secret
    
    To avoid exposure in the logs, auth_encryption_key from heat.conf
    should be marked secret.
    
    Change-Id: Ia4aca067f03317dd8073988cd29d26c2ccc88778
    Closes-Bug: #1446408
    (cherry picked from commit 369789f69c34a6cbc4e4169f23c5cd5a958ef008)

commit 7c0dc01099c0d143c0707130eb4570f36154915e
Author: OpenStack Proposal Bot <openstack-infra@lists.openstack.org>
Date:   Mon Apr 20 17:55:07 2015 +0000

Updated from global requirements
    
    Change-Id: I7797401f72e517f5fac3e43a522edf8e539c18b8

commit bc9a59c902c6269f208bede097967b81c8991a2b
Author: Andreas Jaeger <aj@suse.de>
Date:   Mon Apr 20 11:58:35 2015 +0200

Release Import of Translations from Transifex
    
    Manual import of Translations from Transifex. This change also removes
    all po files that are less than 66 per cent translated since such
    partially translated files will not help users.
    
    This change needs to be done manually since the automatic import does
    not handle the proposed branches and we need to sync with latest
    translations.
    
    Change-Id: I72a675c62c83538a07366cbef3bfa84318eb7bb9

commit f82376800f1e6b51d4c8f5a4f35034ecb8199c06
Author: Ethan Lynn <xjunlin@cn.ibm.com>
Date:   Mon Apr 13 18:43:53 2015 +0800

Fix sqlalchemy migration script 061 for DB2 backend
    
    DB2 doesn't support alter column type from string to text,
    need to add a new column and copy data from old column, and
    later remove the old column.
    
    Closes-Bug: #1443252
    Change-Id: I7b3edf6bf7ea0efaf96398a83dad9ebe61caaa23
    (cherry picked from commit 7b25a9d6471a1a2d2d6441f7385d6d2bd00537ea)

commit 6e3f367525632d8acaa95530d236b5eb886d2ec9
Author: Thierry Carrez <thierry@openstack.org>
Date:   Fri Apr 17 10:33:47 2015 +0200

update .gitreview for stable/kilo
    
    Change-Id: I93d957bc06375cdcbc0dfe89ea75420cf58552ce

commit e2b89e2d58ea4aa31c4f3d666c0fd4bf3b0f37ff
Author: Steven Hardy <shardy@redhat.com>
Date:   Tue Apr 14 19:14:23 2015 +0100

Make SoftwareDeployments work for non-CREATE actions
    
    Currently we only create the deployment if the actions list includes
    CREATE, but its valid to create e.g DELETE only deployments, which
    curently won't work.  So determine whether to create the deployment
    on the resource_id instead of via the action, which resolves this problem.
    
    Change-Id: Ia563f4130c31baa4dcee3be3786ea3c49b6bad98
    Closes-Bug: #1444087
    (cherry picked from commit 0d4235015065a814f7e76f4a34d422146c58ec72)

Revision history for this message

Zane Bitter (zaneb) wrote on 2015-10-07:

Looks like this patch was merged right at the beginning of Liberty but the bug was never targeted for that.

Changed in heat:
milestone:	next → liberty-rc2
status:	Fix Committed → Fix Released

Thierry Carrez (ttx) on 2015-10-15

Changed in heat:
milestone:	liberty-rc2 → 5.0.0

Revision history for this message

Chris (tollwilly) wrote on 2016-05-23:

#10

I tried the workaround mentioned (image_update_policy: REBUILD) on Kilo and i aways get the port in use HTTP 409 error on my first stack-update attempt. Running the exact same stack-update command again results in a successful update (Update Complete). I am only changing user_data in the template. Looking at the neutron port that it complains about before and after the first attempt at an update the data is identical, but still the second stack-update attempt goes through while the first does not. I won't be updating to Liberty for some time so any suggestions on how to resolve this in Kilo are much appreciated!

-Chris

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.