Cannot delete the stack after replacing image during stack-update
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
| OpenStack Heat |
Fix Released
|
High
|
Deliang Fan | |
| Kilo |
Invalid
|
High
|
Steve Baker |
Bug Description
Steps to Reproduce:
* create a stack with with a single server (see attached example template)
* update the stack and replace the image the instance is using
- This will fail with the error "Conflict: Port <uuid> is still in use. (HTTP 409) "
* Try to delete the stack
- This will fail with the error "Failed to DELETE : Error deleting backup resources: Resource DELETE failed: Forbidden: You are not authorized to perform the requested action, identity:
Jacques Uber (uberj) wrote : | #1 |
Jacques Uber (uberj) wrote : | #2 |
summary: |
- Cannot delete the stack after repalcing image during stack-update + Cannot delete the stack after replacing image during stack-update |
Jacques Uber (uberj) wrote : | #3 |
Changed in heat: | |
assignee: | nobody → Nikunj Aggarwal (nikunj2512) |
Zane Bitter (zaneb) wrote : | #5 |
Could this be a duplicate of bug 1334514? It looks very similar, and the fix for that went in only a week ago.
Zane Bitter (zaneb) wrote : | #6 |
Looking closer, there are two parts here: the initial failure and then the failure to delete. The second part could be due to bug 1334514, but the first obviously isn't.
Sergey Kraynev (skraynev) wrote : | #7 |
There are some short ideas about this bug, according to last meeting in IRC:
The problem reason:
<skraynev>>during replace we create second server and try to use old port
<skraynev>>but according to neutron model it's not possible
<skraynev>>to have two servers with same port
Some possible way to fix it:
......
<shardy>
<shardy>>--instead of replacing the server?
.....
<stevebaker>>--we should be able to detach the port from the old server at some point in the update process
<mspreitz>
<stevebaker>
<pas-ha>>--may be smth like post/pre-replace
.....
<stevebaker>>--or we deprecate the port resource and make it possible to specify all port details in the server networks list
<mspreitz>>yes, +1 on stevebaker's suggestion
<zaneb>
<stevebaker>
<stevebaker>>--I was going to suggest a summit session on a rich server networks property
<zaneb>>---I'd like to have a summit session on a sensible API for neutron
<stevebaker>
<zaneb>>---when it's easier to store state in the client(s) than your server, you know you've taken a wrong turn
<stevebaker>
<zaneb>>---because nobody ever uses clients on different machines...
So As you can see some ways are possible: workarounds related with port detaching and attaching to new server or huge changes with deprecating whole resource....
Unfortunately we have not end solution, but it possibly will be discussed during mid-cycle meetup.
.
Changed in heat: | |
importance: | Undecided → Medium |
status: | New → Triaged |
Changed in heat: | |
assignee: | Nikunj Aggarwal (nikunj2512) → Anant Patil (ananta) |
Steve Baker (steve-stevebaker) wrote : | #8 |
The description of this bug includes:
Resource DELETE failed: Forbidden: You are not authorized to perform the requested action, identity:
which has nothing to do with the ports issue.
I believe the issue occurs in the following scenario:
- A resource subclass of StackUser (eg, Server, SoftwareDeployment) which has created a user is replaced on update
- For some reason a Forbidden is raised when an attempt is made to delete the resource user for the old replaced resource
- The replaced resource is marked CREATED_COMPLETE, but the stack goes to UPDATE_FAILED
- The stack becomes undeleteable, always going to DELETE_FAILED with the same error:
Resource DELETE failed: Forbidden: You are not authorized to perform the requested action, identity:
I'll mark this as High and come up with a simple template
Steve Baker (steve-stevebaker) wrote : | #9 |
heat stack-create -f bug-1356084.yaml a
heat stack-show bug-1356084 # CREATE_COMPLETE
heat stack-update -f bug-1356084.yaml -P user_data=two bug-1356084
heat stack-show bug-1356084 # UPDATE_FAILED
heat stack-delete bug-1356084
heat stack-show bug-1356084 # DELETE_FAILED
Changed in heat: | |
importance: | Medium → High |
milestone: | none → kilo-2 |
Steve Baker (steve-stevebaker) wrote : | #10 |
Full stack trace
2014-12-30 10:05:05.729 INFO heat.engine.
2014-12-30 10:05:05.729 TRACE heat.engine.
2014-12-30 10:05:05.729 TRACE heat.engine.
2014-12-30 10:05:05.729 TRACE heat.engine.
2014-12-30 10:05:05.729 TRACE heat.engine.
2014-12-30 10:05:05.729 TRACE heat.engine.
2014-12-30 10:05:05.729 TRACE heat.engine.
2014-12-30 10:05:05.729 TRACE heat.engine.
2014-12-30 10:05:05.729 TRACE heat.engine.
2014-12-30 10:05:05.729 TRACE heat.engine.
2014-12-30 10:05:05.729 TRACE heat.engine.
2014-12-30 10:05:05.729 TRACE heat.engine.
2014-12-30 10:05:05.729 TRACE heat.engine.
2014-12-30 10:05:05.729 TRACE heat.engine.
2014-12-30 10:05:05.729 TRACE heat.engine.
2014-12-30 10:05:05.729 TRACE heat.engine.
2014-12-30 10:05:05.729 TRACE heat.engine.
2014-12-30 10:05:05.729 TRACE heat.engine.
2014-12-30 10:05:05.729 TRACE heat.engine.
2014-12-30 10:05:05.729 TRACE heat.engine.
2014-12-30 10:05:05.729 TRACE heat.engine.
2014-12-30 10:05:05.729 TRACE heat.engine.
2014-12-30 10:05:05.729 TRACE heat.engine.
2014-12-30 10:05:05.729 TRACE heat.engine.
2014-12-30 10:05:05.729 TRACE heat.engine.
2014-12-30 10:05:05.729 TRACE heat.engine.
2014-12-30 10:05:05.729 TRACE heat.engine.res...
tags: | added: juno-backport-potential |
Steve Baker (steve-stevebaker) wrote : | #11 |
The stacks can be deleted by switching to an an admin user for that tenant
Changed in heat: | |
assignee: | Anant Patil (ananta) → Deliang Fan (vanderliang) |
Changed in heat: | |
milestone: | kilo-2 → kilo-3 |
Changed in heat: | |
milestone: | kilo-3 → kilo-rc1 |
Changed in heat: | |
status: | Triaged → In Progress |
Deliang Fan (vanderliang) wrote : | #12 |
@Steve Baker (steve-stevebaker) Hi, would you plz tell me which auth method do you use? trust or user/passwd? thank you!
Steve Baker (steve-stevebaker) wrote : | #13 |
devstack defaults (deferred_
Fix proposed to branch: master
Review: https:/
Deliang Fan (vanderliang) wrote : | #15 |
1. heat stack-create -f bug-1356084.yaml a
an instance and a first user is created.
2. heat stack-update -f bug-1356084.yaml -P user_data=two bug-1356084
After a new instance and the second user have been created, the primary instance and the first user will be deleted. But there comes the 403 failure from keystone during deleting the first user, which cause the failure of updating stack.
The primary cause of failure when updating the userdata of instance during stack update is that stack.stack_
def _delete_user(self):
user_id = self._get_user_id()
if user_id is None:
return
try:
except kc_exception.
pass
except ValueError:
# FIXME(shardy): This is a legacy delete path for backwards
# compatibility with resources created before the migration
# to stack_user.
# transitional period, this should be removed.
try:
except kc_exception.
Because self.stack.
So pass the stack_user_
Deliang Fan (vanderliang) wrote : | #16 |
@Steve Baker (steve-stevebaker) I think I have fix this bug after some test, would you please have a try too~
Fix proposed to branch: master
Review: https:/
Change abandoned by Deliang Fan (<email address hidden>) on branch: master
Review: https:/
Reason: duplicated
Reviewed: https:/
Committed: https:/
Submitter: Jenkins
Branch: master
commit a6d65581f60a740
Author: Deliang Fan <email address hidden>
Date: Mon Mar 30 14:33:15 2015 +0800
Correctly initialize copies of stack during updating stack
Pass stack_user_
oldstack to make sure the success when deleting stack domain user.
Create a common method to get the kwargs to create a stack from
an existing stack.
Co-Authored-By: Angus Salkeld <email address hidden>
Change-Id: Ieb7726ed738d5a
Closes-Bug: #1356084
Changed in heat: | |
status: | In Progress → Fix Committed |
Changed in heat: | |
status: | Fix Committed → Fix Released |
Changed in heat: | |
milestone: | kilo-rc1 → 2015.1.0 |
tags: | added: kilo-backport-potential |
tags: | removed: juno-backport-potential kilo-backport-potential |
Angus Salkeld (asalkeld) wrote : | #20 |
Note: this patch is already in Kilo.
no longer affects: | heat/juno |
+------ ------- ------- -+----- ------- ------- ------- ------- -----+- ------- ------- ------- ------- ------- ------- ------- ------- ------- ------- ------- ------- ------- ------- ------- ------- ------- ------- ------- --+---- ------- ------- --+---- ------- ------- ----+ status_ reason | resource_status | event_time | ------- ------- -+----- ------- ------- ------- ------- -----+- ------- ------- ------- ------- ------- ------- ------- ------- ------- ------- ------- ------- ------- ------- ------- ------- ------- ------- ------- --+---- ------- ------- --+---- ------- ------- ----+ 7d48-4e4b- 942b-7707b84094 e8 | state changed | CREATE_IN_PROGRESS | 2014-08- 12T23:10: 43Z | b5e5-459a- 8e88-5e493828aa 63 | state changed | CREATE_IN_PROGRESS | 2014-08- 12T23:10: 43Z | 8294-428c- a140-975b392f84 78 | state changed | CREATE_COMPLETE | 2014-08- 12T23:10: 43Z | 8195-4cc6- b678-0fc9b37533 5a | state changed | CREATE_COMPLETE | 2014-08- 12T23:10: 43Z | d0fe-4b7e- bfe1-7b0e20f46c 07 | state changed | CREATE_IN_PROGRESS | 2014-08- 12T23:10: 43Z | 0905-4a6d- 9ad0-accff83066 c7 | state changed | CREATE_COMPLETE | 2014-08- 12T23:10: 45Z | 5ec6-4332- b7fd-a43b1f83b0 07 | state changed | CREATE_IN_PROGRESS | 2014-08- 12T23:10: 45Z | a2aa-4a54- bf6f-dba722128d 8d | state changed | CREATE_IN_PROGRESS | 2014-08- 12T23:10: 45Z | 2ed3-4d47- b0b5-cbf4d5b889 a6 | state changed | CREATE_COMPLETE | 2014-08- 12T23:10: 46Z | 8453-483b- 87f4-e8b19f1c76 6f | state changed | CREATE_COMPLETE | 2014-08- 12T23:10: 46Z |
| resource_name | id | resource_
+------
| router | fbd01003-
| private_net | 6b8f66d6-
| private_net | e7981326-
| router | 90c40cf0-
| private_subnet | dc7212aa-
| private_subnet | e891cc34-
| server1_port | bc01b808-
| router_interface | 0a3da608-
| server1_port | d4dd1714-
| router_interface | 4bb83d47-
| server1 | 51ee...