stack rollback fails

[Margarita Mazepa]

TROUBLE DESCRIPTION
1.1. Trouble summary
Stack rollback fails for different reasons, 2 examples of heat stack-delete failure are:
1.1.1) 'Resource DELETE failed: Conflict: resources.sg_neil: Security Group 68e71d10-cc84-4b83-8fcf-154d00f7775e in use.'
1.1.2) 'Resource DELETE failed: Unauthorized: resources.pk: Unauthorized (HTTP 401) (Request-ID: req-ccbbc2f8-ee37-47f6-96d2-e0db7a54920d)'

1.2. Detailed trouble description
This detailed trouble description is described for example (1.1.1) above. The same steps are used also for example (1.1.2) also.
1.2.1 Deploying from ECM, the attached hierarchical heat-templates.

a) The 'parent.yaml' heat-template, creates a security group, and then uses a resource-group to spin-up 'n' instances of the 'child.yaml' template
b) The 'child.yaml' templates defines the OS::Nova::Server resource to create the VM instances
c) When the VM instances are spun-up, each VM has 2 neutron-ports.
d) One neutron port is connected to L3VPN network with dhcp, the second neutron-port is connected to L2GW network with dhcp.
e) Each VM performs ping on both networks, and reports results using heat-signals

1.2.2) The problem occurs (see stack-failure.txt) when a failed heat-signal is sent to the stack, the stack rollback fails as follows:

a) The stack reports that VM resource delete completes successfully, but it seems some neutron-ports were not deleted (see point 3 below)
b) The security-group resource delete fails 'Resource DELETE failed: Conflict: resources.sg_neil: Security Group 68e71d10-cc84-4b83-8fcf-154d00f7775e in use.'
c) It was noted in the cic (see attached failed-ports.txt) that some neutron ports both on the L3VPN and L2GW networks could not be deleted. On both networks 1xdhcp-port and 1xvhostuser-port could not be deleted.
d) These failed ports need to be manually deleted, then the security group needs to be manually deleted, then the heat-stack can be manually deleted.

The parent.yaml and child.yaml were used to create the VM's.The VM's are connected to the neutron ports.When the failed heat-signal is sent to the stack, the stack rollback fails.In the process of rollback the VM resource gets deleted succesfully but the neutron ports were not deleted,they have to be manually deleted.

it has happened twice during stability test.

[Oleksiy Molchanov (omolchanov)]

To proceed we need the following:

1) Neutron logs that contains log entries during the time issue happens (for SG deletion issue) and Keystone logs (for unauthorized issue)

2) Heat yamls for reproduction.

[Denis Meltsaykin (dmeltsaykin)]

no feedback over a month, closing as invalid