Deleting stuck build instance may leak allocations
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Compute (nova) |
Fix Released
|
Undecided
|
Alexandre arents |
Bug Description
Description
===========
After issues in control plane during instance creation,
Instance may stay stuck in BUILD state.
Even after deleting them, placement allocation may remain,
and compute host log is complaining that:
Instance eba20a0f-
Steps to reproduce
==================
On a fresh devstack master install
1) open a terminal that display entry in placement.
while true ; do date ; mysql -e "select * from placement.
2) Trigguer a spawn of 50 instances & kill rabbit after 5sec to simulate issue on control plane:
openstack server create --flavor m1.tiny --image cirros-
Note: To reach the bug, goal is to get instances Allocated by scheduler, but not let the time to conductor to create entry in nova_cell1.
You should see allocations appearing in allocations:
+------
| created_at | updated_at | id | resource_
+------
| 2020-01-13 11:02:51 | NULL | 1727 | 1 | 8d0a42fe-
| 2020-01-13 11:02:51 | NULL | 1728 | 1 | 8d0a42fe-
| 2020-01-13 11:02:51 | NULL | 1729 | 1 | 8d0a42fe-
| 2020-01-13 11:02:51 | NULL | 1730 | 1 | 3cd1b8be-
| 2020-01-13 11:02:51 | NULL | 1731 | 1 | 3cd1b8be-
.....
instances are all stuck in BUILD at this stage
3) delete instances:
openstack server list | awk '/m1.tiny/ {print $2}' | xargs openstack server delete
4) service rabbitmq-server start
5) openstack server list
<display nothing>
6) mysql -e "select count(*) from placement.
+----------+
| count(*) |
+----------+
| 150 |
+----------+
Allocation remains
7) nova-compute logs complaining that:
Instance eba20a0f-
Expected result
===============
placement allocation of instance have to be cleanup after deletion
Actual result
=============
placement allocation of instance are leaked.
Environment
===========
At least stein to master seems impacted
Fix proposed to branch: master /review. opendev. org/702368
Review: https:/