Conflict when deleting allocations for an instance that hasn't finished building

Bug #1836754 reported by Artom Lifshitz on 2019-07-16
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Medium
Unassigned

Bug Description

Description
===========

When deleting an instance that hasn't finished building, we'll sometimes get a 409 from placement as such:

Failed to delete allocations for consumer 6494d4d3-013e-478f-9ac1-37ca7a67b776. Error: {"errors": [{"status": 409, "title": "Conflict", "detail": "There was a conflict when trying to complete your request.\\\\n\\\\n Inventory and/or allocations changed while attempting to allocate: Another thread concurrently updated the data. Please retry your update ", "code": "placement.concurrent_update", "request_id": "req-6dcd766b-f5d3-49fa-89f3-02e64079046a"}]}

Steps to reproduce
==================

1. Boot an instance
2. Don't wait for it to become active
3. Delete it immediately

Expected result
===============

The instance deletes successfully.

Actual result
=============

Nova bubbles up that error from Placement.

Logs & Configs
==============

This is being hit at a low rate in various CI tests, logstash query is here:

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Inventory%20and%2For%20allocations%20changed%20while%20attempting%20to%20allocate%3A%20Another%20thread%20concurrently%20updated%20the%20data%5C%22%20AND%20filename%3A%5C%22job-output.txt%5C%22

Eric Fried (efried) on 2019-07-16
Changed in nova:
status: New → Confirmed
importance: Undecided → Medium
tags: added: placement
Chris Dent (cdent) wrote :

Matt's logstash is happening because the host is being deleted which causes the api server to call delete_resource_provider (on the report client) with cascade=True, which deletes allocations but has no retry handling.

At least it seems that way, I'm rushing or I'd leave more info. Hope this is a clue.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers