renaming availability zone doesn't modify host's availability zone

Bug #1378904 reported by Guillaume Winter on 2014-10-08
46
This bug affects 9 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Medium
Matt Riedemann

Bug Description

Hi,

After renaming our availability zones via Horizon Dashboard, we couldn't migrate any "old" instance anymore, the scheduler returning "No valid Host found"...

After searching, we found in the nova DB `instances` table, the "availability_zone" field contains the name of the availability zone, instead of the ID ( or maybe it is intentional ;) ).

So renaming AZ leaves the hosts created prior to this rename orphan and the scheduler cannot find any valid host for them...

Our openstack install is on debian wheezy, with the icehouse "official" repository from archive.gplhost.com/debian/, up to date.

If you need any more infos, I'd be glad to help.

Cheers

Changed in nova:
status: New → Confirmed
tags: added: compute db
Fan Guo (faguo) on 2014-10-09
Changed in nova:
assignee: nobody → Fan Guo (faguo)
xens (r-aviolat) wrote :

I faced the same problem on my setup:

OS: Ubuntu 14.04
OpenStack: Kilo from the ubuntu-cloud repo

I could rename the availability-zone from the dashboard or the API but I couldn't migrate the instances anymore. I had to edit the following fields:

db->nova,table->instances,row->availability_zone
db->neutron,table->ports,row->device_owner

after the manually update the VMs could be migrated again.

leehom (feli5) wrote :

I also faced the same problem on my setup:

OS: CentOS Linux release 7.1.1503 (Core)
OpenStack: Juno from rdo-release

I have two compute nodes, compute1 in AZ1 and compute2 in AZ2.
I created one instance on each compute, which are instance1 in compute1 and instance2 in compute2.

I can live migrate instance1 to compute2, after migration, available zone was changed from AZ1 to AZ2 on dashboard, but when check the instances table in nova database, instance1's available zone is still AZ1.
So though compute1 and compute2 are in different AZ, I can cold migrate instance1 back to compute1.

Change Host Aggregates is just like live migration, instances AZ is changed on dashboard, but not really changed in dashboard.

BTW, when get instance's detail by using "nova --debug list", it also prints the wrong result

This is an automated cleanup. This bug report has been closed because it
is older than 18 months and there is no open code change to fix this.
After this time it is unlikely that the circumstances which lead to
the observed issue can be reproduced.

If you can reproduce the bug, please:
* reopen the bug report (set to status "New")
* AND add the detailed steps to reproduce the issue (if applicable)
* AND leave a comment "CONFIRMED FOR: <RELEASE_NAME>"
  Only still supported release names are valid (LIBERTY, MITAKA, OCATA, NEWTON).
  Valid example: CONFIRMED FOR: LIBERTY

Changed in nova:
assignee: Fan Guo (faguo) → nobody
status: Confirmed → Expired
JK (jiggernaut) wrote :

CONFIRMED FOR: MITAKA
$ nova-manage --version
13.0.0

# Reproduce Steps:
1) Create a HA/AZ. ie., "AZ1"
2) Add compute nodes to "AZ1" (Admin->System->Host Aggregates->Manage Hosts)
3) Launch VM in this AZ.
4) Live migrate/migrate VM - will succeed
5) Create a new HA/AZ. ie., "AZ2"
6) Remove compute nodes from "AZ1"
7) Add compute nodes to "AZ2"
8) Try to migrate VM

Fails with ERROR: Error: No valid host was found. There are not enough hosts available. compute-1: (AvailabilityZoneFilter) avail zone az1 not in host AZ: set([u'az2'])

# nova-scheduler.log
2016-11-17 21:08:38.690 168453 INFO nova.filters [req-e9cede77-e888-4553-83d6-4e112a8e44a7 59d4a769c88545acb86f646b2464f4d1 93dd4afc2ddb4bfd88d8b5d13d348998 - - -] AvailabilityZoneFilter: (compute-1) REJECT: avail zone az1 not in host AZ: set([u'az2'])

# nova show <uuid> displays correct AZ for VM
| OS-EXT-AZ:availability_zone | az2

# however nova --debug list displays in the RESP BODY:
"OS-EXT-AZ:availability_zone": "az1"

# check the VM in DB, availability_zone is still listed as 'az1' as well.

Changed in nova:
status: Expired → New
Sean Dague (sdague) wrote :

This looks like a real issue. The short term fix for this would be to also update instances when availability_zone name is updated over the API.

Changed in nova:
status: New → Triaged
importance: Undecided → Low
tags: added: api availability-zones
tags: added: low-hanging-fruit
Changed in nova:
assignee: nobody → Anusha Unnam (anusha-unnam)

Fix proposed to branch: master
Review: https://review.openstack.org/419502

Changed in nova:
assignee: Anusha Unnam (anusha-unnam) → Radoslav Gerganov (rgerganov)
status: Triaged → In Progress
Sean Dague (sdague) wrote :

Automatically discovered version icehouse in description. If this is incorrect, please update the description to include 'nova version: ...'

tags: added: openstack-version.icehouse
Sayali Lunkad (sayalilunkad) wrote :

CONFIRMED FOR NEWTON. After renaming the availability zone which has some instances running in it manual changes to the nova database need to be made in order to update the availability_zone. After that if live migration is carried out without specifying a host or anything which requires the AvailabilityZoneFilter returns no host found and this error message in the nova-scheduler.log
 INFO nova.filters [req-f0377310-f38a-47d8-a01b-65d90d79f202 7682adbc49f741ae8555915f95776a7b 07d042659b65465794e46ceafb1a1397 - - -] Filter AvailabilityZoneFilter returned 0 hosts
2017-08-24 10:04:17.687 20388 INFO nova.filters [req-f0377310-f38a-47d8-a01b-65d90d79f202 7682adbc49f741ae8555915f95776a7b 07d042659b65465794e46ceafb1a1397 - - -] Filtering removed all hosts for the request with
 instance ID 'd08b2dd4-039b-4de0-98f4-28c00c0c8cee'. Filter results: ['RetryFilter: (start: 2, end: 2)', 'AvailabilityZoneFilter: (start: 2, end: 0)']

Adding some addition info to the log point to this:
Availability Zone 'texas' requested. (d52-54-77-77-01-04, d52-54-77-77-01-04.c14.cloud.suse.de) ram: 1991MB disk: 14336MB io_ops: 0 instances: 0 has AZs: set([u'austin'])
for all the available hosts.

Sayali Lunkad (sayalilunkad) wrote :

After renaming availability zones via horizon the live migrate option for instances in horizon does not work unless a host is specified. This is due to the AvailibilityZone filter which removes all hosts because the availibility_zone of the host is obtained from the RequestSpec object which is stored in the request_specs table in the nova_api database. When availability_zone is renamed it is not updated in this table hence the mismatch. Not sure if we need to file another bug for this.

Sylvain Bauza (sylvain-bauza) wrote :

There is clear upstream consensus on the fact that since availability zones are for end-users, updating an AZ from an operator point-of-view would confuse their users.

Let me explain further : say you'd like to change AZ "foo" into "bar". For end-users looking at the AZ API before booting their instances, they can see "foo" as a valid target. So they just use --availability_zone foo in their instance boot calls and they expect to see their instances in AZ foo.

Now, what if operator turns "foo" into "bar" ? If I'm an end-user, I'd be very surprised to see my instances being now in "bar" while I explicitely asked "foo"!

As a clear design decision, we really want to make it explicit that renaming an AZ should be forbidden if there are active instances hosted within that AZ.
Closing that bug as Wontfix since I feel not being able to modify an AZ is not a bug but rather a design decision, but I feel we also need to modify the aggregates API to return a HTTP40x if someone is wanting to update an aggregate metadata containing AZ information when there are instances attached to it.

Changed in nova:
status: In Progress → Won't Fix

Change abandoned by Radoslav Gerganov (<email address hidden>) on branch: master
Review: https://review.openstack.org/419502

Fix proposed to branch: master
Review: https://review.openstack.org/509206

Changed in nova:
assignee: Radoslav Gerganov (rgerganov) → Andrey Volkov (avolkov)
status: Won't Fix → In Progress

Change abandoned by Andrey Volkov (<email address hidden>) on branch: master
Review: https://review.openstack.org/509206

Andrey Volkov (avolkov) on 2018-07-18
Changed in nova:
assignee: Andrey Volkov (avolkov) → nobody
Jacolex (jacolex) wrote :

Hi
You should also change spec in request_specs table like this (starting from ocata):

use nova_api
update request_specs set spec = replace (spec,'"availability_zone": "old-name"','"availability_zone": "new-name"') where instance_uuid = 'xxxxxxxxxxxxxxx';

Maybe it will help someone, who want to use this unsupported scenario... ;)

Changed in nova:
assignee: nobody → Andrey Volkov (avolkov)
Matt Riedemann (mriedem) on 2019-02-06
Changed in nova:
importance: Low → Medium
Changed in nova:
assignee: Andrey Volkov (avolkov) → Matt Riedemann (mriedem)
Matt Riedemann (mriedem) on 2019-02-15
Changed in nova:
assignee: Matt Riedemann (mriedem) → Andrey Volkov (avolkov)

Reviewed: https://review.openstack.org/635315
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=5dc1ed3c5cc20173eecd071b0804c469bdc9422f
Submitter: Zuul
Branch: master

commit 5dc1ed3c5cc20173eecd071b0804c469bdc9422f
Author: Matt Riedemann <email address hidden>
Date: Wed Feb 6 16:11:40 2019 -0500

    api-ref: warn about changing/unsetting AZ name with instances

    It is currently possible to rename or unset the availability_zone
    metadata value on a host aggregate which can adversely impact
    instances that were created in that specific AZ since later
    attempts to migrate or unshelve those instances will fail if the
    AZ with the original name no longer exists.

    This adds a warning to the API reference for updating the AZ
    name and also fixes a grammar typo in the 'metadata' response
    parameter description.

    Change-Id: Ie9d4a1ff1a23827490fe51350c11292c6efc4eb2
    Related-Bug: #1378904

Changed in nova:
assignee: Andrey Volkov (avolkov) → Matt Riedemann (mriedem)
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers