[SRU] nova scheduler should ignore removed groups

[Fabian Zimmermann]

[Impact]
Fixes nova to not fail when scheduling a vm that previously belonged to a server group that has been deleted.

[Test Plan]
* deploy Openstack Yoga with shared storage e.g. Ceph
* create a server group (e.g. affinity) and boot one or more instances within that group:

  openstack server group create --policy affinity sg1
  openstack server create --image jammy --flavor m1.small --key-name testkey --nic net-id=private vm1 --hint group=e29105e1-06a2-40fa-85fa-899db944a82 --wait

* delete the server group:

  openstack server group delete sg1

* poweroff the compute host where the vms are running
* evacuate the vms from the powered down host:

  nova evacuate vm1 juju-882778-lp1890244-focal-yoga-13.cloud.sts

* check vms are ACTIVE and reachable on new compute host(s)

[Regression Potential]
No regressions are expected as a result of this fix.

-------------------------------------------------------------------------

Description
===========
We created a server group and started some instances in it.

Later we removed the server group.

Some time later, we had to evacuate these instances, but this failed, because the
scheduler removed all available hosts during filtering.

Steps to reproduce
==================
* create a server group
* start some instances in this group
* delete the server group
* ( hard poweroff your hypervisor )
* evacuate the instances

Expected result
===============
The instances are evacuated

Actual result
=============
The instances run into ERROR-state, because the server group is not found.

Environment
===========
* Kolla deployed OpenStack Train
* Ubuntu 18.04 / KVM + Libvirt

Logs & Configs
==============

scheduler tells:

 Filtering removed all hosts for the request with instance ID 'adddf2c9-0252-4463-a97c-f1ec209d9f49'. Filter results: ['AvailabilityZoneFilter: (start: 2, end: 2)', 'ComputeFilter: (start: 2, end: 2)', 'ComputeCapabilitiesFilter: (start: 2, end: 2)', 'ImagePropertiesFilter: (start: 2, end: 2)', 'ServerGroupAntiAffinityFilter: (start: 2, end: 2)', 'ServerGroupAffinityFilter: (start: 2, end: 0)']

instance show:

 | fault | {'code': 404, 'created': '2020-08-04T06:13:41Z', 'message': 'Instance group 7e84dc57-de05-4c92-9e3b-6e2d06c1d85b could not be found.'} |

[Balazs Gibizer (balazs-gibizer)]

Interesting, I tried to reproduce it on latest master with a two node devstack and evacuate worked with affinity or anti-affinity group after the group was deleted.

We does not store the group uuid on the instance but store the instance_uuid in the group so when the group is deleted there should be no way to found that group based on the instance_uuid.

This is the place where the scheduler gets the group, if any, for the instance being scheduled [1].

We also store the group uuid in the RequestSpec but loading a RequestSpec back from the DB also guarded against deleted groups. [2]

Could you execute

$openstack server group show <grp>

before you delete the group?

Could you also grep the nova logs for with the group uuid to see if any interesting log shows up?

[1] https://github.com/openstack/nova/blob/8ecc29bfccc64e6036d068f9bcbeb0d8e0748776/nova/scheduler/utils.py#L1076

[2] https://github.com/openstack/nova/blob/8ecc29bfccc64e6036d068f9bcbeb0d8e0748776/nova/objects/request_spec.py#L595

[Balazs Gibizer (balazs-gibizer)]

I'm marking this Incomplete until we get more information about the reproduction. Please set it back to New when you provide more information.

[Launchpad Janitor (janitor)]

[Expired for OpenStack Compute (nova) because there has been no activity for 60 days.]

[Gabriel Hartmann (gagaha)]

Hi,

we have a similar problem on instance resizes.
We are currently running on OpenStack Victoria.

Steps to reproduce
==================
* create a server group
* start some instances in this group
* delete the server group
* resize an instance of the former server group

This also results in the error message 'Instance group ad661086-ca64-47ee-ad16-3226734de605 could not be found.' and vm_state 'error'.

I traced the error down and it seems to originate from nova-compute on the destination host candidates.
Here are the error logs from nova-compute on one of the destinations candidates when trying to do the resize:

2021-11-02 14:24:09.560 7 ERROR nova.compute.manager [req-64de0df0-e144-4dba-ada4-1cb048e82706 4e9bb6ded0ed4a248be15edf71ea8944 f7c7480026ff49fcb5d439f66ea05203 - default default] [instance: b9a023da-26ae-4cef-af6e-20b549c0236f] Error trying to reschedule: nova.exception_Remote.NoValidHost_Remote: No valid host
was found. No valid host found for resize
Traceback (most recent call last):

  File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/conductor/manager.py", line 353, in _cold_migrate
    task.execute()

  File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/conductor/tasks/base.py", line 26, in wrap
    self.rollback(ex)

  File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__
    self.force_reraise()

  File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
    raise self.value

  File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/conductor/tasks/base.py", line 23, in wrap
    return original(self)

  File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/conductor/tasks/base.py", line 40, in execute
    return self._execute()

  File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/conductor/tasks/migrate.py", line 300, in _execute
    selection = self._reschedule()

  File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/conductor/tasks/migrate.py", line 382, in _reschedule
    raise exception.MaxRetriesExceeded(reason=reason)

nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance b9a023da-26ae-4cef-af6e-20b549c0236f.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

  File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/rpc/server.py", line 241, in inner
    return func(*args, **kwargs)

  File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/conductor/manager.py", line 98, in wrapper
    return fn(self, context, *args, **kwargs)

  File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/compute/utils.py", line 1434, in decorated_function
    return function(self, context, *args, **kwargs)

  File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/conductor/manager.py", line 299, in migrate_server
    self._cold_migrate(context, instance, flavor,

  File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/conductor/manager.py", line 368, in _cold_migrate
    raise exception.NoValidHost(reason=msg)

nova.exceptio...

[Sylvain Bauza (sylvain-bauza)]

I was able to reproduce this bug.
This is due to the fact we look at the RequestSpec https://github.com/openstack/nova/blob/ebe08834f311e8e22bfd9685d7e6e91dab967382/nova/compute/manager.py#L3657

and then we direcly ask to get the group by the hint there https://github.com/openstack/nova/blob/ebe08834f311e8e22bfd9685d7e6e91dab967382/nova/compute/manager.py#L1733

Eventually, if the group is deleted, we'll return a InstanceGroupNotFound exception by https://github.com/openstack/nova/blob/ebe08834f311e8e22bfd9685d7e6e91dab967382/nova/objects/instance_group.py#L223

[Balazs Gibizer (balazs-gibizer)]

So the missing pieces to a successful reproduction on master were:

1) in devstack the [workarounds]disable_group_policy_check_upcall is set to True by default as devstack is deployed with proper cell isolation hence no upcall is actually possible. So first I had to set [workarounds]disable_group_policy_check_upcall to False to trigger the upcall

2) Then I needed to promote the cell1 conductor in devstack to a super conductor by adding api db access to its config
  [api_database]
  connection = mysql+pymysql://root:admin@127.0.0.1/nova_api?charset=utf8

After this the reported issue is reproducible:

Jun 21 10:25:57 gibi-devstack-aio nova-compute[2131543]: ERROR nova.compute.manager [None req-22e98c46-328a-4dc1-9381-550e44a44cee admin admin] [instance: 251e19f6-431a-4df9-aab3-79d8cdf65f28] Setting instance vm_state to ERROR: nova.exception_Remote.InstanceGroupNotFound_Remote: Instance group 7eea7e6b-8277-4811-a27e-7a0ea39874ab could not be found.
Jun 21 10:25:57 gibi-devstack-aio nova-compute[2131543]: ERROR nova.compute.manager [instance: 251e19f6-431a-4df9-aab3-79d8cdf65f28] Traceback (most recent call last):
Jun 21 10:25:57 gibi-devstack-aio nova-compute[2131543]: ERROR nova.compute.manager [instance: 251e19f6-431a-4df9-aab3-79d8cdf65f28] File "/opt/stack/nova/nova/compute/manager.py", line 10362, in _error_out_instance_on_exception
Jun 21 10:25:57 gibi-devstack-aio nova-compute[2131543]: ERROR nova.compute.manager [instance: 251e19f6-431a-4df9-aab3-79d8cdf65f28] yield
Jun 21 10:25:57 gibi-devstack-aio nova-compute[2131543]: ERROR nova.compute.manager [instance: 251e19f6-431a-4df9-aab3-79d8cdf65f28] File "/opt/stack/nova/nova/compute/manager.py", line 3555, in rebuild_instance
Jun 21 10:25:57 gibi-devstack-aio nova-compute[2131543]: ERROR nova.compute.manager [instance: 251e19f6-431a-4df9-aab3-79d8cdf65f28] self._do_rebuild_instance_with_claim(
Jun 21 10:25:57 gibi-devstack-aio nova-compute[2131543]: ERROR nova.compute.manager [instance: 251e19f6-431a-4df9-aab3-79d8cdf65f28] File "/opt/stack/nova/nova/compute/manager.py", line 3637, in _do_rebuild_instance_with_claim
Jun 21 10:25:57 gibi-devstack-aio nova-compute[2131543]: ERROR nova.compute.manager [instance: 251e19f6-431a-4df9-aab3-79d8cdf65f28] self._do_rebuild_instance(
Jun 21 10:25:57 gibi-devstack-aio nova-compute[2131543]: ERROR nova.compute.manager [instance: 251e19f6-431a-4df9-aab3-79d8cdf65f28] File "/opt/stack/nova/nova/compute/manager.py", line 3666, in _do_rebuild_instance
Jun 21 10:25:57 gibi-devstack-aio nova-compute[2131543]: ERROR nova.compute.manager [instance: 251e19f6-431a-4df9-aab3-79d8cdf65f28] self._validate_instance_group_policy(context, instance, hints)
Jun 21 10:25:57 gibi-devstack-aio nova-compute[2131543]: ERROR nova.compute.manager [instance: 251e19f6-431a-4df9-aab3-79d8cdf65f28] File "/opt/stack/nova/nova/compute/manager.py", line 1738, in _validate_instance_group_policy
Jun 21 10:25:57 gibi-devstack-aio nova-compute[2131543]: ERROR nova.compute.manager [instance: 251e19f6-431a-4df9-aab3-79d8cdf65f28] group = objects.InstanceGroup.get_by_hint(context, group_hint)
Jun 21 10:25:57 gibi-devstack-aio nova-compute[2131543]: ERROR nov...

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/nova/+/847000

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/nova/+/847001

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/nova/+/847000
Committed: https://opendev.org/openstack/nova/commit/84a84f7f2fff58cf6254d6267af0ca5cee64c53b
Submitter: "Zuul (22348)"
Branch: master

commit 84a84f7f2fff58cf6254d6267af0ca5cee64c53b
Author: Sean Mooney <email address hidden>
Date: Tue Jun 21 12:04:20 2022 +0100

    add repoducer test for bug 1890244

    This change adds a test to simulate validating
    a instnace group policy where the group has been
    deleted but is still referenced in the scheduler hint.

    Change-Id: I803e6286a773d9e53639ab0cd449fc72bb3be613
    Related-Bug: #1890244

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/nova/+/847001
Committed: https://opendev.org/openstack/nova/commit/cd2c2f359bbd4913cfe73199847bc35b2664aaa9
Submitter: "Zuul (22348)"
Branch: master

commit cd2c2f359bbd4913cfe73199847bc35b2664aaa9
Author: Sean Mooney <email address hidden>
Date: Tue Jun 21 12:23:45 2022 +0100

    ignore deleted server groups in validation

    This change simply catches the exception raised when
    we lookup a servergroup via a hint and the validation
    upcall is enabled.

    Change-Id: I858b4da35382a9f4dcf88f4b6db340e1f34eb82d
    Closes-Bug: #1890244

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 26.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/nova/+/867989

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/nova/+/872663

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/nova/+/872663
Committed: https://opendev.org/openstack/nova/commit/f57900ad20d21aac5cb91aed87edc121f9008115
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit f57900ad20d21aac5cb91aed87edc121f9008115
Author: Sean Mooney <email address hidden>
Date: Tue Jun 21 12:04:20 2022 +0100

    add repoducer test for bug 1890244

    This change adds a test to simulate validating
    a instnace group policy where the group has been
    deleted but is still referenced in the scheduler hint.

    Change-Id: I803e6286a773d9e53639ab0cd449fc72bb3be613
    Related-Bug: #1890244
    (cherry picked from commit 84a84f7f2fff58cf6254d6267af0ca5cee64c53b)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/nova/+/867989
Committed: https://opendev.org/openstack/nova/commit/7934b9ec57d7060fbcf27706aa98ebf5a83f920a
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit 7934b9ec57d7060fbcf27706aa98ebf5a83f920a
Author: Sean Mooney <email address hidden>
Date: Tue Jun 21 12:23:45 2022 +0100

    ignore deleted server groups in validation

    This change simply catches the exception raised when
    we lookup a servergroup via a hint and the validation
    upcall is enabled.

    Change-Id: I858b4da35382a9f4dcf88f4b6db340e1f34eb82d
    Closes-Bug: #1890244
    (cherry picked from commit cd2c2f359bbd4913cfe73199847bc35b2664aaa9)

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "lp1890244-jammy-yoga.debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

[Corey Bryant (corey.bryant)]

Upload to jammy unapproved: https://launchpad.net/ubuntu/jammy/+queue?queue_state=1&queue_text=nova