Comment 6 for bug 1890244

So the missing pieces to a successful reproduction on master were:

1) in devstack the [workarounds]disable_group_policy_check_upcall is set to True by default as devstack is deployed with proper cell isolation hence no upcall is actually possible. So first I had to set [workarounds]disable_group_policy_check_upcall to False to trigger the upcall

2) Then I needed to promote the cell1 conductor in devstack to a super conductor by adding api db access to its config
  [api_database]
  connection = mysql+pymysql://root:admin@127.0.0.1/nova_api?charset=utf8

After this the reported issue is reproducible:

Jun 21 10:25:57 gibi-devstack-aio nova-compute[2131543]: ERROR nova.compute.manager [None req-22e98c46-328a-4dc1-9381-550e44a44cee admin admin] [instance: 251e19f6-431a-4df9-aab3-79d8cdf65f28] Setting instance vm_state to ERROR: nova.exception_Remote.InstanceGroupNotFound_Remote: Instance group 7eea7e6b-8277-4811-a27e-7a0ea39874ab could not be found.
Jun 21 10:25:57 gibi-devstack-aio nova-compute[2131543]: ERROR nova.compute.manager [instance: 251e19f6-431a-4df9-aab3-79d8cdf65f28] Traceback (most recent call last):
Jun 21 10:25:57 gibi-devstack-aio nova-compute[2131543]: ERROR nova.compute.manager [instance: 251e19f6-431a-4df9-aab3-79d8cdf65f28] File "/opt/stack/nova/nova/compute/manager.py", line 10362, in _error_out_instance_on_exception
Jun 21 10:25:57 gibi-devstack-aio nova-compute[2131543]: ERROR nova.compute.manager [instance: 251e19f6-431a-4df9-aab3-79d8cdf65f28] yield
Jun 21 10:25:57 gibi-devstack-aio nova-compute[2131543]: ERROR nova.compute.manager [instance: 251e19f6-431a-4df9-aab3-79d8cdf65f28] File "/opt/stack/nova/nova/compute/manager.py", line 3555, in rebuild_instance
Jun 21 10:25:57 gibi-devstack-aio nova-compute[2131543]: ERROR nova.compute.manager [instance: 251e19f6-431a-4df9-aab3-79d8cdf65f28] self._do_rebuild_instance_with_claim(
Jun 21 10:25:57 gibi-devstack-aio nova-compute[2131543]: ERROR nova.compute.manager [instance: 251e19f6-431a-4df9-aab3-79d8cdf65f28] File "/opt/stack/nova/nova/compute/manager.py", line 3637, in _do_rebuild_instance_with_claim
Jun 21 10:25:57 gibi-devstack-aio nova-compute[2131543]: ERROR nova.compute.manager [instance: 251e19f6-431a-4df9-aab3-79d8cdf65f28] self._do_rebuild_instance(
Jun 21 10:25:57 gibi-devstack-aio nova-compute[2131543]: ERROR nova.compute.manager [instance: 251e19f6-431a-4df9-aab3-79d8cdf65f28] File "/opt/stack/nova/nova/compute/manager.py", line 3666, in _do_rebuild_instance
Jun 21 10:25:57 gibi-devstack-aio nova-compute[2131543]: ERROR nova.compute.manager [instance: 251e19f6-431a-4df9-aab3-79d8cdf65f28] self._validate_instance_group_policy(context, instance, hints)
Jun 21 10:25:57 gibi-devstack-aio nova-compute[2131543]: ERROR nova.compute.manager [instance: 251e19f6-431a-4df9-aab3-79d8cdf65f28] File "/opt/stack/nova/nova/compute/manager.py", line 1738, in _validate_instance_group_policy
Jun 21 10:25:57 gibi-devstack-aio nova-compute[2131543]: ERROR nova.compute.manager [instance: 251e19f6-431a-4df9-aab3-79d8cdf65f28] group = objects.InstanceGroup.get_by_hint(context, group_hint)
Jun 21 10:25:57 gibi-devstack-aio nova-compute[2131543]: ERROR nova.compute.manager [instance: 251e19f6-431a-4df9-aab3-79d8cdf65f28] File "/opt/stack/nova/nova/objects/instance_group.py", line 389, in get_by_hint
Jun 21 10:25:57 gibi-devstack-aio nova-compute[2131543]: ERROR nova.compute.manager [instance: 251e19f6-431a-4df9-aab3-79d8cdf65f28] return cls.get_by_uuid(context, hint)
Jun 21 10:25:57 gibi-devstack-aio nova-compute[2131543]: ERROR nova.compute.manager [instance: 251e19f6-431a-4df9-aab3-79d8cdf65f28] File "/usr/local/lib/python3.8/dist-packages/oslo_versionedobjects/base.py", line 175, in wrapper
Jun 21 10:25:57 gibi-devstack-aio nova-compute[2131543]: ERROR nova.compute.manager [instance: 251e19f6-431a-4df9-aab3-79d8cdf65f28] result = cls.indirection_api.object_class_action_versions(
Jun 21 10:25:57 gibi-devstack-aio nova-compute[2131543]: ERROR nova.compute.manager [instance: 251e19f6-431a-4df9-aab3-79d8cdf65f28] File "/opt/stack/nova/nova/conductor/rpcapi.py", line 240, in object_class_action_versions
Jun 21 10:25:57 gibi-devstack-aio nova-compute[2131543]: ERROR nova.compute.manager [instance: 251e19f6-431a-4df9-aab3-79d8cdf65f28] return cctxt.call(context, 'object_class_action_versions',
Jun 21 10:25:57 gibi-devstack-aio nova-compute[2131543]: ERROR nova.compute.manager [instance: 251e19f6-431a-4df9-aab3-79d8cdf65f28] File "/opt/stack/oslo.messaging/oslo_messaging/rpc/client.py", line 189, in call
Jun 21 10:25:57 gibi-devstack-aio nova-compute[2131543]: ERROR nova.compute.manager [instance: 251e19f6-431a-4df9-aab3-79d8cdf65f28] result = self.transport._send(
Jun 21 10:25:57 gibi-devstack-aio nova-compute[2131543]: ERROR nova.compute.manager [instance: 251e19f6-431a-4df9-aab3-79d8cdf65f28] File "/opt/stack/oslo.messaging/oslo_messaging/transport.py", line 123, in _send
Jun 21 10:25:57 gibi-devstack-aio nova-compute[2131543]: ERROR nova.compute.manager [instance: 251e19f6-431a-4df9-aab3-79d8cdf65f28] return self._driver.send(target, ctxt, message,
Jun 21 10:25:57 gibi-devstack-aio nova-compute[2131543]: ERROR nova.compute.manager [instance: 251e19f6-431a-4df9-aab3-79d8cdf65f28] File "/opt/stack/oslo.messaging/oslo_messaging/_drivers/amqpdriver.py", line 689, in send
Jun 21 10:25:57 gibi-devstack-aio nova-compute[2131543]: ERROR nova.compute.manager [instance: 251e19f6-431a-4df9-aab3-79d8cdf65f28] return self._send(target, ctxt, message, wait_for_reply, timeout,
Jun 21 10:25:57 gibi-devstack-aio nova-compute[2131543]: ERROR nova.compute.manager [instance: 251e19f6-431a-4df9-aab3-79d8cdf65f28] File "/opt/stack/oslo.messaging/oslo_messaging/_drivers/amqpdriver.py", line 681, in _send
Jun 21 10:25:57 gibi-devstack-aio nova-compute[2131543]: ERROR nova.compute.manager [instance: 251e19f6-431a-4df9-aab3-79d8cdf65f28] raise result
Jun 21 10:25:57 gibi-devstack-aio nova-compute[2131543]: ERROR nova.compute.manager [instance: 251e19f6-431a-4df9-aab3-79d8cdf65f28] nova.exception_Remote.InstanceGroupNotFound_Remote: Instance group 7eea7e6b-8277-4811-a27e-7a0ea39874ab could not be found.

https://paste.opendev.org/show/btK22SLw4UH6oDDu5wKw/