nova-multi-cell fails on stable branches with: Failed to compute_task_migrate_server: 'NoneType' object has no attribute 'version'

[melanie witt]

When we get a request to migrate a server that has minimum bandwidth
resource requests, we check whether the service on the target host is
new enough to support resource requests. However, we are not targeting
the target host's cell database when we do the lookup. This works OK
when the migration is occuring within the same cell, but if the target
host is in another cell, we will fail to lookup the service record in
the source cell with the following error [1]:

  Failed to compute_task_migrate_server: 'NoneType' object has no
  attribute 'version'

This is because when Service.get_by_host_and_binary fails to find a
matching record, it returns (and thus returns None).

This failure has recently emerged in the nova-multi-cell gate job
because tempest tests were recently added for minimum bandwidth
resource requests in change:

https://review.opendev.org/c/openstack/tempest/+/694539

and these tests include migration tests.

To fix this, we just need to target the cell database for the target
host's service record when we lookup the service.

This change is stable-only because the service version check was
recently removed in change:

https://review.opendev.org/c/openstack/nova/+/735570

and only exists in stable branches.

[1] https://zuul.opendev.org/t/openstack/build/2ee6a4a30c8f403b911386a6cd80fc51/log/controller/logs/screen-n-super-cond.txt#3778

[melanie witt (melwitt)]

Bug doesn't exist on master branch.

[melanie witt (melwitt)]

WIP patch proposed at https://review.opendev.org/c/openstack/nova/+/766364

[melanie witt (melwitt)]

Logstash query:

8 hits in the past 7 days, check queue, all failures, stable/victoria and stable/ussuri branches

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Failed%20to%20compute_task_migrate_server%3A%20'NoneType'%20object%20has%20no%20attribute%20'version'%5C%22%20AND%20filename%3A%5C%22controller%2Flogs%2Fscreen-n-super-cond.txt%5C%22&from=7d

[Balazs Gibizer (balazs-gibizer)]

A gate unblocking patch has been proposed to avoid this failure in https://review.opendev.org/c/openstack/nova/+/765749

I will mark this bug closed for victoria once https://review.opendev.org/c/openstack/nova/+/765749 lands and keep handling the issue as part of the more generic bug https://bugs.launchpad.net/nova/+bug/1907522

[Balazs Gibizer (balazs-gibizer)]

https://review.opendev.org/c/openstack/nova/+/765749 has been merged so closing this for victoria

[Balazs Gibizer (balazs-gibizer)]

To further explain the closure of this bug.

There are two open bugs for the same root case. This and https://bugs.launchpad.net/nova/+bug/1907522
The fix https://review.opendev.org/c/openstack/nova/+/766925 proposed for https://bugs.launchpad.net/nova/+bug/1907522 will be backported to stable branches and it will make this bug disappear too. So I'm closing this bug and tracking the progress in https://bugs.launchpad.net/nova/+bug/1907522 instead.

[melanie witt (melwitt)]

Adding a comment with the explanation as to why we will not ever fix cell targeting for migrations with qos ports on stable/victoria:

We are going to disallow cross-cell migrations for instances with qos ports:

https://review.opendev.org/c/openstack/nova/+/766925

and backport it, so cell targeting will not be necessary as the code will never be reached.

When I opened this bug, I had thought there was simply a bug in cross-cell migrations for instances with qos ports, but it turned out that support was never added. So it was more a lack of a feature than it was a bug.

For this reason, cross-cell migration support will be added on the master branch as a feature and stable branches will remain without cross-cell migration support. As such, the cell targeting bug opened here is invalidated.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 22.1.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 21.2.0 release.