Migration for instance refers to another host's instance!

[Dmitry Goloshubov]

Detailed bug description:
A lot of messages (thousands per day) in the nova-compute.log like this:

"2016-09-30 15:13:56.874 34172 ERROR nova.compute.resource_tracker [req-9c2f211b-55a4-4b38-bf9f-f103fc15affa - - - - -] Migration for instance 06556ed6-17e1-403d-9fd2-b5795392b8cf refers to another host's instance!"

"2016-09-30 15:13:57.042 34172 WARNING nova.compute.resource_tracker [req-9c2f211b-55a4-4b38-bf9f-f103fc15affa - - - - -] [instance: 06556ed6-17e1-403d-9fd2-b5795392b8cf] Instance not resizing, skipping migration."

Steps to reproduce:
N/A

Description of the environment:
Mirantis OpenStack 9.0, NFV

[Dmitry Goloshubov (dgoloshubov)]

https://bugs.launchpad.net/nova/+bug/1555320
Looks similar, but no progress

[Dmitry Goloshubov (dgoloshubov)]

Also there were some error messages about NUMA (not so many) at some point, and instance status was changing with different NUMA config, so probably it might be related.

<179>Sep 27 18:59:42 node-9 nova-conductor: 2016-09-27 18:59:42.439 16135 ERROR nova.scheduler.utils [req-e00204d3-862d-4775-8470-1fca68f367a1 a011812680284decaea981b06dd9a022 f31ad662e4d74a3cbc10ae341e1b4876 - - -] [instance: 06556ed6-17e1-403d-9fd2-b5795392b8cf] Error from last host: node-13.example.com (node node-13.example.com): [u'ComputeResourcesUnavailable: Insufficient compute resources: Requested instance NUMA topology cannot fit the given host NUMA topology.\n']

[Roman Podoliaka (rpodolyaka)]

Dmitry,

We managed to reproduce this on 9.1, but it seems to be harmless (yet annoying): basically, this error is printed to the log by a periodic task that is executed every minute (by default), when there are unconfirmed cold migrations or resizes from other compute hosts. This should not cause any other damage, though, thus setting the bug importance to Medium.

Just make sure you complete all migrations by doing "nova resize-confirm <instance>" or enable automatic confirmation in nova.conf on the compute nodes:

    cfg.IntOpt("resize_confirm_window",
                default=0,
                help="Automatically confirm resizes and cold migrations "
                     "after N seconds. Set to 0 to disable.")

[Dmitry Goloshubov (dgoloshubov)]

Roman,
In my case instances are in active state and 'nova list' do not show that resize is required.
Is there any way to avoid intensive logging?

[Roman Podoliaka (rpodolyaka)]

You can apply the following patch manually: https://review.openstack.org/#/c/384910/ - we'll back port it to 9.2

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/27614
Submitter: Pkgs Jenkins <email address hidden>
Branch: 9.0/mitaka

Commit: 205493ce207aaf9cf1a87edd01317731efcaf00e
Author: Timofey Durakov <email address hidden>
Date: Wed Nov 9 10:58:40 2016

Change log level to debug for migrations pairing

For resize/cold-migration it's possible that instance
already changed host to destination, but no confirm/revert
has happened yet. In that case resource tracked starts spamming
errors, because it's impossible to match migration and instance.
It's safe to lower log level to debug in that case.

Change-Id: I70cb7426e0e2849ee7d01205ce7b2d883a126d66
Closes-Bug: #1629938
(cherry picked from commit f085fbd7d3bfdf016a37ccc7e6e28786425f0e4e)

[Ekaterina Shutova (eshutova)]

Now it's only one DEBUG message in nova-compute.log:
root@node-1:~# grep -i 'migration for instance' /var/log/nova/nova-compute.log
2016-11-16 08:37:29.546 21792 DEBUG nova.compute.resource_tracker [req-c2161420-a402-4833-a916-560cb098f9ac - - - - -] Migration for instance ecba89ca-02fa-469b-9c4b-17fa7240477b refers to another host's instance! _pair_instances_to_migrations /usr/lib/python2.7/dist-packages/nova/compute/resource_tracker.py:545

Verified on 9.0 snapshot #495