Server's host not changed but actually on dest node after live-migrating

[Eric Xie]

Description
===========
The insances has been migrating for more than two hours. Then it got error 'Unauthorized'.
The host of output of CLI `openstack server show` was stall the old one.
But the instances had already been running on dest node.

Steps to reproduce
==================
1. Create one instance with large mem
2. Run some application which cosume mem, like `memtester`
3. Execute live-migrate

Expected result
===============
Rollback instance to old one, or update instance's host to dest node

Actual result
=============
Instance on dest node but the host is src node in DB

Environment
===========
$ git log -1
commit ee6af34437069a23284f4521330057a95f86f9b7 (HEAD -> stable/rocky, origin/stable/rocky)
Author: Luigi Toscano <email address hidden>
Date: Wed Dec 18 00:28:15 2019 +0100

    Zuul v3: use devstack-plugin-nfs-tempest-full

    ... and replace its legacy ancestor.

    Change-Id: Ifd4387a02b3103e1258e146e63c73be1ad10030c
    (cherry picked from commit e7e39b8c2e20f5d7b5e70020f0e42541dc772e68)
    (cherry picked from commit e82e1704caa1c2baea29f05e8d426337e8de7a3c)
    (cherry picked from commit 99aa8ebc12949f9bba76f22e877b07d02791bf5b)

Logs & Configs
==============
2020-04-02 21:08:32,890.890 6358 INFO nova.virt.libvirt.driver [req-b8d694f5-f60a-4866-bcd2-c107b2caa809 bdb83637364c4db4ba1a01f6ea879ff1 496db91424
254a85a4130a26801447c9 - default default] [instance: 8e76d7a1-e7f4-4476-94b3-724db6bfd467] Migration running for 30 secs, memory 80% remaining; (byt
es processed=3503551373, remaining=27653689344, total=34364792832)
2020-04-02 23:08:05,165.165 6358 INFO nova.virt.libvirt.driver [req-f22d9bee-9c1f-47a6-a2d5-3611f5b2529c bdb83637364c4db4ba1a01f6ea879ff1 496db91424254a85a4130a26801447c9 - default default] [instance: 8e76d7a1-e7f4-4476-94b3-724db6bfd467] Migration operation has completed
2020-04-02 23:08:05,166.166 6358 INFO nova.compute.manager [req-f22d9bee-9c1f-47a6-a2d5-3611f5b2529c bdb83637364c4db4ba1a01f6ea879ff1 496db91424254a85a4130a26801447c9 - default default] [instance: 8e76d7a1-e7f4-4476-94b3-724db6bfd467] _post_live_migration() is started..
2020-04-02 23:08:05,535.535 6358 WARNING nova.virt.libvirt.driver [req-f22d9bee-9c1f-47a6-a2d5-3611f5b2529c bdb83637364c4db4ba1a01f6ea879ff1 496db91424254a85a4130a26801447c9 - default default] [instance: 8e76d7a1-e7f4-4476-94b3-724db6bfd467] Error monitoring migration: The request you have made requires authentication. (HTTP 401): Unauthorized: The request you have made requires authentication. (HTTP 401)
2020-04-02 23:08:05,537.537 6358 ERROR nova.compute.manager [instance: 8e76d7a1-e7f4-4476-94b3-724db6bfd467] Unauthorized: The request you have made requires authentication. (HTTP 401)

[Sylvain Bauza (sylvain-bauza)]

Looks like something bad happened to you when finishing the live migration :

The exception you see comes from https://github.com/openstack/nova/blob/07939afe3e60ef6061aa89ef408dc72785f3d8dd/nova/virt/libvirt/driver.py#L9225-L9228 which means something got bad during the live migration process.

Based on the log above, we can see it was happening in https://github.com/openstack/nova/blob/07939afe3e60ef6061aa89ef408dc72785f3d8dd/nova/virt/libvirt/driver.py#L9130-L9135 which is at the very end of the migration itself.

As you can also see, we call post_live_migration() which runs https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L8384 and later fails when calling either Neutron or Cinder.

Could you please check the neutron-api and cinder-api logs to see a potential HTTP401 failure at the same time, and then you'll see that probably there is some misconfiguration.

[Launchpad Janitor (janitor)]

[Expired for OpenStack Compute (nova) because there has been no activity for 60 days.]