Containers: live migration does not honour server group (anti_affinity) policy

[Peng Peng]

Brief Description
-----------------
Boot up 2 VMs with server group (anti_affinity) policy, after 2 VMs live migrated, 2 VMs are landed in same host.

Severity
--------
Major

Steps to Reproduce
------------------
Boot up 2 VMs with server group (anti_affinity) policy,
live migration 2 VMs,

Expected Behaviour
------------------
VMs honour to server group (anti_affinity) policy and landed in different hosts

Actual Behaviour
----------------
2 VMs are landed in same host.

Reproducibility
---------------
Reproducible
7 failed in 10 runs

System Configuration
--------------------
Multi-node system

Branch/Pull Time/Commit
-----------------------
master as of 20190318T233000Z

Timestamp/Logs
--------------

[2019-03-19 11:56:17,442] 262 DEBUG MainThread ssh.send :: Send 'nova --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne show ef730c32-b656-450a-865c-f76f40a52557'
[2019-03-19 11:56:18,938] 387 DEBUG MainThread ssh.expect :: Output:
+--------------------------------------+----------------------------------------------------------------------------------+
| Property | Value |
+--------------------------------------+----------------------------------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | compute-2

[2019-03-19 11:56:20,664] 262 DEBUG MainThread ssh.send :: Send 'nova --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne live-migration ef730c32-b656-450a-865c-f76f40a52557'

[2019-03-19 11:56:41,377] 262 DEBUG MainThread ssh.send :: Send 'nova --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne show ef730c32-b656-450a-865c-f76f40a52557'
[2019-03-19 11:56:42,921] 387 DEBUG MainThread ssh.expect :: Output:
+--------------------------------------+----------------------------------------------------------------------------------+
| Property | Value |
+--------------------------------------+----------------------------------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | compute-0

[2019-03-19 11:57:42,875] 262 DEBUG MainThread ssh.send :: Send 'nova --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne show ca9768b3-8146-464f-a830-d942f34cdfa0'
[2019-03-19 11:57:44,443] 387 DEBUG MainThread ssh.expect :: Output:
+--------------------------------------+----------------------------------------------------------------------------------+
| Property | Value |
+--------------------------------------+----------------------------------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | compute-1 |

[2019-03-19 11:57:46,186] 262 DEBUG MainThread ssh.send :: Send 'nova --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne live-migration ca9768b3-8146-464f-a830-d942f34cdfa0'

[2019-03-19 11:58:06,465] 262 DEBUG MainThread ssh.send :: Send 'nova --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne show ca9768b3-8146-464f-a830-d942f34cdfa0'
[2019-03-19 11:58:07,976] 387 DEBUG MainThread ssh.expect :: Output:
+--------------------------------------+----------------------------------------------------------------------------------+
| Property | Value |
+--------------------------------------+----------------------------------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | compute-0 |

[Ghada Khalil (gkhalil)]

This is a nova issue and will need to be reported as a nova launchpad.

[Bruce Jones (brucej)]

See https://bugs.launchpad.net/nova/+bug/1820018 for the related Nova upstream bug

[Ghada Khalil (gkhalil)]

Marking as release gating for stx given the severity of the issue (live migration not working with anti-affinity groups).
Needs follow-up with the nova community.

[yong hu (yhu6)]

I think this LP is duplicated to https://bugs.launchpad.net/bugs/1821755, which was confirmed a Nova upstream issue.
@Ghada, if you don't mind, I can duplicate this one to 1821755.

[yong hu (yhu6)]

The live-migrations on 2 VMs were taking place within short interval (< 2 mins), and it triggered a Nova upstream issue confirmed in https://bugs.launchpad.net/bugs/1821755.

[Boxiang Zhu (bxzhu-5355)]

I have test this case in stx.2.0 tag. And the issue still remains. I have post a patch to fix this issue to nova upstream and the patch is still in review.

[Matt Riedemann (mriedem)]

Has anyone from starlingx tested the proposed fix for bug 1821755 (of which this bug is a duplicate?): https://review.opendev.org/#/c/651969/ (note that the patch [PS10] is currently failing unit test jobs).