affinity filter not work on master branch

[Jiang]

I deploy openstack with devstack, use master branchs. But the affinity filter not work.

There is only one compute node in my environment. I create a server group with anti-affinity. And then, I create a vm with anti-affinify, the vm create success. But, when I create the second vm, it should be failed, but it success.

I print the spec_obj.instance_group.hosts to log, but I get empty list. So, when I create the second vm, antiaffinity should return False, but return True. This is a problem.

The operations see nova.log.

[Matt Riedemann (mriedem)]

Tempest tests against the ServerGroup(Anti)AffinityFilters, so I'm not sure what you're seeing here. Can you give a specific set of recreate steps for what you're trying to do?

[Matt Riedemann (mriedem)]

@Jiang, I marked this as incomplete because I need more information / answers to my questions in comment 2.

[Jiang (jiangpf)]

[root@greenvm-j14070v2 nova]# openstack image list
+--------------------------------------+--------------------------+--------+
| ID | Name | Status |
+--------------------------------------+--------------------------+--------+
| b11c3607-a353-4d1a-880d-5ee117d19a8a | cirros-0.3.5-x86_64-disk | active |
+--------------------------------------+--------------------------+--------+
[root@greenvm-j14070v2 nova]# openstack flavor list
+----+-----------+-------+------+-----------+-------+-----------+
| ID | Name | RAM | Disk | Ephemeral | VCPUs | Is Public |
+----+-----------+-------+------+-----------+-------+-----------+
| 1 | m1.tiny | 512 | 1 | 0 | 1 | True |
| 2 | m1.small | 2048 | 20 | 0 | 1 | True |
| 3 | m1.medium | 4096 | 40 | 0 | 2 | True |
| 4 | m1.large | 8192 | 80 | 0 | 4 | True |
| 42 | m1.nano | 64 | 0 | 0 | 1 | True |
| 5 | m1.xlarge | 16384 | 160 | 0 | 8 | True |
| 84 | m1.micro | 128 | 0 | 0 | 1 | True |
| c1 | cirros256 | 256 | 0 | 0 | 1 | True |
| d1 | ds512M | 512 | 5 | 0 | 1 | True |
| d2 | ds1G | 1024 | 10 | 0 | 1 | True |
| d3 | ds2G | 2048 | 10 | 0 | 2 | True |
| d4 | ds4G | 4096 | 20 | 0 | 4 | True |
+----+-----------+-------+------+-----------+-------+-----------+
[root@greenvm-j14070v2 nova]# openstack network list
+--------------------------------------+---------+----------------------------------------------------------------------------+
| ID | Name | Subnets |
+--------------------------------------+---------+----------------------------------------------------------------------------+
| dca2e6cc-d3e7-49a5-8a0b-ec17acf9034b | private | 5ccfe577-8128-432c-8980-ca2362ef0ab8, 7dc5ce3e-18fa-417b-8eeb-3e9dc341b481 |
| f6b11b0c-b555-45c2-b0ea-ff5b9d22cbed | public | 398607a4-c906-4807-a667-7c83258aabe0, d101eadc-4092-42f6-8ac9-11d668a0374b |
+--------------------------------------+---------+----------------------------------------------------------------------------+
[root@greenvm-j14070v2 nova]# openstack server group list
+--------------------------------------+---------------+---------------+
| ID | Name | Policies |
+--------------------------------------+---------------+---------------+
| ed739005-fa01-4911-b7be-0f5657a62614 | anti-affinity | anti-affinity |
+--------------------------------------+---------------+---------------+
[root@greenvm-j14070v2 nova]# openstack server create --network dca2e6cc-d3e7-49a5-8a0b-ec17acf9034b --image b11c3607-a353-4d1a-880d-5ee117d19a8a --flavor m1.nano --hint group=ed739005-fa01-4911-b7be-0f5657a62614 cirros-01
+-------------------------------------+-----------------------------------------------------------------+
| Field | Value ...

[liuzhuangzhuang (liuzhuangzhuang)]

@Jiang, Does the enabled_filters option in the controller node's nova configuration file contains the ServerGroupAntiAffinityFilter?

[Jiang (jiangpf)]

@liuzhuangzhuang, I deployed openstack with devstack. The enabled_filters is bellow:
enabled_filters = RetryFilter,AvailabilityZoneFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,ServerGroupAntiAffinityFilter,ServerGroupAffinityFilter,SameHostFilter,DifferentHostFilter

[Balazs Gibizer (balazs-gibizer)]

I think I was able to reproduce the problem in a fresh single node devstack [1].

The GroupAntiAffinityFilter printed the following when scheduled the second instance:

Group anti affinity: check if ubuntu not in [] {{(pid=25524) host_passes /opt/stack/nova/nova/scheduler/filters/affinity_filter.py:105}}

This means that the instance_group.hosts field is empty during the second scheduling towards the same host the first instance is already running on [2].

It felt like I'm booting the two instances two quickly and hitting a race. So I waited couple of minutes and tried a 3rd instance. But that also placed to the same (and only) compute host.

[1] http://paste.openstack.org/show/725339/
[2] https://github.com/openstack/nova/blob/fcf406003bcde2b60a515f694f2640e27a9d65fd/nova/scheduler/filters/affinity_filter.py#L101-L105

[Balazs Gibizer (balazs-gibizer)]

After some debugging it seems that scheduler_utils.setup_instance_group(context, request_spec) in the conductor [3] returns empty host list because the instance query in [4] returns 0 instances for a non empty uuids filter.

Jul 08 19:43:18 ubuntu nova-conductor[7138]: INFO nova.objects.instance_group [None req-6f7cea32-7573-41ee-8eea-0383c5ade138 admin admin] !!! filters {'deleted': False, 'uuid': [u'bbb51826-a9fa-4b66-b976-e9cf1c75175d', u'e6d4be63-8e4e-4be7-ba41-4b56d234a1b4', u'60eb2895-b098-4d5e-b5b2-d9a0686f39bd', u'4c2abe15-40a1-49f7-ada4-11fa9ad7dfef', u'e869bc16-40f8-4fd1-a738-c3015b999cef', u'eeefa697-95ef-445e-985c-da2ae9e6f9e2', u'65614545-9425-4531-8b17-442c6a18dffd', u'9a2c1e5e-0f84-49de-b42e-7baf7935a139', u'c1e22c58-307a-4a67-8e58-90651d42d8e9', u'6680b18b-6e3b-4934-9679-97a61cee35b2', u'd2ac7593-48e0-4aea-813b-b4133acacc53', u'5085065c-a514-4d2b-8060-9ed12f4cb947', u'45b5c17f-b65f-4e1a-abd7-b5b079a4eb60']}
Jul 08 19:43:18 ubuntu nova-conductor[7138]: INFO nova.objects.instance_group [None req-6f7cea32-7573-41ee-8eea-0383c5ade138 admin admin] !!! instances: 0
Jul 08 19:43:18 ubuntu nova-conductor[7138]: INFO nova.scheduler.utils [None req-6f7cea32-7573-41ee-8eea-0383c5ade138 admin admin] !!! group_hosts set([])
Jul 08 19:43:18 ubuntu nova-conductor[7138]: INFO nova.scheduler.utils [None req-6f7cea32-7573-41ee-8eea-0383c5ade138 admin admin] !!! group_info GroupDetails(hosts=set([]), policies=[u'anti-affinity'], members=[u'bbb51826-a9fa-4b66-b976-e9cf1c75175d', u'e6d4be63-8e4e-4be7-ba41-4b56d234a1b4', u'60eb2895-b098-4d5e-b5b2-d9a0686f39bd', u'4c2abe15-40a1-49f7-ada4-11fa9ad7dfef', u'e869bc16-40f8-4fd1-a738-c3015b999cef', u'eeefa697-95ef-445e-985c-da2ae9e6f9e2', u'65614545-9425-4531-8b17-442c6a18dffd', u'9a2c1e5e-0f84-49de-b42e-7baf7935a139', u'c1e22c58-307a-4a67-8e58-90651d42d8e9', u'6680b18b-6e3b-4934-9679-97a61cee35b2', u'd2ac7593-48e0-4aea-813b-b4133acacc53', u'5085065c-a514-4d2b-8060-9ed12f4cb947', u'45b5c17f-b65f-4e1a-abd7-b5b079a4eb60'])
Jul 08 19:43:18 ubuntu nova-conductor[7138]: INFO nova.conductor.manager [None req-6f7cea32-7573-41ee-8eea-0383c5ade138 admin admin] group members=[u'bbb51826-a9fa-4b66-b976-e9cf1c75175d', u'e6d4be63-8e4e-4be7-ba41-4b56d234a1b4', u'60eb2895-b098-4d5e-b5b2-d9a0686f39bd', u'4c2abe15-40a1-49f7-ada4-11fa9ad7dfef', u'e869bc16-40f8-4fd1-a738-c3015b999cef', u'eeefa697-95ef-445e-985c-da2ae9e6f9e2', u'65614545-9425-4531-8b17-442c6a18dffd', u'9a2c1e5e-0f84-49de-b42e-7baf7935a139', u'c1e22c58-307a-4a67-8e58-90651d42d8e9', u'6680b18b-6e3b-4934-9679-97a61cee35b2', u'd2ac7593-48e0-4aea-813b-b4133acacc53', u'5085065c-a514-4d2b-8060-9ed12f4cb947', u'45b5c17f-b65f-4e1a-abd7-b5b079a4eb60'], group hosts=[]

[3] https://github.com/openstack/nova/blob/c0350da4a1607d7aa113caceaefb5d29303c7eed/nova/conductor/manager.py#L711-L713
[4] https://github.com/openstack/nova/blob/c0350da4a1607d7aa113caceaefb5d29303c7eed/nova/objects/instance_group.py#L422

[Balazs Gibizer (balazs-gibizer)]

I went back to 18.0.0b1 and still able to reproduce the problem. It feels strange.

[Balazs Gibizer (balazs-gibizer)]

Regarding the tempest test test_create_server_with_scheduler_hint_group_anti_affinity it does pass but it boots the two instance with a single requests[1]. This means there is a single scheduling attempt not two.

In my devstack the single request case works properly so I conclude that the tempest test does not exercise the same scenario that the reporter and I see failing.

[1] https://github.com/openstack/tempest/blob/bcb6cc6184b6c847f6e72305f52b886835a6ca90/tempest/api/compute/admin/test_servers_on_multinodes.py#L50

[Balazs Gibizer (balazs-gibizer)]

I pushed the tempest modification that does the two server create in two separate request: https://review.openstack.org/#/c/581349/

[Balazs Gibizer (balazs-gibizer)]

and the new tempest test reproduces the problem reported in this bug: http://logs.openstack.org/49/581349/1/check/tempest-multinode-full/0ac326a/testr_results.html.gz

[Matt Riedemann (mriedem)]

This might overlap with bug 1746863.

[Balazs Gibizer (balazs-gibizer)]

There is a pretty similar bug that is already being worked on https://bugs.launchpad.net/nova/+bug/1746863