VMs are going into error state when nova host aggregation is used to boot VMs

[chandrasekaran natarajan]

VMs are going into error state when nova host aggregation is used to boot VMs.i have added host aggregation and added compute hosts to it.The meta data key-value is added in ext_specs of flavor.

Setup used: Devstack-master branch

In nova.conf, have the below config for filters

scheduler_default_filters = RetryFilter,AvailabilityZoneFilter,RamFilter,DiskFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,ServerGroupAntiAffinityFilter,ServerGroupAffinityFilter,SameHostFilter,DifferentHostFilter,AggregateInstanceExtraSpecsFilter
scheduler_driver = filter_scheduler

Steps:

1. Create an aggregate

stack@devstack:~/devstack$ nova aggregate-create test
+----+------+-------------------+-------+----------+
| Id | Name | Availability Zone | Hosts | Metadata |
+----+------+-------------------+-------+----------+
| 4 | test | - | | |
+----+------+-------------------+-------+----------+

2.Set a meta-data “family=intel” to the created aggregate

stack@devstack:~/devstack$ nova aggregate-set-metadata 4 family=intel
Metadata has been successfully updated for aggregate 4.
+----+------+-------------------+-------+----------------+
| Id | Name | Availability Zone | Hosts | Metadata |
+----+------+-------------------+-------+----------------+
| 4 | test | - | | 'family=intel' |
+----+------+-------------------+-------+----------------+

3. Add a host(devstack) to this aggregate

stack@devstack:~/devstack$ nova aggregate-add-host 4 devstack
Host devstack has been successfully added for aggregate 4
+----+------+-------------------+------------+----------------+
| Id | Name | Availability Zone | Hosts | Metadata |
+----+------+-------------------+------------+----------------+
| 4 | test | - | 'devstack' | 'family=intel' |
+----+------+-------------------+------------+----------------+

4. Create a new flavor

stack@devstack:~/devstack$ nova flavor-create testflavor1 auto 4096 10 2 --is-public true
+--------------------------------------+-------------+-----------+------+-----------+------+-------+-------------+-----------+
| ID | Name | Memory_MB | Disk | Ephemeral | Swap | VCPUs | RXTX_Factor | Is_Public |
+--------------------------------------+-------------+-----------+------+-----------+------+-------+-------------+-----------+
| 75a142d8-3124-4fc9-97b3-9b1db1469d76 | testflavor1 | 4096 | 10 | 0 | | 2 | 1.0 | True |
+--------------------------------------+-------------+-----------+------+-----------+------+-------+-------------+-----------+

5.Create flavor key-value to match meta-data of the aggregate

stack@devstack:~/devstack$ nova flavor-key testflavor1 set family=intel

stack@devstack:~/devstack$ nova flavor-list
+--------------------------------------+-------------+-----------+------+-----------+------+-------+-------------+-----------+
| ID | Name | Memory_MB | Disk | Ephemeral | Swap | VCPUs | RXTX_Factor | Is_Public |
+--------------------------------------+-------------+-----------+------+-----------+------+-------+-------------+-----------+
| 1 | m1.tiny | 512 | 1 | 0 | | 1 | 1.0 | True |
| 2 | m1.small | 2048 | 20 | 0 | | 1 | 1.0 | True |
| 3 | m1.medium | 4096 | 40 | 0 | | 2 | 1.0 | True |
| 4 | m1.large | 8192 | 80 | 0 | | 4 | 1.0 | True |
| 42 | m1.nano | 64 | 0 | 0 | | 1 | 1.0 | True |
| 5 | m1.xlarge | 16384 | 160 | 0 | | 8 | 1.0 | True |
| 75a142d8-3124-4fc9-97b3-9b1db1469d76 | testflavor1 | 4096 | 10 | 0 | | 2 | 1.0 | True |
| 84 | m1.micro | 128 | 0 | 0 | | 1 | 1.0 | True |
| c1 | cirros256 | 256 | 0 | 0 | | 1 | 1.0 | True |
| d1 | ds512M | 512 | 5 | 0 | | 1 | 1.0 | True |
| d2 | ds1G | 1024 | 10 | 0 | | 1 | 1.0 | True |
| d3 | ds2G | 2048 | 10 | 0 | | 2 | 1.0 | True |
| d4 | ds4G | 4096 | 20 | 0 | | 4 | 1.0 | True |
+--------------------------------------+-------------+-----------+------+-----------+------+-------+-------------+-----------+

6.From the nova flavor-show, we can see the key-value is in “extra_specs” already

stack@devstack:~/devstack$ nova flavor-show testflavor1
+----------------------------+--------------------------------------+
| Property | Value |
+----------------------------+--------------------------------------+
| OS-FLV-DISABLED:disabled | False |
| OS-FLV-EXT-DATA:ephemeral | 0 |
| disk | 10 |
| extra_specs | {"family": "intel"} |
| id | 75a142d8-3124-4fc9-97b3-9b1db1469d76 |
| name | testflavor1 |
| os-flavor-access:is_public | True |
| ram | 4096 |
| rxtx_factor | 1.0 |
| swap | |
| vcpus | 2 |
+----------------------------+--------------------------------------+
stack@devstack:~/devstack$

7. Now boot a VM with the flavor

stack@devstack:~/devstack$ nova boot --image cirros-0.3.4-x86_64-uec --flavor testflavor1 --nic net-name=private vm-test +--------------------------------------+----------------------------------------------------------------+
| Property | Value |
+--------------------------------------+----------------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | |
| OS-EXT-SRV-ATTR:host | - |
| OS-EXT-SRV-ATTR:hostname | vm-test |
| OS-EXT-SRV-ATTR:hypervisor_hostname | - |
| OS-EXT-SRV-ATTR:instance_name | instance-00000016 |
| OS-EXT-SRV-ATTR:kernel_id | 6c74a0f6-b54a-4604-9d6f-9e3886a811ee |
| OS-EXT-SRV-ATTR:launch_index | 0 |
| OS-EXT-SRV-ATTR:ramdisk_id | 2728b402-2af0-4b01-91ae-0472ecb34a01 |
| OS-EXT-SRV-ATTR:reservation_id | r-6n0igzwj |
| OS-EXT-SRV-ATTR:root_device_name | - |
| OS-EXT-SRV-ATTR:user_data | - |
| OS-EXT-STS:power_state | 0 |
| OS-EXT-STS:task_state | scheduling |
| OS-EXT-STS:vm_state | building |
| OS-SRV-USG:launched_at | - |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| adminPass | T8PaX8G9F6rF |
| config_drive | |
| created | 2016-08-30T10:24:03Z |
| description | - |
| flavor | testflavor1 (75a142d8-3124-4fc9-97b3-9b1db1469d76) |
| hostId | |
| host_status | |
| id | 238b7482-8c94-427c-b073-dd1c922c572c |
| image | cirros-0.3.4-x86_64-uec (bc8b111e-e312-4124-a3ba-64e7a2376612) |
| key_name | - |
| locked | False |
| metadata | {} |
| name | vm-test |
| os-extended-volumes:volumes_attached | [] |
| progress | 0 |
| security_groups | default |
| status | BUILD |
| tags | [] |
| tenant_id | b26794c2d89748e9996d767a4ccfc5d0 |
| updated | 2016-08-30T10:24:04Z |
| user_id | 2398670e99844bd5ba30d6b7d3cdeaa5 |
+--------------------------------------+----------------------------------------------------------------+
stack@devstack:~/devstack$ nova show vm-test
+--------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Property | Value |
+--------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | |
| OS-EXT-SRV-ATTR:host | - |
| OS-EXT-SRV-ATTR:hostname | vm-test |
| OS-EXT-SRV-ATTR:hypervisor_hostname | - |
| OS-EXT-SRV-ATTR:instance_name | instance-00000016 |
| OS-EXT-SRV-ATTR:kernel_id | 6c74a0f6-b54a-4604-9d6f-9e3886a811ee |
| OS-EXT-SRV-ATTR:launch_index | 0 |
| OS-EXT-SRV-ATTR:ramdisk_id | 2728b402-2af0-4b01-91ae-0472ecb34a01 |
| OS-EXT-SRV-ATTR:reservation_id | r-6n0igzwj |
| OS-EXT-SRV-ATTR:root_device_name | - |
| OS-EXT-SRV-ATTR:user_data | - |
| OS-EXT-STS:power_state | 0 |
| OS-EXT-STS:task_state | - |
| OS-EXT-STS:vm_state | error |
| OS-SRV-USG:launched_at | - |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| config_drive | |
| created | 2016-08-30T10:24:03Z |
| description | - |
| fault | {"message": "No valid host was found. There are not enough hosts available.", "code": 500, "details": " File \"/opt/stack/nova/nova/conductor/manager.py\", line 484, in build_instances |
| | context, request_spec, filter_properties) |
| | File \"/opt/stack/nova/nova/conductor/manager.py\", line 541, in _schedule_instances |
| | hosts = self.scheduler_client.select_destinations(context, spec_obj) |
| | File \"/opt/stack/nova/nova/scheduler/utils.py\", line 370, in wrapped |
| | return func(*args, **kwargs) |
| | File \"/opt/stack/nova/nova/scheduler/client/__init__.py\", line 51, in select_destinations |
| | return self.queryclient.select_destinations(context, spec_obj) |
| | File \"/opt/stack/nova/nova/scheduler/client/__init__.py\", line 37, in __run_method |
| | return getattr(self.instance, __name)(*args, **kwargs) |
| | File \"/opt/stack/nova/nova/scheduler/client/query.py\", line 32, in select_destinations |
| | return self.scheduler_rpcapi.select_destinations(context, spec_obj) |
| | File \"/opt/stack/nova/nova/scheduler/rpcapi.py\", line 126, in select_destinations |
| | return cctxt.call(ctxt, 'select_destinations', **msg_args) |
| | File \"/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py\", line 169, in call |
| | retry=self.retry) |
| | File \"/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py\", line 97, in _send |
| | timeout=timeout, retry=retry) |
| | File \"/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py\", line 464, in send |
| | retry=retry) |
| | File \"/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py\", line 455, in _send |
| | raise result |
| | ", "created": "2016-08-30T10:24:04Z"} |
| flavor | testflavor1 (75a142d8-3124-4fc9-97b3-9b1db1469d76) |
| hostId | |
| host_status | |
| id | 238b7482-8c94-427c-b073-dd1c922c572c |
| image | cirros-0.3.4-x86_64-uec (bc8b111e-e312-4124-a3ba-64e7a2376612) |
| key_name | - |
| locked | False |
| metadata | {} |
| name | vm-test |
| os-extended-volumes:volumes_attached | [] |
| status | ERROR |
| tags | [] |
| tenant_id | b26794c2d89748e9996d767a4ccfc5d0 |
| updated | 2016-08-30T10:24:04Z |
| user_id | 2398670e99844bd5ba30d6b7d3cdeaa5 |
+--------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
stack@devstack:~/devstack$

Attached devstack logs

[melanie witt (melwitt)]

There are a number of reasons scheduling could fail to find a valid host and we need to see the n-sch.log and n-cpu.log to see the details of what's going on. If you can attach those to this bug, we can look into it further.

[chandrasekaran natarajan (chandrasekaran-natarajan)]

Please find the log attached

[Sean Dague (sdague)]

There are no currently open reviews on this bug, changing
the status back to the previous state and unassigning. If
there are active reviews related to this bug, please include
links in comments.

[Launchpad Janitor (janitor)]

[Expired for OpenStack Compute (nova) because there has been no activity for 60 days.]