Scheduler RetryFilter returns 0 hosts

[Ed Cranford]

Clean vagrant up, tried to build examples/plans/ghost.yaml and deploy failed. Here's a snippet from nova-scheduler, showing RetryFilter kicking our one host out of the running:

2014-08-04 18:02:15.866 DEBUG nova.filters [req-4693fa95-501c-4d5f-b106-8e9cf0ea56de demo demo] Starting with 1 host(s) from (pid=6840) get_filtered_objects /opt/stack/nova/nova/filters.py:70
2014-08-04 18:02:15.867 DEBUG nova.filters [req-4693fa95-501c-4d5f-b106-8e9cf0ea56de demo demo] Filter RetryFilter returned 1 host(s) from (pid=6840) get_filtered_objects /opt/stack/nova/nova/filters.py:88
2014-08-04 18:02:15.867 DEBUG nova.filters [req-4693fa95-501c-4d5f-b106-8e9cf0ea56de demo demo] Filter AvailabilityZoneFilter returned 1 host(s) from (pid=6840) get_filtered_objects /opt/stack/nova/nova/filters.py:88
2014-08-04 18:02:15.869 DEBUG nova.filters [req-4693fa95-501c-4d5f-b106-8e9cf0ea56de demo demo] Filter RamFilter returned 1 host(s) from (pid=6840) get_filtered_objects /opt/stack/nova/nova/filters.py:88
2014-08-04 18:02:15.870 DEBUG nova.filters [req-4693fa95-501c-4d5f-b106-8e9cf0ea56de demo demo] Filter ComputeFilter returned 1 host(s) from (pid=6840) get_filtered_objects /opt/stack/nova/nova/filters.py:88
2014-08-04 18:02:15.870 DEBUG nova.filters [req-4693fa95-501c-4d5f-b106-8e9cf0ea56de demo demo] Filter ComputeCapabilitiesFilter returned 1 host(s) from (pid=6840) get_filtered_objects /opt/stack/nova/nova/filters.py:88
2014-08-04 18:02:15.871 DEBUG nova.filters [req-4693fa95-501c-4d5f-b106-8e9cf0ea56de demo demo] Filter ImagePropertiesFilter returned 1 host(s) from (pid=6840) get_filtered_objects /opt/stack/nova/nova/filters.py:88
2014-08-04 18:02:16.000 INFO nova.openstack.common.periodic_task [req-4693fa95-501c-4d5f-b106-8e9cf0ea56de demo demo] Skipping periodic task _periodic_update_dns because its interval is negative
2014-08-04 18:02:16.022 DEBUG nova.filters [req-4693fa95-501c-4d5f-b106-8e9cf0ea56de demo demo] Filter ServerGroupAntiAffinityFilter returned 1 host(s) from (pid=6840) get_filtered_objects /opt/stack/nova/nova/filters.py:88
2014-08-04 18:02:16.023 DEBUG nova.filters [req-4693fa95-501c-4d5f-b106-8e9cf0ea56de demo demo] Filter ServerGroupAffinityFilter returned 1 host(s) from (pid=6840) get_filtered_objects /opt/stack/nova/nova/filters.py:88
2014-08-04 18:02:16.023 DEBUG nova.scheduler.filter_scheduler [req-4693fa95-501c-4d5f-b106-8e9cf0ea56de demo demo] Filtered [(devstack, devstack) ram:5458 disk:27648 io_ops:0 instances:0] from (pid=6840) _schedule /opt/stack/nova/nova/scheduler/filter_scheduler.py:280
2014-08-04 18:02:16.024 DEBUG nova.scheduler.filter_scheduler [req-4693fa95-501c-4d5f-b106-8e9cf0ea56de demo demo] Weighed [WeighedHost [host: devstack, weight: 1.0]] from (pid=6840) _schedule /opt/stack/nova/nova/scheduler/filter_scheduler.py:285
2014-08-04 18:02:16.024 INFO oslo.messaging._drivers.impl_rabbit [req-4693fa95-501c-4d5f-b106-8e9cf0ea56de demo demo] Connecting to AMQP server on 10.0.2.15:5672
2014-08-04 18:02:16.033 INFO oslo.messaging._drivers.impl_rabbit [req-4693fa95-501c-4d5f-b106-8e9cf0ea56de demo demo] Connected to AMQP server on 10.0.2.15:5672
2014-08-04 18:02:17.492 DEBUG nova.filters [req-4693fa95-501c-4d5f-b106-8e9cf0ea56de demo demo] Starting with 1 host(s) from (pid=6840) get_filtered_objects /opt/stack/nova/nova/filters.py:70
2014-08-04 18:02:17.492 DEBUG nova.scheduler.filters.retry_filter [req-4693fa95-501c-4d5f-b106-8e9cf0ea56de demo demo] Host [u'devstack', u'devstack'] fails. Previously tried hosts: [[u'devstack', u'devstack']] from (pid=6840) host_passes /opt/stack/nova/nova/scheduler/filters/retry_filter.py:42
2014-08-04 18:02:17.492 INFO nova.filters [req-4693fa95-501c-4d5f-b106-8e9cf0ea56de demo demo] Filter RetryFilter returned 0 hosts

[Adrian Otto (aotto)]

This sounds like the nova-compute service did not start. This sounds like what Dimitry mentioned this afternoon. Do you have reason to believe this is a Solum bug? If not, we should find a way to submit this as a devstack bug, right?

As soon as we can tie this back to a Solum issue, I'm happy to continue with triage on it.

[Ed Cranford (ed--cranford)]

I'll work to reproduce it again.

I don't think nova-compute failed to start, because the filters that happen before RetryFilter are each returning 1 valid host.
I'm not certain yet if it is a Solum problem or not, but I'll try to get to the bottom of it.

[Maxim Nestratov (mnestratov)]

Hi Ed,

I've just encountered exactly the same issue on my devstack environment. Did you have a chance to sort it out? Please share details if any.

[Ed Cranford (ed--cranford)]

Unfortunately I haven't found the time to try to reproduce or solve this yet.

[Devdatta Kulkarni (devdatta-kulkarni)]

Marking it as invalid as I haven't encountered this in a while.