Scheduler RetryFilter returns 0 hosts

Bug #1352496 reported by Ed Cranford on 2014-08-04
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Solum
Undecided
Unassigned

Bug Description

Clean vagrant up, tried to build examples/plans/ghost.yaml and deploy failed. Here's a snippet from nova-scheduler, showing RetryFilter kicking our one host out of the running:

2014-08-04 18:02:15.866 DEBUG nova.filters [req-4693fa95-501c-4d5f-b106-8e9cf0ea56de demo demo] Starting with 1 host(s) from (pid=6840) get_filtered_objects /opt/stack/nova/nova/filters.py:70
2014-08-04 18:02:15.867 DEBUG nova.filters [req-4693fa95-501c-4d5f-b106-8e9cf0ea56de demo demo] Filter RetryFilter returned 1 host(s) from (pid=6840) get_filtered_objects /opt/stack/nova/nova/filters.py:88
2014-08-04 18:02:15.867 DEBUG nova.filters [req-4693fa95-501c-4d5f-b106-8e9cf0ea56de demo demo] Filter AvailabilityZoneFilter returned 1 host(s) from (pid=6840) get_filtered_objects /opt/stack/nova/nova/filters.py:88
2014-08-04 18:02:15.869 DEBUG nova.filters [req-4693fa95-501c-4d5f-b106-8e9cf0ea56de demo demo] Filter RamFilter returned 1 host(s) from (pid=6840) get_filtered_objects /opt/stack/nova/nova/filters.py:88
2014-08-04 18:02:15.870 DEBUG nova.filters [req-4693fa95-501c-4d5f-b106-8e9cf0ea56de demo demo] Filter ComputeFilter returned 1 host(s) from (pid=6840) get_filtered_objects /opt/stack/nova/nova/filters.py:88
2014-08-04 18:02:15.870 DEBUG nova.filters [req-4693fa95-501c-4d5f-b106-8e9cf0ea56de demo demo] Filter ComputeCapabilitiesFilter returned 1 host(s) from (pid=6840) get_filtered_objects /opt/stack/nova/nova/filters.py:88
2014-08-04 18:02:15.871 DEBUG nova.filters [req-4693fa95-501c-4d5f-b106-8e9cf0ea56de demo demo] Filter ImagePropertiesFilter returned 1 host(s) from (pid=6840) get_filtered_objects /opt/stack/nova/nova/filters.py:88
2014-08-04 18:02:16.000 INFO nova.openstack.common.periodic_task [req-4693fa95-501c-4d5f-b106-8e9cf0ea56de demo demo] Skipping periodic task _periodic_update_dns because its interval is negative
2014-08-04 18:02:16.022 DEBUG nova.filters [req-4693fa95-501c-4d5f-b106-8e9cf0ea56de demo demo] Filter ServerGroupAntiAffinityFilter returned 1 host(s) from (pid=6840) get_filtered_objects /opt/stack/nova/nova/filters.py:88
2014-08-04 18:02:16.023 DEBUG nova.filters [req-4693fa95-501c-4d5f-b106-8e9cf0ea56de demo demo] Filter ServerGroupAffinityFilter returned 1 host(s) from (pid=6840) get_filtered_objects /opt/stack/nova/nova/filters.py:88
2014-08-04 18:02:16.023 DEBUG nova.scheduler.filter_scheduler [req-4693fa95-501c-4d5f-b106-8e9cf0ea56de demo demo] Filtered [(devstack, devstack) ram:5458 disk:27648 io_ops:0 instances:0] from (pid=6840) _schedule /opt/stack/nova/nova/scheduler/filter_scheduler.py:280
2014-08-04 18:02:16.024 DEBUG nova.scheduler.filter_scheduler [req-4693fa95-501c-4d5f-b106-8e9cf0ea56de demo demo] Weighed [WeighedHost [host: devstack, weight: 1.0]] from (pid=6840) _schedule /opt/stack/nova/nova/scheduler/filter_scheduler.py:285
2014-08-04 18:02:16.024 INFO oslo.messaging._drivers.impl_rabbit [req-4693fa95-501c-4d5f-b106-8e9cf0ea56de demo demo] Connecting to AMQP server on 10.0.2.15:5672
2014-08-04 18:02:16.033 INFO oslo.messaging._drivers.impl_rabbit [req-4693fa95-501c-4d5f-b106-8e9cf0ea56de demo demo] Connected to AMQP server on 10.0.2.15:5672
2014-08-04 18:02:17.492 DEBUG nova.filters [req-4693fa95-501c-4d5f-b106-8e9cf0ea56de demo demo] Starting with 1 host(s) from (pid=6840) get_filtered_objects /opt/stack/nova/nova/filters.py:70
2014-08-04 18:02:17.492 DEBUG nova.scheduler.filters.retry_filter [req-4693fa95-501c-4d5f-b106-8e9cf0ea56de demo demo] Host [u'devstack', u'devstack'] fails. Previously tried hosts: [[u'devstack', u'devstack']] from (pid=6840) host_passes /opt/stack/nova/nova/scheduler/filters/retry_filter.py:42
2014-08-04 18:02:17.492 INFO nova.filters [req-4693fa95-501c-4d5f-b106-8e9cf0ea56de demo demo] Filter RetryFilter returned 0 hosts

Adrian Otto (aotto) wrote :

This sounds like the nova-compute service did not start. This sounds like what Dimitry mentioned this afternoon. Do you have reason to believe this is a Solum bug? If not, we should find a way to submit this as a devstack bug, right?

As soon as we can tie this back to a Solum issue, I'm happy to continue with triage on it.

Changed in solum:
status: New → Triaged
status: Triaged → Incomplete
Ed Cranford (ed--cranford) wrote :

I'll work to reproduce it again.

I don't think nova-compute failed to start, because the filters that happen before RetryFilter are each returning 1 valid host.
I'm not certain yet if it is a Solum problem or not, but I'll try to get to the bottom of it.

Maxim Nestratov (mnestratov) wrote :

Hi Ed,

I've just encountered exactly the same issue on my devstack environment. Did you have a chance to sort it out? Please share details if any.

Ed Cranford (ed--cranford) wrote :

Unfortunately I haven't found the time to try to reproduce or solve this yet.

Marking it as invalid as I haven't encountered this in a while.

Changed in solum:
status: Incomplete → Invalid
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers