Failing to launch instances : Filter ComputeCapabilitiesFilter returned 0 hosts

[Derek Higgins]

Failure started between 1 and 2 AM UTC

Running nova in debug mode shows the problem

Jun 04 09:15:55 localhost nova-scheduler[9605]: 2014-06-04 09:15:55.259 9605 DEBUG nova.filters [req-c37d26da-66de-4658-ba6f-a06a775f1a28 None] Filter ComputeFilter returned 1 host(s) get_filtered_objects /opt/stack/venvs/nova/lib/python2.7/site-packages/nova/filters.py:88
Jun 04 09:15:55 localhost nova-scheduler[9605]: 2014-06-04 09:15:55.259 9605 DEBUG nova.scheduler.filters.compute_capabilities_filter [req-c37d26da-66de-4658-ba6f-a06a775f1a28 None] (seed, 8f3d2259-ef0b-44fc-a0c4-4d5cc2ef1443) ram:3072 disk:40960 io_ops:0 instances:0 fails instance_type extra_specs requirements host_passes /opt/stack/venvs/nova/lib/python2.7/site-packages/nova/scheduler/filters/compute_capabilities_filter.py:72
Jun 04 09:15:55 localhost nova-scheduler[9605]: 2014-06-04 09:15:55.260 9605 INFO nova.filters [req-c37d26da-66de-4658-ba6f-a06a775f1a28 None] Filter ComputeCapabilitiesFilter returned 0 hosts

[Derek Higgins (derekh)]

A test revert local to tripleo has passed CI (first pass we've seen in 10 hours about 30 runs ) https://review.openstack.org/#/c/97732/

[Hans Lindgren (hanlind)]

From the debug above it looks like some host stats are not reported back to the scheduler. Host_state only have
ram:3072 disk:40960 io_ops:0 instances:0

It should have stats with extra_specs like cpu_arch in it as well.

Is this using ironic or nova-baremetal driver?

[Derek Higgins (derekh)]

Its nova-baremetal

[Hans Lindgren (hanlind)]

Likely cause:

The capabilities cleanup that merged in Nova resolved the issue with wrong HostState class being used. Now that BaremetalNodeState is used, it must provide the same stats info HostState does, which is currently missing.

(btw. Ignore the comment on the debug log as the full host_state is not shown in the log)

[Hans Lindgren (hanlind)]

Hopefully this patch will solve this
https://review.openstack.org/97806

[Gregory Haynes (greghaynes)]

https://review.openstack.org/97806 solves the issue in TripleO for me.

[aeva black (tenbrae)]

Nova hasn't approved that yet, and is instead moving to revert hanlind's original patch to unbreak nova_bm and ironic in the short term.

See https://review.openstack.org/#/c/97757/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/97757
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=61453f3b991c7955cb7321c6ef85f3fa659c2a82
Submitter: Jenkins
Branch: master

commit 61453f3b991c7955cb7321c6ef85f3fa659c2a82
Author: Derek Higgins <email address hidden>
Date: Wed Jun 4 12:05:04 2014 +0100

    Revert "Remove traces of now unused host capabilities from scheduler"

    This reverts commit ce3f9e5fa9cd05f3ee3bb0cc7d06521d05901cf4.

    This commit caused problems in tripleo, nova-bm isn't able to launch
    instance as they are being filtered out by ComputeCapabilitiesFilter.

    Fixes bug #1326289

    Change-Id: Icaae95eea78eff11cee34e8a3875d62b3cce0b60