"No valid host" error in periodic CI jobs

Bug #1567395 reported by Sagi (Sergey) Shnaidman
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
High
Unassigned

Bug Description

Starting from Mar 22 most periodic jobs fail because "No valid host" error in overcloud deployment.
It's true for ha, nonha, upgrades jobs, liberty and mitaka jobs are passing.

For example:
https://jenkins02.openstack.org/job/periodic-tripleo-ci-f22-nonha/21/
http://logs.openstack.org/periodic/periodic-tripleo-ci-f22-nonha/4f4e040/

In /var/log/nova/nova-scheduler.log you can see filtering out nodes because of memory.

016-04-06 07:17:29.377 18537 DEBUG nova.scheduler.filters.ram_filter [req-0f6fccae-3306-4b25-aa1a-a72883b9f351 b4b327e5307a4a4b80ac78f80d467e29 372d29d4f54444cbb8c076c05dcb23a6 - - -] (instack.localdomain, dd26ed80-35ec-4e7a-888a-f83592576b43) ram: 1024MB disk: 1024MB io_ops: 0 instances: 0 does not have 4096 MB usable ram, it only has 1024.0 MB usable ram. host_passes /usr/lib/python2.7/site-packages/nova/scheduler/filters/ram_filter.py:59
2016-04-06 07:35:03.386 18537 DEBUG nova.scheduler.filters.ram_filter [req-bf253f05-4618-4b33-967c-6c96e85cdfe6 b4b327e5307a4a4b80ac78f80d467e29 372d29d4f54444cbb8c076c05dcb23a6 - - -] (instack.localdomain, bfc1
048c-96e1-45d2-aae0-3090480c6c65) ram: 0MB disk: 1024MB io_ops: 0 instances: 0 does not have 4096 MB usable ram, it only has 0.0 MB usable ram. host_passes /usr/lib/python2.7/site-packages/nova/scheduler/filters/ram_filter.py:59

Although from http://logs.openstack.org/periodic/periodic-tripleo-ci-f22-nonha/4f4e040/console.html:
you can see the configurations of hosts that have 5120 MB memory:

2016-04-06 06:10:50.373 | "cpu": "1",
2016-04-06 06:10:50.373 | "memory": "5120",
2016-04-06 06:10:50.373 | "disk": "40",
2016-04-06 06:10:50.373 | "arch": "amd64",
2016-04-06 06:10:50.373 | "pm_user": "root",
2016-04-06 06:10:50.374 | "pm_addr": "192.168.1.95"
2016-04-06 06:10:50.374 | },
2016-04-06 06:10:50.374 | {
2016-04-06 06:10:50.374 | "pm_type": "pxe_ssh",
2016-04-06 06:10:50.374 | "mac": [
2016-04-06 06:10:50.374 | "00:ea:f0:b7:ca:0c"
2016-04-06 06:10:50.375 | ],

There are many errors "InstanceDeployFailure: Failed to provision instance" in nova-compute.log:
https://paste.fedoraproject.org/350971/30745146/

also found erros, but not sure they're fatal:

https://paste.fedoraproject.org/350968/
ironic-inspector/ironic-inspector.log :
Preprocessing hook validate_interfaces: No suitable interfaces found

Tags: ci Edit Tag help
Revision history for this message
Sagi (Sergey) Shnaidman (sshnaidm) wrote :
Revision history for this message
Derek Higgins (derekh) wrote :
Changed in tripleo:
status: New → Fix Released
importance: Undecided → High
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers