tripleo

did not reach state "available", the state is "clean wait", error: None'

Bug #1786109 reported by wes hayutin on 2018-08-08

This bug affects 3 people

Affects		Status	Importance	Assigned to	Milestone
	tripleo	Incomplete	Medium	wes hayutin

Bug Description

2018-08-08 17:49:37 | Introspection completed.
2018-08-08 17:49:37 | + openstack overcloud node provide --all-manageable
2018-08-08 17:49:40 | Waiting for messages on queue 'tripleo' with no timeout.
2018-08-08 19:07:06 | Connection is already closed.
2018-08-08 19:07:06 | [{}, {}, {}, {u'result': u'Node 28ff84cb-a833-48da-93ab-8ac7aea5e140 did not reach state "available", the state is "clean wait", error: None'}]

https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/3e961ea/logs/undercloud/home/zuul/overcloud_prep_images.log.txt.gz#_2018-08-08_19_07_06

2018-08-08 16:49:38.027118 | tripleo-ovb-centos-ovb-centos-7 | 2018-08-08 16:49:38.027239 | tripleo-ovb-centos-7 | 2018-08-08 16:49:38.027290 | tripleo-ovb-centos-7 | 2018-08-08 16:49:38.027339 | tripleo-ovb-centos-7 | 2018-08-08 16:49:38.027377 | tripleo-ovb-centos-7 | 2018-08-08 16:49:38.027428 | tripleo-ovb-centos-7 | 2018-08-08 16:49:38.027457 | tripleo-ovb-centos-7 | 2018-08-08 16:49:38.027500 | tripleo-ovb-centos-7 | 2018-08-08 16:49:38.027524 | tripleo-ovb-centos-7 | 2018-08-08 16:49:38.027583 | tripleo-ovb-centos-7 | 2018-08-08 16:49:38.027661 | tripleo-ovb-centos-7 | 2018-08-08 16:49:38.027700 | tripleo-ovb-centos-7 | 2018-08-08 16:49:38.027743 | tripleo-ovb-centos-7 | 2018-08-08 16:49:38.027777 | tripleo-ovb-centos-7 | 2018-08-08 16:49:38.027819 | tripleo-ovb-centos-7 | 2018-08-08 16:49:38.027843 | tripleo-ovb-centos-7 | 2018-08-08 16:49:38.027864 | tripleo-ovb-centos-7 | 2018-08-08 16:49:38.027917 | tripleo-ovb-centos-7 | 2018-08-08 16:49:38.027998 | tripleo-ovb-centos-7 | 2018-08-08 16:49:38.028060 | tripleo-ovb-centos-7 | 2018-08-08 16:49:38.028117 | tripleo-ovb-centos-7 | 2018-08-08 16:49:38.028149 | tripleo-ovb-centos-7 | 2018-08-08 16:49:38.028195 | tripleo-ovb-centos-7 | 2018-08-08 16:49:38.028235 | tripleo-ovb-centos-7 | 2018-08-08 16:49:38.028298 | tripleo-ovb-centos-7 | 2018-08-08 16:49:38.028350 | tripleo-ovb-centos-7 | 2018-08-08 16:49:38.028384 | tripleo-ovb-centos-7 | 2018-08-08 16:49:38.028426 | tripleo-ovb-centos-7 | 2018-08-08 16:49:38.028459 | tripleo-ovb-centos-7 | 2018-08-08 16:49:38.028501 | tripleo-ovb-centos-7 | 2018-08-08 16:49:38.028523 | tripleo-ovb-centos-7 | 2018-08-08 16:49:38.028544 | tripleo-ovb-centos-7 | 2018-08-08 16:49:38.028595 | tripleo-ovb-centos-7 | 2018-08-08 16:49:38.028671 | tripleo-ovb-centos-7 | 2018-08-08 16:49:38.028714 | tripleo-ovb-centos-7 | 2018-08-08 16:49:38.028770 | tripleo-ovb-centos-7 | 2018-08-08 16:49:38.028802 | tripleo-ovb-centos-7 | 2018-08-08 16:49:38.028857 | tripleo-ovb-centos-7 | 2018-08-08 16:49:38.028883 | tripleo-ovb-centos-7 | 2018-08-08 16:49:38.028956 | tripleo-ovb-centos-7 | 2018-08-08 16:49:38.029012 | tripleo-ovb-centos-7 | 2018-08-08 16:49:38.029048 | tripleo-ovb-centos-7 | 2018-08-08 16:49:38.029090 | tripleo-ovb-centos-7 | 2018-08-08 16:49:38.029123 | tripleo-ovb-centos-7 | 2018-08-08 16:49:38.029165 | tripleo-ovb-centos-7 | 2018-08-08 16:49:38.029187 | tripleo-ovb-centos-7 | 2018-08-08 16:49:38.029208 | tripleo-ovb-centos-7 | 2018-08-08 16:49:38.029260 | tripleo-ovb-centos-7 | 2018-08-08 16:49:38.029327 | tripleo-ovb-centos-7 | 2018-08-08 16:49:38.029366 | tripleo-ovb-centos-7 | 2018-08-08 16:49:38.029432 | tripleo-ovb-centos-7 | 2018-08-08 16:49:38.029460 | tripleo-ovb-centos-7 | 2018-08-08 16:49:38.029502 | tripleo-ovb-centos-7 | 2018-08-08 16:49:38.029524 | tripleo-ovb-centos-7 | 2018-08-08 16:49:38.029582 | tripleo-ovb-centos-7 | 2018-08-08 16:49:38.029670 | tripleo-ovb-centos-7 | 2018-08-08 16:49:38.029706 | tripleo-ovb-centos-7 | 2018-08-08 16:49:38.029749 | tripleo-ovb-centos-7 | 2018-08-08 16:49:38.029782 | tripleo-ovb-centos-7 | 2018-08-08 16:49:38.029823 | tripleo-ovb-centos-7 | />7 | 2018-08-08 16:49:38,024 - testenv-client - INFO - Received job : {
"nodes": [
{
"pm_password": "password",
"name": "baremetal-917-0",
"memory": 8192,
"pm_addr": "192.168.102.29",
"mac": [
"fa:16:3e:ef:83:f9"
],
"capabilities": "boot_option:local",
"pm_type": "pxe_ipmitool",
"disk": 80,
"arch": "x86_64",
"cpu": 4,
"pm_user": "admin"
},
{
"pm_password": "password",
"name": "baremetal-917-1",
"memory": 8192,
"pm_addr": "192.168.100.114",
"mac": [
"fa:16:3e:da:c8:2a"
],
"capabilities": "boot_option:local",
"pm_type": "pxe_ipmitool",
"disk": 80,
"arch": "x86_64",
"cpu": 4,
"pm_user": "admin"
},
{
"pm_password": "password",
"name": "baremetal-917-2",
"memory": 8192,
"pm_addr": "192.168.102.86",
"mac": [
"fa:16:3e:49:0f:12"
],
"capabilities": "boot_option:local",
"pm_type": "pxe_ipmitool",
"disk": 80,
"arch": "x86_64",
"cpu": 4,
"pm_user": "admin"
},
{
"pm_password": "password",
"name": "baremetal-917-3",
"memory": 8192,
"pm_addr": "192.168.102.46",
"mac": [
"fa:16:3e:6a:85:01"
],
"capabilities": "boot_option:local",
"pm_type": "pxe_ipmitool",
"disk": 80,
"arch": "x86_64",
"cpu": 4,
"pm_user": "admin"

2018-08-08 17:48:25.554 ERROR /var/log/containers/nova/nova-compute.log: 8 ERROR nova.scheduler.client.report [req-e98338fa-b9d9-4c20-bb0a-c71217ab2f3e - - - - -] [req-8e785e03-96b4-4abb-b0e8-0e81fec0474d] Failed to update inventory to [{'VCPU': {'allocation_ratio': 1.0, 'total': 4, 'reserved': 4, 'step_size': 1, 'min_unit': 1, 'max_unit': 4}, 'MEMORY_MB': {'allocation_ratio': 1.0, 'total': 8192, 'reserved': 8192, 'step_size': 1, 'min_unit': 1, 'max_unit': 8192}, u'CUSTOM_BAREMETAL': {'allocation_ratio': 1.0, 'total': 1, 'reserved': 1, 'step_size': 1, 'min_unit': 1, 'max_unit': 1}, 'DISK_GB': {'allocation_ratio': 1.0, 'total': 80, 'reserved': 80, 'step_size': 1, 'min_unit': 1, 'max_unit': 80}}] for resource provider with UUID 5d298fd9-6e2c-4778-9580-65aa38d1ac31. Got 400: {"errors": [{"status": 400, "request_id": "req-8e785e03-96b4-4abb-b0e8-0e81fec0474d", "detail": "The server could not comply with the request since it is either malformed or otherwise incorrect.\n\n Unable to update inventory for resource provider 5d298fd9-6e2c-4778-9580-65aa38d1ac31: Invalid inventory for 'VCPU' on resource provider '5d298fd9-6e2c-4778-9580-65aa38d1ac31'. The reserved value is greater than or equal to total. ", "title": "Bad Request"}]}
2018-08-08 17:48:25.555 ERROR /var/log/containers/nova/nova-compute.log: 8 ERROR nova.compute.manager [req-e98338fa-b9d9-4c20-bb0a-c71217ab2f3e - - - - -] Error updating resources for node 5d298fd9-6e2c-4778-9580-65aa38d1ac31.: ResourceProviderSyncFailed: Failed to synchronize the placement service with resource provider information supplied by the compute host.

https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/3e961ea/logs/undercloud/var/log/extra/errors.txt.gz#_2018-08-08_17_48_25_554

Tags:

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-08-08: Related fix proposed to tripleo-quickstart-extras (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/590036

Revision history for this message

wes hayutin (weshayutin) wrote on 2018-08-09:

Download full text (19.2 KiB)

https://logs.rdoproject.org/36/590036/1/openstack-check/legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/0d2eae7/logs/undercloud/home/zuul/flavor_list.log.txt.gz

Alex Schultz (alex-schultz) on 2018-08-14

Changed in tripleo:
milestone:	rocky-rc1 → stein-1

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-08-20: Related fix merged to tripleo-quickstart-extras (master)

Reviewed: https://review.openstack.org/590036
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/commit/?id=9ea6018d3b776b625c948afab8e2c264e67ddbe8
Submitter: Zuul
Branch: master

commit 9ea6018d3b776b625c948afab8e2c264e67ddbe8
Author: Wes Hayutin <email address hidden>
Date: Wed Aug 8 16:41:58 2018 -0400

list and show all the available flavors prior to introspection

    It would be helpful for debugging if all the available
    OpenStack flavors were listed out prior to running
    introspection

Related-Bug: #1786109
Change-Id: I13543fdd1aa1d08c8ad3df6430be2d1e74d060c9

Juan Antonio Osorio Robles (juan-osorio-robles) on 2018-10-30

Changed in tripleo:
milestone:	stein-1 → stein-2

Revision history for this message

Sudeep Batra (sudeep.batra) wrote on 2019-01-12:

overcloud_prep_images.log Edit (26.9 KiB, text/plain)

I also got this error running the below command :
./tripleo-quickstart/quickstart.sh --clean --release master --teardown all --tags all -e undercloud_undercloud_ntp_servers=147.117.251.28 -e @/home/toor/deploy-config.yaml 127.0.0.2

TASK [overcloud-prep-images : Prepare the overcloud images for deploy] ********************************************************
task path: /home/toor/.quickstart/usr/local/share/ansible/roles/overcloud-prep-images/tasks/overcloud-prep-images.yml:2
Saturday 12 January 2019 14:44:16 -0500 (0:00:04.097) 1:14:43.245 ******
fatal: [undercloud]: FAILED! => {
    "changed": true,
    "cmd": "set -o pipefail && /home/stack/overcloud-prep-images.sh 2>&1 | awk '{ print strftime(\"%Y-%m-%d %H:%M:%S |\"), $0; fflush(); }' > /home/stack/overcloud_prep_images.log",
    "delta": "1:21:36.673992",
    "end": "2019-01-12 21:04:24.204992",
    "rc": 1,
    "start": "2019-01-12 19:42:47.531000"
}

MSG:

non-zero return code

PLAY RECAP ********************************************************************************************************************
127.0.0.2 : ok=147 changed=71 unreachable=0 failed=0
localhost : ok=19 changed=8 unreachable=0 failed=0
undercloud : ok=35 changed=19 unreachable=0 failed=1

overcloud_prep_images.log attached.

Hoping the Tripleo quickstart deployment becomes seamless as openstack-helm (so fast and I can directly get into the next stage of operations).

Emilien Macchi (emilienm) on 2019-01-13

Changed in tripleo:
milestone:	stein-2 → stein-3

Alex Schultz (alex-schultz) on 2019-03-13

Changed in tripleo:
milestone:	stein-3 → train-1

Alex Schultz (alex-schultz) on 2019-06-07

Changed in tripleo:
milestone:	train-1 → train-2

Alex Schultz (alex-schultz) on 2019-07-29

Changed in tripleo:
milestone:	train-2 → train-3

Alex Schultz (alex-schultz) on 2019-09-11

Changed in tripleo:
milestone:	train-3 → ussuri-1

Emilien Macchi (emilienm) on 2019-12-19

Changed in tripleo:
milestone:	ussuri-1 → ussuri-2

wes hayutin (weshayutin) on 2020-02-10

Changed in tripleo:
milestone:	ussuri-2 → ussuri-3

wes hayutin (weshayutin) on 2020-04-07

Changed in tripleo:
status:	Triaged → Incomplete

wes hayutin (weshayutin) on 2020-04-13

Changed in tripleo:
milestone:	ussuri-3 → ussuri-rc3

wes hayutin (weshayutin) on 2020-05-26

Changed in tripleo:
milestone:	ussuri-rc3 → victoria-1

Emilien Macchi (emilienm) on 2020-07-28

Changed in tripleo:
milestone:	victoria-1 → victoria-3

Marios Andreou (marios-b) on 2020-11-03

Changed in tripleo:
milestone:	victoria-3 → wallaby-1

Marios Andreou (marios-b) on 2020-12-08

Changed in tripleo:
milestone:	wallaby-1 → wallaby-2

Marios Andreou (marios-b) on 2021-01-29

Changed in tripleo:
milestone:	wallaby-2 → wallaby-3

Revision history for this message

Marios Andreou (marios-b) wrote on 2021-02-17:

This is an automated action. Bug status has been set to 'Incomplete' and target milestone has been removed due to inactivity. If you disagree please re-set these values and reach out to us on freenode #tripleo