CI reports wrong number of nodes and fails

[Steven Hardy]

Seeing this intermittently:

2015-11-18 11:28:58.582 | Deployment failed: Not enough nodes - available: 0, requested: 2
2015-11-18 11:28:58.631 | #################
2015-11-18 11:28:58.631 | tripleo.sh -- Overcloud deployment started - DONE.
2015-11-18 11:28:58.631 | #################
2015-11-18 11:28:58.636 | + source /home/stack/stackrc
2015-11-18 11:28:58.636 | ++ export NOVA_VERSION=1.1
2015-11-18 11:28:58.637 | ++ NOVA_VERSION=1.1
2015-11-18 11:28:58.637 | +++ sudo hiera admin_password
2015-11-18 11:28:58.752 | ++ export OS_PASSWORD=69f206613b9e1297333042a22003f40b25f8103c
2015-11-18 11:28:58.752 | ++ OS_PASSWORD=69f206613b9e1297333042a22003f40b25f8103c
2015-11-18 11:28:58.752 | ++ export OS_AUTH_URL=http://192.0.2.1:5000/v2.0
2015-11-18 11:28:58.752 | ++ OS_AUTH_URL=http://192.0.2.1:5000/v2.0
2015-11-18 11:28:58.752 | ++ export OS_USERNAME=admin
2015-11-18 11:28:58.753 | ++ OS_USERNAME=admin
2015-11-18 11:28:58.753 | ++ export OS_TENANT_NAME=admin
2015-11-18 11:28:58.753 | ++ OS_TENANT_NAME=admin
2015-11-18 11:28:58.753 | ++ export COMPUTE_API_VERSION=1.1
2015-11-18 11:28:58.753 | ++ COMPUTE_API_VERSION=1.1
2015-11-18 11:28:58.753 | ++ export OS_NO_CACHE=True
2015-11-18 11:28:58.753 | ++ OS_NO_CACHE=True
2015-11-18 11:28:58.754 | ++ export OS_CLOUDNAME=undercloud
2015-11-18 11:28:58.754 | ++ OS_CLOUDNAME=undercloud
2015-11-18 11:28:58.754 | ++ export OS_IMAGE_API_VERSION=1
2015-11-18 11:28:58.754 | ++ OS_IMAGE_API_VERSION=1
2015-11-18 11:28:58.755 | ++ grep ACTIVE
2015-11-18 11:28:58.756 | ++ wc -l
2015-11-18 11:28:58.756 | ++ cut -f1 -d ' '
2015-11-18 11:28:58.757 | ++ nova list
2015-11-18 11:29:00.343 | Wrong number of nodes deployed
2015-11-18 11:29:00.343 | + '[' 2 '!=' 0 ']'
2015-11-18 11:29:00.343 | + echo 'Wrong number of nodes deployed'
2015-11-18 11:29:00.343 | + exit 1

[Steven Hardy (shardy)]

2015-11-18 13:22:57.580 | #################
2015-11-18 13:22:57.580 | tripleo.sh -- Introspect nodes
2015-11-18 13:22:57.580 | #################
2015-11-18 13:22:57.580 | You must source a stackrc file for the Undercloud.
2015-11-18 13:22:57.580 | Attempting to source stackrc at /home/stack/stackrc
2015-11-18 13:22:57.696 | Done.
2015-11-18 13:25:02.763 | Setting available nodes to manageable...
2015-11-18 13:25:02.763 | Starting introspection of node: a25e0ee7-d922-4baa-875f-9738c59c2e67
2015-11-18 13:25:02.764 | Starting introspection of node: cecdc1f7-3a85-46ac-98c9-ae7ea461b714
2015-11-18 13:25:02.764 | Waiting for introspection to finish...
2015-11-18 13:25:02.764 | Introspection for UUID a25e0ee7-d922-4baa-875f-9738c59c2e67 finished successfully.
2015-11-18 13:25:02.764 | Introspection for UUID cecdc1f7-3a85-46ac-98c9-ae7ea461b714 finished successfully.
2015-11-18 13:25:02.764 | Setting manageable nodes to available...
2015-11-18 13:25:02.764 | FAIL: State not updated for Node a25e0ee7-d922-4baa-875f-9738c59c2e67
2015-11-18 13:25:02.764 | FAIL: State not updated for Node cecdc1f7-3a85-46ac-98c9-ae7ea461b714
2015-11-18 13:25:02.764 | Introspection completed.
2015-11-18 13:25:02.834 | #################

Discussion with trown on IRC uncovers that the error is actually interaction between the introspection (which was recently re-enabled in CI) and the setting of the nodes to available.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/246955

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/246964

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/246955
Committed: https://git.openstack.org/cgit/openstack/instack-undercloud/commit/?id=246d1e8595945f74095b3ec70c63f493f615eed6
Submitter: Jenkins
Branch: master

commit 246d1e8595945f74095b3ec70c63f493f615eed6
Author: John Trowbridge <email address hidden>
Date: Wed Nov 18 09:36:11 2015 -0500

    Disable clean_nodes by default

    This patch sets the conductor/clean_nodes configuration option
    for Ironic to False by default. To maintain the same behavior
    of the old ramdisk.

    The tripleoclient `openstack baremetal introspection bulk`
    command needs to be fixed to account for the time it takes
    to move from manageable to available in order to have cleaning
    turned on by default.

    Change-Id: I23c3e7ada3da8085957e86406527e9e5202bdd69
    Closes-Bug: 1517449

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/246964
Committed: https://git.openstack.org/cgit/openstack/instack-undercloud/commit/?id=7a52bf775ada395a96648f063970a9142e4bb73f
Submitter: Jenkins
Branch: stable/liberty

commit 7a52bf775ada395a96648f063970a9142e4bb73f
Author: John Trowbridge <email address hidden>
Date: Wed Nov 18 09:36:11 2015 -0500

    Disable clean_nodes by default

    This patch sets the conductor/clean_nodes configuration option
    for Ironic to False by default. To maintain the same behavior
    of the old ramdisk.

    The tripleoclient `openstack baremetal introspection bulk`
    command needs to be fixed to account for the time it takes
    to move from manageable to available in order to have cleaning
    turned on by default.

    Change-Id: I23c3e7ada3da8085957e86406527e9e5202bdd69
    Closes-Bug: 1517449
    (cherry picked from commit 246d1e8595945f74095b3ec70c63f493f615eed6)