Ovb jobs stack creation is failing with "Resource CREATE failed: WaitConditionTimeout: resources.baremetal_env.resources.bmc.resources.bmc_wait_condition: 0 of 1 received"

[Sandeep Yadav]

Description:

Ovb jobs stack creation is failing with

"Resource CREATE failed: WaitConditionTimeout: resources.baremetal_env.resources.bmc.resources.bmc_wait_condition: 0 of 1 received"

This is affecting all the releases

Build history:-
https://review.rdoproject.org/zuul/builds?job_name=tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001

Logs snippet:-

https://logserver.rdoproject.org/96/792196/2/openstack-check/tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001/ff0526a/logs/failed_ovb_stack.log
https://logserver.rdoproject.org/openstack-regular/opendev.org/openstack/tripleo-ci/master/tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001/802fc1f/logs/failed_ovb_stack.log
https://logserver.rdoproject.org/openstack-periodic-integration-stable2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-victoria/81b491f/logs/failed_ovb_stack.log

~~~
baremetal_50397.baremetal_env.bmc.bmc_wait_condition:
  resource_type: OS::Heat::WaitCondition
  physical_resource_id:
  status: CREATE_FAILED
  status_reason: |
    WaitConditionTimeout: resources.bmc_wait_condition: 0 of 1 received
~~~

[Rabi Mishra (rabi)]

Has the ovb bmc image changed and is possibly broken?

[Sandeep Yadav (sandeepyadav93)]

Hello Rabi,

We have reached out to infra they have opened a ticket with #vexxhost

ticket: #ECY-867988

[Harald Jensås (harald-jensas)]

Can we get to the console log from the ovb bmc instance?

[Harald Jensås (harald-jensas)]

Here is a patch to the ovb-manage role that will make it try to capture nova instance console logs on stack failure.

https://review.rdoproject.org/r/c/config/+/33818

[Sagi (Sergey) Shnaidman (sshnaidm)]

Merged this one: https://review.rdoproject.org/r/c/config/+/25789

[Rabi Mishra (rabi)]

I guess vexxhost has some issue.

[ 11.479294] cloud-init[893]: 2021-05-24 13:06:15,580 - util.py[WARNING]: No active metadata service found

[Rabi Mishra (rabi)]

This BZ seems related https://bugzilla.redhat.com/show_bug.cgi?id=1952692

[Harald Jensås (harald-jensas)]

Yeah, seems a metadata issue. Cloud-init does not run the install_openstackbmc.sh script in the CI instances.

https://logserver.rdoproject.org/86/791486/16/openstack-check/tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001/ab25d0c/logs/bmc_16_36009-console.log
[ 13.554485] cloud-init[1154]: Cloud-init v. 18.5 running 'modules:final' at Mon, 24 May 2021 13:06:17 +0000. Up 13.49 seconds.

< .... SNIP .... >

[ 13.701092] cloud-init[1154]: Cloud-init v. 18.5 finished at Mon, 24 May 2021 13:06:17 +0000. Datasource DataSourceNone. Up 13.69 seconds
[ 13.724129] cloud-init[1154]: 2021-05-24 13:06:17,817 - cc_final_message.py[WARNING]: Used fallback datasource

Comparing to my working lab environment which also use ovb and ovb-manage the 'modules:final' run's the ovb bmc setup script[1]:

[ 69.989960] cloud-init[1264]: Cloud-init v. 18.5 running 'modules:final' at Mon, 24 May 2021 14:27:53 +0000. Up 69.84 seconds.
[ 70.052925] cloud-init[1264]: + required_packages='python-pip os-net-config git jq python2-os-client-config python2-openstackclient'
[ 70.058100] cloud-init[1264]: + have_packages
[ 70.061107] cloud-init[1264]: + for i in '$required_packages'
[ 70.068318] cloud-init[1264]: + rpm -qa
[ 70.076076] cloud-init[1264]: + grep -q python-pip

< .... SNIP .... >

[ 201.778123] cloud-init[1264]: nullCloud-init v. 18.5 finished at Mon, 24 May 2021 14:30:05 +0000. Datasource DataSourceOpenStack [net,ver=2]. Up 201.76 seconds
[[32m OK [0m] Started Execute cloud user/final scripts.

[1] https://opendev.org/openstack/openstack-virtual-baremetal/src/branch/master/bin/install_openstackbmc.sh#L8-L15

[Sandeep Yadav (sandeepyadav93)]

Thanks to Harald, he suggested it may be a viable workaround to use config-drive for the UC and BMC resources.

Ovb jobs are back to green after we moved to using config-drive for BMC node:-

~~~
https://review.opendev.org/c/openstack/openstack-virtual-baremetal/+/792884
https://review.opendev.org/c/openstack/openstack-virtual-baremetal/+/792907
https://review.opendev.org/c/openstack/openstack-virtual-baremetal/+/792880
https://review.opendev.org/c/openstack/openstack-virtual-baremetal/+/792823
https://review.rdoproject.org/r/c/config/+/33831
~~~

[Sandeep Yadav (sandeepyadav93)]

fs039 is still affected:-

Rabi have posted patches:_
https://review.rdoproject.org/r/c/config/+/33867
https://review.opendev.org/c/openstack/openstack-virtual-baremetal/+/793147

[Sandeep Yadav (sandeepyadav93)]

Thanks to rabi and Harald,

This is fixed after the below patches merged:-

https://review.opendev.org/c/openstack/openstack-virtual-baremetal/+/793689
https://review.opendev.org/c/openstack/openstack-virtual-baremetal/+/793775

We have a green run for fs039:-

https://logserver.rdoproject.org/67/793867/5/openstack-check/tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039/eb28959/