Heat stack creation failing for featureset039 in Master branch with "ResourceInError: resources.NovaCompute: Went to status ERROR due to "Message: MessagingTimeout, Code: 500"

[Sandeep Yadav]

Description:-

Heat stack creation failing for featureset039 in Master branch with "ResourceInError: resources.NovaCompute: Went to status ERROR due to "Message: MessagingTimeout, Code: 500". Issue started on 19th Feb'21.

Build history:-
https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-master

Log snippet:-

https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-master/93d42a0/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz
~~~
2021-02-21 18:55:37 | overcloud.Compute.0.NovaCompute:
2021-02-21 18:55:37 | resource_type: OS::TripleO::ComputeServer
2021-02-21 18:55:37 | physical_resource_id: 6483d2aa-105b-4c5c-a2e5-c941ed123ef4
2021-02-21 18:55:37 | status: CREATE_FAILED
2021-02-21 18:55:37 | status_reason: |
2021-02-21 18:55:37 | ResourceInError: resources.NovaCompute: Went to status ERROR due to "Message: MessagingTimeout, Code: 500"
2021-02-21 18:55:37 | 2021-02-21 18:55:37.970 148902 DEBUG osc_lib.shell [-] clean_up ListStackFailures: clean_up /usr/lib/python3.6/site-packages/osc_lib/shell.py:496
2021-02-21 18:55:37 | 2021-02-21 18:55:37.971 148902 INFO osc_lib.shell [-] END return value: 0
2021-02-21 18:55:37 | 2021-02-21 18:55:37.973 148902 WARNING tripleoclient.utils.safe_write [-] The output file /home/zuul/overcloud-deployment_status.yaml will be overriden
2021-02-21 18:55:37 | /usr/lib/python3.6/site-packages/barbicanclient/__init__.py:61: UserWarning: The secrets module is moved to barbicanclient/v1 directory, direct import of barbicanclient.secrets will be deprecated. Please import barbicanclient.v1.secrets instead.
2021-02-21 18:55:37 | % (name, name, name))
2021-02-21 18:55:37 | The output file /home/zuul/overcloud-deployment_status.yaml will be overriden
2021-02-21 18:55:37 | 2021-02-21 18:55:37.973 148902 DEBUG tripleoclient.v1.overcloud_deploy.DeployOvercloud [-] Preserving deployment artifacts archive_deploy_artifacts /usr/lib/python3.6/site-packages/tripleoclient/utils.py:1653
2021-02-21 18:55:38 | 2021-02-21 18:55:38.799 148902 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud [-] Exception occured while running the command: ValueError: Failed to deploy: Heat Stack update failed.
2021-02-21 18:55:38 | 2021-02-21 18:55:38.799 148902 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud Traceback (most recent call last):
2021-02-21 18:55:38 | 2021-02-21 18:55:38.799 148902 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", line 417, in _try_overcloud_deploy_with_compat_yaml
2021-02-21 18:55:38 | 2021-02-21 18:55:38.799 148902 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud deployment_options=deployment_options)
2021-02-21 18:55:38 | 2021-02-21 18:55:38.799 148902 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", line 245, in _heat_deploy
2021-02-21 18:55:38 | 2021-02-21 18:55:38.799 148902 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud self.log)
2021-02-21 18:55:38 | 2021-02-21 18:55:38.799 148902 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.6/site-packages/tripleoclient/workflows/deployment.py", line 97, in deploy_without_plan
2021-02-21 18:55:38 | 2021-02-21 18:55:38.799 148902 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud raise exceptions.DeploymentError("Heat Stack update failed.")
2021-02-21 18:55:38 | 2021-02-21 18:55:38.799 148902 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud tripleoclient.exceptions.DeploymentError: Heat Stack update failed.
2021-02-21 18:55:38 | 2021-02-21 18:55:38.799 148902 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud
2021-02-21 18:55:38 | 2021-02-21 18:55:38.799 148902 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud During handling of the above exception, another exception occurred:
2021-02-21 18:55:38 | 2021-02-21 18:55:38.799 148902 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud
2021-02-21 18:55:38 | 2021-02-21 18:55:38.799 148902 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud Traceback (most recent call last):
2021-02-21 18:55:38 | 2021-02-21 18:55:38.799 148902 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.6/site-packages/tripleoclient/command.py", line 34, in run
2021-02-21 18:55:38 | 2021-02-21 18:55:38.799 148902 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud super(Command, self).run(parsed_args)
2021-02-21 18:55:38 | 2021-02-21 18:55:38.799 148902 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.6/site-packages/osc_lib/command/command.py", line 39, in run
2021-02-21 18:55:38 | 2021-02-21 18:55:38.799 148902 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud return super(Command, self).run(parsed_args)
2021-02-21 18:55:38 | 2021-02-21 18:55:38.799 148902 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.6/site-packages/cliff/command.py", line 185, in run
2021-02-21 18:55:38 | 2021-02-21 18:55:38.799 148902 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud return_code = self.take_action(parsed_args) or 0
2021-02-21 18:55:38 | 2021-02-21 18:55:38.799 148902 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", line 922, in take_action
2021-02-21 18:55:38 | 2021-02-21 18:55:38.799 148902 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud self._deploy_tripleo_heat_templates_tmpdir(stack, parsed_args)
2021-02-21 18:55:38 | 2021-02-21 18:55:38.799 148902 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", line 261, in _deploy_tripleo_heat_templates_tmpdir
2021-02-21 18:55:38 | 2021-02-21 18:55:38.799 148902 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud new_tht_root, tht_root)
2021-02-21 18:55:38 | 2021-02-21 18:55:38.799 148902 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", line 380, in _deploy_tripleo_heat_templates
2021-02-21 18:55:38 | 2021-02-21 18:55:38.799 148902 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud deployment_options=deployment_options)
2021-02-21 18:55:38 | 2021-02-21 18:55:38.799 148902 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", line 420, in _try_overcloud_deploy_with_compat_yaml
2021-02-21 18:55:38 | 2021-02-21 18:55:38.799 148902 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud raise ValueError(messages)
2021-02-21 18:55:38 | 2021-02-21 18:55:38.799 148902 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud ValueError: Failed to deploy: Heat Stack update failed.
2021-02-21 18:55:38 | 2021-02-21 18:55:38.799 148902 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud 
2021-02-21 18:55:38 | Exception occured while running the command
2021-02-21 18:55:38 | 2021-02-21 18:55:38.799 148902 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud Traceback (most recent call last):
2021-02-21 18:55:38 | 2021-02-21 18:55:38.799 148902 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", line 417, in _try_overcloud_deploy_with_compat_yaml
2021-02-21 18:55:38 | 2021-02-21 18:55:38.799 148902 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud deployment_options=deployment_options)
2021-02-21 18:55:38 | 2021-02-21 18:55:38.799 148902 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", line 245, in _heat_deploy
2021-02-21 18:55:38 | 2021-02-21 18:55:38.799 148902 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud self.log)
2021-02-21 18:55:38 | 2021-02-21 18:55:38.799 148902 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.6/site-packages/tripleoclient/workflows/deployment.py", line 97, in deploy_without_plan
2021-02-21 18:55:38 | 2021-02-21 18:55:38.799 148902 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud raise exceptions.DeploymentError("Heat Stack update failed.")
2021-02-21 18:55:38 | 2021-02-21 18:55:38.799 148902 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud tripleoclient.exceptions.DeploymentError: Heat Stack update failed.
2021-02-21 18:55:38 | 2021-02-21 18:55:38.799 148902 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud
2021-02-21 18:55:38 | 2021-02-21 18:55:38.799 148902 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud During handling of the above exception, another exception occurred:
2021-02-21 18:55:38 | 2021-02-21 18:55:38.799 148902 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud
2021-02-21 18:55:38 | 2021-02-21 18:55:38.799 148902 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud Traceback (most recent call last):
2021-02-21 18:55:38 | 2021-02-21 18:55:38.799 148902 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.6/site-packages/tripleoclient/command.py", line 34, in run
2021-02-21 18:55:38 | 2021-02-21 18:55:38.799 148902 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud super(Command, self).run(parsed_args)
2021-02-21 18:55:38 | 2021-02-21 18:55:38.799 148902 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.6/site-packages/osc_lib/command/command.py", line 39, in run
2021-02-21 18:55:38 | 2021-02-21 18:55:38.799 148902 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud return super(Command, self).run(parsed_args)
2021-02-21 18:55:38 | 2021-02-21 18:55:38.799 148902 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.6/site-packages/cliff/command.py", line 185, in run
2021-02-21 18:55:38 | 2021-02-21 18:55:38.799 148902 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud return_code = self.take_action(parsed_args) or 0
2021-02-21 18:55:38 | 2021-02-21 18:55:38.799 148902 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", line 922, in take_action
2021-02-21 18:55:38 | 2021-02-21 18:55:38.799 148902 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud self._deploy_tripleo_heat_templates_tmpdir(stack, parsed_args)
2021-02-21 18:55:38 | 2021-02-21 18:55:38.799 148902 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", line 261, in _deploy_tripleo_heat_templates_tmpdir
2021-02-21 18:55:38 | 2021-02-21 18:55:38.799 148902 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud new_tht_root, tht_root)
2021-02-21 18:55:38 | 2021-02-21 18:55:38.799 148902 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", line 380, in _deploy_tripleo_heat_templates
2021-02-21 18:55:38 | 2021-02-21 18:55:38.799 148902 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud deployment_options=deployment_options)
2021-02-21 18:55:38 | 2021-02-21 18:55:38.799 148902 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", line 420, in _try_overcloud_deploy_with_compat_yaml
2021-02-21 18:55:38 | 2021-02-21 18:55:38.799 148902 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud raise ValueError(messages)
2021-02-21 18:55:38 | 2021-02-21 18:55:38.799 148902 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud ValueError: Failed to deploy: Heat Stack update failed.
2021-02-21 18:55:38 | 2021-02-21 18:55:38.799 148902 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud
2021-02-21 18:55:38 | 2021-02-21 18:55:38.805 148902 ERROR openstack [-] Failed to deploy: Heat Stack update failed.
2021-02-21 18:55:38 | Failed to deploy: Heat Stack update failed.
2021-02-21 18:55:38 | 2021-02-21 18:55:38.805 148902 DEBUG osc_lib.shell [-] clean_up DeployOvercloud: Failed to deploy: Heat Stack update failed. clean_up /usr/lib/python3.6/site-packages/osc_lib/shell.py:496
~~~

Another example:
https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-master/9253ddc/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz

[Rabi Mishra (rabi)]

Looks like broken by https://review.opendev.org/c/openstack/tripleo-common/+/775964 as removed the nova filters used by these jobs that still use nova.

[Rabi Mishra (rabi)]

2021-02-19 18:28:52.468 8 ERROR nova Traceback (most recent call last):
2021-02-19 18:28:52.468 8 ERROR nova File "/usr/bin/nova-scheduler", line 10, in <module>
2021-02-19 18:28:52.468 8 ERROR nova sys.exit(main())
2021-02-19 18:28:52.468 8 ERROR nova File "/usr/lib/python3.6/site-packages/nova/cmd/scheduler.py", line 48, in main
2021-02-19 18:28:52.468 8 ERROR nova topic=scheduler_rpcapi.RPC_TOPIC)
2021-02-19 18:28:52.468 8 ERROR nova File "/usr/lib/python3.6/site-packages/nova/service.py", line 256, in create
2021-02-19 18:28:52.468 8 ERROR nova periodic_interval_max=periodic_interval_max)
2021-02-19 18:28:52.468 8 ERROR nova File "/usr/lib/python3.6/site-packages/nova/service.py", line 116, in __init__
2021-02-19 18:28:52.468 8 ERROR nova self.manager = manager_class(host=self.host, *args, **kwargs)
2021-02-19 18:28:52.468 8 ERROR nova File "/usr/lib/python3.6/site-packages/nova/scheduler/manager.py", line 62, in __init__
2021-02-19 18:28:52.468 8 ERROR nova invoke_on_load=True
2021-02-19 18:28:52.468 8 ERROR nova File "/usr/lib/python3.6/site-packages/stevedore/driver.py", line 62, in __init__
2021-02-19 18:28:52.468 8 ERROR nova warn_on_missing_entrypoint=warn_on_missing_entrypoint
2021-02-19 18:28:52.468 8 ERROR nova File "/usr/lib/python3.6/site-packages/stevedore/named.py", line 81, in __init__
2021-02-19 18:28:52.468 8 ERROR nova verify_requirements)
2021-02-19 18:28:52.468 8 ERROR nova File "/usr/lib/python3.6/site-packages/stevedore/extension.py", line 233, in _load_plugins
2021-02-19 18:28:52.468 8 ERROR nova self._on_load_failure_callback(self, ep, err)
2021-02-19 18:28:52.468 8 ERROR nova File "/usr/lib/python3.6/site-packages/stevedore/extension.py", line 225, in _load_plugins
2021-02-19 18:28:52.468 8 ERROR nova verify_requirements,
2021-02-19 18:28:52.468 8 ERROR nova File "/usr/lib/python3.6/site-packages/stevedore/named.py", line 158, in _load_one_plugin
2021-02-19 18:28:52.468 8 ERROR nova verify_requirements,
2021-02-19 18:28:52.468 8 ERROR nova File "/usr/lib/python3.6/site-packages/stevedore/extension.py", line 257, in _load_one_plugin
2021-02-19 18:28:52.468 8 ERROR nova obj = plugin(*invoke_args, **invoke_kwds)
2021-02-19 18:28:52.468 8 ERROR nova File "/usr/lib/python3.6/site-packages/nova/scheduler/filter_scheduler.py", line 44, in __init__
2021-02-19 18:28:52.468 8 ERROR nova super(FilterScheduler, self).__init__(*args, **kwargs)
2021-02-19 18:28:52.468 8 ERROR nova File "/usr/lib/python3.6/site-packages/nova/scheduler/driver.py", line 45, in __init__
2021-02-19 18:28:52.468 8 ERROR nova self.host_manager = host_manager.HostManager()
2021-02-19 18:28:52.468 8 ERROR nova File "/usr/lib/python3.6/site-packages/nova/scheduler/host_manager.py", line 340, in __init__
2021-02-19 18:28:52.468 8 ERROR nova CONF.filter_scheduler.available_filters)
2021-02-19 18:28:52.468 8 ERROR nova File "/usr/lib/python3.6/site-packages/nova/loadables.py", line 106, in get_matching_classes
2021-02-19 18:28:52.468 8 ERROR nova obj = importutils.import_class(cls_name)
2021-02-19 18:28:52.468 8 ERROR nova File "/usr/lib/python3.6/site-packages/oslo_u...

[Rabi Mishra (rabi)]

https://review.opendev.org/c/openstack/tripleo-common/+/776870 should fix this.

[Sandeep Yadav (sandeepyadav93)]

fs039 uses novajoin:-

~~~
https://github.com/openstack/tripleo-quickstart/blob/master/config/general_config/featureset039.yml#L80
~~~

novajoin is no longer supported:-
https://docs.openstack.org/project-deploy-guide/tripleo-docs/latest/features/tls-introduction.html
~~~
As of Victoria, novajoin is not longer supported. If you are updating from Ussuri, tripleo will automatically migrate your deployment from novajoin to tripleo-ipa.
~~~

For the master branch as we are reducing undercloud footprint by removing services like nova/swift. We will check further with the security team if we can drop fs039 or migrate this job. We already have a standalone job that test deployment with IPA:-

periodic-tripleo-ci-centos-8-standalone-on-multinode-ipa

[Sandeep Yadav (sandeepyadav93)]

Fix [1] Already merged upstream (waiting to hit integration line.) Testproject[2] passes[3] with depends-on [1].

We need confirmation from security dfg about fs039 as novajoin is not supported in wallaby

[1] https://review.opendev.org/c/openstack/tripleo-common/+/776870
[2] https://review.rdoproject.org/r/#/c/28537/11
[3] periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-master https://review.rdoproject.org/zuul/build/f9961d92e2fc4bf5ab37ab77079ced76 : SUCCESS in 4h 17m 37s

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-common 15.0.0 release.