quickstart containers minimal failing w/ error configuring heat_api

[wes hayutin]

https://ci.centos.org/artifacts/rdo/jenkins-tqe-containers-gate-master-tripleo-ci-delorean-full-containers_minimal-2294/undercloud/home/stack/failed_deployment_list.log.gz

            "2018-01-09 10:01:46,225 ERROR: 20546 -- Failed running docker-puppet.py for heat_api_cfn",
            "2018-01-09 10:01:46,225 ERROR: 20546 -- Notice: hiera(): Cannot load backend module_data: cannot load such file -- hiera/backend/module_data_backend",
            "Notice: Compiled catalog for overcloud-controller-0.localdomain in environment production in 9.89 seconds",
            "Notice: /Stage[main]/Heat::Api_cfn/Heat_config[heat_api_cfn/bind_host]/ensure: created",
            "Notice: /Stage[main]/Apache/Concat[/etc/httpd/conf/ports.conf]/File[/etc/httpd/conf/ports.conf]/ensure: defined content as '{md5}58ba1003052908ca2def9d3a0b90636e'",
            "Notice: /Stage[main]/Apache::Mod::Headers/Apache::Mod[headers]/File[headers.load]/ensure: defined content as '{md5}96094c96352002c43ada5bdf8650ff38'",
            "Notice: /Stage[main]/Heat::Wsgi::Apache_api_cfn/Heat::Wsgi::Apache[api_cfn]/Openstacklib::Wsgi::Apache[heat_api_cfn_wsgi]/File[/var/www/cgi-bin/heat]/ensure: created",
            "Notice: /Stage[main]/Heat::Wsgi::Apache_api_cfn/Heat::Wsgi::Apache[api_cfn]/Openstacklib::Wsgi::Apache[heat_api_cfn_wsgi]/File[heat_api_cfn_wsgi]/ensure: defined content as '{md5}c3ae61ab87649c8cdfab8977da2b194b'",
            "Notice: /Stage[main]/Heat::Wsgi::Apache_api_cfn/Heat::Wsgi::Apache[api_cfn]/Openstacklib::Wsgi::Apache[heat_api_cfn_wsgi]/Apache::Vhost[heat_api_cfn_wsgi]/Concat[10-heat_api_cfn_wsgi.conf]/File[/etc/httpd/conf.d/10-heat_api_cfn_wsgi.conf]/ensure: defined content as '{md5}da8f4bd0ceefdaebc9127cc711ee8a8e'",
            "Notice: Applied catalog in 6.93 seconds",
            "2018-01-09 10:01:46,226 ERROR: 20546 -- + mkdir -p /etc/puppet",
            " with Stdlib::Compat::Integer. There is further documentation for validate_legacy function in the README. at [\"/etc/puppet/modules/heat/manifests/wsgi/apache_api_cfn.pp\", 125]:[\"/etc/config.pp\", 2]",
            "2018-01-09 10:01:46,226 INFO: 20546 -- Finished processing puppet configs for heat_api_cfn",
            "2018-01-09 10:01:46,226 INFO: 20546 -- Starting configuration of cinder using image 192.168.24.1:8787/tripleomaster/centos-binary-cinder-api:current-tripleo",
            "2018-01-09 10:01:46,227 INFO: 20546 -- Removing container: docker-puppet-cinder",
            "2018-01-09 10:01:46,286 INFO: 20546 -- Pulling image: 192.168.24.1:8787/tripleomaster/centos-binary-cinder-api:current-tripleo",
            "2018-01-09 10:01:52,194 INFO: 20545 -- Removing container: docker-puppet-memcached",
            "2018-01-09 10:01:52,301 INFO: 20545 -- Finished processing puppet configs for memcached",
            "2018-01-09 10:01:52,301 INFO: 20545 -- Starting configuration of crond using image 192.168.24.1:8787/tripleomaster/centos-binary-cron:current-tripleo",
            "2018-01-09 10:01:52,302 INFO: 20545 -- Removing container: docker-puppet-crond",
            "2018-01-09 10:01:52,350 INFO: 20545 -- Pulling image: 192.168.24.1:8787/tripleomaster/centos-binary-cron:current-tripleo",
            "2018-01-09 10:02:12,124 INFO: 20545 -- Removing container: docker-puppet-crond",
            "2018-01-09 10:02:12,319 INFO: 20545 -- Finished processing puppet configs for crond",
            "2018-01-09 10:02:12,320 INFO: 20545 -- Starting configuration of neutron using image 192.168.24.1:8787/tripleomaster/centos-binary-neutron-server:current-tripleo",
            "2018-01-09 10:02:12,321 INFO: 20545 -- Removing container: docker-puppet-neutron",
            "2018-01-09 10:02:12,372 INFO: 20545 -- Pulling image: 192.168.24.1:8787/tripleomaster/centos-binary-neutron-server:current-tripleo",
            "2018-01-09 10:02:40,590 INFO: 20547 -- Removing container: docker-puppet-horizon",
            "2018-01-09 10:02:40,808 INFO: 20547 -- Finished processing puppet configs for horizon",
            "2018-01-09 10:02:55,303 INFO: 20546 -- Removing container: docker-puppet-cinder",
            "2018-01-09 10:02:55,382 INFO: 20546 -- Finished processing puppet configs for cinder",
            "2018-01-09 10:02:57,322 INFO: 20545 -- Removing container: docker-puppet-neutron",
            "2018-01-09 10:02:57,372 INFO: 20545 -- Finished processing puppet configs for neutron",
            "2018-01-09 10:02:57,373 ERROR: 20544 -- ERROR configuring heat_api",
            "2018-01-09 10:02:57,374 ERROR: 20544 -- ERROR configuring heat_api_cfn"
        ]

[Sofer Athlan-Guyot (sofer-athlan-guyot)]

Doing debug on a live env I got this in puppet logs:

Error: /Stage[main]/Heat::Policy/Openstacklib::Policy::Base[context_is_admin]/Augeas[/etc/heat/policy.json-context_is_admin-role:admin-add]: Could not evaluate: Saving failed, see debug

Notice: /Stage[main]/Heat::Policy/Openstacklib::Policy::Base[context_is_admin]/Augeas[/etc/heat/policy.json-context_is_admin-role:admin]: Dependency Augeas[/etc/heat/policy.json-context_is_admin-role:admin-add] has failures: true
Debug: /Stage[main]/Heat::Policy/Openstacklib::Policy::Base[context_is_admin]/Augeas[/etc/heat/policy.json-context_is_admin-role:admin]: Resource is being skipped, unscheduling all events
Debug: Openstacklib::Policy::Base[context_is_admin]: Not tagged with file, file_line, concat, augeas, cron, heat_config
Debug: Openstacklib::Policy::Base[context_is_admin]: Resource is being skipped, unscheduling all events
Notice: /Stage[main]/Keystone::Deps/Anchor[keystone::config::end]: Dependency Augeas[/etc/heat/policy.json-context_is_admin-role:admin-add] has failures: true
Debug: /Stage[main]/Keystone::Deps/Anchor[keystone::config::end]: Resource is being skipped, unscheduling all events
Notice: /Stage[main]/Keystone::Deps/Anchor[keystone::db::begin]: Dependency Augeas[/etc/heat/policy.json-context_is_admin-role:admin-add] has failures: true
Debug: /Stage[main]/Keystone::Deps/Anchor[keystone::db::begin]: Resource is being skipped, unscheduling all events
Notice: /Stage[main]/Keystone::Deps/Anchor[keystone::db::end]: Dependency Augeas[/etc/heat/policy.json-context_is_admin-role:admin-add] has failures: true
Debug: /Stage[main]/Keystone::Deps/Anchor[keystone::db::end]: Resource is being skipped, unscheduling all events
Notice: /Stage[main]/Keystone::Deps/Anchor[keystone::dbsync::begin]: Dependency Augeas[/etc/heat/policy.json-context_is_admin-role:admin-add] has failures: true
Debug: /Stage[main]/Keystone::Deps/Anchor[keystone::dbsync::begin]: Resource is being skipped, unscheduling all events
Notice: /Stage[main]/Keystone::Deps/Anchor[keystone::dbsync::end]: Dependency Augeas[/etc/heat/policy.json-context_is_admin-role:admin-add] has failures: true
Debug: /Stage[main]/Keystone::Deps/Anchor[keystone::dbsync::end]: Resource is being skipped, unscheduling all events
Warning: /Stage[main]/Heat::Policy/Openstacklib::Policy::Base[context_is_admin]/Augeas[/etc/heat/policy.json-context_is_admin-role:admin]: Skipping because of failed dependencies
Warning: /Stage[main]/Keystone::Deps/Anchor[keystone::config::end]: Skipping because of failed dependencies
Warning: /Stage[main]/Keystone::Deps/Anchor[keystone::db::begin]: Skipping because of failed dependencies
Warning: /Stage[main]/Keystone::Deps/Anchor[keystone::db::end]: Skipping because of failed dependencies
Warning: /Stage[main]/Keystone::Deps/Anchor[keystone::dbsync::begin]: Skipping because of failed dependencies
Warning: /Stage[main]/Keystone::Deps/Anchor[keystone::dbsync::end]: Skipping because of failed dependencies
Warning: /Stage[main]/Keystone::Deps/Anchor[keystone::service::begin]: Skipping because of failed dependencies
Notice: /Stage[main]/Keystone::Deps/Anchor[keystone::service::begin]: Dependency Augeas[/etc/heat/policy.json-context_i...

[Sofer Athlan-Guyot (sofer-athlan-guyot)]

Same kind of error for heat-api (previous was heat-api-cfn)

Error: /Stage[main]/Heat::Policy/Openstacklib::Policy::Base[context_is_admin]/Augeas[/etc/heat/policy.json-context_is_admin-role:admin-add]: Could not evaluate: Saving failed, see debug
Notice: /Stage[main]/Heat::Policy/Openstacklib::Policy::Base[context_is_admin]/Augeas[/etc/heat/policy.json-context_is_admin-role:admin]: Dependency Augeas[/etc/heat/policy.json-context_is_admin-role:admin-add] has failures: true
Debug: /Stage[main]/Heat::Policy/Openstacklib::Policy::Base[context_is_admin]/Augeas[/etc/heat/policy.json-context_is_admin-role:admin]: Resource is being skipped, unscheduling all events
Debug: Openstacklib::Policy::Base[context_is_admin]: Not tagged with file, file_line, concat, augeas, cron, heat_config
Debug: Openstacklib::Policy::Base[context_is_admin]: Resource is being skipped, unscheduling all events
Notice: /Stage[main]/Keystone::Deps/Anchor[keystone::config::end]: Dependency Augeas[/etc/heat/policy.json-context_is_admin-role:admin-add] has failures: true
Debug: /Stage[main]/Keystone::Deps/Anchor[keystone::config::end]: Resource is being skipped, unscheduling all events

[Thomas Herve (therve)]

Should have been fixed by https://review.openstack.org/#/c/530225/ but it doesn't seem to be present in the build.

[Sofer Athlan-Guyot (sofer-athlan-guyot)]

Should be fixed by https://review.openstack.org/#/c/530225/, verifying now

[Sofer Athlan-Guyot (sofer-athlan-guyot)]

Oki, so the error is manifesting during stable/pike deployment and the patch above hasn't been backport. Testing the deployment with the patch applied.

[Sofer Athlan-Guyot (sofer-athlan-guyot)]

So the env was wrongly using a master overcloud image. But that confirm that the review fix that issue.