tripleo-ci-centos-7-scenario001-multinode-oooq-container job is failing on noop jobs at Deploy the overcloud step while keystone_init_tasks.
http://logs.openstack.org/45/560445/78/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/a37efa0/job-output.txt.gz#_2018-07-05_02_29_42_057193
from http://logs.openstack.org/45/560445/78/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/a37efa0/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2018-07-05_02_29_29 log and this log:
http://logs.openstack.org/45/560445/78/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/a37efa0/logs/undercloud/var/lib/mistral/5654fc3f-9375-439a-b53e-25df469b290f/ansible.log.txt.gz
Below is the traceback we got:
"Notice: /Stage[main]/Apache/File[/etc/httpd/conf.modules.d/10-wsgi.conf]/ensure: removed",
"Notice: /Stage[main]/Keystone::Wsgi::Apache/Openstacklib::Wsgi::Apache[keystone_wsgi_main]/Apache::Vhost[keystone_wsgi_main]/Concat[10-keystone_wsgi_main.conf]/File[/etc/httpd/conf.d/10-keystone_wsgi_main.conf]/ensure: defined content as '{md5}da79296ac5505df71c0ce6a87fc2dbc2'",
"Notice: /Stage[main]/Keystone::Wsgi::Apache/Openstacklib::Wsgi::Apache[keystone_wsgi_admin]/Apache::Vhost[keystone_wsgi_admin]/Concat[10-keystone_wsgi_admin.conf]/File[/etc/httpd/conf.d/10-keystone_wsgi_admin.conf]/ensure: defined content as '{md5}2917603a32f1c98f96eb5d1e3e306dc6'",
"Notice: /Stage[main]/Tripleo::Profile::Base::Keystone/Keystone_domain[heat_stack]/ensure: created",
"Notice: /Stage[main]/Tripleo::Profile::Base::Keystone/Keystone_user[heat_stack_domain_admin::heat_stack]/ensure: created",
"Notice: /Stage[main]/Keystone::Roles::Admin/Keystone_tenant[service]/ensure: created",
"Notice: /Stage[main]/Keystone::Roles::Admin/Keystone_tenant[admin]/description: description changed 'Bootstrap project for initializing the cloud.' to 'admin tenant'",
"Notice: /Stage[main]/Keystone::Roles::Admin/Keystone_user[admin]/email: defined 'email' as '<email address hidden>'",
"Notice: /Stage[main]/Tripleo::Profile::Base::Keystone/Keystone_user_role[heat_stack_domain_admin::heat_stack@::heat_stack]/ensure: created",
"Notice: /Stage[main]/Ceilometer::Keystone::Auth/Keystone_role[ResellerAdmin]/ensure: created",
"Notice: /Stage[main]/Heat::Keystone::Auth/Keystone_role[heat_stack_user]/ensure: created",
"Notice: /Stage[main]/Keystone::Endpoint/Keystone::Resource::Service_identity[keystone]/Keystone_service[keystone::identity]/ensure: created",
"Notice: /Stage[main]/Keystone::Endpoint/Keystone::Resource::Service_identity[keystone]/Keystone_endpoint[regionOne/keystone::identity]/ensure: created",
"Notice: /Stage[main]/Aodh::Keystone::Auth/Keystone::Resource::Service_identity[aodh]/Keystone_user[aodh]/ensure: created",
"Notice: /Stage[main]/Aodh::Keystone::Auth/Keystone::Resource::Service_identity[aodh]/Keystone_user_role[aodh@service]/ensure: created",
"Notice: /Stage[main]/Aodh::Keystone::Auth/Keystone::Resource::Service_identity[aodh]/Keystone_service[aodh::alarming]/ensure: created",
"Notice: /Stage[main]/Aodh::Keystone::Auth/Keystone::Resource::Service_identity[aodh]/Keystone_endpoint[regionOne/aodh::alarming]/ensure: created",
"Notice: /Stage[main]/Ceilometer::Keystone::Auth/Keystone::Resource::Service_identity[ceilometer]/Keystone_user[ceilometer]/ensure: created",
"Notice: /Stage[main]/Ceilometer::Keystone::Auth/Keystone::Resource::Service_identity[ceilometer]/Keystone_user_role[ceilometer@service]/ensure: created",
"Notice: /Stage[main]/Ceilometer::Keystone::Auth/Keystone::Resource::Service_identity[ceilometer]/Keystone_service[ceilometer::metering]/ensure: created",
"Notice: /Stage[main]/Cinder::Keystone::Auth/Keystone::Resource::Service_identity[cinder]/Keystone_user[cinder]/ensure: created",
"Notice: /Stage[main]/Cinder::Keystone::Auth/Keystone::Resource::Service_identity[cinder]/Keystone_user_role[cinder@service]/ensure: created",
"Notice: /Stage[main]/Cinder::Keystone::Auth/Keystone::Resource::Service_identity[cinder]/Keystone_service[cinder::volume]/ensure: created",
"Notice: /Stage[main]/Cinder::Keystone::Auth/Keystone::Resource::Service_identity[cinder]/Keystone_endpoint[regionOne/cinder::volume]/ensure: created",
"Notice: /Stage[main]/Cinder::Keystone::Auth/Keystone::Resource::Service_identity[cinderv2]/Keystone_service[cinderv2::volumev2]/ensure: created",
"Notice: /Stage[main]/Cinder::Keystone::Auth/Keystone::Resource::Service_identity[cinderv3]/Keystone_service[cinderv3::volumev3]/ensure: created",
"Notice: /Stage[main]/Cinder::Keystone::Auth/Keystone::Resource::Service_identity[cinderv3]/Keystone_endpoint[regionOne/cinderv3::volumev3]/ensure: created",
"Notice: /Stage[main]/Glance::Keystone::Auth/Keystone::Resource::Service_identity[glance]/Keystone_user[glance]/ensure: created",
"Notice: /Stage[main]/Glance::Keystone::Auth/Keystone::Resource::Service_identity[glance]/Keystone_user_role[glance@service]/ensure: created",
"Notice: /Stage[main]/Glance::Keystone::Auth/Keystone::Resource::Service_identity[glance]/Keystone_service[glance::image]/ensure: created",
"Notice: /Stage[main]/Glance::Keystone::Auth/Keystone::Resource::Service_identity[glance]/Keystone_endpoint[regionOne/glance::image]/ensure: created",
"Notice: /Stage[main]/Gnocchi::Keystone::Auth/Keystone::Resource::Service_identity[gnocchi]/Keystone_user[gnocchi]/ensure: created",
"Notice: /Stage[main]/Gnocchi::Keystone::Auth/Keystone::Resource::Service_identity[gnocchi]/Keystone_user_role[gnocchi@service]/ensure: created",
"Notice: /Stage[main]/Gnocchi::Keystone::Auth/Keystone::Resource::Service_identity[gnocchi]/Keystone_service[gnocchi::metric]/ensure: created",
"Notice: /Stage[main]/Gnocchi::Keystone::Auth/Keystone::Resource::Service_identity[gnocchi]/Keystone_endpoint[regionOne/gnocchi::metric]/ensure: created",
"Notice: /Stage[main]/Heat::Keystone::Auth/Keystone::Resource::Service_identity[heat]/Keystone_user[heat]/ensure: created",
"Notice: /Stage[main]/Heat::Keystone::Auth/Keystone::Resource::Service_identity[heat]/Keystone_user_role[heat@service]/ensure: created",
"Notice: /Stage[main]/Heat::Keystone::Auth/Keystone::Resource::Service_identity[heat]/Keystone_service[heat::orchestration]/ensure: created",
"Notice: /Stage[main]/Heat::Keystone::Auth/Keystone::Resource::Service_identity[heat]/Keystone_endpoint[regionOne/heat::orchestration]/ensure: created",
"Notice: /Stage[main]/Heat::Keystone::Auth_cfn/Keystone::Resource::Service_identity[heat-cfn]/Keystone_user[heat-cfn]/ensure: created",
"Notice: /Stage[main]/Heat::Keystone::Auth_cfn/Keystone::Resource::Service_identity[heat-cfn]/Keystone_user_role[heat-cfn@service]/ensure: created",
"Notice: /Stage[main]/Heat::Keystone::Auth_cfn/Keystone::Resource::Service_identity[heat-cfn]/Keystone_service[heat-cfn::cloudformation]/ensure: created",
"Notice: /Stage[main]/Heat::Keystone::Auth_cfn/Keystone::Resource::Service_identity[heat-cfn]/Keystone_endpoint[regionOne/heat-cfn::cloudformation]/ensure: created",
"Notice: /Stage[main]/Neutron::Keystone::Auth/Keystone::Resource::Service_identity[neutron]/Keystone_user[neutron]/ensure: created",
"Notice: /Stage[main]/Neutron::Keystone::Auth/Keystone::Resource::Service_identity[neutron]/Keystone_user_role[neutron@service]/ensure: created",
"Notice: /Stage[main]/Neutron::Keystone::Auth/Keystone::Resource::Service_identity[neutron]/Keystone_service[neutron::network]/ensure: created",
"Notice: /Stage[main]/Neutron::Keystone::Auth/Keystone::Resource::Service_identity[neutron]/Keystone_endpoint[regionOne/neutron::network]/ensure: created",
"Notice: /Stage[main]/Nova::Keystone::Auth/Keystone::Resource::Service_identity[nova]/Keystone_user[nova]/ensure: created",
"Notice: /Stage[main]/Nova::Keystone::Auth/Keystone::Resource::Service_identity[nova]/Keystone_user_role[nova@service]/ensure: created",
"Notice: /Stage[main]/Nova::Keystone::Auth/Keystone::Resource::Service_identity[nova]/Keystone_service[nova::compute]/ensure: created",
"Notice: /Stage[main]/Nova::Keystone::Auth/Keystone::Resource::Service_identity[nova]/Keystone_endpoint[regionOne/nova::compute]/ensure: created",
"Notice: /Stage[main]/Nova::Keystone::Auth_placement/Keystone::Resource::Service_identity[placement]/Keystone_user[placement]/ensure: created",
"Notice: /Stage[main]/Nova::Keystone::Auth_placement/Keystone::Resource::Service_identity[placement]/Keystone_user_role[placement@service]/ensure: created",
"Notice: /Stage[main]/Nova::Keystone::Auth_placement/Keystone::Resource::Service_identity[placement]/Keystone_service[placement::placement]/ensure: created",
"Notice: /Stage[main]/Nova::Keystone::Auth_placement/Keystone::Resource::Service_identity[placement]/Keystone_endpoint[regionOne/placement::placement]/ensure: created",
"Notice: /Stage[main]/Panko::Keystone::Auth/Keystone::Resource::Service_identity[panko]/Keystone_user[panko]/ensure: created",
"Notice: /Stage[main]/Panko::Keystone::Auth/Keystone::Resource::Service_identity[panko]/Keystone_user_role[panko@service]/ensure: created",
"Notice: /Stage[main]/Panko::Keystone::Auth/Keystone::Resource::Service_identity[panko]/Keystone_service[panko::event]/ensure: created",
"Notice: /Stage[main]/Panko::Keystone::Auth/Keystone::Resource::Service_identity[panko]/Keystone_endpoint[regionOne/panko::event]/ensure: created",
"Notice: Applied catalog in 443.45 seconds",
"Changes:",
" Total: 176",
"Events:",
" Success: 176",
" Total: 178",
" Failure: 2",
"Resources:",
" Failed: 1",
" Skipped: 130",
" Changed: 176",
" Out of sync: 177",
" Total: 472",
"Time:",
" Resources: 0.00",
" Package: 0.00",
" Concat file: 0.00",
" Concat fragment: 0.00",
" File: 0.59",
" Keystone config: 1.73",
" Keystone user role: 107.44",
" Last run: 1530757763",
" Keystone endpoint: 187.23",
" Keystone domain: 2.69",
" Keystone service: 35.38",
" Total: 435.83",
" Keystone role: 5.20",
" Keystone tenant: 5.22",
" Keystone user: 80.78",
" Config retrieval: 9.57",
"Version:",
" Config: 1530757310",
" Puppet: 4.8.2",
"",
"2018-07-05 02:29:25,404 ERROR: 106252 -- + mkdir -p /etc/puppet",
"+ cp -a /tmp/puppet-etc/auth.conf /tmp/puppet-etc/hiera.yaml /tmp/puppet-etc/hieradata /tmp/puppet-etc/modules /tmp/puppet-etc/puppet.conf /tmp/puppet-etc/ssl /etc/puppet",
"+ rm -Rf /etc/puppet/ssl",
"+ echo '{\"step\": 3}'",
"+ TAGS=",
"+ '[' -n file,file_line,concat,augeas,cron,keystone_config,keystone_domain_config,keystone_endpoint,keystone_identity_provider,keystone_paste_ini,keystone_role,keystone_service,keystone_tenant,keystone_user,keystone_user_role,keystone_domain ']'",
"+ TAGS='--tags file,file_line,concat,augeas,cron,keystone_config,keystone_domain_config,keystone_endpoint,keystone_identity_provider,keystone_paste_ini,keystone_role,keystone_service,keystone_tenant,keystone_user,keystone_user_role,keystone_domain'",
"+ origin_of_time=/var/lib/config-data/keystone_init_tasks.origin_of_time",
"+ touch /var/lib/config-data/keystone_init_tasks.origin_of_time",
"+ sync",
"+ set +e",
"+ FACTER_hostname=centos-7-inap-mtl01-0000555891",
"+ FACTER_uuid=docker",
"+ /usr/bin/puppet apply --summarize --detailed-exitcodes --color=false --logdest syslog --logdest console --modulepath=/etc/puppet/modules:/usr/share/openstack-puppet/modules --tags file,file_line,concat,augeas,cron,keystone_config,keystone_domain_config,keystone_endpoint,keystone_identity_provider,keystone_paste_ini,keystone_role,keystone_service,keystone_tenant,keystone_user,keystone_user_role,keystone_domain /etc/config.pp",
"Failed to get D-Bus connection: Operation not permitted",
"Warning: Facter: Could not retrieve fact='nic_alias', resolution='<anonymous>': Could not execute '/usr/bin/os-net-config -i': command not found",
"Warning: Undefined variable 'deploy_config_name'; ",
" (file & line not available)",
"Warning: ModuleLoader: module 'keystone' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules",
"Warning: This method is deprecated, please use the stdlib validate_legacy function,",
" with Stdlib::Compat::Hash. There is further documentation for validate_legacy function in the README. at [\"/etc/puppet/modules/keystone/manifests/policy.pp\", 34]:[\"/etc/puppet/modules/keystone/manifests/init.pp\", 757]",
" (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:28:in `deprecation')",
" with Pattern[]. There is further documentation for validate_legacy function in the README. at [\"/etc/puppet/modules/keystone/manifests/init.pp\", 760]:[\"/etc/config.pp\", 2]",
"Warning: ModuleLoader: module 'openstacklib' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules",
" with Stdlib::Compat::String. There is further documentation for validate_legacy function in the README. at [\"/etc/puppet/modules/keystone/manifests/init.pp\", 1108]:[\"/etc/config.pp\", 2]",
"Warning: ModuleLoader: module 'aodh' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules",
"Warning: ModuleLoader: module 'ceilometer' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules",
"Warning: ModuleLoader: module 'cinder' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules",
"Warning: ModuleLoader: module 'glance' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules",
"Warning: ModuleLoader: module 'gnocchi' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules",
"Warning: ModuleLoader: module 'heat' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules",
"Warning: ModuleLoader: module 'neutron' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules",
"Warning: ModuleLoader: module 'nova' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules",
"Warning: ModuleLoader: module 'panko' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules",
"Warning: ModuleLoader: module 'oslo' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules",
"Warning: ModuleLoader: module 'mysql' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules",
"Warning: Puppet::Type::Keystone_tenant::ProviderOpenstack: Support for a resource without the domain set is deprecated in Liberty cycle. It will be dropped in the M-cycle. Currently using 'Default' as default domain name while the default domain id is '36905d55288241cf87ba14894ecbdd0f'.",
"Error: Execution of '/usr/bin/openstack endpoint create --format shell d3f224ccae2b46d587460d0baa1c7d08 admin http://192.168.24.10:8776/v2/%(tenant_id)s --region regionOne' returned 1: An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-b52baa11-a9ee-49d5-b395-0b65409c0af9)",
"Error: /Stage[main]/Cinder::Keystone::Auth/Keystone::Resource::Service_identity[cinderv2]/Keystone_endpoint[regionOne/cinderv2::volumev2]: Could not evaluate: Puppet::Util::Log requires a message",
"+ rc=6",
"+ set -e",
"+ '[' 6 -ne 2 -a 6 -ne 0 ']'",
"+ exit 6",
"2018-07-05 02:29:25,404 INFO: 106252 -- Finished processing puppet configs for keystone_init_tasks",
"2018-07-05 02:29:25,406 ERROR: 106249 -- ERROR configuring keystone_init_tasks"
]
}
2018-07-05 02:29:25,902 p=8381 u=mistral | NO MORE HOSTS LEFT *************************************************************
2018-07-05 02:29:25,903 p=8381 u=mistral | PLAY RECAP *********************************************************************
2018-07-05 02:29:25,903 p=8381 u=mistral | centos-7-inap-mtl01-0000555891 : ok=188 changed=59 unreachable=0 failed=1
2018-07-05 02:29:25,903 p=8381 u=mistral | undercloud : ok=21 changed=10 unreachable=0 failed=0
2018-07-05 02:29:25,905 p=8381 u=mistral | Thursday 05 July 2018 02:29:25 +0000 (0:00:00.220) 0:43:17.189 *********
2018-07-05 02:29:25,905 p=8381 u=mistral | ===============================================================================
This failure has happened recently in master noop jobs, looks like a bug in keystone init step
From keystone.log: http://logs.openstack.org/45/560445/78/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/a37efa0/logs/undercloud/var/log/keystone/keystone.log.txt.gz#_2018-07-05_01_03_36_417
2018-07-05 01:03:36.417 1182 ERROR keystone.assignment.core [req-0ce9a356-d50f-4d36-8f5b-b129f0ff5dc9 03412e950dfd48f894b03dc009141cf3 e3d0c70bb2954e03ac88a04a545c9c61 - default default] Circular reference found role inference rules - 778b27a98cbf484dad80c97cb6365126.: NeedRegenerationException
let's check if we hit this again