Detailed bug description:
Deploying a cluster HA fails with "All nodes are finished. Failed tasks: Task[openstack-network-networks/7] Stopping the deployment process!"
Steps to reproduce:
3 controllers
3 ceph + compute
Not L3 HA selected.
Expected results:
Openstack environment in an HA cluster with 3 controllers and 3 nodes (ceph and compute)
Actual result:
Deployment fails. The failing node shows this on puppet.log:
<code>
2017-10-31 15:05:07 +0000 Scope(Class[Openstack_tasks::Openstack_network::Networks]) (notice): MODULAR: openstack_network/networks.pp
2017-10-31 15:05:08 +0000 Puppet (notice): Compiled catalog for ha-ctrl02.demolab.local in environment production in 0.33 seconds
2017-10-31 15:05:11 +0000 Puppet::Type::Neutron_network::ProviderNeutron (notice): Unable to complete neutron request due to non-fatal error: "Execution of '/usr/bin/neutron net-list --format=csv --column=id --quote=none' returned 1: neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead.
Connection to neutron failed: Failed to connect Neutron server". Retrying for 8 sec.
2017-10-31 15:05:15 +0000 Puppet::Type::Neutron_network::ProviderNeutron (notice): Unable to complete neutron request due to non-fatal error: "Execution of '/usr/bin/neutron net-list --format=csv --column=id --quote=none' returned 1: neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead.
Connection to neutron failed: Failed to connect Neutron server". Retrying for 4 sec.
2017-10-31 15:05:19 +0000 Puppet::Type::Neutron_network::ProviderNeutron (notice): Unable to complete neutron request due to non-fatal error: "Execution of '/usr/bin/neutron net-list --format=csv --column=id --quote=none' returned 1: neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead.
Connection to neutron failed: Failed to connect Neutron server". Retrying for 0 sec.
2017-10-31 15:05:23 +0000 Puppet (err): Could not prefetch neutron_network provider 'neutron': Can't retrieve net-list because Neutron or Keystone API is not available.
/etc/puppet/modules/neutron/lib/puppet/provider/neutron.rb:121:in `list_neutron_resources'
/etc/puppet/modules/neutron/lib/puppet/provider/neutron_network/neutron.rb:39:in `instances'
/etc/puppet/modules/neutron/lib/puppet/provider/neutron_network/neutron.rb:60:in `prefetch'
/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:307:in `prefetch'
/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:197:in `prefetch_if_necessary'
/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:96:in `block in evaluate'
/usr/lib/ruby/vendor_ruby/puppet/graph/relationship_graph.rb:116:in `traverse'
/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:138:in `evaluate'
/usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:169:in `block in apply'
/usr/lib/ruby/vendor_ruby/puppet/util/log.rb:149:in `with_destination'
/usr/lib/ruby/vendor_ruby/puppet/transaction/report.rb:112:in `as_logging_destination'
/usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:168:in `apply'
/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:120:in `block in apply_catalog'
/usr/lib/ruby/vendor_ruby/puppet/util.rb:161:in `block in benchmark'
/usr/lib/ruby/2.3.0/benchmark.rb:308:in `realtime'
/usr/lib/ruby/vendor_ruby/puppet/util.rb:160:in `benchmark'
/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:119:in `apply_catalog'
/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:227:in `run_internal'
/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:134:in `block in run'
/usr/lib/ruby/vendor_ruby/puppet/context.rb:64:in `override'
/usr/lib/ruby/vendor_ruby/puppet.rb:246:in `override'
/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:133:in `run'
/usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:302:in `apply_catalog'
/usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:236:in `block in main'
/usr/lib/ruby/vendor_ruby/puppet/context.rb:64:in `override'
/usr/lib/ruby/vendor_ruby/puppet.rb:246:in `override'
/usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:198:in `main'
/usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:159:in `run_command'
/usr/lib/ruby/vendor_ruby/puppet/application.rb:381:in `block (2 levels) in run'
/usr/lib/ruby/vendor_ruby/puppet/application.rb:507:in `plugin_hook'
/usr/lib/ruby/vendor_ruby/puppet/application.rb:381:in `block in run'
/usr/lib/ruby/vendor_ruby/puppet/util.rb:496:in `exit_on_fail'
/usr/lib/ruby/vendor_ruby/puppet/application.rb:381:in `run'
/usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:146:in `run'
/usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:92:in `execute'
/usr/bin/puppet:8:in `<main>'
2017-10-31 15:05:23 +0000 Puppet (err): Not managing Neutron_network[admin_floating_net] due to earlier Neutron API failures.
/usr/lib/ruby/vendor_ruby/puppet/util/errors.rb:106:in `fail'
/etc/puppet/modules/neutron/lib/puppet/provider/neutron_network/neutron.rb:74:in `create'
/usr/lib/ruby/vendor_ruby/puppet/property/ensure.rb:16:in `block in defaultvalues'
/usr/lib/ruby/vendor_ruby/puppet/property.rb:197:in `call_valuemethod'
/usr/lib/ruby/vendor_ruby/puppet/property.rb:498:in `set'
/usr/lib/ruby/vendor_ruby/puppet/property.rb:581:in `sync'
/usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:204:in `sync'
/usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:128:in `sync_if_needed'
/usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:81:in `perform_changes'
/usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:20:in `evaluate'
/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:204:in `apply'
/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:217:in `eval_resource'
/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:147:in `call'
/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:147:in `block (2 levels) in evaluate'
/usr/lib/ruby/vendor_ruby/puppet/util.rb:335:in `block in thinmark'
/usr/lib/ruby/2.3.0/benchmark.rb:308:in `realtime'
/usr/lib/ruby/vendor_ruby/puppet/util.rb:334:in `thinmark'
/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:147:in `block in evaluate'
/usr/lib/ruby/vendor_ruby/puppet/graph/relationship_graph.rb:118:in `traverse'
/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:138:in `evaluate'
/usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:169:in `block in apply'
/usr/lib/ruby/vendor_ruby/puppet/util/log.rb:149:in `with_destination'
/usr/lib/ruby/vendor_ruby/puppet/transaction/report.rb:112:in `as_logging_destination'
/usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:168:in `apply'
/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:120:in `block in apply_catalog'
/usr/lib/ruby/vendor_ruby/puppet/util.rb:161:in `block in benchmark'
/usr/lib/ruby/2.3.0/benchmark.rb:308:in `realtime'
/usr/lib/ruby/vendor_ruby/puppet/util.rb:160:in `benchmark'
/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:119:in `apply_catalog'
/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:227:in `run_internal'
/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:134:in `block in run'
/usr/lib/ruby/vendor_ruby/puppet/context.rb:64:in `override'
/usr/lib/ruby/vendor_ruby/puppet.rb:246:in `override'
/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:133:in `run'
/usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:302:in `apply_catalog'
/usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:236:in `block in main'
/usr/lib/ruby/vendor_ruby/puppet/context.rb:64:in `override'
/usr/lib/ruby/vendor_ruby/puppet.rb:246:in `override'
/usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:198:in `main'
/usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:159:in `run_command'
/usr/lib/ruby/vendor_ruby/puppet/application.rb:381:in `block (2 levels) in run'
/usr/lib/ruby/vendor_ruby/puppet/application.rb:507:in `plugin_hook'
/usr/lib/ruby/vendor_ruby/puppet/application.rb:381:in `block in run'
/usr/lib/ruby/vendor_ruby/puppet/util.rb:496:in `exit_on_fail'
/usr/lib/ruby/vendor_ruby/puppet/application.rb:381:in `run'
/usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:146:in `run'
/usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:92:in `execute'
/usr/bin/puppet:8:in `<main>'
2017-10-31 15:05:23 +0000 /Stage[main]/Openstack_tasks::Openstack_network::Networks/Neutron_network[admin_floating_net]/ensure (err): change from absent to present failed: Not managing Neutron_network[admin_floating_net] due to earlier Neutron API failures.
2017-10-31 15:05:23 +0000 Puppet (err): Not managing Neutron_network[admin_internal_net] due to earlier Neutron API failures.
/usr/lib/ruby/vendor_ruby/puppet/util/errors.rb:106:in `fail'
/etc/puppet/modules/neutron/lib/puppet/provider/neutron_network/neutron.rb:74:in `create'
/usr/lib/ruby/vendor_ruby/puppet/property/ensure.rb:16:in `block in defaultvalues'
/usr/lib/ruby/vendor_ruby/puppet/property.rb:197:in `call_valuemethod'
/usr/lib/ruby/vendor_ruby/puppet/property.rb:498:in `set'
/usr/lib/ruby/vendor_ruby/puppet/property.rb:581:in `sync'
/usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:204:in `sync'
/usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:128:in `sync_if_needed'
/usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:81:in `perform_changes'
/usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:20:in `evaluate'
/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:204:in `apply'
/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:217:in `eval_resource'
/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:147:in `call'
/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:147:in `block (2 levels) in evaluate'
/usr/lib/ruby/vendor_ruby/puppet/util.rb:335:in `block in thinmark'
/usr/lib/ruby/2.3.0/benchmark.rb:308:in `realtime'
/usr/lib/ruby/vendor_ruby/puppet/util.rb:334:in `thinmark'
/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:147:in `block in evaluate'
/usr/lib/ruby/vendor_ruby/puppet/graph/relationship_graph.rb:118:in `traverse'
/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:138:in `evaluate'
/usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:169:in `block in apply'
/usr/lib/ruby/vendor_ruby/puppet/util/log.rb:149:in `with_destination'
/usr/lib/ruby/vendor_ruby/puppet/transaction/report.rb:112:in `as_logging_destination'
/usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:168:in `apply'
/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:120:in `block in apply_catalog'
/usr/lib/ruby/vendor_ruby/puppet/util.rb:161:in `block in benchmark'
/usr/lib/ruby/2.3.0/benchmark.rb:308:in `realtime'
/usr/lib/ruby/vendor_ruby/puppet/util.rb:160:in `benchmark'
/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:119:in `apply_catalog'
/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:227:in `run_internal'
/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:134:in `block in run'
/usr/lib/ruby/vendor_ruby/puppet/context.rb:64:in `override'
/usr/lib/ruby/vendor_ruby/puppet.rb:246:in `override'
/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:133:in `run'
/usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:302:in `apply_catalog'
/usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:236:in `block in main'
/usr/lib/ruby/vendor_ruby/puppet/context.rb:64:in `override'
/usr/lib/ruby/vendor_ruby/puppet.rb:246:in `override'
/usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:198:in `main'
/usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:159:in `run_command'
/usr/lib/ruby/vendor_ruby/puppet/application.rb:381:in `block (2 levels) in run'
/usr/lib/ruby/vendor_ruby/puppet/application.rb:507:in `plugin_hook'
/usr/lib/ruby/vendor_ruby/puppet/application.rb:381:in `block in run'
/usr/lib/ruby/vendor_ruby/puppet/util.rb:496:in `exit_on_fail'
/usr/lib/ruby/vendor_ruby/puppet/application.rb:381:in `run'
/usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:146:in `run'
/usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:92:in `execute'
/usr/bin/puppet:8:in `<main>'
2017-10-31 15:05:23 +0000 /Stage[main]/Openstack_tasks::Openstack_network::Networks/Neutron_network[admin_internal_net]/ensure (err): change from absent to present failed: Not managing Neutron_network[admin_internal_net] due to earlier Neutron API failures.
2017-10-31 15:05:24 +0000 Puppet::Type::Neutron_subnet::ProviderNeutron (notice): Unable to complete neutron request due to non-fatal error: "Execution of '/usr/bin/neutron subnet-list --format=csv --column=id --quote=none' returned 1: neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead.
Connection to neutron failed: Failed to connect Neutron server". Retrying for 9 sec.
2017-10-31 15:05:28 +0000 Puppet::Type::Neutron_subnet::ProviderNeutron (notice): Unable to complete neutron request due to non-fatal error: "Execution of '/usr/bin/neutron subnet-list --format=csv --column=id --quote=none' returned 1: neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead.
Connection to neutron failed: Failed to connect Neutron server". Retrying for 5 sec.
2017-10-31 15:05:31 +0000 Puppet::Type::Neutron_subnet::ProviderNeutron (notice): Unable to complete neutron request due to non-fatal error: "Execution of '/usr/bin/neutron subnet-list --format=csv --column=id --quote=none' returned 1: neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead.
Connection to neutron failed: Failed to connect Neutron server". Retrying for 2 sec.
2017-10-31 15:05:35 +0000 Puppet (err): Could not prefetch neutron_subnet provider 'neutron': Can't retrieve subnet-list because Neutron or Keystone API is not available.
/etc/puppet/modules/neutron/lib/puppet/provider/neutron.rb:121:in `list_neutron_resources'
/etc/puppet/modules/neutron/lib/puppet/provider/neutron_subnet/neutron.rb:31:in `instances'
/etc/puppet/modules/neutron/lib/puppet/provider/neutron_subnet/neutron.rb:55:in `prefetch'
/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:307:in `prefetch'
/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:197:in `prefetch_if_necessary'
/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:96:in `block in evaluate'
/usr/lib/ruby/vendor_ruby/puppet/graph/relationship_graph.rb:116:in `traverse'
/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:138:in `evaluate'
/usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:169:in `block in apply'
/usr/lib/ruby/vendor_ruby/puppet/util/log.rb:149:in `with_destination'
/usr/lib/ruby/vendor_ruby/puppet/transaction/report.rb:112:in `as_logging_destination'
/usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:168:in `apply'
/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:120:in `block in apply_catalog'
/usr/lib/ruby/vendor_ruby/puppet/util.rb:161:in `block in benchmark'
/usr/lib/ruby/2.3.0/benchmark.rb:308:in `realtime'
/usr/lib/ruby/vendor_ruby/puppet/util.rb:160:in `benchmark'
/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:119:in `apply_catalog'
/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:227:in `run_internal'
/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:134:in `block in run'
/usr/lib/ruby/vendor_ruby/puppet/context.rb:64:in `override'
/usr/lib/ruby/vendor_ruby/puppet.rb:246:in `override'
/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:133:in `run'
/usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:302:in `apply_catalog'
/usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:236:in `block in main'
/usr/lib/ruby/vendor_ruby/puppet/context.rb:64:in `override'
/usr/lib/ruby/vendor_ruby/puppet.rb:246:in `override'
/usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:198:in `main'
/usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:159:in `run_command'
/usr/lib/ruby/vendor_ruby/puppet/application.rb:381:in `block (2 levels) in run'
/usr/lib/ruby/vendor_ruby/puppet/application.rb:507:in `plugin_hook'
/usr/lib/ruby/vendor_ruby/puppet/application.rb:381:in `block in run'
/usr/lib/ruby/vendor_ruby/puppet/util.rb:496:in `exit_on_fail'
/usr/lib/ruby/vendor_ruby/puppet/application.rb:381:in `run'
/usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:146:in `run'
/usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:92:in `execute'
/usr/bin/puppet:8:in `<main>'
2017-10-31 15:05:35 +0000 /Stage[main]/Openstack_tasks::Openstack_network::Networks/Neutron_subnet[admin_internal_net__subnet] (notice): Dependency Neutron_network[admin_internal_net] has failures: true
2017-10-31 15:05:35 +0000 /Stage[main]/Openstack_tasks::Openstack_network::Networks/Neutron_subnet[admin_internal_net__subnet] (warning): Skipping because of failed dependencies
2017-10-31 15:05:35 +0000 /Stage[main]/Openstack_tasks::Openstack_network::Networks/Neutron_subnet[admin_floating_net__subnet] (notice): Dependency Neutron_network[admin_floating_net] has failures: true
2017-10-31 15:05:35 +0000 /Stage[main]/Openstack_tasks::Openstack_network::Networks/Neutron_subnet[admin_floating_net__subnet] (warning): Skipping because of failed dependencies
2017-10-31 15:05:35 +0000 Puppet (notice): Finished catalog run in 26.34 seconds
</code>
It seems some dependencies problems, but I cannot find out what they are...
Reproducibility:
Declare an openstack env with 3 controllers and 3 nodes for ceph and compute.
Description of the environment:
6 VMs acting as servers
Fuel 11.0 #956
Here are the detailed snapshot:
https://drive.ingenia.es/index.php/s/JpMsetzmzjGkBaz
Thanks,
