Fuel for OpenStack

Fuel 11 fails to deploy cluster due to errors on openstack-network-networks module

Bug #1729035 reported by Jorge Rodriguez
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Incomplete
Undecided
Unassigned

Bug Description

Detailed bug description:

Deploying a cluster HA fails with "All nodes are finished. Failed tasks: Task[openstack-network-networks/7] Stopping the deployment process!"

Steps to reproduce:

3 controllers
3 ceph + compute
Not L3 HA selected.

Expected results:
Openstack environment in an HA cluster with 3 controllers and 3 nodes (ceph and compute)

Actual result:
Deployment fails. The failing node shows this on puppet.log:

2017-10-31 15:05:07 +0000 Scope(Class[Openstack_tasks::Openstack_network::Networks]) (notice): MODULAR: openstack_network/networks.pp
2017-10-31 15:05:08 +0000 Puppet (notice): Compiled catalog for ha-ctrl02.demolab.local in environment production in 0.33 seconds
2017-10-31 15:05:11 +0000 Puppet::Type::Neutron_network::ProviderNeutron (notice): Unable to complete neutron request due to non-fatal error: "Execution of '/usr/bin/neutron net-list --format=csv --column=id --quote=none' returned 1: neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead.
Connection to neutron failed: Failed to connect Neutron server". Retrying for 8 sec.
2017-10-31 15:05:15 +0000 Puppet::Type::Neutron_network::ProviderNeutron (notice): Unable to complete neutron request due to non-fatal error: "Execution of '/usr/bin/neutron net-list --format=csv --column=id --quote=none' returned 1: neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead.
Connection to neutron failed: Failed to connect Neutron server". Retrying for 4 sec.
2017-10-31 15:05:19 +0000 Puppet::Type::Neutron_network::ProviderNeutron (notice): Unable to complete neutron request due to non-fatal error: "Execution of '/usr/bin/neutron net-list --format=csv --column=id --quote=none' returned 1: neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead.
Connection to neutron failed: Failed to connect Neutron server". Retrying for 0 sec.
2017-10-31 15:05:23 +0000 Puppet (err): Could not prefetch neutron_network provider 'neutron': Can't retrieve net-list because Neutron or Keystone API is not available.
/etc/puppet/modules/neutron/lib/puppet/provider/neutron.rb:121:in `list_neutron_resources'
/etc/puppet/modules/neutron/lib/puppet/provider/neutron_network/neutron.rb:39:in `instances'
/etc/puppet/modules/neutron/lib/puppet/provider/neutron_network/neutron.rb:60:in `prefetch'
/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:307:in `prefetch'
/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:197:in `prefetch_if_necessary'
/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:96:in `block in evaluate'
/usr/lib/ruby/vendor_ruby/puppet/graph/relationship_graph.rb:116:in `traverse'
/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:138:in `evaluate'
/usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:169:in `block in apply'
/usr/lib/ruby/vendor_ruby/puppet/util/log.rb:149:in `with_destination'
/usr/lib/ruby/vendor_ruby/puppet/transaction/report.rb:112:in `as_logging_destination'
/usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:168:in `apply'
/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:120:in `block in apply_catalog'
/usr/lib/ruby/vendor_ruby/puppet/util.rb:161:in `block in benchmark'
/usr/lib/ruby/2.3.0/benchmark.rb:308:in `realtime'
/usr/lib/ruby/vendor_ruby/puppet/util.rb:160:in `benchmark'
/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:119:in `apply_catalog'
/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:227:in `run_internal'
/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:134:in `block in run'
/usr/lib/ruby/vendor_ruby/puppet/context.rb:64:in `override'
/usr/lib/ruby/vendor_ruby/puppet.rb:246:in `override'
/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:133:in `run'
/usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:302:in `apply_catalog'
/usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:236:in `block in main'
/usr/lib/ruby/vendor_ruby/puppet/context.rb:64:in `override'
/usr/lib/ruby/vendor_ruby/puppet.rb:246:in `override'
/usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:198:in `main'
/usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:159:in `run_command'
/usr/lib/ruby/vendor_ruby/puppet/application.rb:381:in `block (2 levels) in run'
/usr/lib/ruby/vendor_ruby/puppet/application.rb:507:in `plugin_hook'
/usr/lib/ruby/vendor_ruby/puppet/application.rb:381:in `block in run'
/usr/lib/ruby/vendor_ruby/puppet/util.rb:496:in `exit_on_fail'
/usr/lib/ruby/vendor_ruby/puppet/application.rb:381:in `run'
/usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:146:in `run'
/usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:92:in `execute'
/usr/bin/puppet:8:in `<main>'
2017-10-31 15:05:23 +0000 Puppet (err): Not managing Neutron_network[admin_floating_net] due to earlier Neutron API failures.
/usr/lib/ruby/vendor_ruby/puppet/util/errors.rb:106:in `fail'
/etc/puppet/modules/neutron/lib/puppet/provider/neutron_network/neutron.rb:74:in `create'
/usr/lib/ruby/vendor_ruby/puppet/property/ensure.rb:16:in `block in defaultvalues'
/usr/lib/ruby/vendor_ruby/puppet/property.rb:197:in `call_valuemethod'
/usr/lib/ruby/vendor_ruby/puppet/property.rb:498:in `set'
/usr/lib/ruby/vendor_ruby/puppet/property.rb:581:in `sync'
/usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:204:in `sync'
/usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:128:in `sync_if_needed'
/usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:81:in `perform_changes'
/usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:20:in `evaluate'
/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:204:in `apply'
/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:217:in `eval_resource'
/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:147:in `call'
/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:147:in `block (2 levels) in evaluate'
/usr/lib/ruby/vendor_ruby/puppet/util.rb:335:in `block in thinmark'
/usr/lib/ruby/2.3.0/benchmark.rb:308:in `realtime'
/usr/lib/ruby/vendor_ruby/puppet/util.rb:334:in `thinmark'
/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:147:in `block in evaluate'
/usr/lib/ruby/vendor_ruby/puppet/graph/relationship_graph.rb:118:in `traverse'
/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:138:in `evaluate'
/usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:169:in `block in apply'
/usr/lib/ruby/vendor_ruby/puppet/util/log.rb:149:in `with_destination'
/usr/lib/ruby/vendor_ruby/puppet/transaction/report.rb:112:in `as_logging_destination'
/usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:168:in `apply'
/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:120:in `block in apply_catalog'
/usr/lib/ruby/vendor_ruby/puppet/util.rb:161:in `block in benchmark'
/usr/lib/ruby/2.3.0/benchmark.rb:308:in `realtime'
/usr/lib/ruby/vendor_ruby/puppet/util.rb:160:in `benchmark'
/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:119:in `apply_catalog'
/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:227:in `run_internal'
/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:134:in `block in run'
/usr/lib/ruby/vendor_ruby/puppet/context.rb:64:in `override'
/usr/lib/ruby/vendor_ruby/puppet.rb:246:in `override'
/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:133:in `run'
/usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:302:in `apply_catalog'
/usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:236:in `block in main'
/usr/lib/ruby/vendor_ruby/puppet/context.rb:64:in `override'
/usr/lib/ruby/vendor_ruby/puppet.rb:246:in `override'
/usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:198:in `main'
/usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:159:in `run_command'
/usr/lib/ruby/vendor_ruby/puppet/application.rb:381:in `block (2 levels) in run'
/usr/lib/ruby/vendor_ruby/puppet/application.rb:507:in `plugin_hook'
/usr/lib/ruby/vendor_ruby/puppet/application.rb:381:in `block in run'
/usr/lib/ruby/vendor_ruby/puppet/util.rb:496:in `exit_on_fail'
/usr/lib/ruby/vendor_ruby/puppet/application.rb:381:in `run'
/usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:146:in `run'
/usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:92:in `execute'
/usr/bin/puppet:8:in `<main>'
2017-10-31 15:05:23 +0000 /Stage[main]/Openstack_tasks::Openstack_network::Networks/Neutron_network[admin_floating_net]/ensure (err): change from absent to present failed: Not managing Neutron_network[admin_floating_net] due to earlier Neutron API failures.
2017-10-31 15:05:23 +0000 Puppet (err): Not managing Neutron_network[admin_internal_net] due to earlier Neutron API failures.
/usr/lib/ruby/vendor_ruby/puppet/util/errors.rb:106:in `fail'
/etc/puppet/modules/neutron/lib/puppet/provider/neutron_network/neutron.rb:74:in `create'
/usr/lib/ruby/vendor_ruby/puppet/property/ensure.rb:16:in `block in defaultvalues'
/usr/lib/ruby/vendor_ruby/puppet/property.rb:197:in `call_valuemethod'
/usr/lib/ruby/vendor_ruby/puppet/property.rb:498:in `set'
/usr/lib/ruby/vendor_ruby/puppet/property.rb:581:in `sync'
/usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:204:in `sync'
/usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:128:in `sync_if_needed'
/usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:81:in `perform_changes'
/usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:20:in `evaluate'
/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:204:in `apply'
/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:217:in `eval_resource'
/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:147:in `call'
/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:147:in `block (2 levels) in evaluate'
/usr/lib/ruby/vendor_ruby/puppet/util.rb:335:in `block in thinmark'
/usr/lib/ruby/2.3.0/benchmark.rb:308:in `realtime'
/usr/lib/ruby/vendor_ruby/puppet/util.rb:334:in `thinmark'
/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:147:in `block in evaluate'
/usr/lib/ruby/vendor_ruby/puppet/graph/relationship_graph.rb:118:in `traverse'
/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:138:in `evaluate'
/usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:169:in `block in apply'
/usr/lib/ruby/vendor_ruby/puppet/util/log.rb:149:in `with_destination'
/usr/lib/ruby/vendor_ruby/puppet/transaction/report.rb:112:in `as_logging_destination'
/usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:168:in `apply'
/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:120:in `block in apply_catalog'
/usr/lib/ruby/vendor_ruby/puppet/util.rb:161:in `block in benchmark'
/usr/lib/ruby/2.3.0/benchmark.rb:308:in `realtime'
/usr/lib/ruby/vendor_ruby/puppet/util.rb:160:in `benchmark'
/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:119:in `apply_catalog'
/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:227:in `run_internal'
/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:134:in `block in run'
/usr/lib/ruby/vendor_ruby/puppet/context.rb:64:in `override'
/usr/lib/ruby/vendor_ruby/puppet.rb:246:in `override'
/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:133:in `run'
/usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:302:in `apply_catalog'
/usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:236:in `block in main'
/usr/lib/ruby/vendor_ruby/puppet/context.rb:64:in `override'
/usr/lib/ruby/vendor_ruby/puppet.rb:246:in `override'
/usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:198:in `main'
/usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:159:in `run_command'
/usr/lib/ruby/vendor_ruby/puppet/application.rb:381:in `block (2 levels) in run'
/usr/lib/ruby/vendor_ruby/puppet/application.rb:507:in `plugin_hook'
/usr/lib/ruby/vendor_ruby/puppet/application.rb:381:in `block in run'
/usr/lib/ruby/vendor_ruby/puppet/util.rb:496:in `exit_on_fail'
/usr/lib/ruby/vendor_ruby/puppet/application.rb:381:in `run'
/usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:146:in `run'
/usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:92:in `execute'
/usr/bin/puppet:8:in `<main>'
2017-10-31 15:05:23 +0000 /Stage[main]/Openstack_tasks::Openstack_network::Networks/Neutron_network[admin_internal_net]/ensure (err): change from absent to present failed: Not managing Neutron_network[admin_internal_net] due to earlier Neutron API failures.
2017-10-31 15:05:24 +0000 Puppet::Type::Neutron_subnet::ProviderNeutron (notice): Unable to complete neutron request due to non-fatal error: "Execution of '/usr/bin/neutron subnet-list --format=csv --column=id --quote=none' returned 1: neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead.
Connection to neutron failed: Failed to connect Neutron server". Retrying for 9 sec.
2017-10-31 15:05:28 +0000 Puppet::Type::Neutron_subnet::ProviderNeutron (notice): Unable to complete neutron request due to non-fatal error: "Execution of '/usr/bin/neutron subnet-list --format=csv --column=id --quote=none' returned 1: neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead.
Connection to neutron failed: Failed to connect Neutron server". Retrying for 5 sec.
2017-10-31 15:05:31 +0000 Puppet::Type::Neutron_subnet::ProviderNeutron (notice): Unable to complete neutron request due to non-fatal error: "Execution of '/usr/bin/neutron subnet-list --format=csv --column=id --quote=none' returned 1: neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead.
Connection to neutron failed: Failed to connect Neutron server". Retrying for 2 sec.
2017-10-31 15:05:35 +0000 Puppet (err): Could not prefetch neutron_subnet provider 'neutron': Can't retrieve subnet-list because Neutron or Keystone API is not available.
/etc/puppet/modules/neutron/lib/puppet/provider/neutron.rb:121:in `list_neutron_resources'
/etc/puppet/modules/neutron/lib/puppet/provider/neutron_subnet/neutron.rb:31:in `instances'
/etc/puppet/modules/neutron/lib/puppet/provider/neutron_subnet/neutron.rb:55:in `prefetch'
/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:307:in `prefetch'
/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:197:in `prefetch_if_necessary'
/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:96:in `block in evaluate'
/usr/lib/ruby/vendor_ruby/puppet/graph/relationship_graph.rb:116:in `traverse'
/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:138:in `evaluate'
/usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:169:in `block in apply'
/usr/lib/ruby/vendor_ruby/puppet/util/log.rb:149:in `with_destination'
/usr/lib/ruby/vendor_ruby/puppet/transaction/report.rb:112:in `as_logging_destination'
/usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:168:in `apply'
/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:120:in `block in apply_catalog'
/usr/lib/ruby/vendor_ruby/puppet/util.rb:161:in `block in benchmark'
/usr/lib/ruby/2.3.0/benchmark.rb:308:in `realtime'
/usr/lib/ruby/vendor_ruby/puppet/util.rb:160:in `benchmark'
/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:119:in `apply_catalog'
/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:227:in `run_internal'
/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:134:in `block in run'
/usr/lib/ruby/vendor_ruby/puppet/context.rb:64:in `override'
/usr/lib/ruby/vendor_ruby/puppet.rb:246:in `override'
/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:133:in `run'
/usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:302:in `apply_catalog'
/usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:236:in `block in main'
/usr/lib/ruby/vendor_ruby/puppet/context.rb:64:in `override'
/usr/lib/ruby/vendor_ruby/puppet.rb:246:in `override'
/usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:198:in `main'
/usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:159:in `run_command'
/usr/lib/ruby/vendor_ruby/puppet/application.rb:381:in `block (2 levels) in run'
/usr/lib/ruby/vendor_ruby/puppet/application.rb:507:in `plugin_hook'
/usr/lib/ruby/vendor_ruby/puppet/application.rb:381:in `block in run'
/usr/lib/ruby/vendor_ruby/puppet/util.rb:496:in `exit_on_fail'
/usr/lib/ruby/vendor_ruby/puppet/application.rb:381:in `run'
/usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:146:in `run'
/usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:92:in `execute'
/usr/bin/puppet:8:in `<main>'
2017-10-31 15:05:35 +0000 /Stage[main]/Openstack_tasks::Openstack_network::Networks/Neutron_subnet[admin_internal_net__subnet] (notice): Dependency Neutron_network[admin_internal_net] has failures: true
2017-10-31 15:05:35 +0000 /Stage[main]/Openstack_tasks::Openstack_network::Networks/Neutron_subnet[admin_internal_net__subnet] (warning): Skipping because of failed dependencies
2017-10-31 15:05:35 +0000 /Stage[main]/Openstack_tasks::Openstack_network::Networks/Neutron_subnet[admin_floating_net__subnet] (notice): Dependency Neutron_network[admin_floating_net] has failures: true
2017-10-31 15:05:35 +0000 /Stage[main]/Openstack_tasks::Openstack_network::Networks/Neutron_subnet[admin_floating_net__subnet] (warning): Skipping because of failed dependencies
2017-10-31 15:05:35 +0000 Puppet (notice): Finished catalog run in 26.34 seconds

It seems some dependencies problems, but I cannot find out what they are...

Reproducibility:
Declare an openstack env with 3 controllers and 3 nodes for ceph and compute.

Description of the environment:
6 VMs acting as servers
Fuel 11.0 #956

Revision history for this message
Jorge Rodriguez (jorgerodriguezing) wrote :
Revision history for this message
Jorge Rodriguez (jorgerodriguezing) wrote :

No reply? :(

Revision history for this message
Oleksiy Molchanov (omolchanov) wrote :

Jorge, please attach full diagnostic snapshot.

Changed in fuel:
status: New → Incomplete
Revision history for this message
Jorge Rodriguez (jorgerodriguezing) wrote :

Snapshot is up to 611 MB, can I upload any way? Maybe a link for you to download is possible?

Revision history for this message
Jorge Rodriguez (jorgerodriguezing) wrote :

Here you can download the snapshot:

https://drive.ingenia.es/index.php/s/JpMsetzmzjGkBaz

Regards.

Revision history for this message
Jorge Rodriguez (jorgerodriguezing) wrote :

No answers to this?

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.