Fuel for OpenStack

Deploy HA with Nova on primary controller failed

Bug #1321662 reported by Nastya Urlapova on 2014-05-21

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	Fuel for OpenStack	Fix Committed	High	Fuel Library (Deprecated)	Fuel for OpenStack 5.1
	5.0.x	Fix Released	Critical	Vladimir Kuklin	Fuel for OpenStack 5.0.1

Bug Description

iso #214
http://jenkins-product.srt.mirantis.net:8080/view/0_0_swarm/job/master_fuelmain.system_test.centos.thread_4/63/testReport/(root)/ha_flat_add_compute/ha_flat_add_compute/

Deploy HA on Centos with 3xController and 2xComputer

On primary controller error in puppet:
2014-05-21 01:07:51 ERR
(/Stage[main]/Osnailyfacter::Cluster_ha/Nova_floating_range[10.108.1.128-10.108.1.254]) /usr/bin/puppet:4
2014-05-21 01:07:51 ERR
(/Stage[main]/Osnailyfacter::Cluster_ha/Nova_floating_range[10.108.1.128-10.108.1.254]) /usr/lib/ruby/site_ruby/1.8/puppet/util/command_line.rb:91:in `execute'
2014-05-21 01:07:51 ERR
(/Stage[main]/Osnailyfacter::Cluster_ha/Nova_floating_range[10.108.1.128-10.108.1.254]) /usr/lib/ruby/site_ruby/1.8/puppet/util/command_line.rb:137:in `run'
2014-05-21 01:07:51 ERR
(/Stage[main]/Osnailyfacter::Cluster_ha/Nova_floating_range[10.108.1.128-10.108.1.254]) /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:364:in `run'
2014-05-21 01:07:51 ERR
(/Stage[main]/Osnailyfacter::Cluster_ha/Nova_floating_range[10.108.1.128-10.108.1.254]) /usr/lib/ruby/site_ruby/1.8/puppet/util.rb:478:in `exit_on_fail'
2014-05-21 01:07:51 ERR
(/Stage[main]/Osnailyfacter::Cluster_ha/Nova_floating_range[10.108.1.128-10.108.1.254]) /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:364:in `run'
2014-05-21 01:07:51 ERR
(/Stage[main]/Osnailyfacter::Cluster_ha/Nova_floating_range[10.108.1.128-10.108.1.254]) /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:470:in `plugin_hook'
2014-05-21 01:07:51 ERR
(/Stage[main]/Osnailyfacter::Cluster_ha/Nova_floating_range[10.108.1.128-10.108.1.254]) /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:364:in `run'
2014-05-21 01:07:51 ERR
(/Stage[main]/Osnailyfacter::Cluster_ha/Nova_floating_range[10.108.1.128-10.108.1.254]) /usr/lib/ruby/site_ruby/1.8/puppet/application/apply.rb:146:in `run_command'
2014-05-21 01:07:51 ERR
(/Stage[main]/Osnailyfacter::Cluster_ha/Nova_floating_range[10.108.1.128-10.108.1.254]) /usr/lib/ruby/site_ruby/1.8/puppet/application/apply.rb:218:in `main'
2014-05-21 01:07:51 ERR
(/Stage[main]/Osnailyfacter::Cluster_ha/Nova_floating_range[10.108.1.128-10.108.1.254]) /usr/lib/ruby/site_ruby/1.8/puppet/application/apply.rb:268:in `apply_catalog'
2014-05-21 01:07:51 ERR
(/Stage[main]/Osnailyfacter::Cluster_ha/Nova_floating_range[10.108.1.128-10.108.1.254]) /usr/lib/ruby/site_ruby/1.8/puppet/configurer.rb:192:in `run'
2014-05-21 01:07:51 ERR
(/Stage[main]/Osnailyfacter::Cluster_ha/Nova_floating_range[10.108.1.128-10.108.1.254]) /usr/lib/ruby/site_ruby/1.8/puppet/configurer.rb:124:in `apply_catalog'
2014-05-21 01:07:51 ERR
(/Stage[main]/Osnailyfacter::Cluster_ha/Nova_floating_range[10.108.1.128-10.108.1.254]) /usr/lib/ruby/site_ruby/1.8/puppet/util.rb:160:in `benchmark'
2014-05-21 01:07:51 ERR
(/Stage[main]/Osnailyfacter::Cluster_ha/Nova_floating_range[10.108.1.128-10.108.1.254]) /usr/lib/ruby/1.8/benchmark.rb:308:in `realtime'
2014-05-21 01:07:51 ERR
(/Stage[main]/Osnailyfacter::Cluster_ha/Nova_floating_range[10.108.1.128-10.108.1.254]) /usr/lib/ruby/site_ruby/1.8/puppet/util.rb:161:in `benchmark'
2014-05-21 01:07:51 ERR
(/Stage[main]/Osnailyfacter::Cluster_ha/Nova_floating_range[10.108.1.128-10.108.1.254]) /usr/lib/ruby/site_ruby/1.8/puppet/configurer.rb:125:in `apply_catalog'
2014-05-21 01:07:51 ERR
(/Stage[main]/Osnailyfacter::Cluster_ha/Nova_floating_range[10.108.1.128-10.108.1.254]) /usr/lib/ruby/site_ruby/1.8/puppet/resource/catalog.rb:163:in `apply'
2014-05-21 01:07:51 ERR
(/Stage[main]/Osnailyfacter::Cluster_ha/Nova_floating_range[10.108.1.128-10.108.1.254]) /usr/lib/ruby/site_ruby/1.8/puppet/transaction/report.rb:108:in `as_logging_destination'
2014-05-21 01:07:51 ERR
(/Stage[main]/Osnailyfacter::Cluster_ha/Nova_floating_range[10.108.1.128-10.108.1.254]) /usr/lib/ruby/site_ruby/1.8/puppet/util/log.rb:149:in `with_destination'
2014-05-21 01:07:51 ERR
(/Stage[main]/Osnailyfacter::Cluster_ha/Nova_floating_range[10.108.1.128-10.108.1.254]) /usr/lib/ruby/site_ruby/1.8/puppet/resource/catalog.rb:164:in `apply'
2014-05-21 01:07:51 ERR
(/Stage[main]/Osnailyfacter::Cluster_ha/Nova_floating_range[10.108.1.128-10.108.1.254]) /usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:108:in `evaluate'
2014-05-21 01:07:51 ERR
(/Stage[main]/Osnailyfacter::Cluster_ha/Nova_floating_range[10.108.1.128-10.108.1.254]) /usr/lib/ruby/site_ruby/1.8/puppet/graph/relationship_graph.rb:118:in `traverse'
2014-05-21 01:07:51 ERR
(/Stage[main]/Osnailyfacter::Cluster_ha/Nova_floating_range[10.108.1.128-10.108.1.254]) /usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:117:in `evaluate'
2014-05-21 01:07:51 ERR
(/Stage[main]/Osnailyfacter::Cluster_ha/Nova_floating_range[10.108.1.128-10.108.1.254]) /usr/lib/ruby/site_ruby/1.8/puppet/util.rb:326:in `thinmark'
2014-05-21 01:07:51 ERR
(/Stage[main]/Osnailyfacter::Cluster_ha/Nova_floating_range[10.108.1.128-10.108.1.254]) /usr/lib/ruby/1.8/benchmark.rb:308:in `realtime'
2014-05-21 01:07:51 ERR
(/Stage[main]/Osnailyfacter::Cluster_ha/Nova_floating_range[10.108.1.128-10.108.1.254]) /usr/lib/ruby/site_ruby/1.8/puppet/util.rb:327:in `thinmark'
2014-05-21 01:07:51 ERR
(/Stage[main]/Osnailyfacter::Cluster_ha/Nova_floating_range[10.108.1.128-10.108.1.254]) /usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:117:in `evaluate'
2014-05-21 01:07:51 ERR
(/Stage[main]/Osnailyfacter::Cluster_ha/Nova_floating_range[10.108.1.128-10.108.1.254]) /usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:117:in `call'
2014-05-21 01:07:51 ERR
(/Stage[main]/Osnailyfacter::Cluster_ha/Nova_floating_range[10.108.1.128-10.108.1.254]) /usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:187:in `eval_resource'
2014-05-21 01:07:51 ERR
(/Stage[main]/Osnailyfacter::Cluster_ha/Nova_floating_range[10.108.1.128-10.108.1.254]) /usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:174:in `apply'
2014-05-21 01:07:51 ERR
(/Stage[main]/Osnailyfacter::Cluster_ha/Nova_floating_range[10.108.1.128-10.108.1.254]) /usr/lib/ruby/site_ruby/1.8/puppet/transaction/resource_harness.rb:17:in `evaluate'
2014-05-21 01:07:51 ERR
(/Stage[main]/Osnailyfacter::Cluster_ha/Nova_floating_range[10.108.1.128-10.108.1.254]) /usr/lib/ruby/site_ruby/1.8/puppet/transaction/resource_harness.rb:223:in `from_resource'
2014-05-21 01:07:51 ERR
(/Stage[main]/Osnailyfacter::Cluster_ha/Nova_floating_range[10.108.1.128-10.108.1.254]) /usr/lib/ruby/site_ruby/1.8/puppet/type.rb:1063:in `retrieve_resource'
2014-05-21 01:07:51 ERR
(/Stage[main]/Osnailyfacter::Cluster_ha/Nova_floating_range[10.108.1.128-10.108.1.254]) /usr/lib/ruby/site_ruby/1.8/puppet/type.rb:1035:in `retrieve'
2014-05-21 01:07:51 ERR
(/Stage[main]/Osnailyfacter::Cluster_ha/Nova_floating_range[10.108.1.128-10.108.1.254]) /usr/lib/ruby/site_ruby/1.8/puppet/property/ensure.rb:81:in `retrieve'
2014-05-21 01:07:51 ERR
(/Stage[main]/Osnailyfacter::Cluster_ha/Nova_floating_range[10.108.1.128-10.108.1.254]) /etc/puppet/modules/nova/lib/puppet/provider/nova_floating_range/nova_manage.rb:15:in `exists?'
2014-05-21 01:07:51 ERR
(/Stage[main]/Osnailyfacter::Cluster_ha/Nova_floating_range[10.108.1.128-10.108.1.254]) /etc/puppet/modules/nova/lib/puppet/provider/nova_floating_range/nova_manage.rb:49:in `operate_range'
2014-05-21 01:07:51 ERR
(/Stage[main]/Osnailyfacter::Cluster_ha/Nova_floating_range[10.108.1.128-10.108.1.254]) /usr/lib/ruby/gems/1.8/gems/openstack-1.1.2/lib/openstack/compute/connection.rb:475:in `get_floating_ips_bulk'
2014-05-21 01:07:51 ERR
(/Stage[main]/Osnailyfacter::Cluster_ha/Nova_floating_range[10.108.1.128-10.108.1.254]) /usr/lib/ruby/gems/1.8/gems/openstack-1.1.2/lib/openstack/compute/connection.rb:502:in `check_extension'
2014-05-21 01:07:51 ERR
(/Stage[main]/Osnailyfacter::Cluster_ha/Nova_floating_range[10.108.1.128-10.108.1.254]) /usr/lib/ruby/gems/1.8/gems/openstack-1.1.2/lib/openstack/compute/connection.rb:242:in `api_extensions'
2014-05-21 01:07:51 ERR
(/Stage[main]/Osnailyfacter::Cluster_ha/Nova_floating_range[10.108.1.128-10.108.1.254]) /usr/lib/ruby/gems/1.8/gems/openstack-1.1.2/lib/openstack/connection.rb:207:in `req'
2014-05-21 01:07:51 ERR
(/Stage[main]/Osnailyfacter::Cluster_ha/Nova_floating_range[10.108.1.128-10.108.1.254]) /usr/lib/ruby/gems/1.8/gems/openstack-1.1.2/lib/openstack/connection.rb:510:in `raise_exception'
2014-05-21 01:07:51 ERR
(/Stage[main]/Osnailyfacter::Cluster_ha/Nova_floating_range[10.108.1.128-10.108.1.254]) /usr/lib/ruby/gems/1.8/gems/openstack-1.1.2/lib/openstack/connection.rb:533:in `deal_with_faulty_error'
2014-05-21 01:07:51 ERR
(/Stage[main]/Osnailyfacter::Cluster_ha/Nova_floating_range[10.108.1.128-10.108.1.254]) '
2014-05-21 01:07:51 ERR
(/Stage[main]/Osnailyfacter::Cluster_ha/Nova_floating_range[10.108.1.128-10.108.1.254]) </body></html>
2014-05-21 01:07:51 ERR
(/Stage[main]/Osnailyfacter::Cluster_ha/Nova_floating_range[10.108.1.128-10.108.1.254]) The server didn't respond in time.
2014-05-21 01:07:51 ERR
(/Stage[main]/Osnailyfacter::Cluster_ha/Nova_floating_range[10.108.1.128-10.108.1.254]) Could not evaluate: Oops - not sure what happened: 757: unexpected token at '<html><body><h1>504 Gateway Time-out</h1>

Revision history for this message

Nastya Urlapova (aurlapova) wrote on 2014-05-21:

fail_ha_flat_add_compute-2014_05_21__01_46_18.tar.gz Edit (4.4 MiB, application/x-tar)

Vladimir Kuklin (vkuklin) on 2014-05-26

Changed in fuel:
status:	New → Confirmed

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-05-26: Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/95586

Changed in fuel:
assignee:	Fuel Library Team (fuel-library) → Vladimir Kuklin (vkuklin)
status:	Confirmed → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-05-26: Fix proposed to fuel-library (stable/5.0)

Fix proposed to branch: stable/5.0
Review: https://review.openstack.org/95587

Revision history for this message

Mike Scherbakov (mihgen) wrote on 2014-05-26:

Fix is risky enough at the current release timing, so we have to keep this one as known issue.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-06-05: Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/95586
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=46f471175285fa430eacc99ebb149c56177f35fa
Submitter: Jenkins
Branch: master

commit 46f471175285fa430eacc99ebb149c56177f35fa
Author: Vladimir Kuklin <email address hidden>
Date: Tue May 27 00:44:21 2014 +0400

Wait for haproxy nova backend to become ready

Change-Id: Id186545879cce9f1ecbfb70dccab85a6d3aa9ec0
Closes-Bug: #1321662

Changed in fuel:
status:	In Progress → Fix Committed

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-06-05: Fix merged to fuel-library (stable/5.0)

Reviewed: https://review.openstack.org/95587
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=12b460c07bdd3a299f97dac028a325ece8778f90
Submitter: Jenkins
Branch: stable/5.0

commit 12b460c07bdd3a299f97dac028a325ece8778f90
Author: Vladimir Kuklin <email address hidden>
Date: Tue May 27 00:44:21 2014 +0400

Wait for haproxy nova backend to become ready

Change-Id: Id186545879cce9f1ecbfb70dccab85a6d3aa9ec0
Closes-Bug: #1321662

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-06-16: Related fix proposed to fuel-library (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/100274

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-06-17: Related fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/100274
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=b846e7ac26b9b58f96c837240b6ad043329c331a
Submitter: Jenkins
Branch: master

commit b846e7ac26b9b58f96c837240b6ad043329c331a
Author: Sergii Golovatiuk <email address hidden>
Date: Mon Jun 16 14:49:51 2014 +0000

Limit HAProxy exec check to primary controller

HAProxy IP is controlled by Pacemaker/Corosync. Exec fails on all Slave controllers.
This patch limits exec check to primary controller only.

    Change-Id: Ic8daace980f2cb01feeb1f8006c9ca22d4a6a8f2
    Closes-Bug: 1329780
    Related-Bug: 1321662

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-06-17: Related fix proposed to fuel-library (stable/5.0)

Related fix proposed to branch: stable/5.0
Review: https://review.openstack.org/100552

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-06-17: Related fix merged to fuel-library (stable/5.0)

#10

Reviewed: https://review.openstack.org/100552
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=3bb37ea2dbf5860f2d21cfdc11c9f23337e98d69
Submitter: Jenkins
Branch: stable/5.0

commit 3bb37ea2dbf5860f2d21cfdc11c9f23337e98d69
Author: Sergii Golovatiuk <email address hidden>
Date: Mon Jun 16 14:49:51 2014 +0000

Limit HAProxy exec check to primary controller

HAProxy IP is controlled by Pacemaker/Corosync. Exec fails on all Slave controllers.
This patch limits exec check to primary controller only.

    Change-Id: Ic8daace980f2cb01feeb1f8006c9ca22d4a6a8f2
    Closes-Bug: 1329780
    Related-Bug: 1321662

Dmitry Pyzhov (dpyzhov) on 2014-08-13

no longer affects:	fuel/5.1.x
Changed in fuel:
importance:	Critical → High
assignee:	Vladimir Kuklin (vkuklin) → Fuel Library Team (fuel-library)

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Bug attachments

fail_ha_flat_add_compute-2014_05_21__01_46_18.tar.gz Edit

Add attachment

Remote bug watches

Bug watches keep track of this bug in other bug trackers.