[vcenter] Deploy is failed due to reason: Haproxy_ocf/Service[p_haproxy]/enable) Execution of '/usr/sbin/crm resource manage clone_p_haproxy' returned 1: Call cib_replace failed (-62): Timer expired

Bug #1391599 reported by Tatyana Dubyk
20
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
New
High
Dmitry Ilyin
6.0.x
New
High
Dmitry Ilyin

Bug Description

Create env from iso :
[root@nailgun ~]# fuel --fuel-version
api: '1.0'
astute_sha: 702af3db6f5bca92525bc8322d7d5d7675ec857e
auth_required: true
build_id: 2014-11-09_21-00-23
build_number: '31'
feature_groups:
- mirantis
fuellib_sha: e5b3de834a400d98d8c6ba416249832a0c16076c
fuelmain_sha: 087aea3a21ecd566a30657c562b9397853059434
nailgun_sha: d6f3a5865307f34d0ece063b87a879f3670780a1
ostf_sha: 64cb59c681658a7a55cc2c09d079072a41beb346
production: docker
release: 5.1.1
release_versions:
  2014.1.1-5.1.1:
    VERSION:
      api: '1.0'
      astute_sha: 702af3db6f5bca92525bc8322d7d5d7675ec857e
      build_id: 2014-11-09_21-00-23
      build_number: '31'
      feature_groups:
      - mirantis
      fuellib_sha: e5b3de834a400d98d8c6ba416249832a0c16076c
      fuelmain_sha: 087aea3a21ecd566a30657c562b9397853059434
      nailgun_sha: d6f3a5865307f34d0ece063b87a879f3670780a1
      ostf_sha: 64cb59c681658a7a55cc2c09d079072a41beb346
      production: docker
      release: 5.1.1

OS: CentOS 6.5
Network: nova with flat
hypervisor: vcenter with settings below
VCENTER_IP= 172.16.0.254
VCENTER_USERNAME= <email address hidden>
VCENTER_PASSWORD= Qwer!1234
VCENTER_CLUSTERS= Cluster1,Cluster2
storages: Cinder LVM over iSCSI for volumes

Networks settings: put Fixed on eth3

Create env with configuration :
-1 controller
-1 controller
-1 controller + cinder

Deploy env
Check that deployment is failed due to reason:
 (/Stage[main]/Cluster::Haproxy_ocf/Service[p_haproxy]/enable) change from true to true failed: Execution of '/usr/sbin/crm resource manage clone_p_haproxy' returned 1: Call cib_replace failed (-62): Timer expired
 /usr/bin/puppet:4
 /usr/lib/ruby/site_ruby/1.8/puppet/util/command_line.rb:91:in `execute'
 /usr/lib/ruby/site_ruby/1.8/puppet/util/command_line.rb:137:in `run'
 /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:364:in `run'
 /usr/lib/ruby/site_ruby/1.8/puppet/util.rb:478:in `exit_on_fail'
 /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:364:in `run'
 /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:470:in `plugin_hook'
 /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:364:in `run'
 /usr/lib/ruby/site_ruby/1.8/puppet/application/apply.rb:146:in `run_command'
 /usr/lib/ruby/site_ruby/1.8/puppet/application/apply.rb:218:in `main'
 /usr/lib/ruby/site_ruby/1.8/puppet/application/apply.rb:268:in `apply_catalog'
 /usr/lib/ruby/site_ruby/1.8/puppet/configurer.rb:192:in `run'
 /usr/lib/ruby/site_ruby/1.8/puppet/configurer.rb:124:in `apply_catalog'
 /usr/lib/ruby/site_ruby/1.8/puppet/util.rb:160:in `benchmark'
 /usr/lib/ruby/1.8/benchmark.rb:308:in `realtime'
 /usr/lib/ruby/site_ruby/1.8/puppet/util.rb:161:in `benchmark'
 /usr/lib/ruby/site_ruby/1.8/puppet/configurer.rb:125:in `apply_catalog'
 /usr/lib/ruby/site_ruby/1.8/puppet/resource/catalog.rb:163:in `apply'
 /usr/lib/ruby/site_ruby/1.8/puppet/transaction/report.rb:108:in `as_logging_destination'
 /usr/lib/ruby/site_ruby/1.8/puppet/util/log.rb:149:in `with_destination'
 /usr/lib/ruby/site_ruby/1.8/puppet/resource/catalog.rb:164:in `apply'
 /usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:108:in `evaluate'
 /usr/lib/ruby/site_ruby/1.8/puppet/graph/relationship_graph.rb:118:in `traverse'
 /usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:117:in `evaluate'
 /usr/lib/ruby/site_ruby/1.8/puppet/util.rb:326:in `thinmark'
 /usr/lib/ruby/1.8/benchmark.rb:308:in `realtime'
 /usr/lib/ruby/site_ruby/1.8/puppet/util.rb:327:in `thinmark'
 /usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:117:in `evaluate'
 /usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:117:in `call'
 /usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:187:in `eval_resource'
 /usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:174:in `apply'
 /usr/lib/ruby/site_ruby/1.8/puppet/transaction/resource_harness.rb:18:in `evaluate'
 /usr/lib/ruby/site_ruby/1.8/puppet/transaction/resource_harness.rb:81:in `perform_changes'
 /usr/lib/ruby/site_ruby/1.8/puppet/transaction/resource_harness.rb:81:in `each'
 /usr/lib/ruby/site_ruby/1.8/puppet/transaction/resource_harness.rb:82:in `perform_changes'
 /usr/lib/ruby/site_ruby/1.8/puppet/transaction/resource_harness.rb:130:in `sync_if_needed'
 /usr/lib/ruby/site_ruby/1.8/puppet/transaction/resource_harness.rb:193:in `sync'
 /usr/lib/ruby/site_ruby/1.8/puppet/property.rb:581:in `sync'
 /usr/lib/ruby/site_ruby/1.8/puppet/property.rb:498:in `set'
 /usr/lib/ruby/site_ruby/1.8/puppet/property.rb:197:in `call_valuemethod'
 /usr/lib/ruby/site_ruby/1.8/puppet/property.rb:197:in `send'
 /usr/lib/ruby/site_ruby/1.8/puppet/type/service.rb:44:in `set_true'
 /etc/puppet/modules/corosync/lib/puppet/provider/service/pacemaker.rb:237:in `enable'
 /usr/lib/ruby/site_ruby/1.8/puppet/provider.rb:444:in `crm'
 /usr/lib/ruby/site_ruby/1.8/puppet/provider.rb:444:in `send'
 /usr/lib/ruby/site_ruby/1.8/puppet/provider.rb:237:in `crm'
 /usr/lib/ruby/site_ruby/1.8/puppet/provider/command.rb:23:in `execute'
 /usr/lib/ruby/site_ruby/1.8/puppet/util/execution.rb:188:in `execute'
 Execution of '/usr/sbin/crm resource manage clone_p_haproxy' returned 1: Call cib_replace failed (-62): Timer expired
 (/Stage[main]/Galera/Service[mysql-service]/ensure) ' returned 70: Call failed: Communication error on send
 (/Stage[main]/Galera/Service[mysql-service]/ensure) </diff>
 (/Stage[main]/Galera/Service[mysql-service]/ensure) </diff-added>
 (/Stage[main]/Galera/Service[mysql-service]/ensure) </cib>
 (/Stage[main]/Galera/Service[mysql-service]/ensure) </configuration>
 (/Stage[main]/Galera/Service[mysql-service]/ensure) </constraints>
 (/Stage[main]/Galera/Service[mysql-service]/ensure) <rsc_location id="clone_p_mysql_on_node-3.test.domain.local" node="node-3.test.domain.local" rsc="clone_p_mysql" score="0"/>
 (/Stage[main]/Galera/Service[mysql-service]/ensure) <constraints>
 (/Stage[main]/Galera/Service[mysql-service]/ensure) <configuration>
 (/Stage[main]/Galera/Service[mysql-service]/ensure) <cib>
 (/Stage[main]/Galera/Service[mysql-service]/ensure) <diff-added>
 (/Stage[main]/Galera/Service[mysql-service]/ensure) change from stopped to running failed: Execution of '/usr/sbin/cibadmin --patch --sync-call --xml-text <diff>
 /usr/bin/puppet:4
 /usr/lib/ruby/site_ruby/1.8/puppet/util/command_line.rb:91:in `execute'
 /usr/lib/ruby/site_ruby/1.8/puppet/util/command_line.rb:137:in `run'
 /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:364:in `run'
 /usr/lib/ruby/site_ruby/1.8/puppet/util.rb:478:in `exit_on_fail'
 /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:364:in `run'
 /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:470:in `plugin_hook'
 /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:364:in `run'
 /usr/lib/ruby/site_ruby/1.8/puppet/application/apply.rb:146:in `run_command'
 /usr/lib/ruby/site_ruby/1.8/puppet/application/apply.rb:218:in `main'
 /usr/lib/ruby/site_ruby/1.8/puppet/application/apply.rb:268:in `apply_catalog'
 /usr/lib/ruby/site_ruby/1.8/puppet/configurer.rb:192:in `run'
 /usr/lib/ruby/site_ruby/1.8/puppet/configurer.rb:124:in `apply_catalog'
 /usr/lib/ruby/site_ruby/1.8/puppet/util.rb:160:in `benchmark'
 /usr/lib/ruby/1.8/benchmark.rb:308:in `realtime'
 /usr/lib/ruby/site_ruby/1.8/puppet/util.rb:161:in `benchmark'
 /usr/lib/ruby/site_ruby/1.8/puppet/configurer.rb:125:in `apply_catalog'
 /usr/lib/ruby/site_ruby/1.8/puppet/resource/catalog.rb:163:in `apply'
 /usr/lib/ruby/site_ruby/1.8/puppet/transaction/report.rb:108:in `as_logging_destination'
 /usr/lib/ruby/site_ruby/1.8/puppet/util/log.rb:149:in `with_destination'
 /usr/lib/ruby/site_ruby/1.8/puppet/resource/catalog.rb:164:in `apply'
 /usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:108:in `evaluate'
 /usr/lib/ruby/site_ruby/1.8/puppet/graph/relationship_graph.rb:118:in `traverse'
 /usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:117:in `evaluate'
 /usr/lib/ruby/site_ruby/1.8/puppet/util.rb:326:in `thinmark'
 /usr/lib/ruby/1.8/benchmark.rb:308:in `realtime'
 /usr/lib/ruby/site_ruby/1.8/puppet/util.rb:327:in `thinmark'
 /usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:117:in `evaluate'
 /usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:117:in `call'
 /usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:187:in `eval_resource'
 /usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:174:in `apply'
 /usr/lib/ruby/site_ruby/1.8/puppet/transaction/resource_harness.rb:18:in `evaluate'
 /usr/lib/ruby/site_ruby/1.8/puppet/transaction/resource_harness.rb:77:in `perform_changes'
 /usr/lib/ruby/site_ruby/1.8/puppet/transaction/resource_harness.rb:107:in `manage_via_ensure_if_possible'
 /usr/lib/ruby/site_ruby/1.8/puppet/transaction/resource_harness.rb:130:in `sync_if_needed'
 /usr/lib/ruby/site_ruby/1.8/puppet/transaction/resource_harness.rb:193:in `sync'
 /usr/lib/ruby/site_ruby/1.8/puppet/type/service.rb:86:in `sync'
 /usr/lib/ruby/site_ruby/1.8/puppet/property.rb:581:in `sync'
 /usr/lib/ruby/site_ruby/1.8/puppet/property.rb:498:in `set'
 /usr/lib/ruby/site_ruby/1.8/puppet/property.rb:197:in `call_valuemethod'
 /usr/lib/ruby/site_ruby/1.8/puppet/property.rb:197:in `send'
 /usr/lib/ruby/site_ruby/1.8/puppet/type/service.rb:75:in `set_running'
 /etc/puppet/modules/corosync/lib/puppet/provider/service/pacemaker.rb:255:in `start'
 /etc/puppet/modules/corosync/lib/puppet/provider/service/pacemaker.rb:314:in `unban'
 /etc/puppet/modules/corosync/lib/puppet/provider/service/pacemaker.rb:306:in `set_location'
 /usr/lib/ruby/site_ruby/1.8/puppet/provider.rb:444:in `cibadmin'
 /usr/lib/ruby/site_ruby/1.8/puppet/provider.rb:444:in `send'
 /usr/lib/ruby/site_ruby/1.8/puppet/provider.rb:237:in `cibadmin'
 /usr/lib/ruby/site_ruby/1.8/puppet/provider/command.rb:23:in `execute'
 /usr/lib/ruby/site_ruby/1.8/puppet/util/execution.rb:188:in `execute'
 ' returned 70: Call failed: Communication error on send
     </diff>
       </diff-added>
         </cib>
           </configuration>
             </constraints>
               <rsc_location id="clone_p_mysql_on_node-3.test.domain.local" node="node-3.test.domain.local" rsc="clone_p_mysql" score="0"/>
             <constraints>
           <configuration>
         <cib>
       <diff-added>

Full log see below in astute's logs attachment.

Tags: vcenter
Revision history for this message
Tatyana Dubyk (tdubyk) wrote :
Tatyana Dubyk (tdubyk)
Changed in fuel:
milestone: none → 5.1.1
Changed in fuel:
assignee: nobody → Fuel Library Team (fuel-library)
Dmitry Ilyin (idv1985)
Changed in fuel:
assignee: Fuel Library Team (fuel-library) → Dmitry Ilyin (idv1985)
importance: Undecided → Medium
importance: Medium → High
Revision history for this message
Vladimir Kuklin (vkuklin) wrote :

This bug is a duplicate and is related to corosync bugs. We are going to address it by corosync upgrade to version 2.x

no longer affects: fuel/5.1.x
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/134964
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=c045ce3078c3d5e0d5df786041875332b8f3fac2
Submitter: Jenkins
Branch: master

commit c045ce3078c3d5e0d5df786041875332b8f3fac2
Author: Dmitry Ilyin <email address hidden>
Date: Thu Nov 27 18:11:16 2014 +0300

    Fix idempotency of cs_resource

    * insync? to drop status metadata from checks
    * code cleanup
    * fix rspec for cs_resource type
    * switch location add implementation from pcs
      to cibadmin --patch to solve problems with
      cib changes not being synced to other nodes

    related-blueprint: pacemaker-improvements
    Related-Bug: 1391599
    Related-Bug: 1390480
    Related-Bug: 1396481

    Change-Id: I5410b91ea01fc8c6805de6becdf0800d0d486188
    Signed-off-by: Sergii Golovatiuk <email address hidden>

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Bug attachments

Remote bug watches

Bug watches keep track of this bug in other bug trackers.