[library] crm commit is done using replace instead if patch

Bug #1338594 reported by Vladimir Kuklin
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Committed
Critical
Vladimir Kuklin

Bug Description

This leads to race conditions and e.g. broken deployment when some of resources are not specified.

Revision history for this message
Vladimir Kuklin (vkuklin) wrote :

it looks like, we need to apply this patch for crmsh:

diff -Naur a/modules/cibconfig.py b/modules/cibconfig.py 1
   --- a/modules/cibconfig.py» 2013-02-12 16:14:02.000000000 +0400 2
   +++ b/modules/cibconfig.py» 2014-07-07 18:39:29.592779754 +0400 3
   @@ -2092,7 +2092,7 @@ 4
   return False 5
   if self.new_schema and not self.commit_schema(): 6
   return False 7
   - cibadmin_opts = force and "-R --force" or "-R" 8
   + cibadmin_opts = force and "-s -R --force" or "-R -s" 9
   rc = pipe_string("%s %s" % (cib_piped,cibadmin_opts), conf_node.toxml()) 10
   if rc != 0: 11
   update_err("cib",cibadmin_opts,conf_node.toprettyxml(), rc)

Revision history for this message
OSCI Robot (oscirobot) wrote :

Package crmsh has been built from changeset: http://gerrit.mirantis.com/17875
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1-stable-17875/centos
You can build an ISO with this package:
make iso EXTRA_RPM_REPOS="osci-testing,http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1-stable-17875/centos"

Changed in fuel:
status: Confirmed → In Progress
Revision history for this message
OSCI Robot (oscirobot) wrote :

Package crmsh has been built from changeset: http://gerrit.mirantis.com/17878
DEB Repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-5.1-stable-17878/ubuntu
You can build an ISO with this package:
make iso EXTRA_DEB_REPOS="http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-5.1-stable-17878/ubuntu /"

Revision history for this message
Vladimir Kuklin (vkuklin) wrote :

it looks the actual problem is that crm does application using replace and in this case the data can be outdated after galera or rabbitmq ocf modification. thus we need to build 1.2.6 version of crmsh along with patch that set's minimum required version of pacemaker to be 1.1.10 to support cibadmin patch option.

Revision history for this message
Vladimir Kuklin (vkuklin) wrote :

the latest ubuntu package is here:

https://launchpad.net/ubuntu/+source/crmsh/1.2.6+git+e77add-1.2ubuntu1

it contains all the stuff for deb package building including the patch and spec file for RPM

I modified it a little bit in order to apply patch during debian package build here:

https://gerrit.mirantis.com/#/c/17878/

I am going to test it and tell if it really helps.

Changed in fuel:
assignee: Fuel Library Team (fuel-library) → Vladimir Kuklin (vkuklin)
Revision history for this message
Vladimir Kuklin (vkuklin) wrote :

this regression could be introduced by commits that disabled crm shadow usage along with merge of galera and rabbitmq ocf scripts that alter CIB attributes during deployment.

Revision history for this message
Vladimir Kuklin (vkuklin) wrote :

adding logs with haproxy resource failing to be committed

summary: - crm commit is done asynchrounously
+ crm commit is done using replace instead if patch
Revision history for this message
OSCI Robot (oscirobot) wrote : Re: crm commit is done using replace instead if patch

Package crmsh has been built from changeset: http://gerrit.mirantis.com/17878
DEB Repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-5.1-stable-17878/ubuntu
You can build an ISO with this package:
make iso EXTRA_DEB_REPOS="http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-5.1-stable-17878/ubuntu /"

Revision history for this message
Vladimir Kuklin (vkuklin) wrote :

newer crmsh does not help at all. it looks like we need to do accurate crm diff application.

Revision history for this message
Vladimir Kuklin (vkuklin) wrote :

it should be worked around by following method:

instead of using live CIB:
1) create temporary shadow cib
2) run crm with this CIB and use configure load update
3) generate diff between live cib and this cib in corresponding scope:
   a) "resources" for resources and groups
   b) "constraints" for orders, locations and colocations
   c) "rsc_defaults" for rsc_defaults
4) generate a patch using crm_diff
5) fix xml patch by appending <cib> and <configuration> tags
6) apply the patch by using cibadmin -P --xml-file
7) clean up everything

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to fuel-library (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/105270

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Related fix proposed to branch: master
Review: https://review.openstack.org/105271

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-library (master)

Change abandoned by Vladimir Kuklin (<email address hidden>) on branch: master
Review: https://review.openstack.org/105270

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Change abandoned by Vladimir Kuklin (<email address hidden>) on branch: master
Review: https://review.openstack.org/105271

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/105432

Revision history for this message
OSCI Robot (oscirobot) wrote : Re: crm commit is done using replace instead if patch

Package crmsh has been built from changeset: http://gerrit.mirantis.com/17875
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1-stable-17875/centos
You can build an ISO with this package:
make iso EXTRA_RPM_REPOS="osci-testing,http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1-stable-17875/centos"

Revision history for this message
OSCI Robot (oscirobot) wrote :

Package crmsh has been built from changeset: http://gerrit.mirantis.com/17878
DEB Repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-5.1-stable/ubuntu
You can build an ISO with this package:
make iso EXTRA_DEB_REPOS="http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-5.1-stable/ubuntu /"

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
OSCI Robot (oscirobot) wrote :

Package crmsh has been built from changeset: http://gerrit.mirantis.com/17875
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1-stable/centos
You can build an ISO with this package:
make iso EXTRA_RPM_REPOS="osci-testing,http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1-stable/centos"

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/105432
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=03342f53fac5cfadb44083b32f46f04b2b1ccbb3
Submitter: Jenkins
Branch: master

commit 03342f53fac5cfadb44083b32f46f04b2b1ccbb3
Author: Vladimir Kuklin <email address hidden>
Date: Tue Jul 8 15:24:07 2014 +0400

    Apply CIB changes using patch method

    Generate XML diff patch and apply it.

    1) generate config changes in shadow
    2) generate diff only for specific scope
    3) apply it

    This will not touch cluster attributes and
    other stuff it should not touch, thus not
    breaking functionality of OCF scripts using
    CIB attributes.

    Change-Id: Ibcfaa82707d1aed01b79c8307d17f4e71fd60906
    Closes-bug: #1338594

Dmitry Ilyin (idv1985)
summary: - crm commit is done using replace instead if patch
+ [library] crm commit is done using replace instead if patch
Revision history for this message
OSCI Robot (oscirobot) wrote :
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.