Add of detached keystone node to existing cluster failed with Call cib_apply_diff failed (-205): Update was older than existing configuration

[Andrey Sledzinskiy]

Steps:
1. Install detach-database and detach-keystone plugins on master node
2. Create next cluster - Ubuntu, HA, Neutron Vlan, enabled separate-database and separate-keystone, 3 controllers, 1 compute, 1 cinder, 3 database+keystone
3. Deploy cluster
4. After deployment add 1 keystone node
5. Start deployment

Actual result - it failed on new keystone node (node-8):
2015-09-10 09:57:48 ERR (/Stage[main]/Openstack::Corosync/Cs_property[no-quorum-policy]) Could not evaluate: Execution of '/usr/sbin/crm configure property $id="cib-bootstrap-options" no-quorum-policy=ignore' returned 1: Call cib_apply_diff failed (-205): Update was older than existing configuration

Issue might affect keystone plugin only.

[Dmitry Ilyin (idv1985)]

The old implementation of cs_property id lacking retries. Perhaps in this case crm did not update the cib before some property have been changed somewhere and there was no retry to fix this situation.

Why do you even try to manage no-quorum-policy property?
We are using this script after the deployment ./deployment/puppet/osnailyfacter/modular/astute/enable_quorum.rb to set thei property

[Dmitry Ilyin (idv1985)]

Actually the "cluster" task that had a failed property have been restarted and passes successfully on the second run.

This deployment have failed because "cluster-haproxy" task have failed 3 times on the node-8. It was trying to start p_haproxy pacemaker resource but it have not been created before. So, either p_haproxy should be created befor started or not started at all if it's not needed here.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/222290

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/7.0
Review: https://review.openstack.org/222538

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/222290
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=de12021f24840ab5b0d104e456b57bdc10e1b642
Submitter: Jenkins
Branch: master

commit de12021f24840ab5b0d104e456b57bdc10e1b642
Author: Dmitry Ilyin <email address hidden>
Date: Thu Sep 10 20:09:30 2015 +0300

    Remove primary-controller condition from haproxy primitive

    * Allow haproxy_ocf to rin on non-controller nodes

    Change-Id: I8ba85dbda5214f8460e39c9c12ea4e78da10fd9a
    Related-Bug: 1494314

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/222538
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=d5bba5c8a9ee5de90e1a5a349ca22f57c361427f
Submitter: Jenkins
Branch: stable/7.0

commit d5bba5c8a9ee5de90e1a5a349ca22f57c361427f
Author: Dmitry Ilyin <email address hidden>
Date: Thu Sep 10 20:09:30 2015 +0300

    Remove primary-controller condition from haproxy primitive

    * Allow haproxy_ocf to rin on non-controller nodes

    Change-Id: I8ba85dbda5214f8460e39c9c12ea4e78da10fd9a
    Related-Bug: 1494314

[Vladimir Khlyunev (vkhlyunev)]

ISO 293 - verified (deployment successful)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/222361
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=a9b06a12317c0ce1a0f4ccff08f08697912624c0
Submitter: Jenkins
Branch: master

commit a9b06a12317c0ce1a0f4ccff08f08697912624c0
Author: Dmitry Ilyin <email address hidden>
Date: Thu Sep 10 22:37:09 2015 +0300

    Refactor haproxy_ocf to use wrappers

    * Refactor harproxy manifest to make use of pacemaker wrappers
    * It will also make it easier to switch to the upstream version
      of the haproxy module

    Related-Bug: 1494314
    Change-Id: I2d11acd6f11dad9b46257c6d1d70993aa8e36568

[Vasily Gorin (vgorin)]

Cannot be verified now due to bug https://bugs.launchpad.net/fuel/+bug/1500743

[Dmitriy Kruglov (dkruglov)]

Verified on MOS 8.0, build 234. Not reproduced.

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  openstack_version: "2015.1.0-8.0"
  api: "1.0"
  build_number: "234"
  build_id: "234"
  fuel-nailgun_sha: "0a2f99530cf7246acf85db643032a0550168aac5"
  python-fuelclient_sha: "b1ffe1cae9ce7b612d3f746c8e2e2fde6f732748"
  fuel-agent_sha: "bd67efbadabfd8242c979c50b7d61a251621621a"
  fuel-nailgun-agent_sha: "a33a58d378c117c0f509b0e7badc6f0910364154"
  astute_sha: "b60624ee2c5f1d6d805619b6c27965a973508da1"
  fuel-library_sha: "ed9c5c91eda71bfcdcd3e08f91fa3627ea87829c"
  fuel-ostf_sha: "a98973482f839554d90cc1c071d625a01e018cfe"
  fuel-createmirror_sha: "3cb98030d4a12992ea1cda1f464f035980569d2f"
  fuelmenu_sha: "fcb15df4fd1a790b17dd78cf675c11c279040941"
  shotgun_sha: "25a0cc461a9fa4f7684f04cef0ff4ad9aa99a64d"
  network-checker_sha: "0b1b94a9685c6471d6911dff7ecac10b7bd2625f"
  fuel-upgrade_sha: "1e894e26d4e1423a9b0d66abd6a79505f4175ff6"
  fuelmain_sha: "2eca6adc33f02e02cd812e1d4be7c70e05fd07db"

[Anastasia Palkina (apalkina)]

Reproduced on ISO #247 in other test case:

1. Create new environment (by default)
2. Add 3 controllers and 2 computes
3. Start deployment. It was successful
4. Run OSTF test. It was successful
5. Add 1 cinder
6. Redeploy environment. It was successful
7. Run OSTF test. It was successful
8. Remove the 'cinder' node and remove a 'controller' node
9. Add 'controller'+'cinder' node
10. Start deployment. It was successful
11. There is error in puppet.log on controller+cinder (node-5):

2015-12-09 15:54:45 ERR (/Stage[main]/Openstack::Corosync/Cs_property[no-quorum-policy]) Could not evaluate: Execution of '/usr/sbin/crm configure property $id="cib-bootstrap-options" no-quorum-policy=ignore' returned 1: Call cib_apply_diff failed (-205): Update was older than existing configuration

Logs are here: https://drive.google.com/a/mirantis.com/file/d/0B6SjzarTGFxaZ3FPX3Uwa1ZVdGc/view?usp=sharing

Controllers: node-7,10
Controller+cinder: node-5
Computes: node-2,1

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/257765

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/257765
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=6134af0c6ec651e4feef45acdc5a291beab405cd
Submitter: Jenkins
Branch: master

commit 6134af0c6ec651e4feef45acdc5a291beab405cd
Author: Dmitry Bilunov <email address hidden>
Date: Tue Dec 15 12:29:07 2015 +0300

    Fix detached keystone node epoch mismatch

    Pacemaker maintains an internal database, which is used for
    configuration storage. Each update of this database increases a counter,
    called "epoch", which should have the same value cluster-wide.

    If an update operation comes to a previously detached node, a conflict
    will occur. Pacemaker does not allow updating this database on a node,
    which epoch value is lower than the epoch value of a cluster leader.

    We should wait the epoch counter to come into sync by perodically
    retrying the update command.

    Closes-Bug: 1494314
    Change-Id: I1f242bcd90264ec45da2aaa6bc030f244511761b

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/8.0
Review: https://review.openstack.org/267462

[OpenStack Infra (hudson-openstack)]

Change abandoned by Dmitry Bilunov (<email address hidden>) on branch: stable/8.0
Review: https://review.openstack.org/267462
Reason: Duplicate of 257765

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/8.0
Review: https://review.openstack.org/267487

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/267487
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=1a947e855f0c30bb20ae6acc0c86b90f4d034a71
Submitter: Jenkins
Branch: stable/8.0

commit 1a947e855f0c30bb20ae6acc0c86b90f4d034a71
Author: Dmitry Bilunov <email address hidden>
Date: Tue Dec 15 12:29:07 2015 +0300

    Fix detached keystone node epoch mismatch

    Pacemaker maintains an internal database, which is used for
    configuration storage. Each update of this database increases a counter,
    called "epoch", which should have the same value cluster-wide.

    If an update operation comes to a previously detached node, a conflict
    will occur. Pacemaker does not allow updating this database on a node,
    which epoch value is lower than the epoch value of a cluster leader.

    We should wait the epoch counter to come into sync by perodically
    retrying the update command.

    Closes-Bug: 1494314
    Change-Id: I1f242bcd90264ec45da2aaa6bc030f244511761b
    (cherry picked from commit 6134af0c6ec651e4feef45acdc5a291beab405cd)

[Vasily Gorin (vgorin)]

Verification on iso 523 is blocked by https://bugs.launchpad.net/fuel/+bug/1540388 .

[Artem Hrechanychenko (agrechanichenko)]

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  api: "1.0"
  build_number: "529"
  build_id: "529"
  fuel-nailgun_sha: "baec8643ca624e52b37873f2dbd511c135d236d9"
  python-fuelclient_sha: "4f234669cfe88a9406f4e438b1e1f74f1ef484a5"
  fuel-agent_sha: "658be72c4b42d3e1436b86ac4567ab914bfb451b"
  fuel-nailgun-agent_sha: "b2bb466fd5bd92da614cdbd819d6999c510ebfb1"
  astute_sha: "b81577a5b7857c4be8748492bae1dec2fa89b446"
  fuel-library_sha: "e2d79330d5d708796330fac67722c21f85569b87"
  fuel-ostf_sha: "3bc76a63a9e7d195ff34eadc29552f4235fa6c52"
  fuel-mirror_sha: "fb45b80d7bee5899d931f926e5c9512e2b442749"
  fuelmenu_sha: "e071216cb214e34b4d861478033425ee6a54a3be"
  shotgun_sha: "63645dea384a37dde5c01d4f8905566978e5d906"
  network-checker_sha: "a43cf96cd9532f10794dce736350bf5bed350e9d"
  fuel-upgrade_sha: "616a7490ec7199f69759e97e42f9b97dfc87e85b"
  fuelmain_sha: "a365f05b903368225da3fea9aa42afc1d50dc9b4"

[root@nailgun ~]# rpm -qp detach-database-1.1-1.1.0-1.noarch.rpm
detach-database-1.1-1.1.0-1.noarch
[root@nailgun ~]# rpm -qp detach-keystone-1.0-1.0.2-1.noarch.rpm
detach-keystone-1.0-1.0.2-1.noarch

[Artem Hrechanychenko (agrechanichenko)]

Steps:
1. Install detach-database and detach-keystone plugins on master node
2. Create next cluster - Ubuntu, HA, Neutron Vlan, enabled separate-database and separate-keystone, 3 controllers, 1 compute, 1 cinder, 3 database+keystone
3. Deploy cluster
4. After deployment add 1 keystone node
5. Start deployment

Actual result:
Deployment successfully finished

[ElenaRossokhina (esolomina)]

blocked by https://bugs.launchpad.net/fuel/+bug/1561489

[ElenaRossokhina (esolomina)]

Blocked again
fuel-9.0-mos-402-2016-05-26_06-18-59.iso
Initial scenario fails on step of new node addition https://bugs.launchpad.net/fuel/+bug/1586287

[TatyanaGladysheva (tgladysheva)]

Verified on 9.0 ISO #465 (fuel-9.0-mos-465-2016-06-09_22-51-38.iso) with initial scenario. Deployment is finished successfully.

[root@nailgun ~]# shotgun2 short-report
cat /etc/fuel_build_id:
 465
cat /etc/fuel_build_number:
 465
cat /etc/fuel_release:
 9.0
cat /etc/fuel_openstack_version:
 mitaka-9.0
rpm -qa | egrep 'fuel|astute|network-checker|nailgun|packetary|shotgun':
 fuel-release-9.0.0-1.mos6349.noarch
 fuel-misc-9.0.0-1.mos8454.noarch
 python-packetary-9.0.0-1.mos140.noarch
 fuel-bootstrap-cli-9.0.0-1.mos285.noarch
 fuel-migrate-9.0.0-1.mos8454.noarch
 shotgun-9.0.0-1.mos90.noarch
 fuel-notify-9.0.0-1.mos8454.noarch
 nailgun-mcagents-9.0.0-1.mos750.noarch
 python-fuelclient-9.0.0-1.mos325.noarch
 fuel-9.0.0-1.mos6349.noarch
 fuel-utils-9.0.0-1.mos8454.noarch
 fuel-setup-9.0.0-1.mos6349.noarch
 fuel-provisioning-scripts-9.0.0-1.mos8742.noarch
 fuel-library9.0-9.0.0-1.mos8454.noarch
 network-checker-9.0.0-1.mos74.x86_64
 fuel-agent-9.0.0-1.mos285.noarch
 fuel-ui-9.0.0-1.mos2717.noarch
 fuel-ostf-9.0.0-1.mos935.noarch
 fuelmenu-9.0.0-1.mos274.noarch
 fuel-nailgun-9.0.0-1.mos8742.noarch
 rubygem-astute-9.0.0-1.mos750.noarch
 fuel-mirror-9.0.0-1.mos140.noarch
 fuel-openstack-metadata-9.0.0-1.mos8742.noarch

[root@nailgun ~]# fuel plugins list
id | name | version | package_version | releases
---+-----------------+---------+-----------------+--------------------
1 | detach-database | 1.1.0 | 3.0.0 | ubuntu (mitaka-9.0)
2 | detach-keystone | 1.0.2 | 3.0.0 | ubuntu (mitaka-9.0)

[root@nailgun ~]# rpm -qp detach-database-1.1-1.1.0-1.noarch.rpm
detach-database-1.1-1.1.0-1.noarch
[root@nailgun ~]# rpm -qp detach-keystone-1.0-1.0.2-1.noarch.rpm
detach-keystone-1.0-1.0.2-1.noarch