[ubuntu] rebuild corosync with pacemaker from vivid mirrors for Fuel mirrors

[Bogdan Dobrelya]

Original bug https://bugs.launchpad.net/fuel/+bug/1439120

The split brain issue https://bugs.launchpad.net/fuel/+bug/1436343 affects original trusty versions of corosync with pacemaker.
As a solution, Fuel should install OpenStack environments on Ubuntu Trusty with packages crmsh pacemaker libqb0 corosync and dependency libraries pinned to any upstream vivid or Fuel repackaged mirror as it shows no split brain issues on failover.

The list of packages to be re-built for Fuel in order to upgrade the corosync cluster stack is here http://pastebin.com/nTRbugKu

[Bogdan Dobrelya (bogdando)]

Here is related system test case https://bugs.launchpad.net/bugs/1439126

[Bogdan Dobrelya (bogdando)]

We have no a decision made yet, hence I set it to opinion

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/145788
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=0e0c6853fbf4937e9b73efec99ab7f6abff449b8
Submitter: Jenkins
Branch: master

commit 0e0c6853fbf4937e9b73efec99ab7f6abff449b8
Author: Bogdan Dobrelya <email address hidden>
Date: Thu Jan 8 15:16:03 2015 +0100

    Sync apt module from puppetlabs

    v1.8.0 47c6f338eda98ed849531d92b63c5e33103f76e4
    source https://github.com/puppetlabs/puppetlabs-apt

    Related blueprint merge-openstack-puppet-modules
    Related-bug: #1443800

    Change-Id: I5f86751ba9b16fdedd209677bb67e23cdb85b44f
    Signed-off-by: Bogdan Dobrelya <email address hidden>

[Alexander Nevenchannyy (anevenchannyy)]

Folks, i'm try'd to deploy cluster with packages from vivid repo:
cluster-glue_1.0.12~rc1+hg2777-1.2_amd64.deb
corosync_2.3.4-0ubuntu1_amd64.deb
crmsh_1.2.6+git+e77add-1.3ubuntu2_all.deb
libcfg6_2.3.4-0ubuntu1_amd64.deb
libcib4_1.1.12-0ubuntu1_amd64.deb
libcmap4_2.3.4-0ubuntu1_amd64.deb
libcorosync-common4_2.3.4-0ubuntu1_amd64.deb
libcpg4_2.3.4-0ubuntu1_amd64.deb
libcrmcluster4_1.1.12-0ubuntu1_amd64.deb
libcrmcommon3_1.1.12-0ubuntu1_amd64.deb
libcrmservice1_1.1.12-0ubuntu1_amd64.deb
libgmp10_2%3a6.0.0+dfsg-6ubuntu1_amd64.deb
libgnutls-deb0-28_3.3.8-3ubuntu3_amd64.deb
libhogweed2_2.7.1-5_amd64.deb
liblrm2_1.0.12~rc1+hg2777-1.2_amd64.deb
liblrmd1_1.1.12-0ubuntu1_amd64.deb
libnettle4_2.7.1-5_amd64.deb
libp11-kit0_0.20.7-1_amd64.deb
libpengine4_1.1.12-0ubuntu1_amd64.deb
libpe-rules2_1.1.12-0ubuntu1_amd64.deb
libpe-status4_1.1.12-0ubuntu1_amd64.deb
libpils2_1.0.12~rc1+hg2777-1.2_amd64.deb
libplumb2_1.0.12~rc1+hg2777-1.2_amd64.deb
libplumbgpl2_1.0.12~rc1+hg2777-1.2_amd64.deb
libqb0_0.17.0-2ubuntu1_amd64.deb
libquorum5_2.3.4-0ubuntu1_amd64.deb
libsam4_2.3.4-0ubuntu1_amd64.deb
libstonith1_1.0.12~rc1+hg2777-1.2_amd64.deb
libstonithd2_1.1.12-0ubuntu1_amd64.deb
libtasn1-6_4.2-2ubuntu1_amd64.deb
libtotem-pg5_2.3.4-0ubuntu1_amd64.deb
libtransitioner2_1.1.12-0ubuntu1_amd64.deb
libvotequorum7_2.3.4-0ubuntu1_amd64.deb
pacemaker_1.1.12-0ubuntu1_amd64.deb
pacemaker-cli-utils_1.1.12-0ubuntu1_amd64.deb

And we have error:
2015-04-16 15:56:34 ERR ERROR: configure.show: CIB is not valid
2015-04-16 15:56:34 ERR ERROR: No CIB!
2015-04-16 15:56:34 ERR ERROR: You may try the upgrade command
2015-04-16 15:56:34 ERR Could not prefetch cs_property provider 'crm': Execution of '/usr/sbin/crm configure show xml' returned 1: ERROR: CIB not supported: validator 'pacemaker-2.0', release '3.0.9'

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: 6.1
Change author: Aleksandr Mogylchenko <email address hidden>
Review: https://review.fuel-infra.org/5906

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: master
Change author: Aleksandr Mogylchenko <email address hidden>
Review: https://review.fuel-infra.org/5917

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/5917
Submitter: Andrey Nikitin <email address hidden>
Branch: master

Commit: 06d46aa96a06d3c483f788a2a8bc06a47c2f65bd
Author: Aleksandr Mogylchenko <email address hidden>
Date: Fri Apr 17 10:22:28 2015

Adding corosync to Trusty package repositories.

Since it was decided that we need to rebuild the whole stack, corosync
should be added (see bugs descriptions for more details).

Change-Id: I8a7e18661f5350e673be9056699f667d6ad60017
Partial-Bug: #1443800
Related-Bug: #1439120
Related-Bug: #1436343

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: 6.1
Change author: Aleksandr Mogylchenko <email address hidden>
Review: https://review.fuel-infra.org/5950

[Aleksander Mogylchenko (amogylchenko)]

This requires some assistance from OSCI, since requires building cross-dependencies:
https://review.fuel-infra.org/#/c/5906/

https://review.fuel-infra.org/#/c/5388/

https://review.fuel-infra.org/#/c/5950/

Dmitriy Burmistrov and me are working on this.

[Sergii Golovatiuk (sgolovatiuk)]

crmsh should be also repackaged from vivid as the previous one cannot work
with new CIB format.

/usr/sbin/crm configure show xml' returned 1: ERROR: CIB not supported:
validator 'pacemaker-2.0', release '3.0.9'

--
Best regards,
Sergii Golovatiuk,
Skype #golserge
IRC #holser

On Fri, Apr 17, 2015 at 5:15 PM, Aleksander Mogylchenko <
<email address hidden>> wrote:

> This requires some assistance from OSCI, since requires building
> cross-dependencies:
> https://review.fuel-infra.org/#/c/5906/
> https://review.fuel-infra.org/#/c/5388/
> https://review.fuel-infra.org/#/c/5950/
>
> Dmitriy Burmistrov and me are working on this.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1443800
>
> Title:
> [ubuntu] rebuild corosync with pacemaker from vivid mirrors for Fuel
> mirrors
>
> Status in Fuel: OpenStack installer that works:
> In Progress
> Status in Fuel for OpenStack 6.0.x series:
> Invalid
>
> Bug description:
> Original bug https://bugs.launchpad.net/fuel/+bug/1439120
>
> The split brain issue https://bugs.launchpad.net/fuel/+bug/1436343
> affects original trusty versions of corosync with pacemaker.
> As a solution, Fuel should install OpenStack environments on Ubuntu
> Trusty with packages crmsh pacemaker libqb0 corosync and dependency
> libraries pinned to any upstream vivid or Fuel repackaged mirror as it
> shows no split brain issues on failover.
>
> The list of packages to be re-built for Fuel in order to upgrade the
> corosync cluster stack is here http://pastebin.com/nTRbugKu
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/fuel/+bug/1443800/+subscriptions
>

[Fuel Devops McRobotson (fuel-devops-robot)]

Related fix proposed to branch: master
Change author: Aleksandr Mogylchenko <email address hidden>
Review: https://review.fuel-infra.org/5972

[Aleksander Mogylchenko (amogylchenko)]

Interesting.

I’ll rebuild it, although I’m not sure that this is the root cause. Both packages (from trusty and vivid) have the same dependency on pacemaker (>= 1.1.8-1), and version difference is minor (1.2.5 vs 1.2.6).

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/5972
Submitter: Pawel Brzozowski <email address hidden>
Branch: master

Commit: a0fdf5f6533670da91ae22aee93d89930c146421
Author: Aleksandr Mogylchenko <email address hidden>
Date: Fri Apr 17 18:14:52 2015

Adding crmsh to Trusty package repositories.

Since it was decided that we need to rebuild the whole stack, crmsh
should be added (see bugs descriptions for more details).

Change-Id: Ice06ecbb8d0e1efcf39a51be0b0e458df8373514
Related-Bug: #1443800

[Sergii Golovatiuk (sgolovatiuk)]

Actually, it's a bug in vivid

https://bugs.launchpad.net/ubuntu/+source/crmsh/+bug/1445616

--
Best regards,
Sergii Golovatiuk,
Skype #golserge
IRC #holser

On Fri, Apr 17, 2015 at 8:51 PM, Aleksander Mogylchenko <
<email address hidden>> wrote:

> Interesting.
>
> I’ll rebuild it, although I’m not sure that this is the root cause. Both
> packages (from trusty and vivid) have the same dependency on pacemaker
> (>= 1.1.8-1), and version difference is minor (1.2.5 vs 1.2.6).
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1443800
>
> Title:
> [ubuntu] rebuild corosync with pacemaker from vivid mirrors for Fuel
> mirrors
>
> Status in Fuel: OpenStack installer that works:
> In Progress
> Status in Fuel for OpenStack 6.0.x series:
> Invalid
>
> Bug description:
> Original bug https://bugs.launchpad.net/fuel/+bug/1439120
>
> The split brain issue https://bugs.launchpad.net/fuel/+bug/1436343
> affects original trusty versions of corosync with pacemaker.
> As a solution, Fuel should install OpenStack environments on Ubuntu
> Trusty with packages crmsh pacemaker libqb0 corosync and dependency
> libraries pinned to any upstream vivid or Fuel repackaged mirror as it
> shows no split brain issues on failover.
>
> The list of packages to be re-built for Fuel in order to upgrade the
> corosync cluster stack is here http://pastebin.com/nTRbugKu
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/fuel/+bug/1443800/+subscriptions
>

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: 6.1
Change author: Aleksandr Mogylchenko <email address hidden>
Review: https://review.fuel-infra.org/5973

[Aleksander Mogylchenko (amogylchenko)]

I've chosen crmsh_2.1:
https://obs-1.mirantis.com/project/monitor/trusty-fuel-6.1-stable-LP1443800?arch_x86_64=1&defaults=0&repo_ubuntu=1&succeeded=1

if it does not suit your needs, please let me know what you need (2.1.x maybe).

[Aleksander Mogylchenko (amogylchenko)]

Can you please test those packages and let me know if they work?

[Bogdan Dobrelya (bogdando)]

@Aleksandr, the durability testing showed no cluster splits after 1500 iterations. So, this looks fine.

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/5906
Submitter: Michael Semenov <email address hidden>
Branch: 6.1

Commit: 3ec75bd020b44c453e7ce64a3749674e5a06508c
Author: Aleksandr Mogylchenko <email address hidden>
Date: Tue Apr 28 12:40:02 2015

Adding libqb 0.17 fom Vivid.

Sources:
http://archive.ubuntu.com/ubuntu/pool/main/libq/libqb/libqb_0.17.0.orig.tar.gz

It was decided that we need to import HA stack (corosync, pacemaker and libqb)
from Vivid, since testing revealed unstability of versions in Trusty:
https://bugs.launchpad.net/ubuntu/+source/libqb/+bug/1341496

Partial-Bug: #1443800
Related-Bug: #1439120
Related-Bug: #1436343
Change-Id: I23bda853b11f7a9b442d31e6ad1ba3cb5beb4e9e

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/5950
Submitter: Michael Semenov <email address hidden>
Branch: 6.1

Commit: bac08f0040e307695003483aa1834872bfcc4ee2
Author: Aleksandr Mogylchenko <email address hidden>
Date: Tue Apr 28 12:38:33 2015

Add corosync 2.3.4 from Vivid

It was decided that we need to import HA stack (corosync, pacemaker and libqb)
from Vivid, since testing revealed unstability of versions in Trusty,
causing split brain issues after node reboot. (see
bugs descriptions for more details).

Sources:
http://archive.ubuntu.com/ubuntu/pool/main/c/corosync/corosync_2.3.4.orig.tar.gz

Partial-Bug: #1443800
Related-Bug: #1439120
Related-Bug: #1436343
Change-Id: I6b6b0e40e6d39cf78d35a0af825d655f9e70ac10

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/5973
Submitter: Michael Semenov <email address hidden>
Branch: 6.1

Commit: 5a3a09665b7b8ccebb00998b4df41962ca26d875
Author: Aleksandr Mogylchenko <email address hidden>
Date: Tue Apr 28 08:42:47 2015

Add crmsh 2.1

It was decided that we need to import HA stack (corosync, pacemaker and libqb)
from Vivid, since testing revealed unstability of versions in Trusty, causing
split brain issues after node reboot (see bugs descriptions for more details).

crmsh prior to 2.1 has problems with 1.3 and 2.0 RNG schemas support:
http://lists.linux-ha.org/pipermail/linux-ha/2014-April/048007.html
https://github.com/ClusterLabs/crmsh/commit/686c714fdd8a7376f93c8bb78be0b8056f4ecd6e.patch

Those were properly fixed in 2.1, hense new release.

Sources:
https://github.com/ClusterLabs/crmsh/archive/2.1.0.tar.gz

Partial-Bug: #1443800
Change-Id: Id5e0b30a2c2c73b5f6fc64086adad593b9310fd8

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/5388
Submitter: Michael Semenov <email address hidden>
Branch: 6.1

Commit: 219f536a9c96c7c332606fdfa3f1832836a3015e
Author: Aleksandr Mogylchenko <email address hidden>
Date: Tue Apr 28 16:13:06 2015

Pacemaker 1.1.12 from Ubuntu Vivid version.

It was decided that we need to import HA stack (corosync, pacemaker and libqb)
from Vivid, since testing revealed unstability of versions in Trusty (see
bugs descriptions for more details).

Original source:
http://archive.ubuntu.com/ubuntu/pool/main/p/pacemaker/pacemaker_1.1.12.orig.tar.gz

debian/control was modified to depend on crmsh (>= 2.1), which supports new CIB

Partial-Bug: #1443800
Change-Id: Icd22ba7f81b1e559008b09c080c6ce0f70ed88f9

[Alexander Nevenchannyy (anevenchannyy)]

Verified on MOS 6.1 ISO #429
Steps to Verify:
dpkg -l

Installed versions of packages:
corosync 2.3.4-0u~u14.04+mos1
pacemaker 1.1.12-0u~u14.04+mos6.1
crmsh 2.1.0-1~u14.04+mos1
libqb0 0.17.0-2~u14.04+mos1