Push the pacemaker DEB version to the stable 1.1.12 for ubuntu

[Bogdan Dobrelya]

Related bug https://bugs.launchpad.net/fuel/+bug/1436343
We need to ship Ubuntu Trusty in the 6.1 release of Fuel with the pacemaker 1.1.12 package as v 1.1.10 looks not stable:

Pacemaker may be unable to communicate with corosync on restart, see details in the duplicating bugs

[Aleksander Mogylchenko (amogylchenko)]

"looks not stable" is not the valid reason for forking packages.

It was decided that 6.1 will have the absolute minimum of modified/recompiled packages, and if you need the package updated, please provide links to bugs that exist in 1.1.10 and are fixed in 1.1.12.

[Davanum Srinivas (DIMS) (dims-v)]

Aleksander,

Please see:
https://bugs.launchpad.net/fuel/+bug/1436343

That has links to other bugs.

[Vladimir Kuklin (vkuklin)]

Folks

I confirm that we need updated pacemaker package which has lots of fixes compared to 1.1.10. Please consider this issue as highly important and build corresponding packages.

[Vladimir Kuklin (vkuklin)]

And here is another issue

https://bugs.launchpad.net/fuel/6.1.x/+bug/1438823

[Aleksander Mogylchenko (amogylchenko)]

Those are fuel bugs. I meant bugs officially published in corosync and fixed in 1.1.12.

[Aleksander Mogylchenko (amogylchenko)]

We might consider backporting those patches instead of providing new package.

[Bogdan Dobrelya (bogdando)]

Here is an upstream bug, hopefully https://bugs.launchpad.net/ubuntu/+source/pacemaker/+bug/1439649

[Alexei Sheplyakov (asheplyakov)]

> Here is an upstream bug, hopefully https://bugs.launchpad.net/ubuntu/+source/pacemaker/+bug/1439649

Which is not confirmed by anyone (except the reporter himself)

[Alexei Sheplyakov (asheplyakov)]

> Sometimes after node rebooted crmd dies almost immediately which leads to the node being offline
> for her own and for neighbors and ps axu | grep pacemaker show all processes except of crmd

Is it really possible to check if the update solves the problem using the above vague description?

[Davanum Srinivas (DIMS) (dims-v)]

Aleksander,

Looking at:
https://github.com/ClusterLabs/pacemaker/releases/

and at:
https://bugs.launchpad.net/ubuntu/+source/pacemaker/+bug/1426761

Gives an idea of the number of things that got fixed. Though i don't know enough to point out exactly which one of those fixes we need :(

[Davanum Srinivas (DIMS) (dims-v)]

Aleksander,

yes, Alex Khivin has a test to fix the update solves the problem. Please see:
https://bugs.launchpad.net/fuel/+bug/1436343

[Davanum Srinivas (DIMS) (dims-v)]

"Alex Khivin has a test that can check if the update fixes the problem." :)

[Aleksander Mogylchenko (amogylchenko)]

My main confusion comes from the fact that the same software worked fine in 6.0. And now all of a sudden we started seeing errors. That is why I want to get to the root of the problem before forking packages.

[Davanum Srinivas (DIMS) (dims-v)]

Aleksander,

Understood. We've tended to just restart stuff including rabbitmq, removing mnesia etc some of it probably triggered by this pacemaker issues and gotten used to it now. That's why Alexey and I were trying hard to get to the bottom of why rabbitmq clusters are going bad and ended up with this theory/suggestion from Bogdan [1]

[1] https://bugs.launchpad.net/fuel/+bug/1436343/comments/12

[Aleksander Mogylchenko (amogylchenko)]

Repo with pacemaker 1.1.12 is here:
http://obs-1.mirantis.com:82/trusty-fuel-6.1-stable-5388/ubuntu/

Please test the package before we merge it, and confirm that it fits your needs (and put +1 on CR :))

[Bogdan Dobrelya (bogdando)]

I tested corosync 2.3.4-0ubuntu1, libqb0 0.17.0-2ubuntu1, pacemaker 1.1.12-0ubuntu1, crmsh 1.2.6+git+e77add-1.3ubuntu2 from deb http://archive.ubuntu.com/ubuntu/ vivid main universe multiverse and there was no split brain issue reproduced after 1000 tries

[Bogdan Dobrelya (bogdando)]

Superseded by https://bugs.launchpad.net/fuel/+bug/1443800

[Fuel Devops McRobotson (fuel-devops-robot)]

Related fix proposed to branch: 6.1
Change author: Aleksandr Mogylchenko <email address hidden>
Review: https://review.fuel-infra.org/5906

[Fuel Devops McRobotson (fuel-devops-robot)]

Related fix proposed to branch: master
Change author: Aleksandr Mogylchenko <email address hidden>
Review: https://review.fuel-infra.org/5917

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/5917
Submitter: Andrey Nikitin <email address hidden>
Branch: master

Commit: 06d46aa96a06d3c483f788a2a8bc06a47c2f65bd
Author: Aleksandr Mogylchenko <email address hidden>
Date: Fri Apr 17 10:22:28 2015

Adding corosync to Trusty package repositories.

Since it was decided that we need to rebuild the whole stack, corosync
should be added (see bugs descriptions for more details).

Change-Id: I8a7e18661f5350e673be9056699f667d6ad60017
Partial-Bug: #1443800
Related-Bug: #1439120
Related-Bug: #1436343

[Fuel Devops McRobotson (fuel-devops-robot)]

Related fix proposed to branch: 6.1
Change author: Aleksandr Mogylchenko <email address hidden>
Review: https://review.fuel-infra.org/5950

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/5906
Submitter: Michael Semenov <email address hidden>
Branch: 6.1

Commit: 3ec75bd020b44c453e7ce64a3749674e5a06508c
Author: Aleksandr Mogylchenko <email address hidden>
Date: Tue Apr 28 12:40:02 2015

Adding libqb 0.17 fom Vivid.

Sources:
http://archive.ubuntu.com/ubuntu/pool/main/libq/libqb/libqb_0.17.0.orig.tar.gz

It was decided that we need to import HA stack (corosync, pacemaker and libqb)
from Vivid, since testing revealed unstability of versions in Trusty:
https://bugs.launchpad.net/ubuntu/+source/libqb/+bug/1341496

Partial-Bug: #1443800
Related-Bug: #1439120
Related-Bug: #1436343
Change-Id: I23bda853b11f7a9b442d31e6ad1ba3cb5beb4e9e

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/5950
Submitter: Michael Semenov <email address hidden>
Branch: 6.1

Commit: bac08f0040e307695003483aa1834872bfcc4ee2
Author: Aleksandr Mogylchenko <email address hidden>
Date: Tue Apr 28 12:38:33 2015

Add corosync 2.3.4 from Vivid

It was decided that we need to import HA stack (corosync, pacemaker and libqb)
from Vivid, since testing revealed unstability of versions in Trusty,
causing split brain issues after node reboot. (see
bugs descriptions for more details).

Sources:
http://archive.ubuntu.com/ubuntu/pool/main/c/corosync/corosync_2.3.4.orig.tar.gz

Partial-Bug: #1443800
Related-Bug: #1439120
Related-Bug: #1436343
Change-Id: I6b6b0e40e6d39cf78d35a0af825d655f9e70ac10