pacemaker compatibility issues when host/container have different versions

[Michele Baldessari]

When the version of pacemaker inside a container differs from the version running on the host, we can encounter the following error:
"""
Error: unable to get crm_config
cibadmin: Connection to local file '/var/lib/pacemaker/cib/puppet-cib-backup20171212-6-1h3l7tq' failed: Update does not conform to the configured schema
Signon to CIB failed: Update does not conform to the configured schema
"""

This happens in one situation, namely host has a pacemaker version which changed the schema (aka a newer version e.g. 1.1.18) and the containers have an older release in them (e.g. 1.1.16).
Note that the issue is not present when the newer version of pacemaker is in the containers and an older one is on the host.

This is an unfortunate consequence of running stuff partly on the host and partly in containers.
It should also be a "one-time" issue only as no other pcmk rebases are planned for Centos/RHEL 7.

In any case we should explore our options to avoid this issue.

Technically there are two slightly separated problems:
1) The temporary spawn containers that create pcs resources or properties
2) The containers that are managed by pacemaker and do call commandline tools (like cibadmin, crm_attribute, etc) in their normal operation

We did try to disable schema validation and also patched 1.1.16 with a bunch of patches:
0001-Low-implement-common-base-for-lib-xml2-xslt-log-libq.patch 0002-Feature-enable-upgrade-XSLTs-to-use-xsl-message-mean.patch
0001-Refactor-use-common-base-for-lib-xml2-xslt-log-libqb.patch 0003-Low-libcrmcommon-correct-spelling-of-pre-1.0-CIB-tag.patch
0004-Fix-libcrmcommon-handle-schema-versions-properly.patch

but the problem still persists.

It seems the only way out would be to actually bindmount *all* the folders and binaries necessary for pcs and all the pcmk-utils to work inside containers (both for 1) and 2) )

[Matt Young (halcyondude)]

Note:

---

# This is another instance of

https://bugs.launchpad.net/tripleo/+bug/1770692
potential for pacemaker version mismatch (BM vs.container) is a risk for OC deploy failures in gates

# RFE to identify this class of issue:

https://bugs.launchpad.net/tripleo/+bug/1771602
RFE: detect and warn when package versions in bare metal vs. container don't match

---

[Matt Young (halcyondude)]

@michele the RFE is tracking identifying and/or flagging package version mismatches (generally, not specific to pacemaker).

Is it the case that if a mismatch occurs in pacemaker *that* is a case that should not (ever) be supported? Or should we allow for some degree of variance between BM / containers?

[Michele Baldessari (michele)]

Hard to say. Right now, I'd say that unless we manage to get a working fix that won't impose this version restriction between containers and host, we only support same versions between host and containers (or later version on containers). Not newer on host and older on containers.

If we can come up (and atm I am not all too positive about it) with a solution, we would be able to relax this restriction.

[Michele Baldessari (michele)]

As an update here. Damien and I are exploring a hybrid solution to this long standing limitation.
Namely:
1. At deployment time and at update time we copy all files belonging to the set of pacemaker-related rpms under /var/lib/tripleo/pacemaker-host
2. We bind mount a bunch of folders (amongst which /var/lib/tripleo/pacemaker-host) inside the pcmk containers and we use custom PATH + LD_LIBRARY_PATH (both the temporary ones created by paunch and the ones managed by pcmk)
3. We eventually remove all pacemaker rpms from the HA kolla containers

[Juan Antonio Osorio Robles (juan-osorio-robles)]

Is this still an issue?

[Marios Andreou (marios-b)]

This is an automated action. Bug status has been set to 'Incomplete' and target milestone has been removed due to inactivity. If you disagree please re-set these values and reach out to us on freenode #tripleo