RFE: detect and warn when package versions in bare metal vs. container don't match

[Matt Young]

This is an RFE identified as part of an escalation related to

https://bugs.launchpad.net/tripleo/+bug/1770692: potential for pacemaker version mismatch (BM vs.container) is a risk for OC deploy failures in gates

In this particular case the version of pacemaker packages in the BM images and in containers did not match, due to the timing / workflow in CI.

This RFE is to add a check to CI to ensure we don't have package version mismatches prior to spending time/resources/cycles both in our clouds running CI jobs, but in the time to triage, diagnose, debug, and resolve issues caused by this class of issue.

Requirements:

- Our CI tooling should fail early
- The identified package mismatch details should be clearly emitted and in the job logs, rendering it discoverable in an obvious way.
- All mismatches should be identified, vs. a "whack a mole" (e.g. iterative, one mismatch per job attempt) to conserve CI resources and human time.
- Both normal deployment / promotion workflows, as well as Gating change workflows should be handled.

[Matt Young (halcyondude)]

An example of human/debug time spent as a result of this class of issue:

https://bugs.launchpad.net/tripleo/+bug/1771612

[Tim Rozet (trozet)]

This also becomes a critical issue with libvirtd. Since libvirtd is containerized in the nova_libvirt container, it is using an older version of libvirt which is incompatible with the one in centos 7.5. Due to this when you deploy the deployment hangs at step 4 and is stuck during the deployment at the virsh secret-define command for creating the ceph secret:

bin/bash -c /usr/bin/virsh secret-define --file /etc/nova/secret.xml && /usr/bin/virsh secret-set-value --secret '7f6eaf28-e29b-42f5-8fcd-9109b85e2768'

This ends up hanging for 30 minutes or so until libvirt finally reports back an error.

[Bogdan Dobrelya (bogdando)]

I think the issue is not CI limited. Those should belong to validations and productized.

[Bogdan Dobrelya (bogdando)]

This affects upgrades as well.

[Bogdan Dobrelya (bogdando)]

I'd really wanted to see this marked a critical issue...

[Matt Young (halcyondude)]

triage: this is not a *failure* event, but perhaps we should *warn* or *inform* when this is the case...

potential solutions:

in collect-logs...

foreach(c in containers)
    generate rpm list

compare rpm-list from BM <-> c.rpms
warn / alert (or just drop artifact for now)

note: could add this to existing container-check module

[Juan Antonio Osorio Robles (juan-osorio-robles)]

is this still needed?