Wsrep recovery doesnt go far enough to recover sequence number

[Benard Okolie]

When recovering the mariadb cluster the play to recover the sequence number runs the below command:

tail -n 200 /tmp/mariadb_tmp.log | awk -F" " '$0~/Recovered position/{print $NF;exit;}' | awk -F":" '{print $1}'

There are times however when the wsrep number is further back than the last 200 lines, in which case the recovery will fail. It would be good to at least double this number, or maybe even cat the entire file when looking for the wsrep number.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/916525

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/916525
Committed: https://opendev.org/openstack/kolla-ansible/commit/984cb0a7543e3b78f8e7e5b3ef33bc3bd3ce690a
Submitter: "Zuul (22348)"
Branch: master

commit 984cb0a7543e3b78f8e7e5b3ef33bc3bd3ce690a
Author: Sven Kieske <email address hidden>
Date: Fri Apr 19 17:17:28 2024 +0200

    mariadb: fix cluster recovery

    sometimes cluster recovery didn't work
    because we only look for the sequence number in the last 200 lines
    of the log file.

    fix this by ingesting the complete file and only register the last
    sequence number we find.

    Closes-Bug: 1821173

    Change-Id: Iea2661c9d5d262cf99edd5f5b567f252607a0003
    Signed-off-by: Sven Kieske <email address hidden>

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2023.2
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/916449

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/916450

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/916451

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: unmaintained/yoga
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/916452

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/916449
Committed: https://opendev.org/openstack/kolla-ansible/commit/5c750e898ecb787030284d7d95b9d7ed02da91c2
Submitter: "Zuul (22348)"
Branch: stable/2023.2

commit 5c750e898ecb787030284d7d95b9d7ed02da91c2
Author: Sven Kieske <email address hidden>
Date: Fri Apr 19 17:17:28 2024 +0200

    mariadb: fix cluster recovery

    sometimes cluster recovery didn't work
    because we only look for the sequence number in the last 200 lines
    of the log file.

    fix this by ingesting the complete file and only register the last
    sequence number we find.

    Closes-Bug: 1821173

    Change-Id: Iea2661c9d5d262cf99edd5f5b567f252607a0003
    Signed-off-by: Sven Kieske <email address hidden>
    (cherry picked from commit 984cb0a7543e3b78f8e7e5b3ef33bc3bd3ce690a)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/916450
Committed: https://opendev.org/openstack/kolla-ansible/commit/92e67409ce3fc9a0557c285c343bdcd89c2d3327
Submitter: "Zuul (22348)"
Branch: stable/2023.1

commit 92e67409ce3fc9a0557c285c343bdcd89c2d3327
Author: Sven Kieske <email address hidden>
Date: Fri Apr 19 17:17:28 2024 +0200

    mariadb: fix cluster recovery

    sometimes cluster recovery didn't work
    because we only look for the sequence number in the last 200 lines
    of the log file.

    fix this by ingesting the complete file and only register the last
    sequence number we find.

    Closes-Bug: 1821173

    Change-Id: Iea2661c9d5d262cf99edd5f5b567f252607a0003
    Signed-off-by: Sven Kieske <email address hidden>
    (cherry picked from commit 984cb0a7543e3b78f8e7e5b3ef33bc3bd3ce690a)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/916452
Committed: https://opendev.org/openstack/kolla-ansible/commit/d48a33c1260afc9a8128c76c6067769629cb92a4
Submitter: "Zuul (22348)"
Branch: unmaintained/yoga

commit d48a33c1260afc9a8128c76c6067769629cb92a4
Author: Sven Kieske <email address hidden>
Date: Fri Apr 19 17:17:28 2024 +0200

    mariadb: fix cluster recovery

    sometimes cluster recovery didn't work
    because we only look for the sequence number in the last 200 lines
    of the log file.

    fix this by ingesting the complete file and only register the last
    sequence number we find.

    Closes-Bug: 1821173

    Change-Id: Iea2661c9d5d262cf99edd5f5b567f252607a0003
    Signed-off-by: Sven Kieske <email address hidden>
    (cherry picked from commit 984cb0a7543e3b78f8e7e5b3ef33bc3bd3ce690a)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/916451
Committed: https://opendev.org/openstack/kolla-ansible/commit/aa0c8f85517838c38dfaf1ce4d7b1b2ebec713a2
Submitter: "Zuul (22348)"
Branch: stable/zed

commit aa0c8f85517838c38dfaf1ce4d7b1b2ebec713a2
Author: Sven Kieske <email address hidden>
Date: Fri Apr 19 17:17:28 2024 +0200

    mariadb: fix cluster recovery

    sometimes cluster recovery didn't work
    because we only look for the sequence number in the last 200 lines
    of the log file.

    fix this by ingesting the complete file and only register the last
    sequence number we find.

    Closes-Bug: 1821173

    Change-Id: Iea2661c9d5d262cf99edd5f5b567f252607a0003
    Signed-off-by: Sven Kieske <email address hidden>
    (cherry picked from commit 984cb0a7543e3b78f8e7e5b3ef33bc3bd3ce690a)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible zed-eom release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 18.0.0.0rc1 release candidate.