mysql pacemaker upgrade fails when running docker

Bug #1898610 reported by Alex Schultz on 2020-10-05
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
High
Unassigned

Bug Description

In stein and train, we only supported running pacemaker with Docker when using centos7. Unfortunately https://review.opendev.org/#/c/693710/ and https://review.opendev.org/#/c/693773/ introduced a breaking task because it assumes podman with k8s-file. k8s-file is not available when running under docker.

2020-10-05 18:33:50.518 636624 WARNING tripleoclient.v1.tripleo_upgrade.Upgrade [-] TASK [Upgrade Mysql database from a temporary container] ******************************************************************************************************
2020-10-05 18:33:50.521 636624 WARNING tripleoclient.v1.tripleo_upgrade.Upgrade [-] fatal: [standalone-0]: FAILED! => {"changed": true, "cmd": "docker run --rm --log-driver=k8s-file --log-opt path=/var/log/containers/mysql/db-upgrade.log -u root --net=host -e \"KOLLA_CONFIG_STRATEGY=COPY_ALWAYS\" -v /etc/hosts:/etc/hosts:ro -v /etc/localtime:/etc/localtime:ro -v /etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro -v /etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro -v /etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro -v /etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro -v /etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro -v /dev/log:/dev/log -v /etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro -v /etc/puppet:/etc/puppet:ro -v /var/lib/kolla/config_files/mysql.json:/var/lib/kolla/config_files/config.json:rw,z -v /var/lib/config-data/puppet-generated/mysql/:/var/lib/kolla/config_files/src:ro,z -v /var/lib/mysql:/var/lib/mysql:rw,z -v /tmp/mariadb-upgrade:/var/log/mariadb:rw,z \"docker.io/tripleostein/centos-binary-mariadb:pcmklatest\" /bin/bash -ecx \"kolla_set_configs; if mysqladmin ping --silent; then exit 0; fi; chown -R mysql:mysql /var/lib/mysql; chown -R mysql:mysql /var/log/mariadb; mysqld_safe --user=mysql --wsrep-provider=none --skip-networking --wsrep-on=off & timeout 60 sh -c 'while ! mysqladmin ping --silent; do sleep 1; done'; mysql_upgrade; mysqladmin shutdown\"", "delta": "0:00:00.066754", "end": "2020-10-05 18:33:50.491301", "msg": "non-zero return code", "rc": 125, "start": "2020-10-05 18:33:50.424547", "stderr": "/usr/bin/docker-current: Error response from daemon: logger: no log driver named 'k8s-file' is registered.\nSee '/usr/bin/docker-current run --help'.", "stderr_lines": ["/usr/bin/docker-current: Error response from daemon: logger: no log driver named 'k8s-file' is registered.", "See '/usr/bin/docker-current run --help'."], "stdout": "", "stdout_lines": []}

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (stable/train)

Fix proposed to branch: stable/train
Review: https://review.opendev.org/756149

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (stable/train)

Reviewed: https://review.opendev.org/756149
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=f35c3455097a91cf6e07456b339829d82c18ad65
Submitter: Zuul
Branch: stable/train

commit f35c3455097a91cf6e07456b339829d82c18ad65
Author: Emilien Macchi <email address hidden>
Date: Mon Oct 5 14:59:11 2020 -0400

    [train] mysql: adjust CLI args based on container cli

    k8s-file log driver only exists for podman, such as log-opt path option.
    For docker, we'll use syslogd log driver.

    Note: this patch is only done for train and stein, where we still
    support Docker.

    Change-Id: Ie47e1cd81b25914d28ac63668817ce6237e2e8a9
    Closes-Bug: #1898610

tags: added: in-stable-train
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (stable/stein)

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/756564

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (stable/stein)

Reviewed: https://review.opendev.org/756564
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=9f9173da38ff81eb5e194590ac5cada212f4433a
Submitter: Zuul
Branch: stable/stein

commit 9f9173da38ff81eb5e194590ac5cada212f4433a
Author: Emilien Macchi <email address hidden>
Date: Mon Oct 5 14:59:11 2020 -0400

    [train] mysql: adjust CLI args based on container cli

    k8s-file log driver only exists for podman, such as log-opt path option.
    For docker, we'll use syslogd log driver.

    Note: this patch is only done for train and stein, where we still
    support Docker.

    Change-Id: Ie47e1cd81b25914d28ac63668817ce6237e2e8a9
    Closes-Bug: #1898610
    (cherry picked from commit f35c3455097a91cf6e07456b339829d82c18ad65)

tags: added: in-stable-stein
Changed in tripleo:
milestone: victoria-rc1 → wallaby-1
Changed in tripleo:
milestone: wallaby-1 → wallaby-2
Changed in tripleo:
milestone: wallaby-2 → wallaby-3
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates 11.4.0

This issue was fixed in the openstack/tripleo-heat-templates 11.4.0 release.

Revision history for this message
Marios Andreou (marios-b) wrote :

Bug status has been set to 'Fix-Released' based on the discussion and/or patches above. If you disagree please re-set 'Triaged' and reach out to us on freenode #tripleo thank you!

Changed in tripleo:
status: Triaged → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers