Mysql Upgrade tasks failing due to podman log-driver not found

[Cédric Jeanneret]

This is a copy of an existing BZ, for better tracking and backporting:
https://bugzilla.redhat.com/show_bug.cgi?id=1769291

Description of problem:

When upgrading from OSP14 to OSP15, the upgrades run step is running in the following task:

TASK [Bind mounts for temporary container] *************************************
Wednesday 06 November 2019 05:23:55 -0500 (0:00:00.087) 0:02:33.612 ****
ok: [lab-controller01] => {"ansible_facts": {"mysql_upgrade_db_bind_mounts": ["/etc/hosts:/etc/hosts:ro", "/etc/localtime:/etc/localtime:ro", "/etc/pki/ca-trust/extracted:/e
tc/pki/ca-trust/extracted:ro", "/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro", "/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro"
, "/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro", "/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro", "/dev/log:/dev/log", "/etc/ssh/ssh_kn
own_hosts:/etc/ssh/ssh_known_hosts:ro", "/etc/puppet:/etc/puppet:ro", "/var/lib/kolla/config_files/mysql.json:/var/lib/kolla/config_files/config.json", "/var/lib/config-data
/puppet-generated/mysql/:/var/lib/kolla/config_files/src:ro", "/var/lib/mysql:/var/lib/mysql"]}, "changed": false}

TASK [Make sure Mysql upgrade temporary directory exists] **********************
Wednesday 06 November 2019 05:23:55 -0500 (0:00:00.084) 0:02:33.696 ****
ok: [lab-controller01] => {"changed": false, "gid": 0, "group": "root", "mode": "0700", "owner": "root", "path": "/tmp/mariadb-upgrade", "secontext": "unconfined_u:object_r:
user_tmp_t:s0", "size": 6, "state": "directory", "uid": 0}

TASK [Upgrade Mysql database from a temporary container] ***********************
Wednesday 06 November 2019 05:23:56 -0500 (0:00:00.331) 0:02:34.028 ****
fatal: [lab-controller01]: FAILED! => {"changed": true, "cmd": "podman run --rm --log-driver=syslog -u root --net=host -e \"KOLLA_CONFIG_STRATEGY=COPY_ALWAYS\" -v /etc/hosts:/etc/hosts:ro -v /etc/localtime:/etc/localtime:ro -v /etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro -v /etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source
/anchors:ro -v /etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro -v /etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro -v /e
tc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro -v /dev/log:/dev/log -v /etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro -v /etc/puppet:/etc/puppet:ro -v /var/lib/kolla/config_files/mysql.json:/var/lib/kolla/config_files/config.json -v /var/lib/config-data/puppet-generated/mysql/:/var/lib/kolla/config_files/src:ro -v /var/lib/mysql:/var/lib/my
sql -v /tmp/mariadb-upgrade:/var/log/mariadb:rw \"172.16.0.1:8787/rhosp15-rhel8/openstack-mariadb:pcmklatest\" /bin/bash -ecx \"kolla_set_configs; if mysqladmin ping --silen
t; then exit 0; fi; chown -R mysql:mysql /var/lib/mysql; chown -R mysql:mysql /var/log/mariadb; mysqld_safe --user=mysql --wsrep-provider=none --skip-networking --wsrep-on=o
ff & timeout 60 sh -c 'while ! mysqladmin ping --silent; do sleep 1; done'; mysql_upgrade; mysqladmin shutdown\"", "delta": "0:00:00.094721", "end": "2019-11-06 10:23:56.225916", "msg": "non-zero return code", "rc": 125, "start": "2019-11-06 10:23:56.131195", "stderr": "Error: error running container create option: invalid log driver: invalid a
rgument", "stderr_lines": ["Error: error running container create option: invalid log driver: invalid argument"], "stdout": "", "stdout_lines": []}

NO MORE HOSTS LEFT *************************************************************

PLAY RECAP *********************************************************************
lab-controller01 : ok=140 changed=44 unreachable=0 failed=1 skipped=256 rescued=0 ignored=1

Wednesday 06 November 2019 05:23:56 -0500 (0:00:00.385) 0:02:34.414 ****
===============================================================================

Ansible failed, check log at /var/lib/mistral/292a8b96-1d22-4919-a071-50bf4e6eed0a/ansible.log.

The problem seems to be caused by a dreprecation of the syslog log-driver in paunch, task: https://github.com/openstack/tripleo-heat-templates/blob/9fbb355289d59a394e410aee0fe7ce675e130127/deployment/database/mysql-pacemaker-puppet.yaml#L587

We need to use k8s-file driver instead and set also the log file to be used.

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
1. Deploy OSP 14
2. Upgrade UC to OSP15
3. Run first Overcloud Controller upgrade steps

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/693317

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/693317
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=8b07a2259bc7e06e1423ee99a411b873dfe98937
Submitter: Zuul
Branch: master

commit 8b07a2259bc7e06e1423ee99a411b873dfe98937
Author: Cédric Jeanneret <email address hidden>
Date: Thu Nov 7 09:42:08 2019 +0100

    Correct mysql-pacemaker upgrade task

    There were two issues in the mysql-pacemaker upgrade tasks:

    - SELinux: since we're using podman, we have proper selinux enforcing on
      the system and proper selinux separation for the containers. Some
      volumes were lacking the "z" flag, making them unaccessible

    - Since we're on podman, we have to correct the "log-driver" in the
      command. This allow to get a dedicated log for debug purpose.

    Change-Id: Ia03e6e8e913198b315c47982c14ed52569ec702c
    Closes-Bug: #1851617
    Resolves: rhbz#1769291

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/693710

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/693710
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=ef5e84f8ea7ace74912aad7a29eccfd3376c4ec0
Submitter: Zuul
Branch: stable/train

commit ef5e84f8ea7ace74912aad7a29eccfd3376c4ec0
Author: Cédric Jeanneret <email address hidden>
Date: Thu Nov 7 09:42:08 2019 +0100

    Correct mysql-pacemaker upgrade task

    There were two issues in the mysql-pacemaker upgrade tasks:

    - SELinux: since we're using podman, we have proper selinux enforcing on
      the system and proper selinux separation for the containers. Some
      volumes were lacking the "z" flag, making them unaccessible

    - Since we're on podman, we have to correct the "log-driver" in the
      command. This allow to get a dedicated log for debug purpose.

    Change-Id: Ia03e6e8e913198b315c47982c14ed52569ec702c
    Closes-Bug: #1851617
    Resolves: rhbz#1769291
    (cherry picked from commit 8b07a2259bc7e06e1423ee99a411b873dfe98937)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/693773

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/693773
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=6e843c5941b1ec7ce079765ac45dfec3a77f4d08
Submitter: Zuul
Branch: stable/stein

commit 6e843c5941b1ec7ce079765ac45dfec3a77f4d08
Author: Cédric Jeanneret <email address hidden>
Date: Thu Nov 7 09:42:08 2019 +0100

    Correct mysql-pacemaker upgrade task

    There were two issues in the mysql-pacemaker upgrade tasks:

    - SELinux: since we're using podman, we have proper selinux enforcing on
      the system and proper selinux separation for the containers. Some
      volumes were lacking the "z" flag, making them unaccessible

    - Since we're on podman, we have to correct the "log-driver" in the
      command. This allow to get a dedicated log for debug purpose.

    Change-Id: Ia03e6e8e913198b315c47982c14ed52569ec702c
    Closes-Bug: #1851617
    Resolves: rhbz#1769291
    (cherry picked from commit 8b07a2259bc7e06e1423ee99a411b873dfe98937)
    (cherry picked from commit ef5e84f8ea7ace74912aad7a29eccfd3376c4ec0)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 12.0.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 10.6.2 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 11.3.1 release.