When running the upgrade from Rocky to Stein it did fail again with :
TASK [Upgrade Mysql database from a temporary container] ***********************
Tuesday 19 November 2019 09:26:01 -0500 (0:00:00.308) 0:09:07.672 ******
fatal: [controller-0]: FAILED! => {"changed": true, "cmd": "podman run --rm --log-driver=k8s-file --log-opt path=LOG_DIR/db-upgrade.log -u root --net=host -e \"KOLLA_CONFIG_
STRATEGY=COPY_ALWAYS\" -v /etc/hosts:/etc/hosts:ro -v /etc/localtime:/etc/localtime:ro -v /etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro -v /etc/pki/ca-trust/sou
rce/anchors:/etc/pki/ca-trust/source/anchors:ro -v /etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro -v /etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tl
s/certs/ca-bundle.trust.crt:ro -v /etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro -v /dev/log:/dev/log -v /etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro -v /etc/puppet:
/etc/puppet:ro -v /var/lib/kolla/config_files/mysql.json:/var/lib/kolla/config_files/config.json:rw,z -v /var/lib/config-data/puppet-generated/mysql/:/var/lib/kolla/config_f
iles/src:ro,z -v /var/lib/mysql:/var/lib/mysql:rw,z -v /tmp/mariadb-upgrade:/var/log/mariadb:rw,z \"192.168.24.1:8787/rh-osbs/rhosp15-openstack-mariadb:pcmklatest\" /bin/bas
h -ecx \"kolla_set_configs; if mysqladmin ping --silent; then exit 0; fi; chown -R mysql:mysql /var/lib/mysql; chown -R mysql:mysql /var/log/mariadb; mysqld_safe --user=mysq
l --wsrep-provider=none --skip-networking --wsrep-on=off & timeout 60 sh -c 'while ! mysqladmin ping --silent; do sleep 1; done'; mysql_upgrade; mysqladmin shutdown\"", "del
ta": "0:00:01.113447", "end": "2019-11-19 14:26:01.884729", "msg": "non-zero return code", "rc": 127, "start": "2019-11-19 14:26:00.771282", "stderr": "[conmon:e]: Failed to
open log file No such file or directory\nError: write child: broken pipe", "stderr_lines": ["[conmon:e]: Failed to open log file No such file or directory", "Error: write c
hild: broken pipe"], "stdout": "", "stdout_lines": []}
NO MORE HOSTS LEFT *************************************************************
PLAY RECAP *********************************************************************
controller-0 : ok=146 changed=59 unreachable=0 failed=1 skipped=245 rescued=0 ignored=1
Tuesday 19 November 2019 09:26:02 -0500 (0:00:01.365) 0:09:09.038 ******
===============================================================================
It seems there is a typo in the fix and the LOG_DIR parameter didn't get replaced: https://review.opendev.org/#/c/693773/2/deployment/database/mysql-pacemaker-puppet.yaml@598 as the parameter was named LOGDIR.
Doing some manual testing in the env passing the right log directory made the command succeed:
[root@controller-0 heat-admin]# podman run --rm --log-driver=k8s-file --log-opt path=/var/log/containers/mysql/db-upgrade.log -u root --net=host -e "KOLLA_CONFIG_STRATEGY=CO
PY_ALWAYS" -v /etc/hosts:/etc/hosts:ro -v /etc/localtime:/etc/localtime:ro -v /etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro -v /etc/pki/ca-trust/source/anchors:
/etc/pki/ca-trust/source/anchors:ro -v /etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro -v /etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-b
undle.trust.crt:ro -v /etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro -v /dev/log:/dev/log -v /etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro -v /etc/puppet:/etc/puppet:
ro -v /var/lib/kolla/config_files/mysql.json:/var/lib/kolla/config_files/config.json:rw,z -v /var/lib/config-data/puppet-generated/mysql/:/var/lib/kolla/config_files/src:ro,
z -v /var/lib/mysql:/var/lib/mysql:rw,z -v /tmp/mariadb-upgrade:/var/log/mariadb:rw,z "192.168.24.1:8787/rh-osbs/rhosp15-openstack-mariadb:pcmklatest" /bin/bash -ecx "kolla_
set_configs; if mysqladmin ping --silent; then exit 0; fi; chown -R mysql:mysql /var/lib/mysql; chown -R mysql:mysql /var/log/mariadb; mysqld_safe --user=mysql --wsrep-provi
der=none --skip-networking --wsrep-on=off & timeout 60 sh -c 'while ! mysqladmin ping --silent; do sleep 1; done'; mysql_upgrade; mysqladmin shutdown"
+ kolla_set_configs
INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json
INFO:__main__:Validating config file
INFO:__main__:Kolla config strategy set to: COPY_ALWAYS
INFO:__main__:Copying service configuration files
INFO:__main__:Copying /dev/null to /etc/libqb/force-filesystem-sockets
INFO:__main__:Setting permission for /etc/libqb/force-filesystem-sockets
INFO:__main__:Deleting /etc/my.cnf.d/galera.cnf
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/my.cnf.d/galera.cnf to /etc/my.cnf.d/galera.cnf
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/sysconfig/clustercheck to /etc/sysconfig/clustercheck
INFO:__main__:Copying /var/lib/kolla/config_files/src/root/.my.cnf to /root/.my.cnf
INFO:__main__:Writing out command to execute
+ mysqladmin ping --silent
+ chown -R mysql:mysql /var/lib/mysql
+ chown -R mysql:mysql /var/log/mariadb
+ timeout 60 sh -c 'while ! mysqladmin ping --silent; do sleep 1; done'
+ mysqld_safe --user=mysql --wsrep-provider=none --skip-networking --wsrep-on=off
191119 15:22:30 mysqld_safe Logging to '/var/log/mariadb/mariadb.log'.
Fix proposed to branch: master
Review: https:/