mysql_wait_bundle timing out in upgrade jobs

[wes hayutin]

635.99s
2021-07-09 09:52:23 | 2021-07-09 09:52:23.594089 | bc764e10-226f-46f2-db0c-000000001eda | TASK | Check containers status
2021-07-09 09:58:09 | [ERROR]: Container(s) which failed to be created by podman_container module:
2021-07-09 09:58:09 | ['mysql_wait_bundle']
2021-07-09 09:58:09 | [ERROR]: Container(s) which did not finish after 300 minutes:
2021-07-09 09:58:09 | ['mysql_wait_bundle']
2021-07-09 09:58:09 | 2021-07-09 09:58:09.648032 | bc764e10-226f-46f2-db0c-000000001eda | FATAL | Check containers status | standalone | error={"changed": false, "msg": "Failed container(s): ['mysql_wait_bundle'], check logs in /var/log/containers/stdouts/"}

https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_8a5/periodic/opendev.org/openstack/puppet-tripleo/stable/victoria/tripleo-ci-centos-8-standalone-upgrade-victoria/8a533b7/logs/undercloud/home/zuul/standalone_upgrade.log

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22mysql_wait_bundle%5C%22%20AND%20tags%3Aconsole%20AND%20voting%3A1%20AND%20build_status%3AFAILURE

https://zuul.openstack.org/builds?job_name=tripleo-ci-centos-8-standalone-upgrade-victoria&job_name=tripleo-ci-centos-8-standalone-upgrade-ussuri
https://zuul.openstack.org/builds?job_name=tripleo-ci-centos-8-standalone-upgrade-victoria

==== HANGS HERE =========
https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_8a5/periodic/opendev.org/openstack/puppet-tripleo/stable/victoria/tripleo-ci-centos-8-standalone-upgrade-victoria/8a533b7/logs/undercloud/var/log/containers/stdouts/mysql_wait_bundle.log

2021-07-09T09:42:26.182432817+00:00 stdout F Notice: /Stage[main]/Mysql::Server::Config/File[mysql-config-file]/content: content changed '{md5}315d95d4a36cd04cdea7be4a039e9b97' to '{md5}eba374e9c0581cb106dd3378809228ce'
2021-07-09T09:42:26.183063855+00:00 stdout F Info: Class[Mysql::Server::Config]: Unscheduling all events on Class[Mysql::Server::Config]
2021-07-09T09:42:26.481294500+00:00 stdout F Notice: /Stage[main]/Mysql::Server::Installdb/File[/var/log/mariadb/mariadb.log]/ensure: created
2021-07-09T09:42:26.482560731+00:00 stdout F Info: Class[Mysql::Server::Installdb]: Unscheduling all events on Class[Mysql::Server::Installdb]

[wes hayutin (weshayutin)]

package diff
http://paste.openstack.org/show/807338/

last promotion results: which passed.. so something came in via tripleo current:
http://paste.openstack.org/show/807341/

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-ci/+/800301

[Alex Schultz (alex-schultz)]

I was unable to reproduce this doing a manually ussuri install + victoria upgrade. Seems like it might be CI specific

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-ci/+/800301
Committed: https://opendev.org/openstack/tripleo-ci/commit/4d7b37b41187ede7650c35239f453042c757dda6
Submitter: "Zuul (22348)"
Branch: master

commit 4d7b37b41187ede7650c35239f453042c757dda6
Author: Wesley Hayutin <email address hidden>
Date: Fri Jul 9 13:08:04 2021 -0600

    standalone-upgrade ussuri/victoria nv

    check -> nv
    remove gate

    Related-Bug: #1935691
    Change-Id: I5b97ccf09fac490bc9c4f7fc3f4ceb59f7ff6e7a

[yatin (yatinkarel)]

<< I was unable to reproduce this doing a manually ussuri install + victoria upgrade. Seems like it might be CI specific

Seems will only reproduce when using different output directory during initial deploy and upgrade, or with missing password file during upgrade to trigger password generation.

Proposed Fix:- https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/800437, tested at https://review.rdoproject.org/r/c/testproject/+/34484

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/800437
Committed: https://opendev.org/openstack/tripleo-quickstart-extras/commit/bf092d64813c79012ceba1840bec6275ce055e5e
Submitter: "Zuul (22348)"
Branch: master

commit bf092d64813c79012ceba1840bec6275ce055e5e
Author: yatinkarel <email address hidden>
Date: Mon Jul 12 13:42:24 2021 +0530

    [Standalone Upgrade] Use output directory same as deploy

    Standalone deploy already switched to <working_dir>/tripleo-deploy
    in [1], for same reasons switch the standalone upgrade too.

    Also when using different output directory, passwords
    get's regenerated during upgrade and causing issues
    like #1935691.

    [1] https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/789764

    Closes-Bug: #1935691
    Change-Id: If6c1892d9f3d4613163dca2a0f7f293bf076a7b6

[chandan kumar (chkumar246)]

Based on this https://zuul.openstack.org/builds?job_name=tripleo-ci-centos-8-standalone-upgrade-victoria the job looks green,