Comment 7 for bug 1913472

We work together with the reported

Further to the above actions, we performed the following

1. The same container to the other controllers can be started/stop without any problem. The command is executed and returns without a problem ( the puppet configurations seems to be the same)

Correct execution (controller1)
puppet apply --verbose --detailed-exitcodes --summarize --color=false --modulepath /etc/puppet/modules:/opt/stack/puppet-modules:/usr/share/openstack-puppet/modules --tags file,file_line,concat,augeas,pacemaker::resource::bundle,pacemaker::property,pacemaker::resource::ocf,pacemaker::constraint::order,pacemaker::constraint::colocation,galera_ready,mysql_database,mysql_grant,mysql_user -e 'noop_resource('\''package'\''); include tripleo::profile::base::pacemaker;include tripleo::profile::pacemaker::database::mysql_bundle'
...
..

2021-02-06T14:00:30.782993941+00:00 stdout F Info: Loading facts
2021-02-06T14:00:30.783025041+00:00 stdout F Info: Loading facts
2021-02-06T14:00:30.783054016+00:00 stdout F Info: Loading facts
2021-02-06T14:00:38.312934917+00:00 stderr F Warning: Found multiple default providers for service: swiftinit, base, pacemaker, pacemaker_xml; using swiftinit
2021-02-06T14:00:39.492883048+00:00 stderr F Warning: /etc/puppet/hiera.yaml: Use of 'hiera.yaml' version 3 is deprecated. It should be converted to version 5
2021-02-06T14:00:39.492883048+00:00 stderr F (file: /etc/puppet/hiera.yaml)
2021-02-06T14:00:39.493806071+00:00 stderr F Warning: Undefined variable '::deploy_config_name';
2021-02-06T14:00:39.493806071+00:00 stderr F (file & line not available)
2021-02-06T14:00:39.524991159+00:00 stderr F Warning: The function 'hiera' is deprecated in favor of using 'lookup'. See https://puppet.com/docs/puppet/6.14/deprecated_language.html
2021-02-06T14:00:39.524991159+00:00 stderr F (file & line not available)
2021-02-06T14:00:39.818677831+00:00 stderr F Warning: This method is deprecated, please use match expressions with Stdlib::Compat::Array instead. They are described at https://docs.puppet.com/puppet/latest/refere
nce/lang_data_type.html#match-expressions. at ["/etc/puppet/modules/tripleo/manifests/profile/pacemaker/database/mysql_bundle.pp", 214]:["unknown", 1]
2021-02-06T14:00:39.818677831+00:00 stderr F (location: /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:34:in `deprecation')
2021-02-06T14:00:39.968930710+00:00 stdout F Notice: Compiled catalog for controller-1.example.com in environment production in 0.50 seconds
2021-02-06T14:00:40.043804586+00:00 stdout F Info: Applying configuration version '1612620039'
2021-02-06T14:00:40.160360183+00:00 stdout F Notice: Applied catalog in 0.12 seconds
2021-02-06T14:00:40.160892547+00:00 stdout F Changes:
2021-02-06T14:00:40.160892547+00:00 stdout F Events:
2021-02-06T14:00:40.160892547+00:00 stdout F Resources:
2021-02-06T14:00:40.160892547+00:00 stdout F Skipped: 24
2021-02-06T14:00:40.160892547+00:00 stdout F Total: 31
2021-02-06T14:00:40.160892547+00:00 stdout F Time:
2021-02-06T14:00:40.160892547+00:00 stdout F File line: 0.00
2021-02-06T14:00:40.160892547+00:00 stdout F File: 0.00
2021-02-06T14:00:40.160892547+00:00 stdout F Transaction evaluation: 0.12
2021-02-06T14:00:40.160892547+00:00 stdout F Catalog application: 0.12
2021-02-06T14:00:40.160892547+00:00 stdout F Config retrieval: 0.58
2021-02-06T14:00:40.160892547+00:00 stdout F Last run: 1612620040
2021-02-06T14:00:40.160892547+00:00 stdout F Total: 0.12
2021-02-06T14:00:40.160892547+00:00 stdout F Version:
2021-02-06T14:00:40.160892547+00:00 stdout F Config: 1612620039
2021-02-06T14:00:40.160892547+00:00 stdout F Puppet: 6.14.0
2021-02-06T14:00:40.267676500+00:00 stderr F + rc=0
2021-02-06T14:00:40.267676500+00:00 stderr F + set -e
2021-02-06T14:00:40.267709129+00:00 stderr F + set +ux

Hanging execution (controller0)

2021-02-06T13:50:25.177946438+00:00 stderr F (file & line not available)
2021-02-06T13:59:39.394178799+00:00 stderr F + STEP=2
2021-02-06T13:59:39.394178799+00:00 stderr F + TAGS=file,file_line,concat,augeas,pacemaker::resource::bundle,pacemaker::property,pacemaker::resource::ocf,pacemaker::constraint::order,pacemaker::constraint::coloca
tion,galera_ready,mysql_database,mysql_grant,mysql_user
2021-02-06T13:59:39.394178799+00:00 stderr F + CONFIG='include tripleo::profile::base::pacemaker;include tripleo::profile::pacemaker::database::mysql_bundle'
2021-02-06T13:59:39.394178799+00:00 stderr F + EXTRA_ARGS=
2021-02-06T13:59:39.394178799+00:00 stderr F + '[' -d /tmp/puppet-etc ']'
2021-02-06T13:59:39.394178799+00:00 stderr F + cp -a /tmp/puppet-etc/auth.conf /tmp/puppet-etc/devices /tmp/puppet-etc/hieradata /tmp/puppet-etc/hiera.yaml /tmp/puppet-etc/modules /tmp/puppet-etc/puppet.conf /tmp
/puppet-etc/ssl /etc/puppet
2021-02-06T13:59:39.401665280+00:00 stderr F + echo '{"step": 2}'
2021-02-06T13:59:39.401768058+00:00 stderr F + export FACTER_deployment_type=containers
2021-02-06T13:59:39.401768058+00:00 stderr F + FACTER_deployment_type=containers
2021-02-06T13:59:39.401779706+00:00 stderr F + set +e
2021-02-06T13:59:39.401870108+00:00 stderr F + puppet apply --verbose --detailed-exitcodes --summarize --color=false --modulepath /etc/puppet/modules:/opt/stack/puppet-modules:/usr/share/openstack-puppet/modules
--tags file,file_line,concat,augeas,pacemaker::resource::bundle,pacemaker::property,pacemaker::resource::ocf,pacemaker::constraint::order,pacemaker::constraint::colocation,galera_ready,mysql_database,mysql_grant,
mysql_user -e 'noop_resource('\''package'\''); include tripleo::profile::base::pacemaker;include tripleo::profile::pacemaker::database::mysql_bundle'
2021-02-06T13:59:46.163347081+00:00 stdout F Info: Loading facts
2021-02-06T13:59:46.163347081+00:00 stdout F Info: Loading facts
2021-02-06T13:59:46.163347081+00:00 stdout F Info: Loading facts
2021-02-06T13:59:46.163347081+00:00 stdout F Info: Loading facts
2021-02-06T13:59:46.168111744+00:00 stdout F Info: Loading facts
2021-02-06T13:59:55.263069075+00:00 stderr F Warning: /etc/puppet/hiera.yaml: Use of 'hiera.yaml' version 3 is deprecated. It should be converted to version 5
2021-02-06T13:59:55.263069075+00:00 stderr F (file: /etc/puppet/hiera.yaml)
2021-02-06T13:59:55.263069075+00:00 stderr F Warning: Undefined variable '::deploy_config_name';
2021-02-06T13:59:55.263069075+00:00 stderr F (file & line not available)
2021-02-06T13:59:55.275251332+00:00 stderr F Warning: The function 'hiera' is deprecated in favor of using 'lookup'. See https://puppet.com/docs/puppet/6.14/deprecated_language.html
2021-02-06T13:59:55.275251332+00:00 stderr F (file & line not available)

2. We deleted the container mysql_init_bundle and we deployed. The behaviour is the same as the container to hang and the above process to hang.

3. Mysql and its replication is working fine & pcs resource show all processes are OK.

4. The system is an active cloud with a number of VMs and without any problem in its functionality (which would have implied a db problem)

5. The system has been updated/extended 1-2 times the last 6 months with the addition of more nodes.