tripleo-ci-centos-8-standalone-on-multinode-ipa is broken due to mariadb 10.5

Bug #1943330 reported by Michele Baldessari
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
Critical
Unassigned

Bug Description

For example see in https://review.opendev.org/c/openstack/tripleo-heat-templates/+/808205:
https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_945/808205/2/check/tripleo-ci-centos-8-standalone-on-multinode-ipa/945f4e4/logs/undercloud/home/zuul/standalone_deploy.log

2021-09-11 09:15:45.302763 | 2816a99b-cf19-4286-bae2-ac7aa21190bf | INCLUDED | /usr/share/ansible/roles/tripleo_container_manage/tasks/create.yml | standalone-0
2021-09-11 09:15:45.338976 | fa163e72-98ca-be9e-c24d-000000001e80 | TASK | Create containers managed by Podman for /var/lib/tripleo-config/container-startup-config/step_2
2021-09-11 09:16:05.857732 | | WARNING | ERROR: Can't run container mysql_wait_bundle
stderr: + STEP=2
+ TAGS=file,file_line,concat,augeas,galera_ready,mysql_database,mysql_grant,mysql_user
+ CONFIG='include tripleo::profile::pacemaker::database::mysql_bundle'
+ EXTRA_ARGS=
+ '[' -d /tmp/puppet-etc ']'
+ cp -a /tmp/puppet-etc/devices /tmp/puppet-etc/hieradata /tmp/puppet-etc/hiera.yaml /tmp/puppet-etc/modules /tmp/puppet-etc/puppet.conf /tmp/puppet-etc/ssl /etc/puppet
+ echo '{"step": 2}'
+ export FACTER_deployment_type=containers
+ FACTER_deployment_type=containers
+ set +e
+ puppet apply --verbose --detailed-exitcodes --summarize --color=false --modulepath /etc/puppet/modules:/opt/stack/puppet-modules:/usr/share/openstack-puppet/modules --tags file,file_line,concat,augeas,galera_ready,mysql_database,mysql_grant,mysql_user -e 'noop_resource('\''package'\''); include tripleo::profile::pacemaker::database::mysql_bundle'
Warning: /etc/puppet/hiera.yaml: Use of 'hiera.yaml' version 3 is deprecated. It should be converted to version 5
   (file: /etc/puppet/hiera.yaml)
Warning: Undefined variable '::deploy_config_name';
   (file & line not available)
Warning: The function 'hiera' is deprecated in favor of using 'lookup'. See https://puppet.com/docs/puppet/7.8/deprecated_language.html
   (file & line not available)
Warning: Unknown variable: '::pacemaker::pcs_010'. (file: /etc/puppet/modules/pacemaker/manifests/resource/bundle.pp, line: 168, column: 19)
Warning: Unknown variable: 'used_promoted_max'. (file: /etc/puppet/modules/pacemaker/manifests/resource/bundle.pp, line: 185, column: 27)
Error: Execution of '/usr/bin/mysql --defaults-extra-file=/root/.my.cnf --database=mysql -e GRANT ALL PRIVILEGES ON `nova_cell0`.* TO 'nova'@'overcloud.ctlplane.ooo.test'' returned 1: ERROR 1133 (28000) at line 1: Can't find any matching row in the user table
Error: /Stage[main]/Nova::Db::Mysql/Openstacklib::Db::Mysql[nova_cell0]/Openstacklib::Db::Mysql::Host_access[nova_cell0_overcloud.ctlplane.ooo.test]/Mysql_grant[<email address hidden>/nova_cell0.*]/ensure: change from 'absent' to 'present' failed: Execution of '/usr/bin/mysql --defaults-extra-file=/root/.my.cnf --database=mysql -e GRANT ALL PRIVILEGES ON `nova_cell0`.* TO 'nova'@'overcloud.ctlplane.ooo.test'' returned 1: ERROR 1133 (28000) at line 1: Can't find any matching row in the user table
+ rc=6
+ set -e
+ set +ux

Reason seems to be a new behavior when creating users with dns hosts associated to them and our default setting of 'skip-name-resolve' => '1' here:
https://github.com/openstack/puppet-tripleo/blob/master/manifests/profile/pacemaker/database/mysql_bundle.pp#L320

Revision history for this message
Michele Baldessari (michele) wrote :

Need to disccuss with mbayer and damien if we go for https://review.opendev.org/c/openstack/puppet-tripleo/+/808085 to fix this or if we should revert https://review.opendev.org/c/openstack/tripleo-common/+/808044 in the meantime as things might take longer

Revision history for this message
Ronelle Landy (rlandy) wrote :
Changed in tripleo:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (master)

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/808745
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/f2015da4b50345359a9a14e403361ce7b0c7fcb1
Submitter: "Zuul (22348)"
Branch: master

commit f2015da4b50345359a9a14e403361ce7b0c7fcb1
Author: Damien Ciabrini <email address hidden>
Date: Mon Sep 13 13:21:51 2021 +0200

    Simplify mysql users creation

    Openstack users are configured with openstacklib, which in turns
    drive puppet-mysql to create several DB user for each db service:
    <service>@'%' <service>@<ip> and <service>@<mysql_vip>.

    We create several users because we use two different parameters
    host and allowed_hosts in openstacklib, which only has the effect
    of creating a list of users per openstack service.

    However since we always create a user '%', this wildcard host
    will always allow connection to the DB, so the other users are
    currently not useful as they don't get any additional grants or
    restrictions.

    Simplify the entire mysql user creation to only generate one
    user per service, with a wildcard host.

    Change-Id: I928b03f06c702a13f4bd957eaa79153aa711cee4
    Closes-Bug: #1943440
    Closes-Bug: #1943330

Changed in tripleo:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (stable/wallaby)

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/809310

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-heat-templates (master)

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/810556

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-heat-templates (stable/wallaby)

Related fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/810748

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-heat-templates (master)

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/810556
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/7fc781083ff091957086699278238375c2e1d626
Submitter: "Zuul (22348)"
Branch: master

commit 7fc781083ff091957086699278238375c2e1d626
Author: Damien Ciabrini <email address hidden>
Date: Thu Sep 23 09:12:50 2021 +0200

    Only delete heat data in the DB if it exists

    The undercloud upgrade script that deletes users may
    not work if re-run or if other scripts clean DB data
    during the upgrade.
    Only delete users and database if it still exists in
    the DB.

    Change-Id: I36e3b33a824f3ae6d1da97ea53527c2b4519ea3c
    Related-Bug: #1943440
    Related-Bug: #1943330

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-heat-templates (stable/wallaby)

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/810748
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/f58e759e18919838ca8f413b0768be15294e7999
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit f58e759e18919838ca8f413b0768be15294e7999
Author: Damien Ciabrini <email address hidden>
Date: Thu Sep 23 09:12:50 2021 +0200

    Only delete heat data in the DB if it exists

    The undercloud upgrade script that deletes users may
    not work if re-run or if other scripts clean DB data
    during the upgrade.
    Only delete users and database if it still exists in
    the DB.

    Change-Id: I36e3b33a824f3ae6d1da97ea53527c2b4519ea3c
    Related-Bug: #1943440
    Related-Bug: #1943330

tags: added: in-stable-wallaby
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (stable/wallaby)

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/809310
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/c4b9f958358717802a4d9ddaf6eb89fe86ff7173
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit c4b9f958358717802a4d9ddaf6eb89fe86ff7173
Author: Damien Ciabrini <email address hidden>
Date: Mon Sep 13 13:21:51 2021 +0200

    Simplify mysql users creation

    Openstack users are configured with openstacklib, which in turns
    drive puppet-mysql to create several DB user for each db service:
    <service>@'%' <service>@<ip> and <service>@<mysql_vip>.

    We create several users because we use two different parameters
    host and allowed_hosts in openstacklib, which only has the effect
    of creating a list of users per openstack service.

    However since we always create a user '%', this wildcard host
    will always allow connection to the DB, so the other users are
    currently not useful as they don't get any additional grants or
    restrictions.

    Simplify the entire mysql user creation to only generate one
    user per service, with a wildcard host.

    Slight context conflicts in:
      deployment/neutron/neutron-api-container-puppet.yaml

    Change-Id: I928b03f06c702a13f4bd957eaa79153aa711cee4
    Closes-Bug: #1943440
    Closes-Bug: #1943330
    (cherry picked from commit f2015da4b50345359a9a14e403361ce7b0c7fcb1)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates 15.1.0

This issue was fixed in the openstack/tripleo-heat-templates 15.1.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.