os-cinder-install.yml fails with conflicting mariadb packages on centos infra hosts

Bug #1742206 reported by Gary Richards
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack-Ansible
Fix Released
Medium
Markos Chandras

Bug Description

During the run of os-conder-install.yml I get the following error during task 'galera_client : Install galera distro packages'.

It's trying to install packages on one of my infra nodes but fails with this message:
fatal: [infra1]: FAILED! => {
    "attempts": 5,
    "changed": true,
    "failed": true,
    "invocation": {
        "module_args": {
            "conf_file": null,
            "disable_gpg_check": false,
            "disablerepo": null,
            "enablerepo": null,
            "exclude": null,
            "install_repoquery": true,
            "installroot": "/",
            "list": null,
            "name": [
                "bzip2-devel",
                "gcc",
                "gnupg2",
                "libaio",
                "libstdc++",
                "libgcc",
                "libgcrypt",
                "MariaDB-client",
                "MariaDB-devel",
                "MariaDB-shared",
                "openssl-devel",
                "python-devel"
            ],
            "skip_broken": false,
            "state": "latest",
            "update_cache": false,
            "validate_certs": true
        }
    },
    "msg": "\n\nTransaction check error:\n file /etc/my.cnf from install of MariaDB-common-10.1.30-1.el7.centos.x86_64 conflicts with file from package mariadb-config-3:10.1.20-2.el7.x86_64\n file /usr/lib64/mysql/plugin/dialog.so from install of MariaDB-common-10.1.30-1.el7.centos.x86_64 conflicts with file from package mariadb-common-3:10.1.20-2.el7.x86_64\n file /usr/lib64/mysql/plugin/mysql_clear_password.so from install of MariaDB-common-10.1.30-1.el7.centos.x86_64 conflicts with file from package mariadb-common-3:10.1.20-2.el7.x86_64\n\nError Summary\n-------------\n\n",
    "rc": 1,
    "results": [
        "All packages providing gcc are up to date",
        "All packages providing gnupg2 are up to date",
        "All packages providing libaio are up to date",
        "All packages providing libstdc++ are up to date",
        "All packages providing libgcc are up to date",
        "All packages providing libgcrypt are up to date",
        "All packages providing openssl-devel are up to date",
        "All packages providing python-devel are up to date",
        "Loaded plugins: fastestmirror, priorities\nLoading mirror speeds from cached hostfile\n * base: mirror.vorboss.net\n * epel: www.mirrorservice.org\n * extras: mirror.vorboss.net\n * updates: mirror.ox.ac.uk\n588 packages excluded due to repository priority protections\nResolving Dependencies\n--> Running transaction check\n---> Package MariaDB-client.x86_64 0:10.1.30-1.el7.centos will be installed\n--> Processing Dependency: MariaDB-common for package: MariaDB-client-10.1.30-1.el7.centos.x86_64\n---> Package MariaDB-devel.x86_64 0:10.1.30-1.el7.centos will be installed\n---> Package MariaDB-shared.x86_64 0:10.1.30-1.el7.centos will be obsoleting\n---> Package bzip2-devel.x86_64 0:1.0.6-13.el7 will be installed\n---> Package mariadb-libs.x86_64 3:10.1.20-2.el7 will be obsoleted\n--> Running transaction check\n---> Package MariaDB-common.x86_64 0:10.1.30-1.el7.centos will be installed\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nInstalling:\n MariaDB-client x86_64 10.1.30-1.el7.centos MariaDB 39 M\n MariaDB-devel x86_64 10.1.30-1.el7.centos MariaDB 6.6 M\n MariaDB-shared x86_64 10.1.30-1.el7.centos MariaDB 1.3 M\n replacing mariadb-libs.x86_64 3:10.1.20-2.el7\n bzip2-devel x86_64 1.0.6-13.el7 base 218 k\nInstalling for dependencies:\n MariaDB-common x86_64 10.1.30-1.el7.centos MariaDB 123 k\n\nTransaction Summary\n================================================================================\nInstall 4 Packages (+1 Dependent package)\n\nTotal size: 48 M\nDownloading packages:\nRunning transaction check\nRunning transaction test\n"
    ]
}

If I try to yum install one of those packages I see:
  ================================================================================
   Package Arch Version Repository Size
  ================================================================================
  Installing:
   MariaDB-client x86_64 10.1.30-1.el7.centos MariaDB 39 M
  Installing for dependencies:
   MariaDB-common x86_64 10.1.30-1.el7.centos MariaDB 123 k

So it's trying to install those from the MariaDB repository

The packages that the new packages are conflicting with are from the @openstack-pike repository

Gary Richards (ashak)
description: updated
Revision history for this message
Markos Chandras (hwoarang) wrote :

Could you please tell us which repositories you have installed on your host? Could you also show us the result of the 'rpm -qi mariadb-common' command? As far as I know mariadb-* is not provided by the default centos7 repos so you probably have another repo installed that pulls mariadb-* in and that conflicts with the repo configured by the galera_server role

Revision history for this message
Gary Richards (ashak) wrote :

$ yum repolist
Loaded plugins: fastestmirror, priorities
Loading mirror speeds from cached hostfile
 * base: mirror.mhd.uk.as44574.net
 * epel: www.mirrorservice.org
 * extras: repo.uk.bigstepcloud.com
 * updates: repo.uk.bigstepcloud.com
588 packages excluded due to repository priority protections
repo id repo name status
!MariaDB MariaDB Repo 13+3
!base/7/x86_64 CentOS-7 - Base 9,591
!epel/x86_64 Extra Packages for Enterprise Linux 7 - x86_64 11,613+585
!extras/7/x86_64 CentOS-7 - Extras 329
!openstack-pike/x86_64 openstack-pike 2,197
!rdo-qemu-ev rdo-qemu-ev 43
!thm-lxc2.0 COPR repository for LXC 2.0 packages on CentOS 8+1
!updates/7/x86_64 CentOS-7 - Updates 1,651

$ rpm -qi mariadb-common
Name : mariadb-common
Epoch : 3
Version : 10.1.20
Release : 2.el7
Architecture: x86_64
Install Date: Sat 06 Jan 2018 08:05:36 GMT
Group : Applications/Databases
Size : 246872
License : GPLv2 with exceptions and LGPLv2 and BSD
Signature : RSA/SHA1, Wed 25 Oct 2017 12:59:29 BST, Key ID f9b9fee7764429e6
Source RPM : mariadb-10.1.20-2.el7.src.rpm
Build Date : Wed 04 Oct 2017 12:41:58 BST
Build Host : c1bd.rdu2.centos.org
Relocations : (not relocatable)
Packager : CBS <email address hidden>
Vendor : CentOS
URL : http://mariadb.org
Summary : The shared files required by server and client
Description :
The package provides the essential shared files for any MariaDB program.
You will need to install this package to use any other MariaDB package.

These were freshly installed CentOS 7 hosts as of a week ago as a minimal install. I would expect any repo beyond the CentOS ones to have been installed by OSA.

Just for completion, this is failing on one of my infra nodes as part of running os-cinder-install.yml in which it's trying to install MariaDB-client. The galera containers built fine.

Revision history for this message
Gary Richards (ashak) wrote :

I did a little more digging into this.

Trying to work out where the installed package comes from and why:

$ repoquery --whatrequires --installed mariadb-config
mariadb-common-3:10.1.20-2.el7.x86_64
mariadb-config-3:10.1.20-2.el7.x86_64
$ repoquery --whatrequires --installed mariadb-common
mariadb-libs-3:10.1.20-2.el7.x86_64
$ repoquery --whatrequires --installed mariadb-libs
mariadb-libs-3:10.1.20-2.el7.x86_64
postfix-2:2.10.1-6.el7.x86_64

$ yum deplist postfix | grep -i mysql
  dependency: libmysqlclient.so.18()(64bit)
  dependency: libmysqlclient.so.18(libmysqlclient_18)(64bit)
But if I look at the full output:
  dependency: libmysqlclient.so.18()(64bit)
   provider: mariadb-libs.x86_64 3:10.1.20-2.el7
   provider: MariaDB-shared.x86_64 10.1.30-1.el7.centos
  dependency: libmysqlclient.so.18(libmysqlclient_18)(64bit)
   provider: mariadb-libs.x86_64 3:10.1.20-2.el7
   provider: MariaDB-shared.x86_64 10.1.30-1.el7.centos

So I think postfix is the reason mariadb packages from the @openstack-pike repository are installed. However, now that other playbooks have also setup the MariaDB repo, it seems happy that they would be capable of providing the dependency

grepping for postfix in various places turns up the ansible-hardening role as the obvious place that's installing postfix, which in turn is causing the mariadb packages to be installed because the MariaDB repo isn't setup at that point.

The package it's trying to install is happening in galera_client role.

vars/redhat-7.yml sets up vars for:
_galera_client_repo_url and galera_client_distro_packages

the repo url ends up being set to the default because galera_client_yum_repo_url isn't set anywhere (and comments suggest it's for backwards compatibility in Newton), so its ends up set to http://yum.mariadb.org/......

the packages contain the MariaDB-client package (and a couple of others).

So this looks sensible as far as I can tell and would be totally fine if installation of postfix wasn't installing the conflicting packages earlier.

What I can't tell is whether there's something stupid i'm doing that's causing my infra nodes to end up applying the galera_client role as part of os-cinder-install.yml when really it shouldn't or whether this is a genuine problem.

Changed in openstack-ansible:
status: New → Confirmed
importance: Undecided → Medium
assignee: nobody → Markos Chandras (hwoarang)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to openstack-ansible-galera_server (master)

Fix proposed to branch: master
Review: https://review.openstack.org/534819

Changed in openstack-ansible:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to openstack-ansible-galera_server (master)

Reviewed: https://review.openstack.org/534819
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-galera_server/commit/?id=72057200ffe740cfd15cf9914540568491901b06
Submitter: Zuul
Branch: master

commit 72057200ffe740cfd15cf9914540568491901b06
Author: Markos Chandras <email address hidden>
Date: Wed Jan 17 11:59:19 2018 +0000

    RedHat: Ensure that conflicting MariaDB packages are removed

    When the galera_server role is deployed along with other roles, some
    of them may pull mariadb packages as a dependency for the services
    they deploy. However, this role uses MariaDB packages from the upstream
    repository which conflict with those provided by the EPEL or the RDO
    ones so we need to ensure that they are gone before we install the
    upstream ones.

    Change-Id: Ie800c1f4628eff24954f392fe6318206adfabddb
    Closes-Bug: #1742206

Changed in openstack-ansible:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/openstack-ansible-galera_server 17.0.0.0b3

This issue was fixed in the openstack/openstack-ansible-galera_server 17.0.0.0b3 development milestone.

Revision history for this message
Vadim Kuznetsov (vakuznet) wrote :

Including mariadb-server into "Conflicting" packages list will not get the desired result. Here is yum history, which shows that MariaDB-server-10.1.30-1.el7.centos.x86_64 from MariaDB repo was removed.

Loaded plugins: fastestmirror, priorities, versionlock
Transaction ID : 38
Begin time : Sat Apr 7 04:46:45 2018
Begin rpmdb : 264:a3268878e18af20cb5b09e0ada48d54a2b455233
End time : 04:46:50 2018 (5 seconds)
End rpmdb : 259:fbcc22a1e3504686ec90a568283ab6fd0d41e528
User : root <root>
Return-Code : Success
Command Line : -d 2 -y remove mariadb-server mariadb-libs percona-xtrabackup
Transaction performed with:
    Installed rpm-4.11.3-25.el7.x86_64 @base
    Installed yum-3.4.3-154.el7.centos.1.noarch @updates
    Installed yum-plugin-fastestmirror-1.1.31-42.el7.noarch @base
Packages Altered:
    Erase MariaDB-server-10.1.30-1.el7.centos.x86_64 @MariaDB
    Erase MariaDB-shared-10.1.30-1.el7.centos.x86_64 @MariaDB
    Erase percona-toolkit-3.0.8-1.el7.x86_64 @percona-release-x86_64
    Erase percona-xtrabackup-2.3.10-1.el7.x86_64 @percona-release-x86_64
    Erase perl-DBD-MySQL-4.023-5.el7.x86_64 @base

which led to reinstall MariaDB-server. However server start was not triggered. So node died.

Revision history for this message
Vadim Kuznetsov (vakuznet) wrote :

This bug is about galera_client. galera_server was fixed.

Revision history for this message
Markos Chandras (hwoarang) wrote :

Sorry this bug was closed a while ago. If there are still problems with this please open a new one providing all the necessary information (branches affected, steps to reproduce etc).

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to openstack-ansible-galera_server (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/578844

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to openstack-ansible-galera_server (master)

Reviewed: https://review.openstack.org/578844
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-galera_server/commit/?id=270eccd82aedf240e5bcc52876b226808f104bba
Submitter: Zuul
Branch: master

commit 270eccd82aedf240e5bcc52876b226808f104bba
Author: Vadim Kuznetsov <email address hidden>
Date: Thu Jun 28 11:18:49 2018 -0400

    Do not remove MariaDB-server

    Removing mariadb-server and mariadb-libs leads to
    removing MariaDB-server and MariaDB-shared and its
    dependencies
    rpmquery --whatprovides mariadb-server
    MariaDB-server-10.1.30-1.el7.centos.x86_64
    rpmquery --whatprovides mariadb-libs
    MariaDB-shared-10.1.30-1.el7.centos.x86_64

    Which in turn shuts down the server.

    Change-Id: I89df8f6b57b1ebe0ce96d63f193da35107d2bad9
    Closes-Bug: #1762421
    Related-Bug: #1742206

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to openstack-ansible-galera_server (stable/queens)

Related fix proposed to branch: stable/queens
Review: https://review.openstack.org/589973

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to openstack-ansible-galera_server (stable/queens)

Reviewed: https://review.openstack.org/589973
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-galera_server/commit/?id=2ca18bb2761f9fe99ea5eb29cba8e345966daf05
Submitter: Zuul
Branch: stable/queens

commit 2ca18bb2761f9fe99ea5eb29cba8e345966daf05
Author: Vadim Kuznetsov <email address hidden>
Date: Thu Jun 28 11:18:49 2018 -0400

    Do not remove MariaDB-server

    Removing mariadb-server and mariadb-libs leads to
    removing MariaDB-server and MariaDB-shared and its
    dependencies
    rpmquery --whatprovides mariadb-server
    MariaDB-server-10.1.30-1.el7.centos.x86_64
    rpmquery --whatprovides mariadb-libs
    MariaDB-shared-10.1.30-1.el7.centos.x86_64

    Which in turn shuts down the server.

    Change-Id: I89df8f6b57b1ebe0ce96d63f193da35107d2bad9
    Closes-Bug: #1762421
    Related-Bug: #1742206
    (cherry picked from commit 270eccd82aedf240e5bcc52876b226808f104bba)

tags: added: in-stable-queens
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to openstack-ansible-galera_server (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/603215

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to openstack-ansible-galera_server (master)

Reviewed: https://review.openstack.org/603215
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-galera_server/commit/?id=72f402d3e969c94754ade02953d2c94ce541b25e
Submitter: Zuul
Branch: master

commit 72f402d3e969c94754ade02953d2c94ce541b25e
Author: Jesse Pretorius <email address hidden>
Date: Mon Sep 17 20:25:02 2018 +0100

    Revisit conflicting package removal

    In https://review.openstack.org/534819 we introduced the removal
    of conflicting packages from the targeted host, but then we deleted
    the list for RedHat in https://review.openstack.org/578844 because
    yum kept removing and re-adding the same packages.

    In https://review.openstack.org/603205 we solve the issue properly
    given that yum is case-insensitive, and the root cause of the repeat
    remove/install.

    As such, in this patch, we restore the removal of conflicting
    packages for RedHat in a different way. Each of the package removal
    tasks are moved into the tasks specific to each package manager so
    that each can be handled differently.

    Change-Id: I70fbfa6eff8796713c6bec32319382273f8281f8
    Related-Bug: #1762421
    Related-Bug: #1742206

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to openstack-ansible-galera_server (stable/rocky)

Related fix proposed to branch: stable/rocky
Review: https://review.openstack.org/603383

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to openstack-ansible-galera_server (stable/queens)

Related fix proposed to branch: stable/queens
Review: https://review.openstack.org/603388

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to openstack-ansible-galera_server (stable/queens)

Reviewed: https://review.openstack.org/603388
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-galera_server/commit/?id=aff86072af40e4895a3f87a1f99aa66cb54a9290
Submitter: Zuul
Branch: stable/queens

commit aff86072af40e4895a3f87a1f99aa66cb54a9290
Author: Jesse Pretorius <email address hidden>
Date: Mon Sep 17 20:25:02 2018 +0100

    Revisit conflicting package removal

    In https://review.openstack.org/534819 we introduced the removal
    of conflicting packages from the targeted host, but then we deleted
    the list for RedHat in https://review.openstack.org/578844 because
    yum kept removing and re-adding the same packages.

    In https://review.openstack.org/603205 we solve the issue properly
    given that yum is case-insensitive, and the root cause of the repeat
    remove/install.

    As such, in this patch, we restore the removal of conflicting
    packages for RedHat in a different way. Each of the package removal
    tasks are moved into the tasks specific to each package manager so
    that each can be handled differently.

    Change-Id: I70fbfa6eff8796713c6bec32319382273f8281f8
    Related-Bug: #1762421
    Related-Bug: #1742206
    (cherry picked from commit 72f402d3e969c94754ade02953d2c94ce541b25e)

tags: added: in-stable-rocky
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to openstack-ansible-galera_server (stable/rocky)

Reviewed: https://review.openstack.org/603383
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-galera_server/commit/?id=943b0360a7c85e157d0c24abe47215462395ae16
Submitter: Zuul
Branch: stable/rocky

commit 943b0360a7c85e157d0c24abe47215462395ae16
Author: Jesse Pretorius <email address hidden>
Date: Mon Sep 17 20:25:02 2018 +0100

    Revisit conflicting package removal

    In https://review.openstack.org/534819 we introduced the removal
    of conflicting packages from the targeted host, but then we deleted
    the list for RedHat in https://review.openstack.org/578844 because
    yum kept removing and re-adding the same packages.

    In https://review.openstack.org/603205 we solve the issue properly
    given that yum is case-insensitive, and the root cause of the repeat
    remove/install.

    As such, in this patch, we restore the removal of conflicting
    packages for RedHat in a different way. Each of the package removal
    tasks are moved into the tasks specific to each package manager so
    that each can be handled differently.

    Change-Id: I70fbfa6eff8796713c6bec32319382273f8281f8
    Related-Bug: #1762421
    Related-Bug: #1742206
    (cherry picked from commit 72f402d3e969c94754ade02953d2c94ce541b25e)

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.