[9.x swarm] cluster deployment with detached haproxy fails due to the broken package "percona-xtrabackup"

[Sergey Novikov]

Detailed bug description:
the issue was found by https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.plugins.thread_separate_haproxy/153/testReport/(root)/separate_haproxy/separate_haproxy/

Steps to reproduce:
            1. install external-lb and detach-haproxy plugins
            2. Bootstrap 3 slaves from default nodegroup
            3. Create cluster with Neutron VXLAN and custom nodegroups
            4. Run 'fuel-mirror' to replace cluster repositories
               with local mirrors
            5. Bootstrap 2 slaves nodes from second nodegroup
               and one node from third node group
            6. Enable plugins for cluster
            7. Add 2 controllers from default nodegroup and 1 controller
               from second node group
            8. Add 1 compute+cinder from default node group
               and 1 compute+cinder from second node group
            9. Add haproxy node from third node group
            10. Verify networks
            11. Deploy cluster

Deployment fails on Task[primary-database]
part of puppet's log http://paste.openstack.org/show/592811/

Additional information:
snapshot #646

[Vladimir Kuklin (vkuklin)]

The log seems to show issues with repository connectivity to the repos. I would firstly wait for a reproduction of the issue.

[Nastya Urlapova (aurlapova)]

Vlamimir, the new fail you can find by link:
https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.plugins.thread_separate_haproxy/153/

>>> (/Stage[main]/Galera/Package[percona-xtrabackup]/ensure) change from purged to present failed: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold -o APT::Get::AllowUnauthenticated=1 install percona-xtrabackup' returned 100: Reading package lists...

[Nastya Urlapova (aurlapova)]

Version 9.2 snapshot #684

[Vladimir Kuklin (vkuklin)]

Okay, the issue is that the package is being installed before we install the client package that replaces the upstream mysql one, i. e. libmysqlclient18 or its replacement should be present before we try to install libdbd-mysql-perl

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/416587

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/417439

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/417440

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/416587
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=7583f5cb17ededd3eccebe12fb04025d8dbe9e6a
Submitter: Jenkins
Branch: master

commit 7583f5cb17ededd3eccebe12fb04025d8dbe9e6a
Author: Vladimir Kuklin <email address hidden>
Date: Wed Jan 4 18:27:58 2017 +0300

    Fix apt discrepancy around mysql client package

    When we run galera/mysql packages installation
    we can get into dependencies issues with mysql
    client package as it is not installed at the time
    when perl dbd module requires it. At the same time
    libperl-dbd-mysql is depending on it, so it fails.
    Instead we need to ensure that mysql client is installed
    along with its libraries prior to installing additional
    packages for galera.

    Change-Id: Idc91a399a46cd63c2201660bb248c0a0ad3fb9b8
    Closes-bug: #1651182

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/417439
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=abeaf1f50a8a93515aa5c6fcdc14b06ed46aa6b5
Submitter: Jenkins
Branch: stable/mitaka

commit abeaf1f50a8a93515aa5c6fcdc14b06ed46aa6b5
Author: Vladimir Kuklin <email address hidden>
Date: Wed Jan 4 18:27:58 2017 +0300

    Fix apt discrepancy around mysql client package

    When we run galera/mysql packages installation
    we can get into dependencies issues with mysql
    client package as it is not installed at the time
    when perl dbd module requires it. At the same time
    libperl-dbd-mysql is depending on it, so it fails.
    Instead we need to ensure that mysql client is installed
    along with its libraries prior to installing additional
    packages for galera.

    Change-Id: Idc91a399a46cd63c2201660bb248c0a0ad3fb9b8
    Closes-bug: #1651182

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/418990

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/417440
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=f9f7ec7c4cf9aea5c2888d76d4cba957c4527c22
Submitter: Jenkins
Branch: stable/newton

commit f9f7ec7c4cf9aea5c2888d76d4cba957c4527c22
Author: Vladimir Kuklin <email address hidden>
Date: Wed Jan 4 18:27:58 2017 +0300

    Fix apt discrepancy around mysql client package

    When we run galera/mysql packages installation
    we can get into dependencies issues with mysql
    client package as it is not installed at the time
    when perl dbd module requires it. At the same time
    libperl-dbd-mysql is depending on it, so it fails.
    Instead we need to ensure that mysql client is installed
    along with its libraries prior to installing additional
    packages for galera.

    Change-Id: Idc91a399a46cd63c2201660bb248c0a0ad3fb9b8
    Closes-bug: #1651182

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/418990
Committed: https://git.openstack.org/cgit/openstack/fuel-mirror/commit/?id=c0d6c904bbcb83e66c1d6ff5a3bdba6e9a407c7b
Submitter: Jenkins
Branch: stable/mitaka

commit c0d6c904bbcb83e66c1d6ff5a3bdba6e9a407c7b
Author: Vladimir Kuklin <email address hidden>
Date: Wed Jan 11 18:07:56 2017 +0300

    Add libmysqlclient18 to the partial mirror pkg list

    Otherwise it will not be mirrored and it will be marked
    as a virtual package by APT, for which there is no ability
    to do provides via mysql-wsrep-libmysqlclient18 package
    which will break mysql library installation dependencies

    Change-Id: I487032cfa2acc9e22c6b8df46fa5d11019debd3b
    Closes-bug: #1651182

[Ekaterina Shutova (eshutova)]

Problem still exists on latest snapshot #771:
https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.plugins.thread_separate_haproxy/176/testReport/(root)/separate_haproxy/separate_haproxy/

[Vladimir Kuklin (vkuklin)]

The issue itself cannot be resolved without changing the mirror creation workflow. We need to download libmysqlclient18 without actually using MOS repos at the first time. Then we need to create mirrors again with MOS mirrors enabled. Then the mirrors will be complete and include all the required packages.

Here is how it should look like to make things work

http://paste.openstack.org/show/595182/

[Vladimir Khlyunev (vkhlyunev)]

QA team can not do anything with missing package in repo - probably we should put it to 9.2 repos instead of creating new spike in tests

[Roman Vyalov (r0mikiam)]

moved to mos linux , because the bug related to build new packages to 9.2 (according to comment from QA)

[Dmitry Teselkin (teselkin-d)]

percona-xtrabackup / libdb-mysql-perl are upstream (Ubuntu) packages, we're not rebuilding them. There is no excuse to adding a new package into our repositories without introducing new functionality / fixing bugs in packages. This should be fixed on fuel-qa / fuel-build side, so moving to fuel-qa because of [1].

[1] https://bugs.launchpad.net/fuel/+bug/1651182/comments/16

[Vladimir Kuklin (vkuklin)]

guys, this bug requires thorough refactoring of how be build repositories. more precisely, we need to do it in a different way as I described in https://bugs.launchpad.net/fuel/+bug/1651182/comments/16

the mirror should be built with 2 passes of fuel-mirror script with different configuration options. Thus, this bug is a won't fix with release notes based on my comment

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/fuel-library 11.0.0.0rc1 release candidate.

[Denis Meltsaykin (dmeltsaykin)]

It's very sad that nobody considered a proper fix for this. After short investigation we've found out that the problem is apparently might be properly solved, so I'm going to re-open it for 9.2-mu-1.

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/30000
Submitter: Pkgs Jenkins <email address hidden>
Branch: 9.0

Commit: 949a6e053f9ce3c75aa81ff10c944629e85898b7
Author: Ivan Suzdal <email address hidden>
Date: Tue Mar 21 13:20:56 2017

Add libmysqlclient18 metapackage

Related-Bug: #1651182

Change-Id: I221beafa7d2760b919a676f70dbcce0e6c370c18

[Fuel Devops McRobotson (fuel-devops-robot)]

Related fix proposed to branch: mcp/1.0/mitaka
Change author: Ivan Suzdal <email address hidden>
Review: https://review.fuel-infra.org/32408

[TatyanaGladysheva (tgladysheva)]

Verified on 9.2 mu1 updates.

Latest run with fix is passed:
https://patching-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.plugins.thread_separate_haproxy/8/testReport/(root)/separate_haproxy/separate_haproxy/

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/35867
Submitter: Pkgs Jenkins <email address hidden>
Branch: 8.0

Commit: 7f7d79e0816961acec585dbfb7f0c4b1f2a089af
Author: Ivan Suzdal <email address hidden>
Date: Fri Jul 14 10:06:26 2017

MySQL client metapackage and break condition

This patch introduces two changes:
 - added the break condition statement for mysql-server-wsrep-core-5.6
   which is needed to allow easy upgrades from pre 5.6 mysql-wsrep
 - added the libmysqlclient18 metapackage which is needed to avoid
   conflicts during installation of 3rd-party packages which depend on
   the real libmysqlclient18 package and cannot be satisfied with the
   virtual one

Closes-Bug: #1651182
Change-Id: Id3794f47ec6d45cdaeaba9943cb89e9eb22c202c