Longer term fix for VIP failover issue and clash with cellsv2

[Michele Baldessari]

With bug https://bugs.launchpad.net/tripleo/+bug/1643487 we fixed the VIP failover speed issue by adding an explicit ?bind_address=<ip> in all DB URIs. This approach clashes with the cellsv2 nova feature which stores the URI in a central DB. Since the URIs with the bind_address parameter are host-dependent, things started to break. In review https://review.openstack.org/430183 we stopped using the bind_address in nova, but by doing so we are regressing in the VIP failover for all nova db connections again.

This bug is to track the work needed to address the original VIP failover issue without impacting the nova cellsv2 work. Setting explicitely the source address to bind to is still mandatory to ensure that socket won't bind to a bad src address and exhibit the failover speed issue.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/431424

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/431425

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/431424
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=f6116ff0f350aeecdaa346e4e49d208be49ce6b9
Submitter: Jenkins
Branch: master

commit f6116ff0f350aeecdaa346e4e49d208be49ce6b9
Author: Michele Baldessari <email address hidden>
Date: Thu Feb 9 10:53:06 2017 +0100

    Create /etc/my.cnf.d/tripleo.cnf with proper bind-address

    When fixing LP#1643487 we added ?bind_address to all DB URIs.
    Since this clashes with Cellsv2 due to the URIs becoming host
    dependent, we need a new approach to pass bind_address to pymysql
    that leaves the DB URIs host-independent.

    We first create a /etc/my.cnf.d/tripleo.cnf file with a [tripleo]
    section and in this section we add the correct bind-address option.
    Note that we use the puppet augeas lens and not the mysql one
    because the mysql one does not support custom sections *and* there
    are older versions around which do not like the /etc/my.cnf.d/* path.

    The reason for not reusing an existing mariadb file (my.cnf or
    galera.cnf) is that pymysql's ini file support is not robust
    enough at the moment: https://github.com/PyMySQL/PyMySQL/issues/548

    The reason for putting this file creation code only on the controller
    nodes the following: The slow VIP failover only happens if a
    service runs where the VIPs exist. The VIPs get created in the
    haproxy profile and that is why in order to have fast VIP failovers
    the MySQLClient profile must live where the Haproxy service is running.

    Co-Authored-By: Damien Ciabrini <email address hidden>

    Partial-Bug: #1663181
    Change-Id: Iff8bd2d9ee85f7bb1445aa2e1b3cfbff1f397b18

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/431425
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=90431683b5927abb066d7964d513828b5488001c
Submitter: Jenkins
Branch: master

commit 90431683b5927abb066d7964d513828b5488001c
Author: Michele Baldessari <email address hidden>
Date: Thu Feb 9 11:14:03 2017 +0100

    Make the DB URIs host-independent for all services

    When fixing LP#1643487 we added ?bind_address to all DB URIs.
    Since this clashes with Cellsv2 due to the URIs becoming host
    dependent, we need a new approach to pass bind_address to pymysql
    that leaves the DB URIs host-independent.

    In change Iff8bd2d9ee85f7bb1445aa2e1b3cfbff1f397b18 we first create a
    /etc/my.cnf.d/tripleo.cnf file with a [tripleo] section with the correct
    bind-address option.

    In this change we make sure that the DB URIs will point to the added
    file and to the specific section containing the necessary bind-address
    option. We do introduce a new MySQLClient profile which will hold all
    this more client-specific configuration so that this change can fit
    better in the composable roles work. Also, in the future it might
    contain the necessary configuration for SSL for example.

    Note that in case the /etc/my.cnf.d/tripleo.cnf file does not exist
    (because it is created via the mysqlclient profile), things keep on
    working as usual and the bind-address option simply won't be set, which
    has no impact on hosts where there are no VIPs.

    Co-Authored-By: Damien Ciabrini <email address hidden>

    Change-Id: Ieac33efe38f32e949fd89545eb1cd8e0fe114a12
    Related-Bug: #1643487
    Closes-Bug: #1663181
    Closes-Bug: #1664524
    Depends-On: Iff8bd2d9ee85f7bb1445aa2e1b3cfbff1f397b18

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/436144

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/436192

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/436144
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=28853dba9d3d1f5f7fcbca58963489789641ae04
Submitter: Jenkins
Branch: stable/ocata

commit 28853dba9d3d1f5f7fcbca58963489789641ae04
Author: Michele Baldessari <email address hidden>
Date: Thu Feb 9 10:53:06 2017 +0100

    Create /etc/my.cnf.d/tripleo.cnf with proper bind-address

    When fixing LP#1643487 we added ?bind_address to all DB URIs.
    Since this clashes with Cellsv2 due to the URIs becoming host
    dependent, we need a new approach to pass bind_address to pymysql
    that leaves the DB URIs host-independent.

    We first create a /etc/my.cnf.d/tripleo.cnf file with a [tripleo]
    section and in this section we add the correct bind-address option.
    Note that we use the puppet augeas lens and not the mysql one
    because the mysql one does not support custom sections *and* there
    are older versions around which do not like the /etc/my.cnf.d/* path.

    The reason for not reusing an existing mariadb file (my.cnf or
    galera.cnf) is that pymysql's ini file support is not robust
    enough at the moment: https://github.com/PyMySQL/PyMySQL/issues/548

    The reason for putting this file creation code only on the controller
    nodes the following: The slow VIP failover only happens if a
    service runs where the VIPs exist. The VIPs get created in the
    haproxy profile and that is why in order to have fast VIP failovers
    the MySQLClient profile must live where the Haproxy service is running.

    Co-Authored-By: Damien Ciabrini <email address hidden>

    Partial-Bug: #1663181
    Change-Id: Iff8bd2d9ee85f7bb1445aa2e1b3cfbff1f397b18
    (cherry picked from commit f6116ff0f350aeecdaa346e4e49d208be49ce6b9)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/436192
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=3a7ed4403fc5f53e88210577e5e912a1057574db
Submitter: Jenkins
Branch: stable/ocata

commit 3a7ed4403fc5f53e88210577e5e912a1057574db
Author: Michele Baldessari <email address hidden>
Date: Thu Feb 9 11:14:03 2017 +0100

    Make the DB URIs host-independent for all services

    When fixing LP#1643487 we added ?bind_address to all DB URIs.
    Since this clashes with Cellsv2 due to the URIs becoming host
    dependent, we need a new approach to pass bind_address to pymysql
    that leaves the DB URIs host-independent.

    In change Iff8bd2d9ee85f7bb1445aa2e1b3cfbff1f397b18 we first create a
    /etc/my.cnf.d/tripleo.cnf file with a [tripleo] section with the correct
    bind-address option.

    In this change we make sure that the DB URIs will point to the added
    file and to the specific section containing the necessary bind-address
    option. We do introduce a new MySQLClient profile which will hold all
    this more client-specific configuration so that this change can fit
    better in the composable roles work. Also, in the future it might
    contain the necessary configuration for SSL for example.

    Note that in case the /etc/my.cnf.d/tripleo.cnf file does not exist
    (because it is created via the mysqlclient profile), things keep on
    working as usual and the bind-address option simply won't be set, which
    has no impact on hosts where there are no VIPs.

    Co-Authored-By: Damien Ciabrini <email address hidden>

    Change-Id: Ieac33efe38f32e949fd89545eb1cd8e0fe114a12
    Related-Bug: #1643487
    Closes-Bug: #1663181
    Closes-Bug: #1664524
    Depends-On: Iff8bd2d9ee85f7bb1445aa2e1b3cfbff1f397b18
    (cherry picked from commit 90431683b5927abb066d7964d513828b5488001c)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 6.0.0.0rc2 release candidate.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/447453

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 7.0.0.0b1 development milestone.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/447453
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=b555dc3ee570dc766da4d25be4d4bc55807141e4
Submitter: Jenkins
Branch: stable/newton

commit b555dc3ee570dc766da4d25be4d4bc55807141e4
Author: Michele Baldessari <email address hidden>
Date: Thu Feb 9 10:53:06 2017 +0100

    Create /etc/my.cnf.d/tripleo.cnf with proper bind-address

    When fixing LP#1643487 we added ?bind_address to all DB URIs.
    Since this clashes with Cellsv2 due to the URIs becoming host
    dependent, we need a new approach to pass bind_address to pymysql
    that leaves the DB URIs host-independent.

    We first create a /etc/my.cnf.d/tripleo.cnf file with a [tripleo]
    section and in this section we add the correct bind-address option.
    Note that we use the puppet augeas lens and not the mysql one
    because the mysql one does not support custom sections *and* there
    are older versions around which do not like the /etc/my.cnf.d/* path.

    The reason for not reusing an existing mariadb file (my.cnf or
    galera.cnf) is that pymysql's ini file support is not robust
    enough at the moment: https://github.com/PyMySQL/PyMySQL/issues/548

    The reason for putting this file creation code only on the controller
    nodes the following: The slow VIP failover only happens if a
    service runs where the VIPs exist. The VIPs get created in the
    haproxy profile and that is why in order to have fast VIP failovers
    the MySQLClient profile must live where the Haproxy service is running.

    Co-Authored-By: Damien Ciabrini <email address hidden>

    Partial-Bug: #1663181
    Change-Id: Iff8bd2d9ee85f7bb1445aa2e1b3cfbff1f397b18
    (cherry picked from commit f6116ff0f350aeecdaa346e4e49d208be49ce6b9)