percona cluster getting wrong private ip

[Narinder Gupta]

Percona cluster charm from master branch is not getting correct private ip when using spaces.

internal CIDR: 192.0.5.0/24
public CIDR: 172.21.159.0/24

wsrep_node_address=172.21.159.137

Below are the logs requested.

root@juju-5f683e-1-lxd-2:/var/log/juju# ls
machine-1-lxd-2.log unit-mysql-0.log
root@juju-5f683e-1-lxd-2:/var/log/juju# pastebinit unit-mysql-0.log
http://paste.ubuntu.com/23819314/
root@juju-5f683e-1-lxd-2:/var/log/juju# pastebinit machine-1-lxd-2.log
http://paste.ubuntu.com/23819315/
root@juju-5f683e-1-lxd-2:/var/log/juju#
root@juju-5f683e-1-lxd-2:/var/log/juju#
root@juju-5f683e-1-lxd-2:/var/log/juju# cd ../mysql
root@juju-5f683e-1-lxd-2:/var/log/mysql# ls
error.log
root@juju-5f683e-1-lxd-2:/var/log/mysql# pastebinit error.log
http://paste.ubuntu.com/23819317/
root@juju-5f683e-1-lxd-2:/var/log/mysql# cd /etc/mysql/
root@juju-5f683e-1-lxd-2:/etc/mysql# ls
conf.d debian.cnf my.cnf my.cnf.fallback percona-xtradb-cluster percona-xtradb-cluster.cnf percona-xtradb-cluster.conf.d
root@juju-5f683e-1-lxd-2:/etc/mysql# cd percona-xtradb-cluster.conf.d/
root@juju-5f683e-1-lxd-2:/etc/mysql/percona-xtradb-cluster.conf.d# ls
client.cnf isamchk.cnf mysqld.cnf mysqld_safe.cnf
root@juju-5f683e-1-lxd-2:/etc/mysql/percona-xtradb-cluster.conf.d# pastebinit mysqld.cnf
http://paste.ubuntu.com/23819320/

[David Ames (thedac)]

According to John Meinel:

'The charms should be updated to use "network-get <bindname> --preferred-address" instead of just "unit-get private-address". unit-get doesn't pass the information to Juju for us to know which bit of the configuration we're supposed to be reporting.'

In the OpenStack charms most of the time we use charmhelpers.contrib.openstack.ip resolve_address() which does the right thing checking network-get first.

However, we have some isolated instances where unit_get(private-address) is defaulted to. Which according to John can be unpredictable.

Specifically, calls to get_host_ip() functions that return the private-address by default in nova-compute and percona-cluster. Leading to unpredictable IPs in configuration files.

We need to also check other charms for this issue.

Nova compute HostIPContext returns the private-address:
class HostIPContext(context.OSContextGenerator):
    def __call__(self):
        ctxt = {}
        if config('prefer-ipv6'):
            host_ip = get_ipv6_addr()[0]
        else:
            host_ip = get_host_ip(unit_get('private-address'))

        if host_ip:
            # NOTE: do not format this even for ipv6 (see bug 1499656)
            ctxt['host_ip'] = host_ip

        return ctxt

percona-cluster in render_config returns the private-address:
    context = {
        'cluster_name': 'juju_cluster',
        'private_address': get_host_ip(),

[David Ames (thedac)]

According to John Meinel:
'The charms should be updated to use "network-get <bindname>
--preferred-address" instead of just "unit-get private-address".
unit-get doesn't pass the information to Juju for us to know which bit
of the configuration we're supposed to be reporting.'

I would make the argument that private-address *should* be
predictable. It *should* be the PXE boot IP if not configurable.
As expressed in this bug:
https://bugs.launchpad.net/juju/+bug/1591962

These two bugs express the same issue. They say they are fix-released
but the fixes only deal with the symptom not the underlying problem.
https://bugs.launchpad.net/juju/+bug/1616098/
https://bugs.launchpad.net/juju/+bug/1603473/

[Narinder Gupta (narindergupta)]

here is the comment from juju team John Meinel:

There is nothing to say that the PXE address is better than the other address for any given *user's* deployment. If a charm isn't updated to support network-get, even if we did make private-address stable, there would be no way to put the application on the second interface when you really wanted it. And if a charm is updated to support network-get, then the stability of private-address is not as important.
FWIW, I believe we actually default private-address to the first address that MAAS returns. I haven't found any particular ordering to their value, as while it seems to be stable per node, it is not stable between nodes. (If I configure 2 nodes with similar network devices, named the same, in similar address ranges, one Node has a 172.* address first, and the second node has a 10.* address first, etc.)

I'm not against making private-address more stable, but I do feel like it is completely papering over the real issue, which is being able to allow a user to specify where they want the application deployed by making the charm ask Juju where it has been configured to run.

John

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Larry Michel (lmic)]

I was able to get the percona-cluster to fetch the correct private-address by using access-network option to specify the correct network where I expected this private-address to be on.

Note that I also was using the access binding to specify space associated with that subnet so not sure if both binding and access-network are needed. However, the access binding by itself did not work.

For example:

With space oil set mapped to 192.0.5.0/24

  percona-cluster:
    charm: cs:trusty/percona-cluster
    num_units: 1
    options:
      sst-password: root
      root-password: root
      max-connections: 1500
      access-network: 192.0.5.0/24
      source: cloud:trusty-mitaka
    bindings:
      access: oil
    to:
    - lxc:3

[James Page (james-page)]

Larry - for percona-cluster, you'll need to provide a binding for the shared-db relation to oil as well; no need to use the access-network configuration option (infact that prevents the use of juju network space binding in the charm).

[James Page (james-page)]

I appreciate the need to move charms to the network-get --primary-address charm hook tool to support use of network spaces, but I still feel that there might be a 'undefined behaviour' use case in this code path.

If a user does not bind a relation or extra-binding, then what gets returned from network-get --primary-address <relation-name>? Does the unbound, default, behaviour mimic what unit-get private-address does?

[James Page (james-page)]

In the scope of the percona-charm, we see two problems:

a) the network space binding for the cluster relation is not being used for the local wsrep address for mysql (although the cluster relation space binding is used to build the list of hosts participating in the cluster). This might not actually impact function

b) for the db and db-admin relations, network space binding is not supported - however this is relatively easy to switch based on the fact that the shared-db relation already dtrt.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/424911
Committed: https://git.openstack.org/cgit/openstack/charm-percona-cluster/commit/?id=881591871513489a67733d593cdb0721d41b5012
Submitter: Jenkins
Branch: master

commit 881591871513489a67733d593cdb0721d41b5012
Author: David Ames <email address hidden>
Date: Tue Jan 24 16:14:23 2017 -0800

    Fix support for network-spaces

    Fix misc issues with use of Juju 2.0 network spaces:

     - Ensure that wsrep address for local unit is correctly
       set using the binding for the cluster peer relation.

     - Correctly set the DB access hostname for db and db-admin
       relation types.

    This includes a little refactoring to support reuse within
    the charm.

    Closes-Bug: 1657305

    Change-Id: Id1a800e2ada6fd196422b003fd8e251cab5ad724

[Anastasia (anastasia-macmood)]

Based on Ryan's assessment and marking of this as 'Invalid' for juju (ubuntu), I am marking this as 'Invalid' for "juju" project too.

[Liam Young (gnuoy)]

I'm going to mark this is invalid against nova-compute as nova-compute does not have a relation with percona anymore (Icehouse+ I believe).