Host 'xx.xx.xx.xx' is not allowed to connect to this MySQL server

[Jason Hobbs]

in a DNS HA deploy, I got this error in my heat/1 unit:
http://paste.ubuntu.com/p/44wk4TJ7P8/

bundle and overlay:
http://paste.ubuntu.com/p/GQBxppGfHT/

juju status:
http://paste.ubuntu.com/p/xSck9GjQQt/

It looks like heat is trying to connect to mysql via mysql-internal.maas host, but it looks like mysql-internal.maas is resolving to the public space IP reather than the internal one.

MAAS logs show it bouncing around:
http://paste.ubuntu.com/p/sDtMd8VFhX/

I wonder if there is an issue with dnsHA when multiple spaces are being used?

[Ryan Beisner (1chb1n)]

Is there timestamp info available to accompany the crm record changes @ http://paste.ubuntu.com/p/sDtMd8VFhX ? This will help align the events if so.

[Jason Hobbs (jason-hobbs)]

http://paste.ubuntu.com/p/m8yg6ytQBF/

Also attaching the complete maas logs incase they're helpful.

[Ryan Beisner (1chb1n)]

First, confirming config:

 - os-access-hostname is set to mysql-internal.maas
 - dns-ha is set to True
 - vips are not set

Check. The config looks sane based on documented usage.

...

Here are all of the charm-driven maas dns events:

rbeisner@rby:~/Downloads/juju-crashdump-ef4ff605-0aa3-490c-ab3b-56b835ca0b19⟫ find -type f -name unit-* | xargs grep "ocf:maas:dns" | pastebinit -a beisner
http://paste.ubuntu.com/p/x5K8t8fDgG/

...

A converged timeline looks like this:

2018-03-17 12:52:46 mysql-0 says: mysql-internal.maas is set to 10.244.40.241

2018-03-17 12:52:49 maas says: ip 10.244.40.241 linked to resource mysql-internal on zone maas

2018-03-17 12:52:49 maas says: ip 192.168.33.130 unlinked from resource mysql-internal on zone maas

2018-03-17 12:58:01 heat-1 says: heat-internal.maas is set to 192.168.33.158

2018-03-17 12:58:03 heat-1 says: heat-public.maas is set to 10.244.40.235

2018-03-17 13:05:55 mysql-1 says: Grant does NOT exist for host '192.168.33.158' on db 'heat'

2018-03-17 13:06:00 mysql-1 says: Grant exists for host '192.168.33.158' on db 'heat'

2018-03-17 13:08:08 The heat-1 unit traces with DEBUG shared-db-relation-changed 2018-03-17 13:08:08.085 148514 ERROR oslo_db.sqlalchemy.exc_filters InternalError: (1130, u"Host '10.244.40.235' is not allowed to connect to this MySQL server")

...

The percona cluster units never check or set a db grant for a 10.244 address, which confirms the error as reported:

rbeisner@rby:~/Downloads/juju-crashdump-ef4ff605-0aa3-490c-ab3b-56b835ca0b19⟫ find -type f -name unit-*mysql* | xargs grep "Grant.*heat.*" | pastebinit -a beisner
http://paste.ubuntu.com/p/4WMRZ7zdx4/

[Ryan Beisner (1chb1n)]

Perhaps the get_allowed_units charm helper may need some love?

[David Ames (thedac)]

I think this is a configuration error.

 mysql:
    options:
      os-access-hostname: mysql-internal.maas
    bindings:
      "": *oam-space
      cluster: *internal-space
      shared-db: *internal-space

The way the DNSHA code works is it pulls the os-{}-hostname bit to determine which space to use, in this case access. The access space is not defined and therefore it uses the default oam-space.

Please test with the following for percona-cluster:

bindings:
      "": *oam-space
      cluster: *internal-space
      shared-db: *internal-space
      access: *internal-space # Or whatever you expect this to be.

When DNSHA checks for the access space it will get the correct address.

[Jason Hobbs (jason-hobbs)]

Ok - It doesn't make any sense for me that there is a separate binding
for dnsHA, but we'll make the suggested change - thanks.

On Tue, Mar 27, 2018 at 11:13 AM, David Ames <email address hidden> wrote:
> I think this is a configuration error.
>
> mysql:
> options:
> os-access-hostname: mysql-internal.maas
> bindings:
> "": *oam-space
> cluster: *internal-space
> shared-db: *internal-space
>
> The way the DNSHA code works is it pulls the os-{}-hostname bit to
> determine which space to use, in this case access. The access space is
> not defined and therefore it uses the default oam-space.
>
> Please test with the following for percona-cluster:
>
> bindings:
> "": *oam-space
> cluster: *internal-space
> shared-db: *internal-space
> access: *internal-space # Or whatever you expect this to be.
>
> When DNSHA checks for the access space it will get the correct address.
>
> ** Changed in: charm-heat
> Status: New => Invalid
>
> ** Changed in: charm-percona-cluster
> Status: New => Incomplete
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1756988
>
> Title:
> Host 'xx.xx.xx.xx' is not allowed to connect to this MySQL server
>
> Status in OpenStack heat charm:
> Invalid
> Status in OpenStack percona-cluster charm:
> Incomplete
>
> Bug description:
> in a DNS HA deploy, I got this error in my heat/1 unit:
> http://paste.ubuntu.com/p/44wk4TJ7P8/
>
> bundle and overlay:
> http://paste.ubuntu.com/p/GQBxppGfHT/
>
> juju status:
> http://paste.ubuntu.com/p/xSck9GjQQt/
>
> It looks like heat is trying to connect to mysql via mysql-
> internal.maas host, but it looks like mysql-internal.maas is resolving
> to the public space IP reather than the internal one.
>
> MAAS logs show it bouncing around:
> http://paste.ubuntu.com/p/sDtMd8VFhX/
>
> I wonder if there is an issue with dnsHA when multiple spaces are
> being used?
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/charm-heat/+bug/1756988/+subscriptions

[Jason Hobbs (jason-hobbs)]

The charm's documentation seems to suggest that shared-db is the correct binding for this:

"The 'cluster' endpoint binding is used to determine which network space units within the
percona-cluster deployment should use for communication with each other; the 'shared-db'
endpoint binding is used to determine which network space should be used for access to
MySQL databases services from other charms.
...
NOTE: Existing deployments using the access-network configuration option will continue to function; this option is preferred over any network space binding provided for the 'shared-db' relation if set."

Given "the 'shared-db' endpoint binding is used to determine which network space should be used for access to MySQL databases services from other charms" it seems like specifying the shared-db binding should be sufficient.

[Ryan Beisner (1chb1n)]

We're working on a doc update to clarify the use of the access binding in the dns-ha use case, as at least two engineers misinterpreted the usage. Thank you all for your effort and information.

[Jason Hobbs (jason-hobbs)]

When would it make sense for shared-db and access bindings to ever have different values? Why do they both exist?

[David Ames (thedac)]

Jason,

Network spaces can work on either relationship names *or* extra bindings. shared-db is the name of a relationship. Because shared-db is one of many possible relationships to percona-cluster (shared-db, db, db-admin) the extra binding, access, is used in the same way internal, admin, and public are used for OpenStack API charms. Because the code paths used for selecting the correct address for DNSHA, VIP or generic relationship may differ, it is recommend to set them explicitly.

The docs are in dire need of clarification on this point, so we have filed Bug#1759356 [0].

To be as clear as possible for this bug, when using DNSHA and network spaces, please set the access space explicitly.

[0] https://bugs.launchpad.net/charm-percona-cluster/+bug/1759356

[Jason Hobbs (jason-hobbs)]

I understand that you require access binding to be set. I did not see an answer to my question "When would it make sense for shared-db and access bindings to ever have different values?" It seems to me that 'access' should take its value from shared-db if 'access' isn't set.

[Ryan Beisner (1chb1n)]

The answer to your question is: "for OpenStack, it does not make sense to set them differently."

Multiple applications can use the percona-cluster charm, not just OpenStack. Multiple applications can consume the shared-db interface, not just OpenStack.

There is no precedent in our charms to assume or inherit a space binding if not explicitly declared. Yes, it could, but we don't think that it should.

[Jason Hobbs (jason-hobbs)]

So what applications does it make sense to have separate shared-db and access bindings for? If there aren't any, then it seems like there is no reason to have both of them, except for backwards compatability, and one could be deprecated.

[Ryan Beisner (1chb1n)]

We have documentation updates in flight to address the assumptions made in the feature usage, as described in this bug.

https://bugs.launchpad.net/charm-percona-cluster/+bug/1759356

Whether or not the feature as it exists today is agreed as too narrow or too broad an implementation (bind a separate space or not bind a separate space), is a separate design discussion.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/563708

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/18.02
Review: https://review.openstack.org/563718

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/563708
Committed: https://git.openstack.org/cgit/openstack/charm-percona-cluster/commit/?id=5f90ad4ca6983048b8acb268086a481853b16268
Submitter: Zuul
Branch: master

commit 5f90ad4ca6983048b8acb268086a481853b16268
Author: Ryan Beisner <email address hidden>
Date: Mon Apr 23 17:22:23 2018 +0000

    Clarify dns-ha and access binding usage as implemented

    Change-Id: I699611b529f3cddc9d76028219f913ec1391d6f5
    Partial-Bug: #1759356
    Related-Bug: #1756988

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/563718
Committed: https://git.openstack.org/cgit/openstack/charm-percona-cluster/commit/?id=14dd106ebf71b21f8453883326baa94a55cc5bf0
Submitter: Zuul
Branch: stable/18.02

commit 14dd106ebf71b21f8453883326baa94a55cc5bf0
Author: Ryan Beisner <email address hidden>
Date: Mon Apr 23 17:22:23 2018 +0000

    Clarify dns-ha and access binding usage as implemented

    Change-Id: I699611b529f3cddc9d76028219f913ec1391d6f5
    Partial-Bug: #1759356
    Related-Bug: #1756988