[2.3.8] Failed to select hostPort by space

[Felipe Reyes]

[Impact]

We have a juju controller in HA (3 replicas) where jujud is selecting 127.0.0.1:37017 as shown in the following log line:

2019-01-29 13:06:38 DEBUG juju.worker.peergrouper machinetracker.go:105 machine "0" selected address "127.0.0.1:37017" by space "oam" from [X.X.X.0:37017 Y.Y.Y.101:37017 Z.Z.Z.101:37017 127.0.0.1:37017]

This is leading to the following collision:

2019-01-29 15:59:26 ERROR juju.worker.peergrouper worker.go:211 cannot set replicaset: Found two member configurations with same host field, members.0.host == members.1.host == 127.0.0.1:37017

[Other Info]

This environment was upgraded to 2.3.8 to fix this problem, because it was found that https://github.com/juju/juju/pull/8066 landed in 2.3

[Felipe Reyes (freyes)]

I'm compiling a set of logs, they should be ready to download soon.

[Felipe Reyes (freyes)]

I uploaded the machine logs from the controllers, juju status, juju show-controller, and rs.status()

https://private-fileshare.canonical.com/~freyes/lp1813953/

We believe this problem may be related with the subnet where the controllers are.

[Felipe Reyes (freyes)]

SelectPeerHostPortBySpace() should implement "allowMachineLocal = false" as SelectPeerHostPort() did with the PR https://github.com/juju/juju/pull/8066

https://github.com/juju/juju/blob/2.3/mongo/mongo.go#L286

[Joseph Phillips (manadart)]

Can you supply also the output of the "juju spaces"?

The logic around the space used for Mongo peer communication and peer group election changed substantially from 2.3 to 2.4, but not as far as filtering machine-local addresses when selecting from a space.

I'm just wondering how/why the loop-back addresses are in that "oam" space.

[John A Meinel (jameinel)]

Note this entry in the log file:
2019-01-29 10:56:27 ERROR juju.network address.go:316 no hostPorts found in spaces [oam]
2019-01-29 10:56:27 DEBUG juju.mongo mongo.go:293 Failed to select hostPort by space - trying by scope from [X.X.X.X:37017 Y.Y.Y.Y:37017 Z.Z.Z.Z:37017 127.0.0.1:37017]

That indicates that none of the addresses are flagged as being in the right space. It means that we *tried* to look up by space name:
// SelectHostsPortBySpaces picks the first HostPort from the given slice that has
// the given space name associated.
func SelectHostsPortBySpaces(hps []HostPort, spaceNames ...SpaceName) ([]HostPort, bool) {
        if len(spaceNames) == 0 {
                logger.Errorf("host ports not filtered - no spaces given.")
                return hps, false
        }

        var selectedHostPorts []HostPort
        for _, hp := range hps {
                if spaceNameList(spaceNames).IndexOf(hp.SpaceName) >= 0 {
                        logger.Debugf("selected %q as a hostPort in space %q", hp.Value, hp.SpaceName)
                        selectedHostPorts = append(selectedHostPorts, hp)
                }
        }

        if len(selectedHostPorts) > 0 {
                return selectedHostPorts, true
        }

        logger.Errorf("no hostPorts found in spaces %s", spaceNames)
        return hps, false
}

but none of the HostPort entries were flagged with SpaceName = "oam".

So while it is true that we could just never return 127., the real problem is that we don't have an address with the space that you asked us to use.

[John A Meinel (jameinel)]

All of them *are* in "*107*" and "*70*" and "*86*", so those should be possible values. And, in fact, they are currently configured to use the "*107*" addresses. (rs_addr=)

It sounds like the problem is either:

a) We're not properly flagging HostPorts with the right SpaceNames
b) You asked for a space that the existing HostPorts aren't in.

Note that the code that is calling this function is doing:
// SelectPeerHostPortBySpace returns the HostPort to use as the mongo replica set peer
// by selecting it from the given hostPorts.
func SelectPeerHostPortBySpace(hostPorts []network.HostPort, space network.SpaceName) string {
        // ScopeMachineLocal addresses are OK if we can't pick by space.
        suitableHostPorts, foundHostPortsInSpaces := network.SelectMongoHostPortsBySpaces(hostPorts, []network.SpaceName{space})

        if !foundHostPortsInSpaces {
                logger.Debugf("Failed to select hostPort by space - trying by scope from %+v", hostPorts)
                suitableHostPorts = network.SelectMongoHostPortsByScope(hostPorts, true)
        }
        return suitableHostPorts[0]
}

Since we didn't find "oam" by name, we fell back to selecting them by Scope, and *allowing* machineLocal addresses. (I believe we allow local addresses to handle the case where we are not in a replicaset, so we just need to connect to the mongo that is on this machine.

Note that this should all be fixed in 2.4 as we changed it to require setting a space for controllers to be in, and we validate that the spaces are available.

I believe that the bug is probably (a), but that we've fixed that in newer releases.

Is there a specific reason why they can't go to 2.4?

[Tilman Baumann (tilmanbaumann)]

I also first suspected an actual problem with the spaces. And it still might in some way.

But as you can see from the juju machines output, theoretically the spaces are correct.

  "76":
    juju-status:
      current: started
      since: 29 Nov 2018 22:53:35Z
      version: 2.3.8
    dns-name: 100.107.3.117
    ip-addresses:
    - 100.107.3.117
    - 100.70.0.137
    - 100.86.0.167
    instance-id: 6txmyg
    machine-status:
      current: running
      message: Deployed
      since: 23 Nov 2017 16:56:09Z
    series: xenial
    network-interfaces:
      bondA:
        ip-addresses:
        - 100.86.0.167
        mac-address: 3c:fd:fe:9d:02:08
        space: clo
        is-up: true
      bondB:
        ip-addresses:
        - 100.70.0.137
        mac-address: 3c:fd:fe:9d:02:09
        space: ovr
        is-up: true
      bondM:
        ip-addresses:
        - 100.107.3.117
        mac-address: 94:57:a5:5b:2d:50
        gateway: 100.107.3.254
        space: oam
        is-up: true
    constraints: tags=juju
    hardware: arch=amd64 cores=48 mem=262136M tags=management,FC1,cmp,juju availability-zone=FC1
    controller-member-status: adding-vote

[Tilman Baumann (tilmanbaumann)]

Upgrade to 2.4 is certainly an option. If we know the bug is gone there.

The whole reason we upgraded to 2.3.8 in the first place, was pretty much exactly the same problem class.
API addresses for peers were selected incorrectly. But in that version, the mongo addresses were used right. In 2017 we had the same problem with the replica addresses before.

Scepticism is high. :D

[Joseph Phillips (manadart)]

It is behaving as if there was a network topology change.

In this version of Juju, we have the auto-determination of the Mongo space, which has settled on "oam". This doesn't change after it is is set.

Now, no addresses on the controller machines are found in the space, and falling back to selection by scope and allowing machine-local addresses causes such addresses to be used over cloud-local and public ones.

It doesn't appear that we are in a broken state, as the replica-set is healthy - we're just looping trying to apply a bad config, which is failing.

I'd still like to see the "juju spaces" output.

[Tilman Baumann (tilmanbaumann)]

$ juju spaces
Space Subnets
cbe 100.88.197.0/24
       100.88.199.0/24
       100.88.200.0/24
       100.88.201.0/24
clo 100.84.1.0/24
       100.84.197.0/24
       100.84.199.0/24
       100.84.2.0/24
       100.84.200.0/24
       100.84.201.0/24
       100.86.0.0/20
help
ipmi 100.100.1.0/24
      100.100.197.0/24
      100.100.199.0/24
      100.100.2.0/24
      100.100.200.0/24
      100.100.201.0/24
      100.102.1.0/24
      100.102.197.0/24
      100.102.199.0/24
      100.102.2.0/24
      100.102.200.0/24
      100.102.3.0/24
      100.106.231.0/24
      100.106.232.0/24
      100.106.233.0/24
oam 100.101.1.0/24
      100.101.197.0/24
      100.101.199.0/24
      100.101.2.0/24
      100.101.200.0/24
      100.107.0.0/22
      100.107.128.0/22
      100.107.64.0/22
      100.99.1.0/24
      100.99.197.0/24
      100.99.199.0/24
      100.99.2.0/24
      100.99.200.0/24
      100.99.201.0/24
ovr 100.68.1.0/24
      100.68.2.0/24
      100.70.0.0/20

[John A Meinel (jameinel)]

right 100.107.0.0/22 should be the oam addresses that Juju is seeing, so
we aren't labeling the HostPorts correctly to match that subnet.

On Thu, Jan 31, 2019 at 6:01 PM Tilman Baumann <email address hidden>
wrote:

> $ juju spaces
> Space Subnets
> cbe 100.88.197.0/24
> 100.88.199.0/24
> 100.88.200.0/24
> 100.88.201.0/24
> clo 100.84.1.0/24
> 100.84.197.0/24
> 100.84.199.0/24
> 100.84.2.0/24
> 100.84.200.0/24
> 100.84.201.0/24
> 100.86.0.0/20
> help
> ipmi 100.100.1.0/24
> 100.100.197.0/24
> 100.100.199.0/24
> 100.100.2.0/24
> 100.100.200.0/24
> 100.100.201.0/24
> 100.102.1.0/24
> 100.102.197.0/24
> 100.102.199.0/24
> 100.102.2.0/24
> 100.102.200.0/24
> 100.102.3.0/24
> 100.106.231.0/24
> 100.106.232.0/24
> 100.106.233.0/24
> oam 100.101.1.0/24
> 100.101.197.0/24
> 100.101.199.0/24
> 100.101.2.0/24
> 100.101.200.0/24
> 100.107.0.0/22
> 100.107.128.0/22
> 100.107.64.0/22
> 100.99.1.0/24
> 100.99.197.0/24
> 100.99.199.0/24
> 100.99.2.0/24
> 100.99.200.0/24
> 100.99.201.0/24
> ovr 100.68.1.0/24
> 100.68.2.0/24
> 100.70.0.0/20
>
> --
> You received this bug notification because you are subscribed to juju.
> Matching subscriptions: juju bugs
> https://bugs.launchpad.net/bugs/1813953
>
> Title:
> [2.3.8] Failed to select hostPort by space
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju/+bug/1813953/+subscriptions
>

[Joseph Phillips (manadart)]

I think I understand what is happening here.

Juju should not be selecting a machine-local address for peer group usage, as you correctly state. But we should not be falling through to that choice at all, because the machines have addresses in subnets that are on the "oam" space.

I believe the reason they are not chosen by space is due to a bug that we fixed in this patch:
https://github.com/juju/juju/pull/8283

That bug was to do with sourcing machine addresses directly from the machine, instead of through the provider (MAAS) which is space aware; thus the addresses are not decorated with the information required to select them by space.

This could be confirmed by looking directly at the documents for these machines in the MongoDB "machines" collection. If the members of "addresses" don't have a "spacename", then it is definitely the issue.

The best option is to move forward to the latest Juju 2.4 version. This has fixes particular to this area, namely:

- The space for Mongo peer group communication can be user set (juju-mongo-space) and is in fact mandatory if more than one address on the machine is usable for such a purpose.
- There is no longer a fall-back to to choosing a scoped (machine-local) address if a Mongo space is set.
- MAAS machines get addresses via the MAAS provider, which decorates them appropriately with space information so they can be filtered properly.

[Felipe Reyes (freyes)]

On Thu, 2019-01-31 at 15:42 +0000, Joseph Phillips wrote:
> I think I understand what is happening here.
>
> Juju should not be selecting a machine-local address for peer group
> usage, as you correctly state. But we should not be falling through
> to
> that choice at all, because the machines have addresses in subnets
> that
> are on the "oam" space.
>
> I believe the reason they are not chosen by space is due to a bug
> that we fixed in this patch:
> https://github.com/juju/juju/pull/8283
>
> That bug was to do with sourcing machine addresses directly from the
> machine, instead of through the provider (MAAS) which is space aware;
> thus the addresses are not decorated with the information required to
> select them by space.
>
> This could be confirmed by looking directly at the documents for
> these
> machines in the MongoDB "machines" collection. If the members of
> "addresses" don't have a "spacename", then it is definitely the
> issue.
>
> The best option is to move forward to the latest Juju 2.4 version.
Maybe we could add that information manually, so we get the controllers
up-n-running and then upgrade.

> This
> has fixes particular to this area, namely:
>
> - The space for Mongo peer group communication can be user set (juju-
> mongo-space) and is in fact mandatory if more than one address on the
> machine is usable for such a purpose.
> - There is no longer a fall-back to to choosing a scoped (machine-
> local) address if a Mongo space is set.
> - MAAS machines get addresses via the MAAS provider, which decorates
> them appropriately with space information so they can be filtered
> properly.
>
--
Felipe Reyes
Software Sustaining Engineer @ Canonical
# Email: <email address hidden> (GPG:0x9B1FFF39)
# Launchpad: ~freyes | IRC: freyes

[Joseph Phillips (manadart)]

I tested against MAAS 2.5 with Juju 2.3-latest and edge.

With 2.3, I got this in my machine doc:

        "addresses" : [
                {
                        "value" : "10.0.0.170",
                        "addresstype" : "ipv4",
                        "networkscope" : "local-cloud",
                        "origin" : "provider",
                }
        ],

With the new version I get:

        "addresses" : [
                {
                        "value" : "10.0.0.170",
                        "addresstype" : "ipv4",
                        "networkscope" : "local-cloud",
                        "origin" : "provider",
                        "spacename" : "default"
                }
        ],

[Tilman Baumann (tilmanbaumann)]

As discussed in irc
3:22 PM <tilmanb> manadart: bad controller https://pastebin.canonical.com/p/hG4PRGtsgk/
3:22 PM <tilmanb> manadart: good controller https://pastebin.canonical.com/p/FN9NjYPG7P/

The controller with identical setup and version does not have the mongo-space-name field set.

[Tilman Baumann (tilmanbaumann)]

As discussed in irc
3:22 PM <tilmanb> manadart: bad controller https://pastebin.canonical.com/p/hG4PRGtsgk/
3:22 PM <tilmanb> manadart: good controller https://pastebin.canonical.com/p/FN9NjYPG7P/

The controller with identical setup and version does not have the mongo-space-name field set.

As a short term workaround we set the mongo settings to the same values as on the working one.
juju:PRIMARY> db.controllers.update(
... {"_id" : "e"},
... { $set: {
... "mongo-space-name" : "",
... "mongo-space-state" : "invalid"
... }
... });
WriteResult({ "nMatched" : 1, "nUpserted" : 0, "nModified" : 1 })

This has worked. The IP selection is still not great, for example, localhost is included. But it works a lot better in practice.

Long term we are looking at upgrading...

Thanks for your help.