maas allocated two usable addresses for juju enable-ha

[John George]

When calling 'juju enable-ha' one of the added machines is reported to have 'multiple usable addresses ...'.
Juju is version 2.4.3
MAAS is version 2.4.2

The solutions QA test record is here:
https://solutions.qa.canonical.com/#/qa/testRun/283c67d9-dfb4-481e-a14e-3324ddc0b9a1

'2': {'machine-status': {'current': 'running', 'message': 'Deployed', 'since': '19 Sep 2018 19:54:08Z'}, 'ip-addresses': ['10.244.40.200', '10.244.40.201'], 'juju-status': {'current': 'started', 'message': 'multiple usable addresses found\nrun "juju config juju-ha-space=<name>" to set a space for Mongo peer communication'

Looking at maas.log, indeed it appears two addresses where allocated:
jenkins@juju-c811b4-prod-cdo-oil-8:~/tmp-triage/283c67d9-dfb4-481e-a14e-3324ddc0b9a1/maas/home/ubuntu/project/generated/maas/logs-2018-09-19-20.02.50$ egrep -re 'Allocated automatic IP address' . | grep juju-2
./10.244.40.32/var/log/maas/maas.log:Sep 19 19:48:02 swoobat maas.interface: [info] Allocated automatic IP address 10.244.40.200 for ens6 (physical) on juju-2.
./10.244.40.32/var/log/maas/maas.log:Sep 19 19:48:02 swoobat maas.interface: [info] Allocated automatic IP address 10.244.40.201 for ens6 (physical) on juju-2.
./10.244.40.32/var/log/syslog:Sep 19 10:38:58 swoobat maas.interface: [info] Allocated automatic IP address 10.244.40.200 for ens6 (physical) on juju-2.
./10.244.40.32/var/log/syslog:Sep 19 15:10:47 swoobat maas.interface: [info] Allocated automatic IP address 10.244.41.26 for ens6 (physical) on juju-2.
./10.244.40.32/var/log/syslog:Sep 19 19:48:02 swoobat maas.interface: [info] Allocated automatic IP address 10.244.40.200 for ens6 (physical) on juju-2.
./10.244.40.32/var/log/syslog:Sep 19 19:48:02 swoobat maas.interface: [info] Allocated automatic IP address 10.244.40.201 for ens6 (physical) on juju-2.

Both the juju controller crashdump and the maas logs are attached.

[Joseph Phillips (manadart)]

The MAAS log entries are all for the same interface.

Is there another interface or VLAN on that same host?

[Andres Rodriguez (andreserl)]

From the MAAS perspective, it is completely possible that:

1. A single interface has 2 IP addresses.
2. Multiple interfaces have 2 IP addresses in the same subnet/network.

My guess of what's happening here is really (1). This would present itself (automatically) if for example, a machine would DHCP from MAAS' dynamic range, and while having a lease for that IP, the machine would be assigned another IP address (e.g. an statically assigned IP). Since MAAS keeps tracks of both, it would track both interfaces.

My guess in this case, is that it got into a situation in which MAAS is tracking 2 interfaces for the same interface, one of which is a dynamic IP and another one is a static IP. The dynamic IP should disappear from MAAS automatically once the lease for that expires.

[Andres Rodriguez (andreserl)]

(meh submitted before completing).

So, can you confirm which IP addresses this machine got? and/or whether one of these came from the dynamic range?

That said, this should really only happen in two situations:

1. If this is a VM that boots very fast, where it gets an IP from the dynamic range, but the deployed system ends up with a statically configured IP and by the time it is deployed, the lease has not yet been released.

2. A user manually gets the interface to DHCP even if it already has an interface.

Also, as far as Juju, I do not know how it handles the fact, but it would seem that a safe path would be to do a preference order? For example, always check for 'sticky' IP's and if not fallback to whatever else?

[Joseph Phillips (manadart)]

The current Juju behaviour is to return this error if both of these conditions are true:
- There is more than one usable address on the machine.
- There is no configuration value for juju-ha-space.

By usable addresses we mean cloud-local, but we fall back to a public address if there are none of those.

Prior to 2.4, Juju would attempt to set the space transparently by interrogating the addresses, but this had some edge cases, so we decided to be explicit and require the user to specify a space to use whenever multiple addresses are detected.

If a space is configured for HA, then the first address returned from the space is used. So you can work around this just be setting juju-ha-space.

[Joseph Phillips (manadart)]

... just *by* setting juju-ha-space.

[Andres Rodriguez (andreserl)]

Right. In this case it would seem like the interface has 2 IPs from the
same subnet, but from different ranges (e.g 1 statically assigned, and once
from the dhcp server because it hasn’t yet expired).

On Mon, Oct 1, 2018 at 10:16 AM Joseph Phillips <email address hidden>
wrote:

> The current Juju behaviour is to return this error if both of these
> conditions are true:
> - There is more than one usable address on the machine.
> - There is no configuration value for juju-ha-space.
>
> By usable addresses we mean cloud-local, but we fall back to a public
> address if there are none of those.
>
> Prior to 2.4, Juju would attempt to set the space transparently by
> interrogating the addresses, but this had some edge cases, so we decided
> to be explicit and require the user to specify a space to use whenever
> multiple addresses are detected.
>
> If a space is configured for HA, then the first address returned from
> the space is used. So you can work around this just be setting juju-ha-
> space.
>
> --
> You received this bug notification because you are subscribed to MAAS.
> https://bugs.launchpad.net/bugs/1793564
>
> Title:
> maas allocated two usable addresses for juju enable-ha
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju/+bug/1793564/+subscriptions
>
> Launchpad-Notification-Type: bug
> Launchpad-Bug: product=juju; status=Incomplete; importance=Undecided;
> <email address hidden>;
> Launchpad-Bug: product=maas; status=Incomplete; importance=Undecided;
> assignee=None;
> Launchpad-Bug-Tags: cdo-qa foundation-engine
> Launchpad-Bug-Information-Type: Public
> Launchpad-Bug-Private: no
> Launchpad-Bug-Security-Vulnerability: no
> Launchpad-Bug-Commenters: andreserl jog manadart
> Launchpad-Bug-Reporter: John George (jog)
> Launchpad-Bug-Modifier: Joseph Phillips (manadart)
> Launchpad-Message-Rationale: Subscriber (MAAS)
> Launchpad-Message-For: andreserl
>
--
Andres Rodriguez (RoAkSoAx)
Ubuntu Server Developer
MSc. Telecom & Networking
Systems Engineer

[Jason Hobbs (jason-hobbs)]

The IP addresses are in the description:

ip-addresses': ['10.244.40.200', '10.244.40.201']

They are not from different ranges, and they are not from the dynamic range.

[Chris Gregan (cgregan)]

Tracking with two bugs so dupped the other here. Hitting this issue more often now. Raising to Field-high

[Andres Rodriguez (andreserl)]

If this is a recurring issue, please provide the following information:

1. Step by step to reproduce the issue
2. Logs (or pointer to) the exact time this happens (that the machine is juju deployed). The logs (or pointer thereof) are:

A. Maas.log
B. Regiond.log
C. Juju debug log.

[Andres Rodriguez (andreserl)]

Have you guys been able to reproduce this issue? Are there any updates on this? That said, I think this is a reflection of [1], where a commissioning machine NIC ended with 2 ips, and on deploy, two IP's were provided (because it was interpreted as an alias).

[1]: https://bugs.launchpad.net/maas/+bug/1803188

[Jason Hobbs (jason-hobbs)]

I've attached juju and maas logs for a reproduction that occurred around 2018-11-18-00:54:54

'ip-addresses': ['10.244.40.228', '10.244.41.23']

[Andres Rodriguez (andreserl)]

As per https://bugs.launchpad.net/juju/+bug/1793564/comments/12, I believe the issue is [1], which is fixed on RC1.

I'll keep this as incomplete until you guys are able to test RC1 and confirm whether the bug exists or not.

[1]: https://bugs.launchpad.net/maas/+bug/1803188

[Andres Rodriguez (andreserl)]

RC1 seems to have gone successful in [1], and confirms my suspicion that this is a result of the above bug report. As such marking as duplicate.

[1]: https://solutions.qa.canonical.com/#/qa/testRuns?start_date=10%20Days%20Ago&finish_date=Now&maas=maas_2.5.0~rc1-7419-ge0d239532-0ubuntu1~18.04.1