[manual-machines] ERROR juju-ha-space is not set and a unique usable address was not found for machines: 0

[Peter Jose De Sousa]

Hello,

When bootstrapping manual machines on the same subnet it is not possible to enable HA despite all controllers belonging to the same subnet and space within juju.

[Reproducer]

1. Create three VMs on the same subnet, for example /24.
2. Bootstrap on one VM using manual cloud and add the other two VMs
3. juju enable-ha --to=1,2

ERROR juju-ha-space is not set and a unique usable address was not found for machines: 0
run "juju controller-config juju-ha-space=<name>" to set a space for Mongo peer communication

Attempting to set juju-ha-space to alpha does not appear to work either:

root@juju-client:~# juju controller-config juju-ha-space=alpha
ERROR invalid config "juju-ha-space"="alpha": machines with no addresses in this space: 0

[Additional information]
root@juju-client:~# juju status -m controller
Model Controller Cloud/Region Version SLA Timestamp
controller juju-manual-default juju-manual/default 2.9.34 unsupported 14:11:50Z

Machine State Address Inst id Series AZ Message
0 started 10.66.240.6 manual: focal
1 down 10.66.240.7 manual:10.66.240.7 focal
2 down 10.66.240.222 manual:10.66.240.222 focal

root@juju-client:~# juju show-machine -m controller 0
model: controller
machines:
  "0":
    juju-status:
      current: started
      since: 24 Sep 2022 14:07:24Z
      version: 2.9.34
    hostname: juju-ctl-manual-1
    dns-name: 10.66.240.6
    ip-addresses:
    - 10.66.240.6
    - fd42:9a92:4793:8c96:216:3eff:fecc:8855
    - fd42:9a92:4793:8c96:216:3eff:feee:943f
    instance-id: 'manual:'
    machine-status:
      current: pending
      since: 24 Sep 2022 13:49:27Z
    modification-status:
      current: idle
      since: 24 Sep 2022 13:49:27Z
    series: focal
    network-interfaces:
      enp5s0:
        ip-addresses:
        - 10.66.240.6
        - fd42:9a92:4793:8c96:216:3eff:fecc:8855
        - fd42:9a92:4793:8c96:216:3eff:feee:943f
        mac-address: 00:16:3e:cc:88:55
        gateway: 10.66.240.1 10.66.240.1 10.66.240.1
        space: alpha
        is-up: true
    constraints: arch=amd64 mem=3584M
    hardware: arch=amd64 cores=2 mem=15627M
    controller-member-status: has-vote
root@juju-client:~# juju show-machine -m controller 1
model: controller
machines:
  "1":
    juju-status:
      current: started
      since: 24 Sep 2022 13:52:29Z
      version: 2.9.34
    hostname: juju-ctl-manual-2
    dns-name: 10.66.240.7
    ip-addresses:
    - 10.66.240.7
    - fd42:9a92:4793:8c96:216:3eff:febb:3cc
    - fd42:9a92:4793:8c96:216:3eff:feee:9440
    instance-id: manual:10.66.240.7
    machine-status:
      current: pending
      since: 24 Sep 2022 13:51:22Z
    modification-status:
      current: idle
      since: 24 Sep 2022 13:51:22Z
    series: focal
    network-interfaces:
      enp5s0:
        ip-addresses:
        - 10.66.240.7
        - fd42:9a92:4793:8c96:216:3eff:febb:3cc
        - fd42:9a92:4793:8c96:216:3eff:feee:9440
        mac-address: 00:16:3e:bb:03:cc
        gateway: 10.66.240.1 10.66.240.1 10.66.240.1
        space: alpha
        is-up: true
    hardware: arch=amd64 cores=2 mem=15627M
root@juju-client:~# juju show-machine -m controller 2
model: controller
machines:
  "2":
    juju-status:
      current: started
      since: 24 Sep 2022 13:55:18Z
      version: 2.9.34
    hostname: juju-ctl-manual-0
    dns-name: 10.66.240.222
    ip-addresses:
    - 10.66.240.222
    - fd42:9a92:4793:8c96:216:3eff:feb0:a3e2
    - fd42:9a92:4793:8c96:216:3eff:feb0:a3e3
    instance-id: manual:10.66.240.222
    machine-status:
      current: pending
      since: 24 Sep 2022 13:54:24Z
    modification-status:
      current: idle
      since: 24 Sep 2022 13:54:24Z
    series: focal
    network-interfaces:
      enp5s0:
        ip-addresses:
        - 10.66.240.222
        - fd42:9a92:4793:8c96:216:3eff:feb0:a3e2
        - fd42:9a92:4793:8c96:216:3eff:feb0:a3e3
        mac-address: 00:16:3e:b0:a3:e2
        gateway: 10.66.240.1 10.66.240.1 10.66.240.1
        space: alpha
        is-up: true
    hardware: arch=amd64 cores=2 mem=15627M
r

[Workaround]
None at this time,

Thanks,
Peter

[Joseph Phillips (manadart)]

Can you get me the output of `juju-spaces` in the controller model after you have added the machines?

The error might be misleading regarding machine 0 having no addresses in the space.

How it works is that Juju filters the machine addresses based on a local-cloud scope. If this yields a single address, we proceed. If not we require the operator declare which address to use via spaces.

So if you're just using "alpha" and it happens not to isolate a unique address, we can't proceed.

Does it work if you put the IPv4 subnet alone into a new space, say "beta" and use that as the HA space?

[Bartosz Woronicz (mastier1)]

Not sure whart you mean,
I created 3 seperate spaces now (via juju add-space -m controller)

$ juju spaces -m controller
Name Space ID Subnets
alpha 0
oam-space 1 10.33.27.0/24
storage-access-space 2 10.33.28.0/24
fan-network-space 3 252.0.0.0/8

Yet, still presume machines 0 does not have oam-space. But it has, see the below

$ juju controller-config juju-ha-space=oam-space
ERROR invalid config "juju-ha-space"="oam-space": machines with no addresses in this space: 0

$ juju show-machine -m controller 0
model: controller
machines:
  "0":
    juju-status:
      current: started
      since: 01 Dec 2023 00:44:10Z
      version: 3.1.6
    hostname: k8sddev-juju-0
    dns-name: k8sddev-juju-0
    ip-addresses:
    - k8sddev-juju-0
    - 10.33.27.71
    - 252.16.0.1
    instance-id: 'manual:'
    machine-status:
      current: running
      message: Manually provisioned machine
      since: 01 Dec 2023 00:46:39Z
    modification-status:
      current: idle
      since: 30 Nov 2023 23:53:44Z
    base:
      name: ubuntu
      channel: "22.04"
    network-interfaces:
      ens160:
        ip-addresses:
        - 10.33.27.71
        mac-address: 00:50:56:01:41:94
        gateway: 10.33.27.254
        space: oam-space
        is-up: true
      fan-252:
        ip-addresses:
        - 252.16.0.1
        mac-address: 5e:30:82:9f:ef:be
        space: fan-network-space
        is-up: true
    constraints: arch=amd64 mem=3584M
    hardware: arch=amd64 cores=8 mem=15988M
    controller-member-status: has-vote

$ juju version
3.1.6-genericlinux-amd64

So adding extra spaces does not help

[Bartosz Woronicz (mastier1)]

I remove adress from the second nic (surprisingly not visible here in show-machine), still complains on juju-ha-space

2: ens160: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:50:56:01:41:94 brd ff:ff:ff:ff:ff:ff
    altname enp3s0
    inet 10.33.27.71/24 brd 10.33.27.255 scope global ens160
       valid_lft forever preferred_lft forever
    inet6 fe80::250:56ff:fe01:4194/64 scope link
       valid_lft forever preferred_lft forever
3: ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:50:56:01:41:95 brd ff:ff:ff:ff:ff:ff
    altname enp11s0
    inet6 fe80::250:56ff:fe01:4195/64 scope link
       valid_lft forever preferred_lft forever
4: fan-252: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000
    link/ether 5e:30:82:9f:ef:be brd ff:ff:ff:ff:ff:ff
    inet 252.16.0.1/8 scope global fan-252
       valid_lft forever preferred_lft forever
    inet6 fe80::5c30:82ff:fe9f:efbe/64 scope link
       valid_lft forever preferred_lft forever

but enable-ha works

$ juju controller-config juju-ha-space=oam-space
ERROR invalid config "juju-ha-space"="oam-space": machines with no addresses in this space: 0
130 ubuntu@k8sddev-juju-0:~/cpe-deployments$ juju enable-ha --to 1,2
maintaining machines: 0
converting machines: 1, 2

[Joseph Phillips (manadart)]

This issue has partial overlap with https://bugs.launchpad.net/juju/+bug/2039436.
I believe the issue is known.