Manual provider error on re-configuring HA

[Stamatis Katsaounis]

Similar to https://bugs.launchpad.net/juju/+bug/1990724 but not the same:

Running on 3.4.4, I am trying to boostrap juju with manual cloud and enable ha:

juju enable-ha -n 3 --to 1,2

This is working. Then I add a new machine (3) to the cluster and I have machines 0, 1, 2, 3

Then I want to remove machine 2: juju remove-machine 2
Then I want to re-enable ha but use the machine 3: juju enable-ha -n 3 --to 3
Then I get the same error as the similar bug I linked in the beginning:

ubuntu@maas-1:~$ juju enable-ha -n 3 --to 3
ERROR juju-ha-space is not set and a unique usable address was not found for machines: 0
run "juju controller-config juju-ha-space=<name>" to set a space for Mongo peer communication

ubuntu@maas-1:~$ juju show-machine 0
model: controller
machines:
  "0":
    juju-status:
      current: started
      since: 24 Jul 2024 13:07:38Z
      version: 3.4.4
    hostname: maas-1
    dns-name: 10.20.0.62
    ip-addresses:
    - 10.20.0.62
    - 10.20.0.80
    instance-id: 'manual:'
    machine-status:
      current: running
      message: Manually provisioned machine
      since: 24 Jul 2024 13:07:38Z
    modification-status:
      current: idle
      since: 24 Jul 2024 13:07:30Z
    base:
      name: ubuntu
      channel: "22.04"
    network-interfaces:
      enp5s0:
        ip-addresses:
        - 10.20.0.62
        - 10.20.0.80
        mac-address: 00:16:3e:c4:af:95
        gateway: 10.20.0.1 10.20.0.1
        space: alpha
        is-up: true
    constraints: arch=amd64 mem=3584M
    hardware: arch=amd64 cores=4 mem=7930M
    controller-member-status: has-vote
    ha-primary: true

[Joseph Phillips (manadart)]

Was the second of the addresses added post-bootstrap?

[Stamatis Katsaounis (skatsaounis)]

Indeed. There is a keepalived charm that can rotate this second address on machines that are part of the controller model.

[Joseph Phillips (manadart)]

If you run the "ip" command, I assume it indicates that those addresses are SECONDARY.

Is this correct?

[Stamatis Katsaounis (skatsaounis)]

Hi Joseph,

This is the output of ip a for dynamic, secondary, primary,permanent:

ubuntu@maas-1:~$ ip address show enp5s0 dynamic
ubuntu@maas-1:~$ ip address show enp5s0 secondary

ubuntu@maas-1:~$ ip address show enp5s0 primary
2: enp5s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:16:3e:c4:af:95 brd ff:ff:ff:ff:ff:ff
    inet 10.20.0.10/24 brd 10.20.0.255 scope global enp5s0
       valid_lft forever preferred_lft forever
    inet 10.20.0.80/32 scope global enp5s0
       valid_lft forever preferred_lft forever
    inet6 fe80::216:3eff:fec4:af95/64 scope link
       valid_lft forever preferred_lft forever

ubuntu@maas-1:~$ ip address show enp5s0 permanent
2: enp5s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:16:3e:c4:af:95 brd ff:ff:ff:ff:ff:ff
    inet 10.20.0.10/24 brd 10.20.0.255 scope global enp5s0
       valid_lft forever preferred_lft forever
    inet 10.20.0.80/32 scope global enp5s0
       valid_lft forever preferred_lft forever
    inet6 fe80::216:3eff:fec4:af95/64 scope link
       valid_lft forever preferred_lft forever

And this is how keepalived is configured by the charm to manage this IP:

vrrp_script chk_svc_port {
    # returns 1 if connection is refused
    script "/bin/bash -c '</dev/tcp/127.0.0.1/80'"
    # check every 2 seconds
    interval 2
    # make sure master priority drops below backup priority on failure
    weight 2
}
vrrp_instance VI_1 {
    interface enp5s0
    state MASTER
    priority 101
    virtual_router_id 23

    virtual_ipaddress {
        10.20.0.80
    }

    track_script {
        chk_svc_port
    }
}

So it is displayed as primary, permament and it is there because of VRRP and keepalived