juju agent on the controller does not complete after bootstrap

[Yoshi Kadokawa]

When bootstrapping on a machine(manual provider) with more than 3 networks,
the bootstrap command does complete without any errors, however, the juju status for the controller unit stays with `agent initialising` message, and doesn't complete the bootstrap process.

$ juju status -m controller
Model Controller Cloud/Region Version SLA Timestamp
controller manual-default manual/default 3.1.6 unsupported 11:05:47Z

App Version Status Scale Charm Channel Rev Exposed Message
controller waiting 0/1 juju-controller 3.1/stable 14 no agent initialising

Unit Workload Agent Machine Public address Ports Message
controller/0 waiting allocating 0 192.168.88.50 agent initialising

Machine State Address Inst id Base AZ Message
0 started 192.168.88.50 manual: ubuntu@22.04

$ juju ssh -m controller 0 '
    ip -br a
  '
lo UNKNOWN 127.0.0.1/8 ::1/128
enp1s0 UP 192.168.88.50/24 fe80::5054:ff:fe0a:9e30/64
enp2s0 UP 172.16.10.50/24 fe80::5054:ff:febd:ca9/64
enp3s0 DOWN
enp4s0 DOWN
enp5s0 DOWN
enp6s0 UP 172.16.20.50/24 fe80::5054:ff:fe21:cadf/64
enp7s0 DOWN
Connection to 192.168.88.50 closed.

I have confirmed that this does not happen when bootstrapping on a machine with 2 networks.

For now the workaround is to prepare the machine with less than 2 networks, and add the necessary networks after bootstrap has completed.

[Joseph Phillips (manadart)]

The peer-grouper worker, which maintains the MongoDB cluster needs to identify a unique local-cloud scoped address to which MongoDB is bound.

Where multiple are available, the one to use is specified by supplying config for the juju-ha-space.

This could be made to work on MAAS, where the subnets/spaces are provider-sourced, and therefore known at model creation.

However this is not the case with the manual provider, as there are no space definitions.

As part of the Juju 4.0 work we are moving some of this behaviour to the controller charm itself, so we will endeavour to accommodate this scenario.

Until that time, the stated work-around is the avenue to take.

[Joseph Phillips (manadart)]

The work-around mentioned above creates a latent issue.

Progressive subnet discovery was added for the manual provider under this patch:
https://github.com/juju/juju/pull/11899

The problem is that it will only discover subnets from new *NICs*.

This means that since all devices but one were disabled in order to bootstrap and enter HA, they were already in Juju without addresses. Once we added addresses to them, the subnets for those addresses were not added to Juju.

If they were, we would have been able to carve the different subnets into spaces, and set one of those spaces as configuration for "juju-ha-space", which would have ensured a unique local-cloud address that the peer-grouper could use to maintain the Mongo control plane.

Once we had a restart (soft or hard), the peer-grouper now in an error loop could not broadcast the address information needed to establish the Raft transport. No Raft - no leases - no API.

[Nobuto Murata (nobuto)]

Subscribing ~field-critical.

This is blocking an engagement and a proposed workaround didn't work after restarting jujud. There is no known workaround at this point and this issue invalidates some practical scenarios in customer engagements, namely:

- bootstrapping Juju with procreated VMs
- bootstrapping Juju before having any provider
  - for microcloud charm or microk8s charm where there are only 3 physical hosts without MAAS beforehand

[Joseph Phillips (manadart)]

If the addresses enabled at bootstrap/HA-entry are acceptable for:

1. Mongo control plane.
2. Agent to controller communication.

Then post-bootstrap you can issue:

juju model-config -m controller ignore-machine-addresses=true

This is verified working for the Tamkeen cloud.

[John A Meinel (jameinel)]

Unsubscribing field-critical because we have the work around, feedback as to whether this should be field-high still

[Joseph Phillips (manadart)]

I've created https://bugs.launchpad.net/juju/+bug/2052598 for the specific issue of subsequent subnet discovery.

[Joseph Phillips (manadart)]

The other part of this issue intersects with https://bugs.launchpad.net/juju/+bug/1990724.

[Joseph Phillips (manadart)]

https://github.com/juju/juju/pull/16891