microceph is bound on Calico IP address - Failed executing cmd: ['microceph', 'cluster', 'join' ... Error: Failed to get certificate of cluster member "10.1.32.192:7443": Get "htt ps://10.1.32.192:7443": Unable to connect to: 10.1.32.192:7443 ([dial tcp 10.1.32.192:7443: i/o timeout]

[Nobuto Murata]

After working around LP: #2065470, I tried to add the second node with `sunbeam cluster add` and `sunbeam cluster join`.

However, microceph charm failed to form the two nodes clusters with the following error.

unit-microceph-1: 15:21:51 INFO unit.microceph/1.juju-log peers:1: {'snap-channel': 'reef/stable', 'public_net': IPv4Network('192.168.123.0/24'), 'cluster_net': IPv4Network('192.168
.123.0/24'), 'micro_ip': IPv4Address('192.168.123.173')}
unit-microceph-1: 15:22:01 ERROR unit.microceph/1.juju-log peers:1: Failed executing cmd: ['microceph', 'cluster', 'join', 'eyJuYW1lIjoibWljcm9jZXBoLzEiLCJzZWNyZXQiOiJkMzdjYTY2Y2RhM
2MwYjYwMzZjNWJhMDNlNzZlMGI0ZWM3YTM1ODRjMjkzYWM1NTljNDg1OGZiMGVhZTA2YWQ5IiwiZmluZ2VycHJpbnQiOiJkMGIxMmVmMzE3NjllY2ZmYjYzNTE0MWFkZjEzNDY5NDk0NmFhMGU1NzM3ODgyOTYyNGZmMjhjN2JhOWY1ZDc5Ii
wiam9pbl9hZGRyZXNzZXMiOlsiMTAuMS4zMi4xOTI6NzQ0MyJdfQ==', '--microceph-ip', '192.168.123.173'], error: Error: Failed to get certificate of cluster member "10.1.32.192:7443": Get "htt
ps://10.1.32.192:7443": Unable to connect to: 10.1.32.192:7443 ([dial tcp 10.1.32.192:7443: i/o timeout])

unit-microceph-1: 15:22:01 WARNING unit.microceph/1.juju-log peers:1: Error: Failed to get certificate of cluster member "10.1.32.192:7443": Get "https://10.1.32.192:7443": Unable t
o connect to: 10.1.32.192:7443 ([dial tcp 10.1.32.192:7443: i/o timeout])

sunbeam-1:~$ ip -br a
lo UNKNOWN 127.0.0.1/8 ::1/128
enp1s0 UP 192.168.123.228/24 metric 100 fe80::5054:ff:fe9f:c4fc/64
enp9s0 DOWN
cali2ae07341a2e@if3 UP fe80::ecee:eeff:feee:eeee/64
vxlan.calico UNKNOWN 10.1.32.192/32 fe80::64a8:66ff:fe69:8e0f/64
calib5f1a1d0dc7@if3 UP fe80::ecee:eeff:feee:eeee/64
cali8eb9d73bdf4@if3 UP fe80::ecee:eeff:feee:eeee/64

...

ovs-system DOWN
br-int DOWN
br-ex UNKNOWN 10.20.20.1/24 fe80::4c18:b8ff:fe4e:634e/64

192.168.123.0/24 is the host network and 10.1.32.192 is on the calico VXLAN interface. But the microceph on the initial node is bound on the Calico IP.

$ sudo microceph status
MicroCeph deployment summary:
- sunbeam-1 (10.1.32.192)
  Services: mds, mgr, mon, osd
  Disks: 2
- sunbeam-2 (192.168.123.173)
  Services:
  Disks: 0

$ echo 'eyJuYW1lIjoibWljcm9jZXBoLzEiLCJzZWNyZXQiOiJkMzdjYTY2Y2RhM2MwYjYwMzZjNWJhMDNlNzZlMGI0ZWM3YTM1ODRjMjkzYWM1NTljNDg1OGZiMGVhZTA2YWQ5IiwiZmluZ2VycHJpbnQiOiJkMGIxMmVmMzE3NjllY2ZmYjYzNTE0MWFkZjEzNDY5NDk0NmFhMGU1NzM3ODgyOTYyNGZmMjhjN2JhOWY1ZDc5Iiwiam9pbl9hZGRyZXNzZXMiOlsiMTAuMS4zMi4xOTI6NzQ0MyJdfQ==' | base64 -d
{"name":"microceph/1","secret":"d37ca66cda3c0b6036c5ba03e76e0b4ec7a3584c293ac559c4858fb0eae06ad9","fingerprint":"d0b12ef31769ecffb635141adf134694946aa0e57378829624ff28c7ba9f5d79","join_addresses":["10.1.32.192:7443"]}

[Nobuto Murata (nobuto)]

microceph charm is using the manual provider in this context, and the binding is not used other than the default "alpha". Not sure if Juju is capable to leverage multiple spaces on the manual provider these days or not though.

$ juju spaces
Name Space ID Subnets
alpha 0 10.1.32.192/32
                 192.168.123.0/24

$ juju show-application microceph
microceph:
  charm: microceph
  base: ubuntu@22.04
  channel: latest/edge
  constraints:
    arch: amd64
  principal: true
  exposed: false
  remote: false
  life: alive
  endpoint-bindings:
    "": alpha
    admin: alpha
    ceph: alpha
    cluster: alpha
    mds: alpha
    peers: alpha
    public: alpha
    radosgw: alpha

https://github.com/canonical/charm-microceph/blob/6cd11fa645040d53e7a7e68814151c1c138dda2f/src/microceph.py#L86-L99

[Nobuto Murata (nobuto)]

I had a similar issue in different providers that Juju picked /32 from the flannel interface:
https://bugs.launchpad.net/juju/+bug/1897115

[Nobuto Murata (nobuto)]

public_network and cluster_network are set for the Calico network too. It's going to be tricky to solve this with the manual provider...

ref: https://github.com/canonical/microceph/issues/197

$ sudo microceph.ceph config dump
WHO MASK LEVEL OPTION VALUE RO
global advanced cluster_network 10.1.32.192/32 *
global advanced mon_allow_pool_size_one false
global advanced osd_pool_default_crush_rule 1
global advanced osd_pool_default_size 3

$ sudo microceph.ceph mon dump
epoch 2
fsid 76a3e10e-776c-4df9-a7e3-81d92a818c98
last_changed 2024-05-15T01:55:03.634009+0000
created 2024-05-15T01:55:00.522273+0000
min_mon_release 18 (reef)
election_strategy: 1
0: [v2:10.1.32.192:3300/0,v1:10.1.32.192:6789/0] mon.sunbeam-1
dumped monmap epoch 2

[Nobuto Murata (nobuto)]

Specifying ignore-machine-addresses=true in the manifest file didn't do the trick either.

software:
  juju:
    bootstrap_args:
      - --debug
      - --model-default=test-mode=true
      - --model-default=disable-telemetry=true
      - --model-default=logging-config=<root>=INFO;unit=DEBUG
      # LP: #2065700
      - --model-default=ignore-machine-addresses=true

[Nobuto Murata (nobuto)]

Adding the Juju task as I have no idea how to workaround this.

For the context,

- juju 3.4.2-genericlinux-amd64
- manual provider
- the host has the main 192.168.123.0/24 network
- microk8s is deployed on the host, and it will assign 10.1.X.Y/32 to an additional interface
- Juju keeps adding those /32 IP addresses to the main space and it confuses charms to bind to those unreachable IP addresses from other units
- ignore-machine-addresses=true didn't do the trick. Juju still adds /32 to the space

$ juju spaces
Name Space ID Subnets
alpha 0 10.1.32.192/32
                 192.168.123.0/24

[Nobuto Murata (nobuto)]

I tried to workaround it by bootstrapping the storage service first before having the control plane with microk8s. However, sunbeam always add the control plane for bootstrap so --role storage will be translated to --role storage --role control so no luck.

https://github.com/canonical/snap-openstack/blob/599e01aa263729d8f411241531bc424934b9ce05/sunbeam-python/sunbeam/provider/local/commands.py#L238-L241
> # Bootstrap node must always have the control role
> if Role.CONTROL not in roles:
> LOG.debug("Enabling control role for bootstrap")
> roles.append(Role.CONTROL)

[Nobuto Murata (nobuto)]

With an educated guess that Juju sorts subnets in numerical order, I tried an explicit subnet as 10.0.123.0/24 (within 10.0.0.0/16 range to show up earlier than 10.1.0.0/16 as the default Calico range in microk8s).

The result is as expected. The original issue remains the same that Juju may use the /32 address from vxlan.calico. But Juju returns an address from the 10.0.123.0/24 range so it can work around the issue in one way.

In a real-life world, using 10.0.0.0/16 as the host subnet just for Juju and Sunbeam is practically impossible by eliminating 10.0.0.0/8 minus 10.0.0.0/16, 172.16.0.0/12 and 192.168.0.0/16 though.

$ juju spaces
Name Space ID Subnets
alpha 0 10.0.123.0/24
                 10.1.186.0/32
                 10.1.32.192/32

$ juju exec --unit microceph/0 -- network-get public
bind-addresses:
- mac-address: 52:54:00:51:00:99
  interface-name: enp1s0
  addresses:
  - hostname: ""
    value: 10.0.123.241
    cidr: 10.0.123.0/24
    address: 10.0.123.241
  macaddress: 52:54:00:51:00:99
  interfacename: enp1s0
- mac-address: 66:a8:66:69:8e:0f
  interface-name: vxlan.calico
  addresses:
  - hostname: ""
    value: 10.1.32.192
    cidr: 10.1.32.192/32
    address: 10.1.32.192
  macaddress: 66:a8:66:69:8e:0f
  interfacename: vxlan.calico
egress-subnets:
- 10.0.123.241/32
ingress-addresses:
- 10.0.123.241
- 10.1.32.192

$ sudo microceph status
MicroCeph deployment summary:
- sunbeam-1 (10.0.123.241)
  Services: mds, mgr, mon, osd
  Disks: 2
- sunbeam-2 (10.0.123.234)
  Services: mds, mgr, mon
  Disks: 0

[James Page (james-page)]

I think we're probably going to need to create a space once the juju controller is bootstrapped so that we can specify which subnet(s) should be used for services thus avoiding the random binding issue hit here.

[Andre Ruiz (andre-ruiz)]

I think this is the same as LP: #2049694 mostly -- in my case it was not calico but just another addresses that I have in the machine but wanted to ignore and they are still picked.

[Andre Ruiz (andre-ruiz)]

Also, similar (or the same) as LP: #2056218 that is about microceph timing out on join because it is trying to use the public address for clustering which cannot be used for that (although the network indicated in the manifest was a different one).

[Nobuto Murata (nobuto)]

It's similar indeed, but slightly different. Because this issue happens only with the single network story.

In your case, you picked the ranges in 10.0.0.0/16 so you didn't hit the issue itself so you were ready to hit the next issue as multiple networking support.

> bond0 - has two ips, private/31 and public/31. I'm ignoring those.
> bond0.1001 - has one ip on the 10.0.1.0/24 network
> bond0.1002 - ha no IPs, will use network 10.0.2.0/24 later for Floating IPs.

[Billy Olsen (billy-olsen)]

The manual deployment scenario lacks the juju space support that is available in the maas deployment scenario. We'll likely need to bring the spaces into the manual provider scenario in order to account for this.

[James Page (james-page)]

@billy-olsen agreed - we're going to need todo something just to avoid Juju binding to random networks, and in order to support HA controllers in the future (the DB needs to be bound to a specific space right now).

[Joseph Phillips (manadart)]

Juju has space support on the manual provider, but it works in a slightly esoteric way due to the fact there is no real "provider" to detect the subnets from.

As machines are added, Juju will accumulate any subnets it sees and add them to its collection.

These can subsequently be carved into spaces.

Juju does have hierarchical rules for choosing a preferred address, but if 2 addresses have the same scope (local-cloud for example) there is a lexicographically sorted choice.

Feel free to get hold of me direct on Matrix/Mattermost if it would help to look at something in-situ.