No config flag available to disable OSDs binding to ipv4 addresses

[Federico Bosi]

I'm using juju 3.3 and am on the latest version of this charm.

The prefer-ipv6 flag only checks that the host has a valid dynamic ipv6 address (which I don't understand why, static ones are just as good, but that's for another issue).

My OSDs aren't booting because they try to bind to ipv4 addresses though I have none configured.
The situation is identical to the one in https://tracker.ceph.com/issues/52867 (adding ms_bind_ipv4/6 to ceph.conf).
I added them in the config-flags parameter as "{'global': {'ms_bind_ipv4': False, 'ms_bind_ipv6': True}}" and my OSDs all booted correctly.

I will submit shortly a change to opendev that adds that config flag if prefer-ipv6 is enabled and fixes the issue.

[Federico Bosi (rhxto)]

Change review uploaded here :)
https://review.opendev.org/c/openstack/charm-ceph-osd/+/911592

[Federico Bosi (rhxto)]

The Zuul build is failing on python 3.11 because we can't put mock objects in other mock objects.
The other builds are ok.

[Federico Bosi (rhxto)]

This issue affects ceph mon as well.
I bootstrapped ceph-osd on latest/edge rev 584 with this fix.
Ceph mons (mgrs in this case) are unable to talk to eachother because they listen by default on 0.0.0.0 instead of :: or both.
Here's the netstat:
root@select-dragon:~# netstat -tunpl
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 837/sshd: /usr/sbin
tcp 0 0 127.0.0.53:53 0.0.0.0:* LISTEN 87687/systemd-resol
tcp 0 0 0.0.0.0:6800 0.0.0.0:* LISTEN 10067/ceph-mgr
tcp 0 0 0.0.0.0:6801 0.0.0.0:* LISTEN 10067/ceph-mgr
tcp6 0 0 my-prefix:14:3300 :::* LISTEN 9912/ceph-mon
tcp6 0 0 :::22 :::* LISTEN 837/sshd: /usr/sbin
tcp6 0 0 my-prefix:14:6789 :::* LISTEN 9912/ceph-mon
udp 0 0 127.0.0.53:53 0.0.0.0:* 87687/systemd-resol

All PGs are down as they can't connect via ipv4.
I added the same 2 lines in /etc/ceph/ceph.conf and everything went online.

Before I edited the config:
  cluster:
    id: cb11d088-fb1f-11ee-870e-51f7eebe9d0d
    health: HEALTH_WARN
            Reduced data availability: 129 pgs inactive

  services:
    mon: 3 daemons, quorum subtle-viper,crisp-hawk,select-dragon (age 25h)
    mgr: select-dragon(active, since 25h), standbys: crisp-hawk, subtle-viper
    osd: 3 osds: 3 up (since 30m), 3 in (since 32m)

  data:
    pools: 2 pools, 129 pgs
    objects: 0 objects, 0 B
    usage: 0 B used, 0 B / 0 B avail
    pgs: 100.000% pgs unknown
             129 unknown

After:
  cluster:
    id: cb11d088-fb1f-11ee-870e-51f7eebe9d0d
    health: HEALTH_OK

  services:
    mon: 3 daemons, quorum subtle-viper,crisp-hawk,select-dragon (age 10s)
    mgr: select-dragon(active, since 22s), standbys: subtle-viper, crisp-hawk
    osd: 3 osds: 3 up (since 35m), 3 in (since 37m)

  data:
    pools: 2 pools, 129 pgs
    objects: 2 objects, 577 KiB
    usage: 81 MiB used, 140 GiB / 140 GiB avail
    pgs: 129 active+clean

Correct netstat:
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 837/sshd: /usr/sbin
tcp 0 0 127.0.0.53:53 0.0.0.0:* LISTEN 87687/systemd-resol
tcp6 0 0 my-prefix:14:3300 :::* LISTEN 87808/ceph-mon
tcp6 0 0 :::22 :::* LISTEN 837/sshd: /usr/sbin
tcp6 0 0 my-prefix:14:6800 :::* LISTEN 87807/ceph-mgr
tcp6 0 0 my-prefix:224:14:6801. :::* LISTEN 87807/ceph-mgr
tcp6 0 0 my-prefix:14:6789 :::* LISTEN 87808/ceph-mon
udp 0 0 127.0.0.53:53 0.0.0.0:* 87687/systemd-resol

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-ceph-mon/+/916046
Committed: https://opendev.org/openstack/charm-ceph-mon/commit/7e61d1b8a050998697aee91eb0db6c988c2c397a
Submitter: "Zuul (22348)"
Branch: master

commit 7e61d1b8a050998697aee91eb0db6c988c2c397a
Author: Federico Bosi <email address hidden>
Date: Wed Apr 17 09:39:32 2024 +0200

    Add ipv6 bind flags

    Following https://tracker.ceph.com/issues/52867 we need to tell ceph
    which address family to use via the ms_bind_ipv4/6 config flags.
    I added them to the ceph.conf template and updated the config hook.
    Closes-Bug: #2056337

    Change-Id: Ib735bd4876b6909762288b97857bccaa597c2b80

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/quincy.2
Review: https://review.opendev.org/c/openstack/charm-ceph-mon/+/918066

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-ceph-mon/+/918066
Committed: https://opendev.org/openstack/charm-ceph-mon/commit/0166bd9907341eaef4dd1e06b1cde1a85aaf0293
Submitter: "Zuul (22348)"
Branch: stable/quincy.2

commit 0166bd9907341eaef4dd1e06b1cde1a85aaf0293
Author: Federico Bosi <email address hidden>
Date: Wed Apr 17 09:39:32 2024 +0200

    Add ipv6 bind flags

    Following https://tracker.ceph.com/issues/52867 we need to tell ceph
    which address family to use via the ms_bind_ipv4/6 config flags.
    I added them to the ceph.conf template and updated the config hook.
    Closes-Bug: #2056337

    Change-Id: Ib735bd4876b6909762288b97857bccaa597c2b80
    (cherry picked from commit 7e61d1b8a050998697aee91eb0db6c988c2c397a)