Comment 3 for bug 2056337

Revision history for this message
Federico Bosi (rhxto) wrote (last edit ):

This issue affects ceph mon as well.
I bootstrapped ceph-osd on latest/edge rev 584 with this fix.
Ceph mons (mgrs in this case) are unable to talk to eachother because they listen by default on 0.0.0.0 instead of :: or both.
Here's the netstat:
root@select-dragon:~# netstat -tunpl
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 837/sshd: /usr/sbin
tcp 0 0 127.0.0.53:53 0.0.0.0:* LISTEN 87687/systemd-resol
tcp 0 0 0.0.0.0:6800 0.0.0.0:* LISTEN 10067/ceph-mgr
tcp 0 0 0.0.0.0:6801 0.0.0.0:* LISTEN 10067/ceph-mgr
tcp6 0 0 my-prefix:14:3300 :::* LISTEN 9912/ceph-mon
tcp6 0 0 :::22 :::* LISTEN 837/sshd: /usr/sbin
tcp6 0 0 my-prefix:14:6789 :::* LISTEN 9912/ceph-mon
udp 0 0 127.0.0.53:53 0.0.0.0:* 87687/systemd-resol

All PGs are down as they can't connect via ipv4.
I added the same 2 lines in /etc/ceph/ceph.conf and everything went online.

Before I edited the config:
  cluster:
    id: cb11d088-fb1f-11ee-870e-51f7eebe9d0d
    health: HEALTH_WARN
            Reduced data availability: 129 pgs inactive

  services:
    mon: 3 daemons, quorum subtle-viper,crisp-hawk,select-dragon (age 25h)
    mgr: select-dragon(active, since 25h), standbys: crisp-hawk, subtle-viper
    osd: 3 osds: 3 up (since 30m), 3 in (since 32m)

  data:
    pools: 2 pools, 129 pgs
    objects: 0 objects, 0 B
    usage: 0 B used, 0 B / 0 B avail
    pgs: 100.000% pgs unknown
             129 unknown

After:
  cluster:
    id: cb11d088-fb1f-11ee-870e-51f7eebe9d0d
    health: HEALTH_OK

  services:
    mon: 3 daemons, quorum subtle-viper,crisp-hawk,select-dragon (age 10s)
    mgr: select-dragon(active, since 22s), standbys: subtle-viper, crisp-hawk
    osd: 3 osds: 3 up (since 35m), 3 in (since 37m)

  data:
    pools: 2 pools, 129 pgs
    objects: 2 objects, 577 KiB
    usage: 81 MiB used, 140 GiB / 140 GiB avail
    pgs: 129 active+clean

Correct netstat:
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 837/sshd: /usr/sbin
tcp 0 0 127.0.0.53:53 0.0.0.0:* LISTEN 87687/systemd-resol
tcp6 0 0 my-prefix:14:3300 :::* LISTEN 87808/ceph-mon
tcp6 0 0 :::22 :::* LISTEN 837/sshd: /usr/sbin
tcp6 0 0 my-prefix:14:6800 :::* LISTEN 87807/ceph-mgr
tcp6 0 0 my-prefix:224:14:6801. :::* LISTEN 87807/ceph-mgr
tcp6 0 0 my-prefix:14:6789 :::* LISTEN 87808/ceph-mon
udp 0 0 127.0.0.53:53 0.0.0.0:* 87687/systemd-resol