FAILED - RETRYING: waiting for the monitor(s) to form the quorum

[wes hayutin]

http://logs.rdoproject.org/42/23642/7/check/tripleo-ceph-integration-rhel-8-scenario001-standalone/306007c/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz

   ExecStartPre: ''{ path=/bin/sh ; argv[]=/bin/sh -c "$(command -v mkdir)" -p /etc/ceph /var/lib/ceph/mon ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }'''
2020-01-02 17:37:57 | - ' ExecStop: ''{ path=/usr/bin/podman ; argv[]=/usr/bin/podman stop ceph-mon-standalone ; ignore_errors=yes ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }'''
2020-01-02 17:37:57 | - ' ExecStopPost: ''{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/ceph/ceph-mon.standalone.asok ; ignore_errors=yes ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }'''
2020-01-02 17:37:57 | - ' FragmentPath: /etc/systemd/system/ceph-mon@.service'
2020-01-02 17:37:57 | - ' Id: <email address hidden>'
2020-01-02 17:37:57 | - ' Names: <email address hidden>'
2020-01-02 17:37:57 | - ' PrivateTmp: ''no'''
2020-01-02 17:37:57 | - ' ProtectHome: ''no'''
2020-01-02 17:37:57 | - ' ProtectSystem: ''no'''
2020-01-02 17:37:57 | - ' Requires: sysinit.target system-ceph\x2dmon.slice'
2020-01-02 17:37:57 | - ' Restart: always'
2020-01-02 17:37:57 | - ' RestartUSec: 10s'
2020-01-02 17:37:57 | - ' Slice: system-ceph\x2dmon.slice'
2020-01-02 17:37:57 | - ' TimeoutStartUSec: 2min'
2020-01-02 17:37:57 | - ' TimeoutStopUSec: 15s'
2020-01-02 17:37:57 | - ' Type: simple'
2020-01-02 17:37:57 | - ' UnitFilePreset: disabled'
2020-01-02 17:37:57 | - "\e[Ke30=\e[4D\e[K\e[KeyJ1dWlkIjogIjYwMTMwMmM5LWJjODItNGY1Yy1hMWU1LTBmMzlkNzk1ZmNmMSJ9\e[64D\e[K"
2020-01-02 17:37:57 | - 'TASK [ceph-mon : include_tasks ceph_keys.yml] **********************************'
2020-01-02 17:37:57 | - "\e[Ke30=\e[4D\e[K\e[KeyJ1dWlkIjogIjg1ZTdkYTY2LTliN2EtNDY4OC1hODY2LTBmYjkzZmNiMmMxYyJ9\e[64D\e[KThursday 02 January 2020 17:36:09 +0000 (0:00:00.773) 0:01:17.733 ****** "
2020-01-02 17:37:57 | - "\e[Ke30=\e[4D\e[K\e[KeyJ1dWlkIjogIjJkYzIzZWIyLTUyYjgtNDYyYi04NzFmLTg3ZDZiNjAyYjUwNSJ9\e[64D\e[Kincluded: /usr/share/ceph-ansible/roles/ceph-mon/tasks/ceph_keys.yml for standalone"
2020-01-02 17:37:57 | - "\e[Ke30=\e[4D\e[K\e[KeyJ1dWlkIjogIjJjNTI1MWZkLTA4NzgtNDM1YS05NjY5LTZhYTFjNTM2MDY1MiJ9\e[64D\e[K"
2020-01-02 17:37:57 | - 'TASK [ceph-mon : waiting for the monitor(s) to form the quorum...] *************'
2020-01-02 17:37:57 | - "\e[Ke30=\e[4D\e[K\e[KeyJ1dWlkIjogImM2M2I0M2M4LTQ2N2EtNDJjYi1iNjI5LTVlMzU2M2Q2ZmM0MyJ9\e[64D\e[KThursday 02 January 2020 17:36:09 +0000 (0:00:00.172) 0:01:17.905 ****** "
2020-01-02 17:37:57 | - "\e[Ke30=\e[4D\e[K\e[KeyJ1dWlkIjogImZlN2I1MDA3LTNhMGEtNGQ4Yi1hZTA4LTgwN2FkNTc1MjJmNCJ9\e[64D\e[KFAILED - RETRYING: waiting for the monitor(s) to form the quorum... (10 retries left)."
2020-01-02 17:37:57 | - "\e[Ke30=\e[4D\e[K\e[KeyJ1dWlkIjogIjBiNDFhNjY1LTY1MTYtNDMzZC1hZDc1LTI0MjJjMzBkYTk4NCJ9\e[64D\e[Kok: [standalone] => changed=false "
2020-01-02 17:37:57 | - ' attempts: 2'
2020-01-02 17:37:57 | - ' - daemon'
2020-01-02 17:37:57 | - ' - mon.standalone'
2020-01-02 17:37:57 | - ' - mon_status'
2020-01-02 17:37:57 | - ' - --format'
2020-01-02 17:37:57 | - ' delta: ''0:00:00.381976'''
2020-01-02 17:37:57 | - ' end: ''2020-01-02 17:36:30.498019'''
2020-01-02 17:37:57 | - ' start: ''2020-01-02 17:36:30.116043'''
2020-01-02 17:37:57 | - ' stdout: ''{"name":"standalone","rank":0,"state":"leader","election_epoch":3,"quorum":[0],"quorum_age":13,"features":{"required_con":"2449958747315912708","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus"],"quorum_con":"4611087854031667199","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus"]},"outside_quorum":[],"extra_probe_peers":[{"addrvec":[{"type":"v2","addr":"192.168.24.1:3300","nonce":0}]},{"addrvec":[{"type":"v2","addr":"192.168.24.1:3300","nonce":0},{"type":"v1","addr":"192.168.24.1:6789","nonce":0}]}],"sync_provider":[],"monmap":{"epoch":1,"fsid":"4b5c8c0a-ff60-454b-a1b4-9747aa737d19","modified":"2020-01-02 17:36:07.036897","created":"2020-01-02 17:36:07.036897","min_mon_release":14,"min_mon_release_name":"nautilus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus"],"optional":[]},"mons":[{"rank":0,"name":"standalone","public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.24.1:3300","nonce":0},{"type":"v1","addr":"192.168.24.1:6789","nonce":0}]},"addr":"192.168.24.1:6789/0","public_addr":"192.168.24.1:6789/0"}]},"feature_map":{"mon":[{"features":"0x3ffddff8ffacffff","release":"luminous","num":1}]}}'''
2020-01-02 17:37:57 | - "\e[Ke30=\e[4D\e[K\e[KeyJ1dWlkIjogIjlkY2RjNjUxLTMyYjItNGUwZi04ZGM2LTI1NzQ2ZmVmYzFkYyJ9\e[64D\e[K"
2020-01-02 17:37:57 | - 'TASK [ceph-mon : fetch ceph initial keys] **************************************'
2020-01-02 17:37:57 | - "\e[Ke30=\e[4D\e[K\e[KeyJ1dWlkIjogImQ4ZWEzNTM3LTIzNTgtNGE4Ny1iYTRhLWNmNDY4NWMwNDU5YiJ9\e[64D\e[KThursday 02 January 2020 17:36:30 +0000 (0:00:21.277) 0:01:39.182 ****** "
2020-01-02 17:37:57 | - "\e[Ke30=\e[4D\e[K\e[KeyJ1dWlkIjogIjZlZDdhMzg0LWZiMTUtNGQ5NC04OTRjLWM1NGI4MzRlODhmZCJ9\e[64D\e[Kchanged: [standalone] => changed=true "
2020-01-02 17:37:57 | - ' - --entrypoint=ceph'
2020-01-02 17:37:57 | - ' - -n'
2020-01-02 17:37:57 | - ' - -k'
2020-01-02 17:37:57 | - ' - /var/lib/ceph/mon/ceph-standalone/keyring'
2020-01-02 17:37:57 | - ' - auth'
2020-01-02 17:37:57 | - ' - get'
2020-01-02 17:37:57 | - ' - client.bootstrap-rgw'
2020-01-02 17:37:57 | - ' - plain'
2020-01-02 17:37:57 | - ' - -o'
2020-01-02 17:37:57 | - ' - /var/lib/ceph/bootstrap-rgw/ceph.keyring'
2020-01-02 17:37:57 | - ' delta: ''0:00:06.711774'''
2020-01-02 17:37:57 | - ' end: ''2020-01-02 17:36:37.489395'''
2020-01-02 17:37:57 | - ' start: ''2020-01-02 17:36:30.777621'''
2020-01-02 17:37:57 | - ' stderr: exported keyring for client.bootstrap-rgw'
2020-01-02 17:37:57 | - "\e[Ke30=\e[4D\e[K\e[KeyJ1dWlkIjogIjM5YTA3NTEyLWVmYzEtNDUzZC05ZGI5LTQ5MTI1YTI2ODU4ZSJ9\e[64D\e[K"
2020-01-02 17:37:57 | - 'TASK [ceph-mon : include secure_cluster.yml] ***********************************'
2020-01-02 17:37:57 | - "\e[Ke30=\e[4D\e[K\e[KeyJ1dWlkIjogImZiNWQ3NTYxLTA5ZGYtNGI2Yi1hODBmLTgyZjc3YmVkOWMzMCJ9\e[64D\e[KThursday 02 January 2020 17:36:37 +0000 (0:00:06.995) 0:01:46.178 ****** "
2020-01-02 17:37:57 | - "\e[Ke30=\e[4D\e[K\e[KeyJ1dWlkIjogIjk0NDYyYjMzLTc3ZGUtNDRlYS05MWUzLTVhMzUwOGRhNTkwOSJ9\e[64D\e[Kskipping: [standalone] => changed=false "
2020-01-02 17:37:57 | - "\e[Ke30=\e[4D\e[K\e[KeyJ1dWlkIjogImUyZjhiMGFiLTI3NjItNDIwZS04ZmZhLTRhMTEyMzdhZjA0YSJ9\e[64D\e[K"

Found the same error in an old upgrade bug FYI
https://bugs.launchpad.net/tripleo/+bug/1832597

[Giulio Fidente (gfidente)]

from the logs it looks like we're installing a pretty old version of ceph-ansible

ceph-ansible-4.0.0-0.1.rc16.1.el8.noarch

in cbs, for centos7, we have 4.0.6 final already and that's the one we should be using

from the undercloud config it looks like we're using a temporary location [1] to override the ceph-ansible location

I'll help refreshing that build to 4.0.6 then we can try this again

    http://logs.rdoproject.org/42/23642/7/check/tripleo-ceph-integration-rhel-8-scenario001-standalone/306007c/logs/undercloud/etc/yum.repos.d/ceph-ansible-override.repo.txt.gz

[Giulio Fidente (gfidente)]

there is ceph-ansible from master in http://logs.rdoproject.org/42/23642/7/check/tripleo-ceph-integration-master/036b3a0/buildset/ but it doesn't seem to be getting installed

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/701960
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart/commit/?id=0df6618148cf19689e191053d2a0a75aaf15144f
Submitter: Zuul
Branch: master

commit 0df6618148cf19689e191053d2a0a75aaf15144f
Author: Giulio Fidente <email address hidden>
Date: Fri Jan 10 14:18:56 2020 +0100

    Revert "Use custom ceph-ansible until c8 storage sig ready"

    We can build RPMs on the fly now, from the stable-4 branch,
    see https://review.rdoproject.org/r/#/c/24461/

    This reverts commit d3d57757ae02a709aa2eac935d964b9d602e9128.

    Closes-Bug: 1858123
    Change-Id: I69d4d82c3874b2a66bcd8c666b5d5d85f4f0dd3c