tripleo

"FAILED - RETRYING: waiting for the monitor(s) to form the quorum... (1 retries left).", failing on sc001, sc004

Bug #1918672 reported by Bhagyashri Shewale on 2021-03-11

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	tripleo	Fix Released	High	Francesco Pantano	tripleo wallaby-3

Bug Description

        "TASK [ceph-mon : include_tasks ceph_keys.yml] **********************************",
        "Thursday 11 March 2021 09:56:31 +0000 (0:00:01.664) 0:01:18.462 ******** ",
        "included: /usr/share/ceph-ansible/roles/ceph-mon/tasks/ceph_keys.yml for standalone",
        "TASK [ceph-mon : waiting for the monitor(s) to form the quorum...] *************",
        "Thursday 11 March 2021 09:56:31 +0000 (0:00:00.141) 0:01:18.604 ******** ",
        "FAILED - RETRYING: waiting for the monitor(s) to form the quorum... (10 retries left).",
        "FAILED - RETRYING: waiting for the monitor(s) to form the quorum... (9 retries left).",
        "FAILED - RETRYING: waiting for the monitor(s) to form the quorum... (8 retries left).",
        "FAILED - RETRYING: waiting for the monitor(s) to form the quorum... (7 retries left).",
        "FAILED - RETRYING: waiting for the monitor(s) to form the quorum... (6 retries left).",
        "FAILED - RETRYING: waiting for the monitor(s) to form the quorum... (5 retries left).",
        "FAILED - RETRYING: waiting for the monitor(s) to form the quorum... (4 retries left).",
        "FAILED - RETRYING: waiting for the monitor(s) to form the quorum... (3 retries left).",
        "FAILED - RETRYING: waiting for the monitor(s) to form the quorum... (2 retries left).",
        "FAILED - RETRYING: waiting for the monitor(s) to form the quorum... (1 retries left).",
        "fatal: [standalone]: FAILED! => {\"attempts\": 10, \"changed\": false, \"cmd\": [\"podman\", \"exec\", \"ceph-mon-standalone\", \"ceph\", \"--cluster\", \"ceph\", \"-n\", \"mon.\", \"-k\", \"/var/lib/ceph/mon/ceph-standalone/keyring\", \"mon_status\", \"--format\", \"json\"], \"delta\": \"0:00:00.765132\", \"end\": \"2021-03-11 10:00:08.860193\", \"msg\": \"non-zero return code\", \"rc\": 22, \"start\": \"2021-03-11 10:00:08.095061\", \"stderr\": \"2021-03-11T10:00:08.608+0000 7f375fb80700 -1 auth: unable to find a keyring on /etc/ceph/ceph.mon..keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory\\n2021-03-11T10:00:08.608+0000 7f375fb80700 -1 AuthRegistry(0x7f3758059360) no keyring found at /etc/ceph/ceph.mon..keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, disabling cephx\\nError EINVAL: unrecognized command\\nError: non zero exit code: 22: OCI runtime error\", \"stderr_lines\": [\"2021-03-11T10:00:08.608+0000 7f375fb80700 -1 auth: unable to find a keyring on /etc/ceph/ceph.mon..keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory\", \"2021-03-11T10:00:08.608+0000 7f375fb80700 -1 AuthRegistry(0x7f3758059360) no keyring found at /etc/ceph/ceph.mon..keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, disabling cephx\", \"Error EINVAL: unrecognized command\", \"Error: non zero exit code: 22: OCI runtime error\"], \"stdout\": \"\", \"stdout_lines\": []}",
        "NO MORE HOSTS LEFT *************************************************************",
        "PLAY RECAP *********************************************************************",
        "standalone : ok=115 changed=14 unreachable=0 failed=1 skipped=255 rescued=0 ignored=0 ",
        "INSTALLER STATUS ***************************************************************",

Affected jobs:

1. periodic-tripleo-ci-centos-8-scenario001-standalone-master
2. periodic-tripleo-ci-centos-8-scenario004-standalone-master
3. periodic-tripleo-ci-centos-8-scenario010-standalone-master
4. periodic-tripleo-ci-centos-8-scenario010-ovn-provider-standalone-master