scenario10 and scenario010-ovn wallaby jobs are failing standalone deploy - ceph-ansible undercloud deploy failure

[Douglas Viroel]

scenario010-standalone-wallaby is failing since 08/16
https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-scenario010-standalone-wallaby

scenario010-ovn-provider-standalone-wallaby is failing since 08/17:
https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-scenario010-ovn-provider-standalone-wallaby

Jobs are failing to deploy ceph:
https://logserver.rdoproject.org/86/31586/96/check/periodic-tripleo-ci-centos-8-scenario010-standalone-wallaby/6abaab0/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz

FATAL | print ceph-ansible output in case of failure | undercloud | error={
...
Tuesday 17 August 2021 19:13:40 +0000 (0:00:00.042) 0:00:08.224 ******** ",
        "ok: [standalone] => {\"ansible_facts\": {\"_container_exec_cmd\": \"podman exec ceph-mon-standalone\"}, \"changed\": false}",
        "TASK [ceph-facts : get current fsid if cluster is already running] *************",
        "Tuesday 17 August 2021 19:13:40 +0000 (0:00:00.069) 0:00:08.293 ******** ",
        "ok: [standalone] => {\"changed\": false, \"cmd\": [\"timeout\", \"--foreground\", \"-s\", \"KILL\", \"300s\", \"podman\", \"exec\", \"ceph-mon-standalone\", \"ceph\", \"--cluster\", \"ceph\", \"fsid\"], \"delta\": \"0:00:00.152718\", \"end\": \"2021-08-17 19:13:40.775339\", \"failed_when_result\": false, \"msg\": \"non-zero return code\", \"rc\": 125, \"start\": \"2021-08-17 19:13:40.622621\", \"stderr\": \"Error: no container with name or ID ceph-mon-standalone found: no such container\", \"stderr_lines\": [\"Error: no container with name or ID ceph-mon-standalone found: no such container\"], \"stdout\": \"\", \"stdout_lines\": []}",
...

[wes hayutin (weshayutin)]

https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario010-standalone-wallaby/b65afff/logs/undercloud/var/log/ceph/ceph-volume.log.txt.gz

2021-08-17 23:12:24,885][ceph_volume.process][INFO ] Running command: /usr/sbin/blkid -c /dev/null -s PARTUUID -o value ceph_vg/ceph_lv_wal
[2021-08-17 23:12:24,890][ceph_volume.devices.lvm.prepare][ERROR ] lvm prepare was unable to complete
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 252, in safe_prepare
    self.prepare()
  File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root
    return func(*a, **kw)
  File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 376, in prepare
    self.args.block_wal_slots)
  File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 189, in setup_device
    name_uuid = self.get_ptuuid(device_name)
  File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 135, in get_ptuuid
    raise RuntimeError('unable to use device')
RuntimeError: unable to use device
[2021-08-17 23:12:24,891][ceph_volume.devices.lvm.prepare][INFO ] will rollback OSD ID creation
[2021-08-17 23:12:24,891][ceph_volume.process][INFO ] Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it
[2021-08-17 23:12:25,279][ceph_volume.process][INFO ] stderr purged osd.0
[2021-08-17 23:12:25,293][ceph_volume][ERROR ] exception caught by decorator
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc
    return f(*a, **kw)
  File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 152, in main
    terminal.dispatch(self.mapper, subcommand_args)
  File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch
    instance.main()
  File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 42, in main
    terminal.dispatch(self.mapper, self.argv)
  File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch
    instance.main()
  File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 441, in main
    self.safe_prepare()
  File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 252, in safe_prepare
    self.prepare()
  File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root
    return func(*a, **kw)
  File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 376, in prepare
    self.args.block_wal_slots)
  File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 189, in setup_device
    name_uuid = self.get_ptuuid(device_name)
  File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 135, in get_ptuuid
    raise RuntimeError('unable to use device')
RuntimeError: unable to use device

[yatin (yatinkarel)]

Being fixed with https://review.opendev.org/c/openstack/tripleo-heat-templates/+/805034/

[Ronelle Landy (rlandy)]

https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-scenario010-standalone-wallaby is passing now - closing out