Running /home/stack/overcloud-deploy/overcloud/config-download/overcloud/cephadm/cephadm_command.sh 2022-05-10 10:08:02,684 p=771211 u=stack n=ansible | PLAY [Deploy Ceph with cephadm] ************************************************ 2022-05-10 10:08:02,772 p=771211 u=stack n=ansible | 2022-05-10 10:08:02.771150 | 48d539a1-1679-7650-8e2b-000000000018 | TASK | Install cephadm package 2022-05-10 10:08:05,123 p=771211 u=stack n=ansible | 2022-05-10 10:08:05.121502 | 48d539a1-1679-7650-8e2b-000000000018 | OK | Install cephadm package | overcloud-controller-0 2022-05-10 10:08:05,156 p=771211 u=stack n=ansible | 2022-05-10 10:08:05.155132 | 48d539a1-1679-7650-8e2b-000000000019 | TASK | Stat cephadm file 2022-05-10 10:08:06,786 p=771211 u=stack n=ansible | 2022-05-10 10:08:06.783906 | 48d539a1-1679-7650-8e2b-000000000019 | OK | Stat cephadm file | overcloud-controller-0 2022-05-10 10:08:06,901 p=771211 u=stack n=ansible | 2022-05-10 10:08:06.854662 | 48d539a1-1679-7650-8e2b-00000000001a | TASK | Fail if cephadm is not available 2022-05-10 10:08:07,062 p=771211 u=stack n=ansible | 2022-05-10 10:08:07.060480 | 48d539a1-1679-7650-8e2b-00000000001a | SKIPPED | Fail if cephadm is not available | overcloud-controller-0 2022-05-10 10:08:07,131 p=771211 u=stack n=ansible | 2022-05-10 10:08:07.129736 | 48d539a1-1679-7650-8e2b-00000000001b | TASK | List Ceph daemon instances on this host 2022-05-10 10:08:08,369 p=771211 u=stack n=ansible | 2022-05-10 10:08:08.368122 | 48d539a1-1679-7650-8e2b-00000000001b | CHANGED | List Ceph daemon instances on this host | overcloud-controller-0 2022-05-10 10:08:08,449 p=771211 u=stack n=ansible | 2022-05-10 10:08:08.448451 | 48d539a1-1679-7650-8e2b-00000000001d | TASK | Set list of found FSIDs 2022-05-10 10:08:08,638 p=771211 u=stack n=ansible | 2022-05-10 10:08:08.637395 | 48d539a1-1679-7650-8e2b-00000000001d | SKIPPED | Set list of found FSIDs | overcloud-controller-0 2022-05-10 10:08:08,714 p=771211 u=stack n=ansible | 2022-05-10 10:08:08.713664 | 48d539a1-1679-7650-8e2b-00000000001e | TASK | Fail if >1 FSID was discovered 2022-05-10 10:08:08,870 p=771211 u=stack n=ansible | 2022-05-10 10:08:08.868422 | 48d539a1-1679-7650-8e2b-00000000001e | SKIPPED | Fail if >1 FSID was discovered | overcloud-controller-0 2022-05-10 10:08:08,950 p=771211 u=stack n=ansible | 2022-05-10 10:08:08.948948 | 48d539a1-1679-7650-8e2b-00000000001f | TASK | Set FSID to the discovered value 2022-05-10 10:08:09,110 p=771211 u=stack n=ansible | 2022-05-10 10:08:09.109316 | 48d539a1-1679-7650-8e2b-00000000001f | SKIPPED | Set FSID to the discovered value | overcloud-controller-0 2022-05-10 10:08:09,194 p=771211 u=stack n=ansible | 2022-05-10 10:08:09.193278 | 48d539a1-1679-7650-8e2b-000000000020 | TASK | Set random tripleo_cephadm_fsid if no running ceph containers were found 2022-05-10 10:08:09,334 p=771211 u=stack n=ansible | 2022-05-10 10:08:09.333185 | 48d539a1-1679-7650-8e2b-000000000020 | SKIPPED | Set random tripleo_cephadm_fsid if no running ceph containers were found | overcloud-controller-0 2022-05-10 10:08:09,420 p=771211 u=stack n=ansible | 2022-05-10 10:08:09.374343 | 48d539a1-1679-7650-8e2b-000000000022 | TASK | Set first monitor IP if it was not passed 2022-05-10 10:08:09,574 p=771211 u=stack n=ansible | 2022-05-10 10:08:09.572508 | 48d539a1-1679-7650-8e2b-000000000022 | SKIPPED | Set first monitor IP if it was not passed | overcloud-controller-0 2022-05-10 10:08:09,673 p=771211 u=stack n=ansible | 2022-05-10 10:08:09.627505 | 48d539a1-1679-7650-8e2b-000000000023 | TASK | Ensure tripleo_cephadm_config_home (e.g. /etc/ceph) exists 2022-05-10 10:08:10,458 p=771211 u=stack n=ansible | 2022-05-10 10:08:10.456943 | 48d539a1-1679-7650-8e2b-000000000023 | CHANGED | Ensure tripleo_cephadm_config_home (e.g. /etc/ceph) exists | overcloud-controller-0 2022-05-10 10:08:10,535 p=771211 u=stack n=ansible | 2022-05-10 10:08:10.533940 | 48d539a1-1679-7650-8e2b-000000000024 | TASK | Ensure specs directory exists 2022-05-10 10:08:10,894 p=771211 u=stack n=ansible | 2022-05-10 10:08:10.892661 | 48d539a1-1679-7650-8e2b-000000000024 | CHANGED | Ensure specs directory exists | overcloud-controller-0 2022-05-10 10:08:10,936 p=771211 u=stack n=ansible | 2022-05-10 10:08:10.935034 | 48d539a1-1679-7650-8e2b-000000000025 | TASK | Stat spec file on ansible host 2022-05-10 10:08:11,915 p=771211 u=stack n=ansible | 2022-05-10 10:08:11.913756 | 48d539a1-1679-7650-8e2b-000000000025 | OK | Stat spec file on ansible host | overcloud-controller-0 -> localhost 2022-05-10 10:08:11,919 p=771211 u=stack n=ansible | [WARNING]: ('overcloud-controller-0 -> localhost', '48d539a1-1679-7650-8e2b-000000000025') missing from stats 2022-05-10 10:08:11,954 p=771211 u=stack n=ansible | 2022-05-10 10:08:11.953291 | 48d539a1-1679-7650-8e2b-000000000026 | TASK | push tripleo_cephadm_spec to bootstrap node if spec file exsits 2022-05-10 10:08:13,308 p=771211 u=stack n=ansible | 2022-05-10 10:08:13.306676 | 48d539a1-1679-7650-8e2b-000000000026 | CHANGED | push tripleo_cephadm_spec to bootstrap node if spec file exsits | overcloud-controller-0 2022-05-10 10:08:13,351 p=771211 u=stack n=ansible | 2022-05-10 10:08:13.349982 | 48d539a1-1679-7650-8e2b-000000000028 | TASK | stat tripleo_cephadm_bootstrap_conf file 2022-05-10 10:08:13,498 p=771211 u=stack n=ansible | 2022-05-10 10:08:13.496426 | 48d539a1-1679-7650-8e2b-000000000028 | SKIPPED | stat tripleo_cephadm_bootstrap_conf file | overcloud-controller-0 2022-05-10 10:08:13,530 p=771211 u=stack n=ansible | 2022-05-10 10:08:13.529193 | 48d539a1-1679-7650-8e2b-000000000029 | TASK | push it to tripleo_cephadm_assimilate_conf on bootstrap node 2022-05-10 10:08:13,674 p=771211 u=stack n=ansible | 2022-05-10 10:08:13.673189 | 48d539a1-1679-7650-8e2b-000000000029 | SKIPPED | push it to tripleo_cephadm_assimilate_conf on bootstrap node | overcloud-controller-0 2022-05-10 10:08:13,708 p=771211 u=stack n=ansible | 2022-05-10 10:08:13.706962 | 48d539a1-1679-7650-8e2b-00000000002b | TASK | generate tripleo_cephadm_assimilate_conf on bootstrap node 2022-05-10 10:08:14,571 p=771211 u=stack n=ansible | 2022-05-10 10:08:14.569729 | 48d539a1-1679-7650-8e2b-00000000002b | CHANGED | generate tripleo_cephadm_assimilate_conf on bootstrap node | overcloud-controller-0 2022-05-10 10:08:14,603 p=771211 u=stack n=ansible | 2022-05-10 10:08:14.602738 | 48d539a1-1679-7650-8e2b-000000000056 | TASK | Add spec to necessary file list when using spec to bootstrap 2022-05-10 10:08:14,761 p=771211 u=stack n=ansible | 2022-05-10 10:08:14.760310 | 48d539a1-1679-7650-8e2b-000000000056 | SKIPPED | Add spec to necessary file list when using spec to bootstrap | overcloud-controller-0 2022-05-10 10:08:14,795 p=771211 u=stack n=ansible | 2022-05-10 10:08:14.794303 | 48d539a1-1679-7650-8e2b-000000000057 | TASK | Stat necessary files to bootstrap with cephadm 2022-05-10 10:08:15,223 p=771211 u=stack n=ansible | 2022-05-10 10:08:15.221898 | 48d539a1-1679-7650-8e2b-000000000057 | OK | Stat necessary files to bootstrap with cephadm | overcloud-controller-0 | item=/home/ceph-admin/.ssh/id_rsa 2022-05-10 10:08:15,419 p=771211 u=stack n=ansible | 2022-05-10 10:08:15.418354 | 48d539a1-1679-7650-8e2b-000000000057 | OK | Stat necessary files to bootstrap with cephadm | overcloud-controller-0 | item=/home/ceph-admin/.ssh/id_rsa.pub 2022-05-10 10:08:15,473 p=771211 u=stack n=ansible | 2022-05-10 10:08:15.470543 | 48d539a1-1679-7650-8e2b-000000000058 | TASK | Fail if necessary files are missing 2022-05-10 10:08:15,725 p=771211 u=stack n=ansible | 2022-05-10 10:08:15.723381 | 48d539a1-1679-7650-8e2b-000000000058 | SKIPPED | Fail if necessary files are missing | overcloud-controller-0 | item={'changed': False, 'stat': {'exists': True, 'path': '/home/ceph-admin/.ssh/id_rsa', 'mode': '0600', 'isdir': False, 'ischr': False, 'isblk': False, 'isreg': True, 'isfifo': False, 'islnk': False, 'issock': False, 'uid': 1002, 'gid': 1003, 'size': 2622, 'inode': 1556100405, 'dev': 2051, 'nlink': 1, 'atime': 1652148449.6169398, 'mtime': 1652148449.3339396, 'ctime': 1652148449.6229398, 'wusr': True, 'rusr': True, 'xusr': False, 'wgrp': False, 'rgrp': False, 'xgrp': False, 'woth': False, 'roth': False, 'xoth': False, 'isuid': False, 'isgid': False, 'blocks': 8, 'block_size': 4096, 'device_type': 0, 'readable': True, 'writeable': True, 'executable': False, 'pw_name': 'ceph-admin', 'gr_name': 'ceph-admin', 'checksum': '0649ed9fc53f0156b01548e62a962bdeee432bb7', 'mimetype': 'text/plain', 'charset': 'us-ascii', 'version': '2011066949', 'attributes': [], 'attr_flags': ''}, 'invocation': {'module_args': {'path': '/home/ceph-admin/.ssh/id_rsa', 'follow': False, 'get_md5': False, 'get_checksum': True, 'get_mime': True, 'get_attributes': True, 'checksum_algorithm': 'sha1'}}, 'failed': False, 'item': '/home/ceph-admin/.ssh/id_rsa', 'ansible_loop_var': 'item'} 2022-05-10 10:08:15,747 p=771211 u=stack n=ansible | 2022-05-10 10:08:15.746281 | 48d539a1-1679-7650-8e2b-000000000058 | SKIPPED | Fail if necessary files are missing | overcloud-controller-0 | item={'changed': False, 'stat': {'exists': True, 'path': '/home/ceph-admin/.ssh/id_rsa.pub', 'mode': '0644', 'isdir': False, 'ischr': False, 'isblk': False, 'isreg': True, 'isfifo': False, 'islnk': False, 'issock': False, 'uid': 1002, 'gid': 1003, 'size': 553, 'inode': 1568730618, 'dev': 2051, 'nlink': 1, 'atime': 1652148451.4029424, 'mtime': 1652148451.1399422, 'ctime': 1652148451.4079425, 'wusr': True, 'rusr': True, 'xusr': False, 'wgrp': False, 'rgrp': True, 'xgrp': False, 'woth': False, 'roth': True, 'xoth': False, 'isuid': False, 'isgid': False, 'blocks': 8, 'block_size': 4096, 'device_type': 0, 'readable': True, 'writeable': True, 'executable': False, 'pw_name': 'ceph-admin', 'gr_name': 'ceph-admin', 'checksum': 'b26be79997a7ac1c2386c2cfda15efa3de7dc282', 'mimetype': 'text/plain', 'charset': 'us-ascii', 'version': '3465945859', 'attributes': [], 'attr_flags': ''}, 'invocation': {'module_args': {'path': '/home/ceph-admin/.ssh/id_rsa.pub', 'follow': False, 'get_md5': False, 'get_checksum': True, 'get_mime': True, 'get_attributes': True, 'checksum_algorithm': 'sha1'}}, 'failed': False, 'item': '/home/ceph-admin/.ssh/id_rsa.pub', 'ansible_loop_var': 'item'} 2022-05-10 10:08:15,796 p=771211 u=stack n=ansible | 2022-05-10 10:08:15.795495 | 48d539a1-1679-7650-8e2b-000000000059 | TASK | Stat pre ceph conf file in case we should bootrap with it 2022-05-10 10:08:16,190 p=771211 u=stack n=ansible | 2022-05-10 10:08:16.188601 | 48d539a1-1679-7650-8e2b-000000000059 | OK | Stat pre ceph conf file in case we should bootrap with it | overcloud-controller-0 2022-05-10 10:08:16,225 p=771211 u=stack n=ansible | 2022-05-10 10:08:16.224397 | 48d539a1-1679-7650-8e2b-00000000005b | TASK | Run cephadm bootstrap 2022-05-10 10:09:40,975 p=771211 u=stack n=ansible | 2022-05-10 10:09:40.971301 | 48d539a1-1679-7650-8e2b-00000000005b | CHANGED | Run cephadm bootstrap | overcloud-controller-0 2022-05-10 10:09:41,117 p=771211 u=stack n=ansible | 2022-05-10 10:09:41.116139 | 48d539a1-1679-7650-8e2b-00000000005c | TASK | Show results of bootstrap 2022-05-10 10:09:41,329 p=771211 u=stack n=ansible | 2022-05-10 10:09:41.326994 | 48d539a1-1679-7650-8e2b-00000000005c | SKIPPED | Show results of bootstrap | overcloud-controller-0 2022-05-10 10:09:41,400 p=771211 u=stack n=ansible | 2022-05-10 10:09:41.399516 | 48d539a1-1679-7650-8e2b-00000000005e | TASK | If cephadm bootstrap was not run report the reason 2022-05-10 10:09:41,561 p=771211 u=stack n=ansible | 2022-05-10 10:09:41.560087 | 48d539a1-1679-7650-8e2b-00000000005e | SKIPPED | If cephadm bootstrap was not run report the reason | overcloud-controller-0 2022-05-10 10:09:41,648 p=771211 u=stack n=ansible | 2022-05-10 10:09:41.599946 | 48d539a1-1679-7650-8e2b-000000000075 | TASK | Ensure tripleo_cephadm_ceph_cli is set 2022-05-10 10:09:41,686 p=771211 u=stack n=ansible | 2022-05-10 10:09:41.685057 | 48d539a1-1679-7650-8e2b-000000000075 | SKIPPED | Ensure tripleo_cephadm_ceph_cli is set | overcloud-controller-0 2022-05-10 10:09:41,818 p=771211 u=stack n=ansible | 2022-05-10 10:09:41.797755 | 48d539a1-1679-7650-8e2b-000000000076 | TASK | Assimilate configuration from tripleo_cephadm_assimilate_conf 2022-05-10 10:09:41,889 p=771211 u=stack n=ansible | 2022-05-10 10:09:41.887778 | 48d539a1-1679-7650-8e2b-000000000076 | SKIPPED | Assimilate configuration from tripleo_cephadm_assimilate_conf | overcloud-controller-0 2022-05-10 10:09:42,003 p=771211 u=stack n=ansible | 2022-05-10 10:09:41.963705 | 48d539a1-1679-7650-8e2b-000000000081 | TASK | Stat spec file on bootstrap node 2022-05-10 10:09:42,889 p=771211 u=stack n=ansible | 2022-05-10 10:09:42.888224 | 48d539a1-1679-7650-8e2b-000000000081 | OK | Stat spec file on bootstrap node | overcloud-controller-0 | item=/home/ceph-admin/specs/ceph_spec.yaml 2022-05-10 10:09:42,967 p=771211 u=stack n=ansible | 2022-05-10 10:09:42.966312 | 48d539a1-1679-7650-8e2b-000000000082 | TASK | Fail if spec file is missing 2022-05-10 10:09:43,211 p=771211 u=stack n=ansible | 2022-05-10 10:09:43.210328 | 48d539a1-1679-7650-8e2b-000000000082 | SKIPPED | Fail if spec file is missing | overcloud-controller-0 | item={'changed': False, 'stat': {'exists': True, 'path': '/home/ceph-admin/specs/ceph_spec.yaml', 'mode': '0644', 'isdir': False, 'ischr': False, 'isblk': False, 'isreg': True, 'isfifo': False, 'islnk': False, 'issock': False, 'uid': 1002, 'gid': 1003, 'size': 1248, 'inode': 1660956963, 'dev': 2051, 'nlink': 1, 'atime': 1652148493.2050045, 'mtime': 1652148492.5190034, 'ctime': 1652148493.2100043, 'wusr': True, 'rusr': True, 'xusr': False, 'wgrp': False, 'rgrp': True, 'xgrp': False, 'woth': False, 'roth': True, 'xoth': False, 'isuid': False, 'isgid': False, 'blocks': 8, 'block_size': 4096, 'device_type': 0, 'readable': True, 'writeable': True, 'executable': False, 'pw_name': 'ceph-admin', 'gr_name': 'ceph-admin', 'checksum': '0d872cd184129d186e494eb68428c72a15ee0cb3', 'mimetype': 'text/plain', 'charset': 'us-ascii', 'version': '3210489396', 'attributes': [], 'attr_flags': ''}, 'invocation': {'module_args': {'path': '/home/ceph-admin/specs/ceph_spec.yaml', 'follow': False, 'get_md5': False, 'get_checksum': True, 'get_mime': True, 'get_attributes': True, 'checksum_algorithm': 'sha1'}}, 'failed': False, 'item': '/home/ceph-admin/specs/ceph_spec.yaml', 'ansible_loop_var': 'item'} 2022-05-10 10:09:43,354 p=771211 u=stack n=ansible | 2022-05-10 10:09:43.289718 | 48d539a1-1679-7650-8e2b-000000000083 | TASK | Get ceph_cli 2022-05-10 10:09:43,619 p=771211 u=stack n=ansible | 2022-05-10 10:09:43.618380 | 2922f686-02a8-4457-afe0-4fbfa9d05423 | INCLUDED | /usr/share/ansible/roles/tripleo_cephadm/tasks/ceph_cli.yaml | overcloud-controller-0 2022-05-10 10:09:43,826 p=771211 u=stack n=ansible | 2022-05-10 10:09:43.825238 | 48d539a1-1679-7650-8e2b-0000000001fb | TASK | Set ceph CLI 2022-05-10 10:09:44,045 p=771211 u=stack n=ansible | 2022-05-10 10:09:44.043667 | 48d539a1-1679-7650-8e2b-0000000001fb | OK | Set ceph CLI | overcloud-controller-0 2022-05-10 10:09:44,085 p=771211 u=stack n=ansible | 2022-05-10 10:09:44.083952 | 48d539a1-1679-7650-8e2b-000000000084 | TASK | Get the ceph orchestrator status 2022-05-10 10:09:45,254 p=771211 u=stack n=ansible | 2022-05-10 10:09:45.252027 | 48d539a1-1679-7650-8e2b-000000000084 | CHANGED | Get the ceph orchestrator status | overcloud-controller-0 2022-05-10 10:09:45,331 p=771211 u=stack n=ansible | 2022-05-10 10:09:45.329793 | 48d539a1-1679-7650-8e2b-000000000085 | TASK | Fail if ceph orchestrator is not available 2022-05-10 10:09:45,524 p=771211 u=stack n=ansible | 2022-05-10 10:09:45.522969 | 48d539a1-1679-7650-8e2b-000000000085 | SKIPPED | Fail if ceph orchestrator is not available | overcloud-controller-0 2022-05-10 10:09:45,605 p=771211 u=stack n=ansible | 2022-05-10 10:09:45.556159 | 48d539a1-1679-7650-8e2b-000000000086 | TASK | Set container images used by the applied services 2022-05-10 10:09:45,900 p=771211 u=stack n=ansible | 2022-05-10 10:09:45.899528 | a5be33fe-3516-46eb-9ac6-e08cd99dafae | INCLUDED | /usr/share/ansible/roles/tripleo_cephadm/tasks/cephadm_config_set.yaml | overcloud-controller-0 2022-05-10 10:09:46,067 p=771211 u=stack n=ansible | 2022-05-10 10:09:46.065770 | 48d539a1-1679-7650-8e2b-000000000232 | TASK | set cephadm debug level 2022-05-10 10:09:46,222 p=771211 u=stack n=ansible | 2022-05-10 10:09:46.220637 | 48d539a1-1679-7650-8e2b-000000000232 | SKIPPED | set cephadm debug level | overcloud-controller-0 2022-05-10 10:09:46,253 p=771211 u=stack n=ansible | 2022-05-10 10:09:46.251899 | 48d539a1-1679-7650-8e2b-000000000233 | TASK | set default container image in ceph configuration 2022-05-10 10:09:47,379 p=771211 u=stack n=ansible | 2022-05-10 10:09:47.377711 | 48d539a1-1679-7650-8e2b-000000000233 | OK | set default container image in ceph configuration | overcloud-controller-0 2022-05-10 10:09:47,412 p=771211 u=stack n=ansible | 2022-05-10 10:09:47.411509 | 48d539a1-1679-7650-8e2b-000000000234 | TASK | enforce minimum ceph clients version 2022-05-10 10:09:47,589 p=771211 u=stack n=ansible | 2022-05-10 10:09:47.587414 | 48d539a1-1679-7650-8e2b-000000000234 | SKIPPED | enforce minimum ceph clients version | overcloud-controller-0 2022-05-10 10:09:47,623 p=771211 u=stack n=ansible | 2022-05-10 10:09:47.622426 | 48d539a1-1679-7650-8e2b-000000000235 | TASK | set container image base in ceph configuration 2022-05-10 10:09:48,646 p=771211 u=stack n=ansible | 2022-05-10 10:09:48.644433 | 48d539a1-1679-7650-8e2b-000000000235 | OK | set container image base in ceph configuration | overcloud-controller-0 2022-05-10 10:09:48,679 p=771211 u=stack n=ansible | 2022-05-10 10:09:48.678186 | 48d539a1-1679-7650-8e2b-000000000237 | TASK | set alertmanager container image in ceph configuration 2022-05-10 10:09:48,840 p=771211 u=stack n=ansible | 2022-05-10 10:09:48.838480 | 48d539a1-1679-7650-8e2b-000000000237 | SKIPPED | set alertmanager container image in ceph configuration | overcloud-controller-0 2022-05-10 10:09:48,875 p=771211 u=stack n=ansible | 2022-05-10 10:09:48.873802 | 48d539a1-1679-7650-8e2b-000000000238 | TASK | set grafana container image in ceph configuration 2022-05-10 10:09:49,040 p=771211 u=stack n=ansible | 2022-05-10 10:09:49.038321 | 48d539a1-1679-7650-8e2b-000000000238 | SKIPPED | set grafana container image in ceph configuration | overcloud-controller-0 2022-05-10 10:09:49,073 p=771211 u=stack n=ansible | 2022-05-10 10:09:49.072399 | 48d539a1-1679-7650-8e2b-000000000239 | TASK | set node-exporter container image in ceph configuration 2022-05-10 10:09:49,237 p=771211 u=stack n=ansible | 2022-05-10 10:09:49.235685 | 48d539a1-1679-7650-8e2b-000000000239 | SKIPPED | set node-exporter container image in ceph configuration | overcloud-controller-0 2022-05-10 10:09:49,268 p=771211 u=stack n=ansible | 2022-05-10 10:09:49.267145 | 48d539a1-1679-7650-8e2b-00000000023a | TASK | set prometheus container image in ceph configuration 2022-05-10 10:09:49,465 p=771211 u=stack n=ansible | 2022-05-10 10:09:49.463454 | 48d539a1-1679-7650-8e2b-00000000023a | SKIPPED | set prometheus container image in ceph configuration | overcloud-controller-0 2022-05-10 10:09:49,500 p=771211 u=stack n=ansible | 2022-05-10 10:09:49.499483 | 48d539a1-1679-7650-8e2b-000000000087 | TASK | Apply spec 2022-05-10 10:09:57,853 p=771211 u=stack n=ansible | 2022-05-10 10:09:57.850879 | 48d539a1-1679-7650-8e2b-000000000087 | CHANGED | Apply spec | overcloud-controller-0 2022-05-10 10:09:57,932 p=771211 u=stack n=ansible | 2022-05-10 10:09:57.930717 | 48d539a1-1679-7650-8e2b-000000000088 | TASK | Show results of spec apply 2022-05-10 10:09:58,133 p=771211 u=stack n=ansible | 2022-05-10 10:09:58.129924 | 48d539a1-1679-7650-8e2b-000000000088 | SKIPPED | Show results of spec apply | overcloud-controller-0 2022-05-10 10:09:58,204 p=771211 u=stack n=ansible | 2022-05-10 10:09:58.203418 | 48d539a1-1679-7650-8e2b-000000000089 | TASK | Wait for the expected number of osds to be running 2022-05-10 10:09:58,377 p=775553 u=stack n=ansible | [DEPRECATION WARNING]: evaluating 'tripleo_cephadm_wait_for_osds' as a bare variable, this behaviour will go away and you might need to add |bool to the expression in the future. Also see CONDITIONAL_BARE_VARS configuration toggle. This feature will be removed in version 2.12. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2022-05-10 10:09:58,549 p=771211 u=stack n=ansible | 2022-05-10 10:09:58.541430 | 77ea7f3a-66ce-4ec8-8e09-3c914bd26ff2 | INCLUDED | /usr/share/ansible/roles/tripleo_cephadm/tasks/wait_for_expected_num_osds.yaml | overcloud-controller-0 2022-05-10 10:09:58,800 p=771211 u=stack n=ansible | 2022-05-10 10:09:58.799117 | 48d539a1-1679-7650-8e2b-000000000291 | TASK | Wait for expected number of osds to be running 2022-05-10 10:30:40,591 p=771211 u=stack n=ansible | 2022-05-10 10:30:40.589873 | 48d539a1-1679-7650-8e2b-000000000291 | FATAL | Wait for expected number of osds to be running | overcloud-controller-0 | error={"attempts": 40, "changed": true, "cmd": "podman run --rm --net=host --ipc=host --volume /etc/ceph:/etc/ceph:z --volume /home/ceph-admin/assimilate_ceph.conf:/home/assimilate_ceph.conf:z --volume /home/ceph-admin/specs/ceph_spec.yaml:/home/ceph_spec.yaml:z --entrypoint ceph 172.25.201.68:8787/ceph/daemon:v6.0.6-stable-6.0-pacific-centos-8-x86_64 --fsid 2dbb93bd-e7ae-4a34-94e5-231eb65202dd -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring status --format json | jq .osdmap.num_up_osds", "delta": "0:00:00.749462", "end": "2022-05-10 02:30:40.552493", "rc": 0, "start": "2022-05-10 02:30:39.803031", "stderr": "", "stderr_lines": [], "stdout": "0", "stdout_lines": ["0"]} 2022-05-10 10:30:40,604 p=771211 u=stack n=ansible | PLAY RECAP ********************************************************************* 2022-05-10 10:30:40,605 p=771211 u=stack n=ansible | overcloud-controller-0 : ok=20 changed=8 unreachable=0 failed=1 skipped=23 rescued=0 ignored=0 2022-05-10 10:30:40,608 p=771211 u=stack n=ansible | 2022-05-10 10:30:40.607538 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Summary Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2022-05-10 10:30:40,609 p=771211 u=stack n=ansible | 2022-05-10 10:30:40.608397 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Total Tasks: 44 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2022-05-10 10:30:40,609 p=771211 u=stack n=ansible | 2022-05-10 10:30:40.609200 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Elapsed Time: 0:22:37.934694 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2022-05-10 10:30:40,610 p=771211 u=stack n=ansible | 2022-05-10 10:30:40.609903 | UUID | Info | Host | Task Name | Run Time 2022-05-10 10:30:40,611 p=771211 u=stack n=ansible | 2022-05-10 10:30:40.610668 | 48d539a1-1679-7650-8e2b-000000000291 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Wait for expected number of osds to be running | 1241.79s 2022-05-10 10:30:40,611 p=771211 u=stack n=ansible | 2022-05-10 10:30:40.611424 | 48d539a1-1679-7650-8e2b-00000000005b | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Run cephadm bootstrap | 84.75s 2022-05-10 10:30:40,612 p=771211 u=stack n=ansible | 2022-05-10 10:30:40.612163 | 48d539a1-1679-7650-8e2b-000000000087 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Apply spec | 8.35s 2022-05-10 10:30:40,613 p=771211 u=stack n=ansible | 2022-05-10 10:30:40.612989 | 48d539a1-1679-7650-8e2b-000000000018 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Install cephadm package | 2.35s 2022-05-10 10:30:40,622 p=771211 u=stack n=ansible | 2022-05-10 10:30:40.613694 | 48d539a1-1679-7650-8e2b-000000000019 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Stat cephadm file | 1.63s 2022-05-10 10:30:40,623 p=771211 u=stack n=ansible | 2022-05-10 10:30:40.622850 | 48d539a1-1679-7650-8e2b-000000000026 | SUMMARY | overcloud-controller-0 | push tripleo_cephadm_spec to bootstrap node if spec file exsits | 1.36s 2022-05-10 10:30:40,624 p=771211 u=stack n=ansible | 2022-05-10 10:30:40.623736 | 48d539a1-1679-7650-8e2b-00000000001b | SUMMARY | overcloud-controller-0 | tripleo_cephadm : List Ceph daemon instances on this host | 1.24s 2022-05-10 10:30:40,625 p=771211 u=stack n=ansible | 2022-05-10 10:30:40.624540 | 48d539a1-1679-7650-8e2b-000000000084 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Get the ceph orchestrator status | 1.14s 2022-05-10 10:30:40,625 p=771211 u=stack n=ansible | 2022-05-10 10:30:40.625370 | 48d539a1-1679-7650-8e2b-000000000233 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : set default container image in ceph configuration | 1.13s 2022-05-10 10:30:40,626 p=771211 u=stack n=ansible | 2022-05-10 10:30:40.626180 | 48d539a1-1679-7650-8e2b-000000000235 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : set container image base in ceph configuration | 1.02s 2022-05-10 10:30:40,627 p=771211 u=stack n=ansible | 2022-05-10 10:30:40.626948 | 48d539a1-1679-7650-8e2b-000000000081 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Stat spec file on bootstrap node | 0.89s 2022-05-10 10:30:40,628 p=771211 u=stack n=ansible | 2022-05-10 10:30:40.627697 | 48d539a1-1679-7650-8e2b-00000000002b | SUMMARY | overcloud-controller-0 | generate tripleo_cephadm_assimilate_conf on bootstrap node | 0.86s 2022-05-10 10:30:40,629 p=771211 u=stack n=ansible | 2022-05-10 10:30:40.628488 | 48d539a1-1679-7650-8e2b-000000000023 | SUMMARY | overcloud-controller-0 | Ensure tripleo_cephadm_config_home (e.g. /etc/ceph) exists | 0.79s 2022-05-10 10:30:40,629 p=771211 u=stack n=ansible | 2022-05-10 10:30:40.629310 | 48d539a1-1679-7650-8e2b-000000000057 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Stat necessary files to bootstrap with cephadm | 0.63s 2022-05-10 10:30:40,630 p=771211 u=stack n=ansible | 2022-05-10 10:30:40.630105 | 48d539a1-1679-7650-8e2b-000000000059 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Stat pre ceph conf file in case we should bootrap with it | 0.39s 2022-05-10 10:30:40,631 p=771211 u=stack n=ansible | 2022-05-10 10:30:40.630893 | 48d539a1-1679-7650-8e2b-000000000024 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Ensure specs directory exists | 0.36s 2022-05-10 10:30:40,632 p=771211 u=stack n=ansible | 2022-05-10 10:30:40.631651 | 48d539a1-1679-7650-8e2b-000000000058 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Fail if necessary files are missing | 0.29s 2022-05-10 10:30:40,633 p=771211 u=stack n=ansible | 2022-05-10 10:30:40.632416 | 48d539a1-1679-7650-8e2b-000000000082 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Fail if spec file is missing | 0.25s 2022-05-10 10:30:40,633 p=771211 u=stack n=ansible | 2022-05-10 10:30:40.633186 | 48d539a1-1679-7650-8e2b-0000000001fb | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Set ceph CLI | 0.22s 2022-05-10 10:30:40,634 p=771211 u=stack n=ansible | 2022-05-10 10:30:40.633951 | 48d539a1-1679-7650-8e2b-00000000005c | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Show results of bootstrap | 0.21s 2022-05-10 10:30:40,677 p=771211 u=stack n=ansible | 2022-05-10 10:30:40.634701 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ End Summary Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2022-05-10 10:30:40,678 p=771211 u=stack n=ansible | 2022-05-10 10:30:40.677707 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ State Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2022-05-10 10:30:40,679 p=771211 u=stack n=ansible | 2022-05-10 10:30:40.678558 | ~~~~~~~~~~~~~~~~~~ Number of nodes which did not deploy successfully: 1 ~~~~~~~~~~~~~~~~~ 2022-05-10 10:30:40,679 p=771211 u=stack n=ansible | 2022-05-10 10:30:40.679348 | The following node(s) had failures: overcloud-controller-0 2022-05-10 10:30:40,680 p=771211 u=stack n=ansible | 2022-05-10 10:30:40.680124 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Running /home/stack/overcloud-deploy/overcloud/config-download/overcloud/cephadm/cephadm_command.sh 2022-05-10 12:41:14,625 p=106241 u=stack n=ansible | PLAY [Deploy Ceph with cephadm] ************************************************ 2022-05-10 12:41:14,714 p=106241 u=stack n=ansible | 2022-05-10 12:41:14.713524 | 48d539a1-1679-39d3-e106-000000000018 | TASK | Install cephadm package 2022-05-10 12:41:17,049 p=106241 u=stack n=ansible | 2022-05-10 12:41:17.047807 | 48d539a1-1679-39d3-e106-000000000018 | OK | Install cephadm package | overcloud-controller-0 2022-05-10 12:41:17,080 p=106241 u=stack n=ansible | 2022-05-10 12:41:17.079342 | 48d539a1-1679-39d3-e106-000000000019 | TASK | Stat cephadm file 2022-05-10 12:41:17,841 p=106241 u=stack n=ansible | 2022-05-10 12:41:17.840041 | 48d539a1-1679-39d3-e106-000000000019 | OK | Stat cephadm file | overcloud-controller-0 2022-05-10 12:41:17,958 p=106241 u=stack n=ansible | 2022-05-10 12:41:17.957721 | 48d539a1-1679-39d3-e106-00000000001a | TASK | Fail if cephadm is not available 2022-05-10 12:41:18,154 p=106241 u=stack n=ansible | 2022-05-10 12:41:18.153153 | 48d539a1-1679-39d3-e106-00000000001a | SKIPPED | Fail if cephadm is not available | overcloud-controller-0 2022-05-10 12:41:18,354 p=106241 u=stack n=ansible | 2022-05-10 12:41:18.351942 | 48d539a1-1679-39d3-e106-00000000001b | TASK | List Ceph daemon instances on this host 2022-05-10 12:41:19,816 p=106241 u=stack n=ansible | 2022-05-10 12:41:19.813874 | 48d539a1-1679-39d3-e106-00000000001b | CHANGED | List Ceph daemon instances on this host | overcloud-controller-0 2022-05-10 12:41:19,887 p=106241 u=stack n=ansible | 2022-05-10 12:41:19.886036 | 48d539a1-1679-39d3-e106-00000000001d | TASK | Set list of found FSIDs 2022-05-10 12:41:20,199 p=106241 u=stack n=ansible | 2022-05-10 12:41:20.197098 | 48d539a1-1679-39d3-e106-00000000001d | SKIPPED | Set list of found FSIDs | overcloud-controller-0 2022-05-10 12:41:20,273 p=106241 u=stack n=ansible | 2022-05-10 12:41:20.272115 | 48d539a1-1679-39d3-e106-00000000001e | TASK | Fail if >1 FSID was discovered 2022-05-10 12:41:20,535 p=106241 u=stack n=ansible | 2022-05-10 12:41:20.533425 | 48d539a1-1679-39d3-e106-00000000001e | SKIPPED | Fail if >1 FSID was discovered | overcloud-controller-0 2022-05-10 12:41:20,622 p=106241 u=stack n=ansible | 2022-05-10 12:41:20.621214 | 48d539a1-1679-39d3-e106-00000000001f | TASK | Set FSID to the discovered value 2022-05-10 12:41:20,787 p=106241 u=stack n=ansible | 2022-05-10 12:41:20.785449 | 48d539a1-1679-39d3-e106-00000000001f | SKIPPED | Set FSID to the discovered value | overcloud-controller-0 2022-05-10 12:41:20,880 p=106241 u=stack n=ansible | 2022-05-10 12:41:20.879091 | 48d539a1-1679-39d3-e106-000000000020 | TASK | Set random tripleo_cephadm_fsid if no running ceph containers were found 2022-05-10 12:41:21,034 p=106241 u=stack n=ansible | 2022-05-10 12:41:21.031845 | 48d539a1-1679-39d3-e106-000000000020 | SKIPPED | Set random tripleo_cephadm_fsid if no running ceph containers were found | overcloud-controller-0 2022-05-10 12:41:21,097 p=106241 u=stack n=ansible | 2022-05-10 12:41:21.096386 | 48d539a1-1679-39d3-e106-000000000022 | TASK | Set first monitor IP if it was not passed 2022-05-10 12:41:21,270 p=106241 u=stack n=ansible | 2022-05-10 12:41:21.268865 | 48d539a1-1679-39d3-e106-000000000022 | SKIPPED | Set first monitor IP if it was not passed | overcloud-controller-0 2022-05-10 12:41:21,306 p=106241 u=stack n=ansible | 2022-05-10 12:41:21.304747 | 48d539a1-1679-39d3-e106-000000000023 | TASK | Ensure tripleo_cephadm_config_home (e.g. /etc/ceph) exists 2022-05-10 12:41:22,994 p=106241 u=stack n=ansible | 2022-05-10 12:41:22.992953 | 48d539a1-1679-39d3-e106-000000000023 | OK | Ensure tripleo_cephadm_config_home (e.g. /etc/ceph) exists | overcloud-controller-0 2022-05-10 12:41:23,073 p=106241 u=stack n=ansible | 2022-05-10 12:41:23.071975 | 48d539a1-1679-39d3-e106-000000000024 | TASK | Ensure specs directory exists 2022-05-10 12:41:23,484 p=106241 u=stack n=ansible | 2022-05-10 12:41:23.482441 | 48d539a1-1679-39d3-e106-000000000024 | OK | Ensure specs directory exists | overcloud-controller-0 2022-05-10 12:41:23,620 p=106241 u=stack n=ansible | 2022-05-10 12:41:23.619287 | 48d539a1-1679-39d3-e106-000000000025 | TASK | Stat spec file on ansible host 2022-05-10 12:41:24,611 p=106241 u=stack n=ansible | 2022-05-10 12:41:24.609235 | 48d539a1-1679-39d3-e106-000000000025 | OK | Stat spec file on ansible host | overcloud-controller-0 -> localhost 2022-05-10 12:41:24,614 p=106241 u=stack n=ansible | [WARNING]: ('overcloud-controller-0 -> localhost', '48d539a1-1679-39d3-e106-000000000025') missing from stats 2022-05-10 12:41:24,727 p=106241 u=stack n=ansible | 2022-05-10 12:41:24.725991 | 48d539a1-1679-39d3-e106-000000000026 | TASK | push tripleo_cephadm_spec to bootstrap node if spec file exsits 2022-05-10 12:41:27,226 p=106241 u=stack n=ansible | 2022-05-10 12:41:27.224825 | 48d539a1-1679-39d3-e106-000000000026 | CHANGED | push tripleo_cephadm_spec to bootstrap node if spec file exsits | overcloud-controller-0 2022-05-10 12:41:27,357 p=106241 u=stack n=ansible | 2022-05-10 12:41:27.316585 | 48d539a1-1679-39d3-e106-000000000028 | TASK | stat tripleo_cephadm_bootstrap_conf file 2022-05-10 12:41:27,523 p=106241 u=stack n=ansible | 2022-05-10 12:41:27.521970 | 48d539a1-1679-39d3-e106-000000000028 | SKIPPED | stat tripleo_cephadm_bootstrap_conf file | overcloud-controller-0 2022-05-10 12:41:27,563 p=106241 u=stack n=ansible | 2022-05-10 12:41:27.554114 | 48d539a1-1679-39d3-e106-000000000029 | TASK | push it to tripleo_cephadm_assimilate_conf on bootstrap node 2022-05-10 12:41:27,767 p=106241 u=stack n=ansible | 2022-05-10 12:41:27.765549 | 48d539a1-1679-39d3-e106-000000000029 | SKIPPED | push it to tripleo_cephadm_assimilate_conf on bootstrap node | overcloud-controller-0 2022-05-10 12:41:27,841 p=106241 u=stack n=ansible | 2022-05-10 12:41:27.840315 | 48d539a1-1679-39d3-e106-00000000002b | TASK | generate tripleo_cephadm_assimilate_conf on bootstrap node 2022-05-10 12:41:28,567 p=106241 u=stack n=ansible | 2022-05-10 12:41:28.565119 | 48d539a1-1679-39d3-e106-00000000002b | CHANGED | generate tripleo_cephadm_assimilate_conf on bootstrap node | overcloud-controller-0 2022-05-10 12:41:28,645 p=106241 u=stack n=ansible | 2022-05-10 12:41:28.618418 | 48d539a1-1679-39d3-e106-000000000056 | TASK | Add spec to necessary file list when using spec to bootstrap 2022-05-10 12:41:28,886 p=106241 u=stack n=ansible | 2022-05-10 12:41:28.884125 | 48d539a1-1679-39d3-e106-000000000056 | SKIPPED | Add spec to necessary file list when using spec to bootstrap | overcloud-controller-0 2022-05-10 12:41:28,930 p=106241 u=stack n=ansible | 2022-05-10 12:41:28.928545 | 48d539a1-1679-39d3-e106-000000000057 | TASK | Stat necessary files to bootstrap with cephadm 2022-05-10 12:41:29,637 p=106241 u=stack n=ansible | 2022-05-10 12:41:29.635496 | 48d539a1-1679-39d3-e106-000000000057 | OK | Stat necessary files to bootstrap with cephadm | overcloud-controller-0 | item=/home/ceph-admin/.ssh/id_rsa 2022-05-10 12:41:29,851 p=106241 u=stack n=ansible | 2022-05-10 12:41:29.849882 | 48d539a1-1679-39d3-e106-000000000057 | OK | Stat necessary files to bootstrap with cephadm | overcloud-controller-0 | item=/home/ceph-admin/.ssh/id_rsa.pub 2022-05-10 12:41:29,893 p=106241 u=stack n=ansible | 2022-05-10 12:41:29.892211 | 48d539a1-1679-39d3-e106-000000000058 | TASK | Fail if necessary files are missing 2022-05-10 12:41:30,698 p=106241 u=stack n=ansible | 2022-05-10 12:41:30.695804 | 48d539a1-1679-39d3-e106-000000000058 | SKIPPED | Fail if necessary files are missing | overcloud-controller-0 | item={'changed': False, 'stat': {'exists': True, 'path': '/home/ceph-admin/.ssh/id_rsa', 'mode': '0600', 'isdir': False, 'ischr': False, 'isblk': False, 'isreg': True, 'isfifo': False, 'islnk': False, 'issock': False, 'uid': 1002, 'gid': 1003, 'size': 2622, 'inode': 1556100405, 'dev': 2051, 'nlink': 1, 'atime': 1652151496.6861079, 'mtime': 1652148449.3339396, 'ctime': 1652148574.8021252, 'wusr': True, 'rusr': True, 'xusr': False, 'wgrp': False, 'rgrp': False, 'xgrp': False, 'woth': False, 'roth': False, 'xoth': False, 'isuid': False, 'isgid': False, 'blocks': 8, 'block_size': 4096, 'device_type': 0, 'readable': True, 'writeable': True, 'executable': False, 'pw_name': 'ceph-admin', 'gr_name': 'ceph-admin', 'checksum': '0649ed9fc53f0156b01548e62a962bdeee432bb7', 'mimetype': 'text/plain', 'charset': 'us-ascii', 'version': '2011066949', 'attributes': [], 'attr_flags': ''}, 'invocation': {'module_args': {'path': '/home/ceph-admin/.ssh/id_rsa', 'follow': False, 'get_md5': False, 'get_checksum': True, 'get_mime': True, 'get_attributes': True, 'checksum_algorithm': 'sha1'}}, 'failed': False, 'item': '/home/ceph-admin/.ssh/id_rsa', 'ansible_loop_var': 'item'} 2022-05-10 12:41:30,712 p=106241 u=stack n=ansible | 2022-05-10 12:41:30.711669 | 48d539a1-1679-39d3-e106-000000000058 | SKIPPED | Fail if necessary files are missing | overcloud-controller-0 | item={'changed': False, 'stat': {'exists': True, 'path': '/home/ceph-admin/.ssh/id_rsa.pub', 'mode': '0644', 'isdir': False, 'ischr': False, 'isblk': False, 'isreg': True, 'isfifo': False, 'islnk': False, 'issock': False, 'uid': 1002, 'gid': 1003, 'size': 553, 'inode': 1568730618, 'dev': 2051, 'nlink': 1, 'atime': 1652148575.0601258, 'mtime': 1652148451.1399422, 'ctime': 1652148574.8021252, 'wusr': True, 'rusr': True, 'xusr': False, 'wgrp': False, 'rgrp': True, 'xgrp': False, 'woth': False, 'roth': True, 'xoth': False, 'isuid': False, 'isgid': False, 'blocks': 8, 'block_size': 4096, 'device_type': 0, 'readable': True, 'writeable': True, 'executable': False, 'pw_name': 'ceph-admin', 'gr_name': 'ceph-admin', 'checksum': 'b26be79997a7ac1c2386c2cfda15efa3de7dc282', 'mimetype': 'text/plain', 'charset': 'us-ascii', 'version': '3465945859', 'attributes': [], 'attr_flags': ''}, 'invocation': {'module_args': {'path': '/home/ceph-admin/.ssh/id_rsa.pub', 'follow': False, 'get_md5': False, 'get_checksum': True, 'get_mime': True, 'get_attributes': True, 'checksum_algorithm': 'sha1'}}, 'failed': False, 'item': '/home/ceph-admin/.ssh/id_rsa.pub', 'ansible_loop_var': 'item'} 2022-05-10 12:41:30,878 p=106241 u=stack n=ansible | 2022-05-10 12:41:30.877620 | 48d539a1-1679-39d3-e106-000000000059 | TASK | Stat pre ceph conf file in case we should bootrap with it 2022-05-10 12:41:31,425 p=106241 u=stack n=ansible | 2022-05-10 12:41:31.424425 | 48d539a1-1679-39d3-e106-000000000059 | OK | Stat pre ceph conf file in case we should bootrap with it | overcloud-controller-0 2022-05-10 12:41:31,459 p=106241 u=stack n=ansible | 2022-05-10 12:41:31.458725 | 48d539a1-1679-39d3-e106-00000000005b | TASK | Run cephadm bootstrap 2022-05-10 12:41:31,769 p=106241 u=stack n=ansible | 2022-05-10 12:41:31.767217 | 48d539a1-1679-39d3-e106-00000000005b | SKIPPED | Run cephadm bootstrap | overcloud-controller-0 2022-05-10 12:41:31,816 p=106241 u=stack n=ansible | 2022-05-10 12:41:31.815556 | 48d539a1-1679-39d3-e106-00000000005c | TASK | Show results of bootstrap 2022-05-10 12:41:32,120 p=106241 u=stack n=ansible | 2022-05-10 12:41:32.117632 | 48d539a1-1679-39d3-e106-00000000005c | SKIPPED | Show results of bootstrap | overcloud-controller-0 2022-05-10 12:41:32,206 p=106241 u=stack n=ansible | 2022-05-10 12:41:32.205535 | 48d539a1-1679-39d3-e106-00000000005e | TASK | If cephadm bootstrap was not run report the reason 2022-05-10 12:41:32,522 p=106241 u=stack n=ansible | 2022-05-10 12:41:32.520833 | 48d539a1-1679-39d3-e106-00000000005e | OK | If cephadm bootstrap was not run report the reason | overcloud-controller-0 | result={ "changed": false, "msg": "'cephadm bootstrap' was not run because 'cephadm ls'\nindicates that Ceph containers are already running.\n" } 2022-05-10 12:41:32,626 p=106241 u=stack n=ansible | 2022-05-10 12:41:32.625642 | 48d539a1-1679-39d3-e106-000000000075 | TASK | Ensure tripleo_cephadm_ceph_cli is set 2022-05-10 12:41:32,679 p=106241 u=stack n=ansible | 2022-05-10 12:41:32.677299 | 48d539a1-1679-39d3-e106-000000000075 | SKIPPED | Ensure tripleo_cephadm_ceph_cli is set | overcloud-controller-0 2022-05-10 12:41:32,709 p=106241 u=stack n=ansible | 2022-05-10 12:41:32.708546 | 48d539a1-1679-39d3-e106-000000000076 | TASK | Assimilate configuration from tripleo_cephadm_assimilate_conf 2022-05-10 12:41:32,745 p=106241 u=stack n=ansible | 2022-05-10 12:41:32.743717 | 48d539a1-1679-39d3-e106-000000000076 | SKIPPED | Assimilate configuration from tripleo_cephadm_assimilate_conf | overcloud-controller-0 2022-05-10 12:41:32,775 p=106241 u=stack n=ansible | 2022-05-10 12:41:32.774302 | 48d539a1-1679-39d3-e106-000000000081 | TASK | Stat spec file on bootstrap node 2022-05-10 12:41:33,309 p=106241 u=stack n=ansible | 2022-05-10 12:41:33.307774 | 48d539a1-1679-39d3-e106-000000000081 | OK | Stat spec file on bootstrap node | overcloud-controller-0 | item=/home/ceph-admin/specs/ceph_spec.yaml 2022-05-10 12:41:33,356 p=106241 u=stack n=ansible | 2022-05-10 12:41:33.355048 | 48d539a1-1679-39d3-e106-000000000082 | TASK | Fail if spec file is missing 2022-05-10 12:41:33,770 p=106241 u=stack n=ansible | 2022-05-10 12:41:33.769271 | 48d539a1-1679-39d3-e106-000000000082 | SKIPPED | Fail if spec file is missing | overcloud-controller-0 | item={'changed': False, 'stat': {'exists': True, 'path': '/home/ceph-admin/specs/ceph_spec.yaml', 'mode': '0644', 'isdir': False, 'ischr': False, 'isblk': False, 'isreg': True, 'isfifo': False, 'islnk': False, 'issock': False, 'uid': 1002, 'gid': 1003, 'size': 1248, 'inode': 1396737357, 'dev': 2051, 'nlink': 1, 'atime': 1652157687.1003287, 'mtime': 1652157685.323326, 'ctime': 1652157687.1003287, 'wusr': True, 'rusr': True, 'xusr': False, 'wgrp': False, 'rgrp': True, 'xgrp': False, 'woth': False, 'roth': True, 'xoth': False, 'isuid': False, 'isgid': False, 'blocks': 8, 'block_size': 4096, 'device_type': 0, 'readable': True, 'writeable': True, 'executable': False, 'pw_name': 'ceph-admin', 'gr_name': 'ceph-admin', 'checksum': '0910be9ef664e22b0c17eca3047763bf04e3dfc7', 'mimetype': 'text/plain', 'charset': 'us-ascii', 'version': '993461113', 'attributes': [], 'attr_flags': ''}, 'invocation': {'module_args': {'path': '/home/ceph-admin/specs/ceph_spec.yaml', 'follow': False, 'get_md5': False, 'get_checksum': True, 'get_mime': True, 'get_attributes': True, 'checksum_algorithm': 'sha1'}}, 'failed': False, 'item': '/home/ceph-admin/specs/ceph_spec.yaml', 'ansible_loop_var': 'item'} 2022-05-10 12:41:33,809 p=106241 u=stack n=ansible | 2022-05-10 12:41:33.808543 | 48d539a1-1679-39d3-e106-000000000083 | TASK | Get ceph_cli 2022-05-10 12:41:33,989 p=106241 u=stack n=ansible | 2022-05-10 12:41:33.988199 | 2d4655ca-f276-4764-9d58-724250e3424d | INCLUDED | /usr/share/ansible/roles/tripleo_cephadm/tasks/ceph_cli.yaml | overcloud-controller-0 2022-05-10 12:41:34,026 p=106241 u=stack n=ansible | 2022-05-10 12:41:34.025840 | 48d539a1-1679-39d3-e106-0000000001fb | TASK | Set ceph CLI 2022-05-10 12:41:34,235 p=106241 u=stack n=ansible | 2022-05-10 12:41:34.233477 | 48d539a1-1679-39d3-e106-0000000001fb | OK | Set ceph CLI | overcloud-controller-0 2022-05-10 12:41:34,274 p=106241 u=stack n=ansible | 2022-05-10 12:41:34.273306 | 48d539a1-1679-39d3-e106-000000000084 | TASK | Get the ceph orchestrator status 2022-05-10 12:41:35,400 p=106241 u=stack n=ansible | 2022-05-10 12:41:35.399290 | 48d539a1-1679-39d3-e106-000000000084 | CHANGED | Get the ceph orchestrator status | overcloud-controller-0 2022-05-10 12:41:35,433 p=106241 u=stack n=ansible | 2022-05-10 12:41:35.431787 | 48d539a1-1679-39d3-e106-000000000085 | TASK | Fail if ceph orchestrator is not available 2022-05-10 12:41:35,599 p=106241 u=stack n=ansible | 2022-05-10 12:41:35.598342 | 48d539a1-1679-39d3-e106-000000000085 | SKIPPED | Fail if ceph orchestrator is not available | overcloud-controller-0 2022-05-10 12:41:35,631 p=106241 u=stack n=ansible | 2022-05-10 12:41:35.629948 | 48d539a1-1679-39d3-e106-000000000086 | TASK | Set container images used by the applied services 2022-05-10 12:41:35,831 p=106241 u=stack n=ansible | 2022-05-10 12:41:35.829852 | 47f5779a-fc04-41f4-b5a0-14c7cbb6c0e3 | INCLUDED | /usr/share/ansible/roles/tripleo_cephadm/tasks/cephadm_config_set.yaml | overcloud-controller-0 2022-05-10 12:41:35,877 p=106241 u=stack n=ansible | 2022-05-10 12:41:35.875881 | 48d539a1-1679-39d3-e106-000000000232 | TASK | set cephadm debug level 2022-05-10 12:41:36,033 p=106241 u=stack n=ansible | 2022-05-10 12:41:36.031965 | 48d539a1-1679-39d3-e106-000000000232 | SKIPPED | set cephadm debug level | overcloud-controller-0 2022-05-10 12:41:36,063 p=106241 u=stack n=ansible | 2022-05-10 12:41:36.061958 | 48d539a1-1679-39d3-e106-000000000233 | TASK | set default container image in ceph configuration 2022-05-10 12:41:37,183 p=106241 u=stack n=ansible | 2022-05-10 12:41:37.182336 | 48d539a1-1679-39d3-e106-000000000233 | OK | set default container image in ceph configuration | overcloud-controller-0 2022-05-10 12:41:37,216 p=106241 u=stack n=ansible | 2022-05-10 12:41:37.214692 | 48d539a1-1679-39d3-e106-000000000234 | TASK | enforce minimum ceph clients version 2022-05-10 12:41:37,383 p=106241 u=stack n=ansible | 2022-05-10 12:41:37.382244 | 48d539a1-1679-39d3-e106-000000000234 | SKIPPED | enforce minimum ceph clients version | overcloud-controller-0 2022-05-10 12:41:37,414 p=106241 u=stack n=ansible | 2022-05-10 12:41:37.413469 | 48d539a1-1679-39d3-e106-000000000235 | TASK | set container image base in ceph configuration 2022-05-10 12:41:38,529 p=106241 u=stack n=ansible | 2022-05-10 12:41:38.527656 | 48d539a1-1679-39d3-e106-000000000235 | OK | set container image base in ceph configuration | overcloud-controller-0 2022-05-10 12:41:38,561 p=106241 u=stack n=ansible | 2022-05-10 12:41:38.560540 | 48d539a1-1679-39d3-e106-000000000237 | TASK | set alertmanager container image in ceph configuration 2022-05-10 12:41:38,717 p=106241 u=stack n=ansible | 2022-05-10 12:41:38.716255 | 48d539a1-1679-39d3-e106-000000000237 | SKIPPED | set alertmanager container image in ceph configuration | overcloud-controller-0 2022-05-10 12:41:38,749 p=106241 u=stack n=ansible | 2022-05-10 12:41:38.748282 | 48d539a1-1679-39d3-e106-000000000238 | TASK | set grafana container image in ceph configuration 2022-05-10 12:41:38,969 p=106241 u=stack n=ansible | 2022-05-10 12:41:38.968352 | 48d539a1-1679-39d3-e106-000000000238 | SKIPPED | set grafana container image in ceph configuration | overcloud-controller-0 2022-05-10 12:41:39,007 p=106241 u=stack n=ansible | 2022-05-10 12:41:39.005756 | 48d539a1-1679-39d3-e106-000000000239 | TASK | set node-exporter container image in ceph configuration 2022-05-10 12:41:39,166 p=106241 u=stack n=ansible | 2022-05-10 12:41:39.165330 | 48d539a1-1679-39d3-e106-000000000239 | SKIPPED | set node-exporter container image in ceph configuration | overcloud-controller-0 2022-05-10 12:41:39,196 p=106241 u=stack n=ansible | 2022-05-10 12:41:39.195241 | 48d539a1-1679-39d3-e106-00000000023a | TASK | set prometheus container image in ceph configuration 2022-05-10 12:41:39,358 p=106241 u=stack n=ansible | 2022-05-10 12:41:39.357269 | 48d539a1-1679-39d3-e106-00000000023a | SKIPPED | set prometheus container image in ceph configuration | overcloud-controller-0 2022-05-10 12:41:39,391 p=106241 u=stack n=ansible | 2022-05-10 12:41:39.389938 | 48d539a1-1679-39d3-e106-000000000087 | TASK | Apply spec 2022-05-10 12:41:43,938 p=106241 u=stack n=ansible | 2022-05-10 12:41:43.936604 | 48d539a1-1679-39d3-e106-000000000087 | CHANGED | Apply spec | overcloud-controller-0 2022-05-10 12:41:43,974 p=106241 u=stack n=ansible | 2022-05-10 12:41:43.972952 | 48d539a1-1679-39d3-e106-000000000088 | TASK | Show results of spec apply 2022-05-10 12:41:44,128 p=106241 u=stack n=ansible | 2022-05-10 12:41:44.127328 | 48d539a1-1679-39d3-e106-000000000088 | SKIPPED | Show results of spec apply | overcloud-controller-0 2022-05-10 12:41:44,161 p=106241 u=stack n=ansible | 2022-05-10 12:41:44.159933 | 48d539a1-1679-39d3-e106-000000000089 | TASK | Wait for the expected number of osds to be running 2022-05-10 12:41:44,303 p=107679 u=stack n=ansible | [DEPRECATION WARNING]: evaluating 'tripleo_cephadm_wait_for_osds' as a bare variable, this behaviour will go away and you might need to add |bool to the expression in the future. Also see CONDITIONAL_BARE_VARS configuration toggle. This feature will be removed in version 2.12. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2022-05-10 12:41:44,366 p=106241 u=stack n=ansible | 2022-05-10 12:41:44.365397 | 274942a0-bbee-417c-a0aa-3c4c0aecb6f8 | INCLUDED | /usr/share/ansible/roles/tripleo_cephadm/tasks/wait_for_expected_num_osds.yaml | overcloud-controller-0 2022-05-10 12:41:44,416 p=106241 u=stack n=ansible | 2022-05-10 12:41:44.415536 | 48d539a1-1679-39d3-e106-000000000291 | TASK | Wait for expected number of osds to be running 2022-05-10 13:02:25,900 p=106241 u=stack n=ansible | 2022-05-10 13:02:25.898199 | 48d539a1-1679-39d3-e106-000000000291 | FATAL | Wait for expected number of osds to be running | overcloud-controller-0 | error={"attempts": 40, "changed": true, "cmd": "podman run --rm --net=host --ipc=host --volume /etc/ceph:/etc/ceph:z --volume /home/ceph-admin/assimilate_ceph.conf:/home/assimilate_ceph.conf:z --volume /home/ceph-admin/specs/ceph_spec.yaml:/home/ceph_spec.yaml:z --entrypoint ceph 172.25.201.68:8787/ceph/daemon:v6.0.6-stable-6.0-pacific-centos-8-x86_64 --fsid 2dbb93bd-e7ae-4a34-94e5-231eb65202dd -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring status --format json | jq .osdmap.num_up_osds", "delta": "0:00:00.748517", "end": "2022-05-10 05:02:25.834500", "rc": 0, "start": "2022-05-10 05:02:25.085983", "stderr": "", "stderr_lines": [], "stdout": "0", "stdout_lines": ["0"]} 2022-05-10 13:02:25,917 p=106241 u=stack n=ansible | PLAY RECAP ********************************************************************* 2022-05-10 13:02:25,918 p=106241 u=stack n=ansible | overcloud-controller-0 : ok=20 changed=5 unreachable=0 failed=1 skipped=23 rescued=0 ignored=0 2022-05-10 13:02:25,920 p=106241 u=stack n=ansible | 2022-05-10 13:02:25.920251 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Summary Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2022-05-10 13:02:25,921 p=106241 u=stack n=ansible | 2022-05-10 13:02:25.921205 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Total Tasks: 44 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2022-05-10 13:02:25,922 p=106241 u=stack n=ansible | 2022-05-10 13:02:25.922094 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Elapsed Time: 0:21:11.305989 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2022-05-10 13:02:25,923 p=106241 u=stack n=ansible | 2022-05-10 13:02:25.922845 | UUID | Info | Host | Task Name | Run Time 2022-05-10 13:02:25,924 p=106241 u=stack n=ansible | 2022-05-10 13:02:25.923600 | 48d539a1-1679-39d3-e106-000000000291 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Wait for expected number of osds to be running | 1241.48s 2022-05-10 13:02:25,924 p=106241 u=stack n=ansible | 2022-05-10 13:02:25.924353 | 48d539a1-1679-39d3-e106-000000000087 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Apply spec | 4.55s 2022-05-10 13:02:25,925 p=106241 u=stack n=ansible | 2022-05-10 13:02:25.925097 | 48d539a1-1679-39d3-e106-000000000026 | SUMMARY | overcloud-controller-0 | push tripleo_cephadm_spec to bootstrap node if spec file exsits | 2.50s 2022-05-10 13:02:25,926 p=106241 u=stack n=ansible | 2022-05-10 13:02:25.925799 | 48d539a1-1679-39d3-e106-000000000018 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Install cephadm package | 2.33s 2022-05-10 13:02:25,927 p=106241 u=stack n=ansible | 2022-05-10 13:02:25.926536 | 48d539a1-1679-39d3-e106-000000000023 | SUMMARY | overcloud-controller-0 | Ensure tripleo_cephadm_config_home (e.g. /etc/ceph) exists | 1.64s 2022-05-10 13:02:25,928 p=106241 u=stack n=ansible | 2022-05-10 13:02:25.927466 | 48d539a1-1679-39d3-e106-00000000001b | SUMMARY | overcloud-controller-0 | tripleo_cephadm : List Ceph daemon instances on this host | 1.46s 2022-05-10 13:02:25,928 p=106241 u=stack n=ansible | 2022-05-10 13:02:25.928283 | 48d539a1-1679-39d3-e106-000000000084 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Get the ceph orchestrator status | 1.13s 2022-05-10 13:02:25,929 p=106241 u=stack n=ansible | 2022-05-10 13:02:25.929003 | 48d539a1-1679-39d3-e106-000000000233 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : set default container image in ceph configuration | 1.12s 2022-05-10 13:02:25,930 p=106241 u=stack n=ansible | 2022-05-10 13:02:25.929762 | 48d539a1-1679-39d3-e106-000000000235 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : set container image base in ceph configuration | 1.12s 2022-05-10 13:02:25,931 p=106241 u=stack n=ansible | 2022-05-10 13:02:25.930512 | 48d539a1-1679-39d3-e106-000000000057 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Stat necessary files to bootstrap with cephadm | 0.93s 2022-05-10 13:02:25,931 p=106241 u=stack n=ansible | 2022-05-10 13:02:25.931222 | 48d539a1-1679-39d3-e106-000000000058 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Fail if necessary files are missing | 0.81s 2022-05-10 13:02:25,932 p=106241 u=stack n=ansible | 2022-05-10 13:02:25.931942 | 48d539a1-1679-39d3-e106-000000000019 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Stat cephadm file | 0.76s 2022-05-10 13:02:25,933 p=106241 u=stack n=ansible | 2022-05-10 13:02:25.932664 | 48d539a1-1679-39d3-e106-00000000002b | SUMMARY | overcloud-controller-0 | generate tripleo_cephadm_assimilate_conf on bootstrap node | 0.73s 2022-05-10 13:02:25,933 p=106241 u=stack n=ansible | 2022-05-10 13:02:25.933385 | 48d539a1-1679-39d3-e106-000000000059 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Stat pre ceph conf file in case we should bootrap with it | 0.55s 2022-05-10 13:02:25,934 p=106241 u=stack n=ansible | 2022-05-10 13:02:25.934108 | 48d539a1-1679-39d3-e106-000000000081 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Stat spec file on bootstrap node | 0.54s 2022-05-10 13:02:25,935 p=106241 u=stack n=ansible | 2022-05-10 13:02:25.934977 | 48d539a1-1679-39d3-e106-000000000082 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Fail if spec file is missing | 0.42s 2022-05-10 13:02:25,936 p=106241 u=stack n=ansible | 2022-05-10 13:02:25.935635 | 48d539a1-1679-39d3-e106-000000000024 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Ensure specs directory exists | 0.41s 2022-05-10 13:02:25,936 p=106241 u=stack n=ansible | 2022-05-10 13:02:25.936284 | 48d539a1-1679-39d3-e106-00000000005e | SUMMARY | overcloud-controller-0 | tripleo_cephadm : If cephadm bootstrap was not run report the reason | 0.32s 2022-05-10 13:02:25,937 p=106241 u=stack n=ansible | 2022-05-10 13:02:25.936894 | 48d539a1-1679-39d3-e106-00000000001d | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Set list of found FSIDs | 0.31s 2022-05-10 13:02:25,938 p=106241 u=stack n=ansible | 2022-05-10 13:02:25.937499 | 48d539a1-1679-39d3-e106-00000000005b | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Run cephadm bootstrap | 0.31s 2022-05-10 13:02:25,939 p=106241 u=stack n=ansible | 2022-05-10 13:02:25.938355 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ End Summary Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2022-05-10 13:02:25,939 p=106241 u=stack n=ansible | 2022-05-10 13:02:25.939276 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ State Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2022-05-10 13:02:25,940 p=106241 u=stack n=ansible | 2022-05-10 13:02:25.940172 | ~~~~~~~~~~~~~~~~~~ Number of nodes which did not deploy successfully: 1 ~~~~~~~~~~~~~~~~~ 2022-05-10 13:02:25,941 p=106241 u=stack n=ansible | 2022-05-10 13:02:25.941028 | The following node(s) had failures: overcloud-controller-0 2022-05-10 13:02:25,942 p=106241 u=stack n=ansible | 2022-05-10 13:02:25.941772 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Running /home/stack/overcloud-deploy/overcloud/config-download/overcloud/cephadm/cephadm_command.sh 2022-05-11 13:33:09,518 p=335214 u=stack n=ansible | PLAY [Deploy Ceph with cephadm] ************************************************ 2022-05-11 13:33:09,602 p=335214 u=stack n=ansible | 2022-05-11 13:33:09.600964 | 48d539a1-1679-6875-cb99-000000000018 | TASK | Install cephadm package 2022-05-11 13:33:09,754 p=335214 u=stack n=ansible | 2022-05-11 13:33:09.752695 | 48d539a1-1679-6875-cb99-000000000018 | SKIPPED | Install cephadm package | overcloud-controller-0 2022-05-11 13:33:09,787 p=335214 u=stack n=ansible | 2022-05-11 13:33:09.786096 | 48d539a1-1679-6875-cb99-000000000019 | TASK | Stat cephadm file 2022-05-11 13:33:10,605 p=335214 u=stack n=ansible | 2022-05-11 13:33:10.603242 | 48d539a1-1679-6875-cb99-000000000019 | OK | Stat cephadm file | overcloud-controller-0 2022-05-11 13:33:10,641 p=335214 u=stack n=ansible | 2022-05-11 13:33:10.640239 | 48d539a1-1679-6875-cb99-00000000001a | TASK | Fail if cephadm is not available 2022-05-11 13:33:10,850 p=335214 u=stack n=ansible | 2022-05-11 13:33:10.848885 | 48d539a1-1679-6875-cb99-00000000001a | SKIPPED | Fail if cephadm is not available | overcloud-controller-0 2022-05-11 13:33:10,894 p=335214 u=stack n=ansible | 2022-05-11 13:33:10.890511 | 48d539a1-1679-6875-cb99-00000000001b | TASK | List Ceph daemon instances on this host 2022-05-11 13:33:12,218 p=335214 u=stack n=ansible | 2022-05-11 13:33:12.216244 | 48d539a1-1679-6875-cb99-00000000001b | CHANGED | List Ceph daemon instances on this host | overcloud-controller-0 2022-05-11 13:33:12,252 p=335214 u=stack n=ansible | 2022-05-11 13:33:12.250786 | 48d539a1-1679-6875-cb99-00000000001d | TASK | Set list of found FSIDs 2022-05-11 13:33:12,402 p=335214 u=stack n=ansible | 2022-05-11 13:33:12.400309 | 48d539a1-1679-6875-cb99-00000000001d | SKIPPED | Set list of found FSIDs | overcloud-controller-0 2022-05-11 13:33:12,434 p=335214 u=stack n=ansible | 2022-05-11 13:33:12.431450 | 48d539a1-1679-6875-cb99-00000000001e | TASK | Fail if >1 FSID was discovered 2022-05-11 13:33:12,602 p=335214 u=stack n=ansible | 2022-05-11 13:33:12.600379 | 48d539a1-1679-6875-cb99-00000000001e | SKIPPED | Fail if >1 FSID was discovered | overcloud-controller-0 2022-05-11 13:33:12,634 p=335214 u=stack n=ansible | 2022-05-11 13:33:12.633058 | 48d539a1-1679-6875-cb99-00000000001f | TASK | Set FSID to the discovered value 2022-05-11 13:33:12,801 p=335214 u=stack n=ansible | 2022-05-11 13:33:12.799332 | 48d539a1-1679-6875-cb99-00000000001f | SKIPPED | Set FSID to the discovered value | overcloud-controller-0 2022-05-11 13:33:12,833 p=335214 u=stack n=ansible | 2022-05-11 13:33:12.832170 | 48d539a1-1679-6875-cb99-000000000020 | TASK | Set random tripleo_cephadm_fsid if no running ceph containers were found 2022-05-11 13:33:12,992 p=335214 u=stack n=ansible | 2022-05-11 13:33:12.990813 | 48d539a1-1679-6875-cb99-000000000020 | SKIPPED | Set random tripleo_cephadm_fsid if no running ceph containers were found | overcloud-controller-0 2022-05-11 13:33:13,029 p=335214 u=stack n=ansible | 2022-05-11 13:33:13.028664 | 48d539a1-1679-6875-cb99-000000000022 | TASK | Set first monitor IP if it was not passed 2022-05-11 13:33:13,183 p=335214 u=stack n=ansible | 2022-05-11 13:33:13.181379 | 48d539a1-1679-6875-cb99-000000000022 | SKIPPED | Set first monitor IP if it was not passed | overcloud-controller-0 2022-05-11 13:33:13,216 p=335214 u=stack n=ansible | 2022-05-11 13:33:13.215596 | 48d539a1-1679-6875-cb99-000000000023 | TASK | Ensure tripleo_cephadm_config_home (e.g. /etc/ceph) exists 2022-05-11 13:33:14,029 p=335214 u=stack n=ansible | 2022-05-11 13:33:14.027743 | 48d539a1-1679-6875-cb99-000000000023 | OK | Ensure tripleo_cephadm_config_home (e.g. /etc/ceph) exists | overcloud-controller-0 2022-05-11 13:33:14,064 p=335214 u=stack n=ansible | 2022-05-11 13:33:14.063013 | 48d539a1-1679-6875-cb99-000000000024 | TASK | Ensure specs directory exists 2022-05-11 13:33:14,427 p=335214 u=stack n=ansible | 2022-05-11 13:33:14.425944 | 48d539a1-1679-6875-cb99-000000000024 | OK | Ensure specs directory exists | overcloud-controller-0 2022-05-11 13:33:14,469 p=335214 u=stack n=ansible | 2022-05-11 13:33:14.468303 | 48d539a1-1679-6875-cb99-000000000025 | TASK | Stat spec file on ansible host 2022-05-11 13:33:15,620 p=335214 u=stack n=ansible | 2022-05-11 13:33:15.619176 | 48d539a1-1679-6875-cb99-000000000025 | OK | Stat spec file on ansible host | overcloud-controller-0 -> localhost 2022-05-11 13:33:15,624 p=335214 u=stack n=ansible | [WARNING]: ('overcloud-controller-0 -> localhost', '48d539a1-1679-6875-cb99-000000000025') missing from stats 2022-05-11 13:33:15,655 p=335214 u=stack n=ansible | 2022-05-11 13:33:15.654038 | 48d539a1-1679-6875-cb99-000000000026 | TASK | push tripleo_cephadm_spec to bootstrap node if spec file exsits 2022-05-11 13:33:17,945 p=335214 u=stack n=ansible | 2022-05-11 13:33:17.943785 | 48d539a1-1679-6875-cb99-000000000026 | CHANGED | push tripleo_cephadm_spec to bootstrap node if spec file exsits | overcloud-controller-0 2022-05-11 13:33:17,989 p=335214 u=stack n=ansible | 2022-05-11 13:33:17.988016 | 48d539a1-1679-6875-cb99-000000000028 | TASK | stat tripleo_cephadm_bootstrap_conf file 2022-05-11 13:33:18,153 p=335214 u=stack n=ansible | 2022-05-11 13:33:18.152346 | 48d539a1-1679-6875-cb99-000000000028 | SKIPPED | stat tripleo_cephadm_bootstrap_conf file | overcloud-controller-0 2022-05-11 13:33:18,184 p=335214 u=stack n=ansible | 2022-05-11 13:33:18.183601 | 48d539a1-1679-6875-cb99-000000000029 | TASK | push it to tripleo_cephadm_assimilate_conf on bootstrap node 2022-05-11 13:33:18,338 p=335214 u=stack n=ansible | 2022-05-11 13:33:18.336810 | 48d539a1-1679-6875-cb99-000000000029 | SKIPPED | push it to tripleo_cephadm_assimilate_conf on bootstrap node | overcloud-controller-0 2022-05-11 13:33:18,373 p=335214 u=stack n=ansible | 2022-05-11 13:33:18.371950 | 48d539a1-1679-6875-cb99-00000000002b | TASK | generate tripleo_cephadm_assimilate_conf on bootstrap node 2022-05-11 13:33:19,171 p=335214 u=stack n=ansible | 2022-05-11 13:33:19.169683 | 48d539a1-1679-6875-cb99-00000000002b | CHANGED | generate tripleo_cephadm_assimilate_conf on bootstrap node | overcloud-controller-0 2022-05-11 13:33:19,206 p=335214 u=stack n=ansible | 2022-05-11 13:33:19.205036 | 48d539a1-1679-6875-cb99-000000000056 | TASK | Add spec to necessary file list when using spec to bootstrap 2022-05-11 13:33:19,358 p=335214 u=stack n=ansible | 2022-05-11 13:33:19.357320 | 48d539a1-1679-6875-cb99-000000000056 | SKIPPED | Add spec to necessary file list when using spec to bootstrap | overcloud-controller-0 2022-05-11 13:33:19,391 p=335214 u=stack n=ansible | 2022-05-11 13:33:19.389810 | 48d539a1-1679-6875-cb99-000000000057 | TASK | Stat necessary files to bootstrap with cephadm 2022-05-11 13:33:19,577 p=335214 u=stack n=ansible | 2022-05-11 13:33:19.575412 | 48d539a1-1679-6875-cb99-000000000057 | SKIPPED | Stat necessary files to bootstrap with cephadm | overcloud-controller-0 | item=/home/ceph-admin/.ssh/id_rsa 2022-05-11 13:33:19,593 p=335214 u=stack n=ansible | 2022-05-11 13:33:19.592752 | 48d539a1-1679-6875-cb99-000000000057 | SKIPPED | Stat necessary files to bootstrap with cephadm | overcloud-controller-0 | item=/home/ceph-admin/.ssh/id_rsa.pub 2022-05-11 13:33:19,627 p=335214 u=stack n=ansible | 2022-05-11 13:33:19.625961 | 48d539a1-1679-6875-cb99-000000000058 | TASK | Fail if necessary files are missing 2022-05-11 13:33:19,806 p=335214 u=stack n=ansible | 2022-05-11 13:33:19.804519 | 48d539a1-1679-6875-cb99-000000000058 | SKIPPED | Fail if necessary files are missing | overcloud-controller-0 | item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': '/home/ceph-admin/.ssh/id_rsa', 'ansible_loop_var': 'item'} 2022-05-11 13:33:19,820 p=335214 u=stack n=ansible | 2022-05-11 13:33:19.819041 | 48d539a1-1679-6875-cb99-000000000058 | SKIPPED | Fail if necessary files are missing | overcloud-controller-0 | item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': '/home/ceph-admin/.ssh/id_rsa.pub', 'ansible_loop_var': 'item'} 2022-05-11 13:33:19,854 p=335214 u=stack n=ansible | 2022-05-11 13:33:19.853011 | 48d539a1-1679-6875-cb99-000000000059 | TASK | Stat pre ceph conf file in case we should bootrap with it 2022-05-11 13:33:20,311 p=335214 u=stack n=ansible | 2022-05-11 13:33:20.309517 | 48d539a1-1679-6875-cb99-000000000059 | SKIPPED | Stat pre ceph conf file in case we should bootrap with it | overcloud-controller-0 2022-05-11 13:33:20,346 p=335214 u=stack n=ansible | 2022-05-11 13:33:20.344720 | 48d539a1-1679-6875-cb99-00000000005b | TASK | Run cephadm bootstrap 2022-05-11 13:33:20,884 p=335214 u=stack n=ansible | 2022-05-11 13:33:20.882415 | 48d539a1-1679-6875-cb99-00000000005b | SKIPPED | Run cephadm bootstrap | overcloud-controller-0 2022-05-11 13:33:20,917 p=335214 u=stack n=ansible | 2022-05-11 13:33:20.916547 | 48d539a1-1679-6875-cb99-00000000005c | TASK | Show results of bootstrap 2022-05-11 13:33:21,073 p=335214 u=stack n=ansible | 2022-05-11 13:33:21.072069 | 48d539a1-1679-6875-cb99-00000000005c | SKIPPED | Show results of bootstrap | overcloud-controller-0 2022-05-11 13:33:21,105 p=335214 u=stack n=ansible | 2022-05-11 13:33:21.103754 | 48d539a1-1679-6875-cb99-00000000005e | TASK | If cephadm bootstrap was not run report the reason 2022-05-11 13:33:21,262 p=335214 u=stack n=ansible | 2022-05-11 13:33:21.260354 | 48d539a1-1679-6875-cb99-00000000005e | SKIPPED | If cephadm bootstrap was not run report the reason | overcloud-controller-0 2022-05-11 13:33:21,296 p=335214 u=stack n=ansible | 2022-05-11 13:33:21.295479 | 48d539a1-1679-6875-cb99-000000000075 | TASK | Ensure tripleo_cephadm_ceph_cli is set 2022-05-11 13:33:21,483 p=335214 u=stack n=ansible | 2022-05-11 13:33:21.482355 | ee7cdea7-8435-409c-9f6d-b7a22de31f97 | INCLUDED | /usr/share/ansible/roles/tripleo_cephadm/tasks/ceph_cli.yaml | overcloud-controller-0 2022-05-11 13:33:21,517 p=335214 u=stack n=ansible | 2022-05-11 13:33:21.516407 | 48d539a1-1679-6875-cb99-0000000001f1 | TASK | Set ceph CLI 2022-05-11 13:33:21,857 p=335214 u=stack n=ansible | 2022-05-11 13:33:21.856376 | 48d539a1-1679-6875-cb99-0000000001f1 | OK | Set ceph CLI | overcloud-controller-0 2022-05-11 13:33:21,891 p=335214 u=stack n=ansible | 2022-05-11 13:33:21.889649 | 48d539a1-1679-6875-cb99-000000000076 | TASK | Assimilate configuration from tripleo_cephadm_assimilate_conf 2022-05-11 13:33:23,043 p=335214 u=stack n=ansible | 2022-05-11 13:33:23.041496 | 48d539a1-1679-6875-cb99-000000000076 | OK | Assimilate configuration from tripleo_cephadm_assimilate_conf | overcloud-controller-0 2022-05-11 13:33:23,076 p=335214 u=stack n=ansible | 2022-05-11 13:33:23.074874 | 48d539a1-1679-6875-cb99-000000000081 | TASK | Stat spec file on bootstrap node 2022-05-11 13:33:23,394 p=335214 u=stack n=ansible | 2022-05-11 13:33:23.392992 | 48d539a1-1679-6875-cb99-000000000081 | SKIPPED | Stat spec file on bootstrap node | overcloud-controller-0 | item=/home/ceph-admin/specs/ceph_spec.yaml 2022-05-11 13:33:23,426 p=335214 u=stack n=ansible | 2022-05-11 13:33:23.425387 | 48d539a1-1679-6875-cb99-000000000082 | TASK | Fail if spec file is missing 2022-05-11 13:33:23,756 p=335214 u=stack n=ansible | 2022-05-11 13:33:23.754420 | 48d539a1-1679-6875-cb99-000000000082 | SKIPPED | Fail if spec file is missing | overcloud-controller-0 | item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': '/home/ceph-admin/specs/ceph_spec.yaml', 'ansible_loop_var': 'item'} 2022-05-11 13:33:23,789 p=335214 u=stack n=ansible | 2022-05-11 13:33:23.787876 | 48d539a1-1679-6875-cb99-000000000083 | TASK | Get ceph_cli 2022-05-11 13:33:24,085 p=335214 u=stack n=ansible | 2022-05-11 13:33:24.084176 | 48d539a1-1679-6875-cb99-000000000083 | SKIPPED | Get ceph_cli | overcloud-controller-0 2022-05-11 13:33:24,120 p=335214 u=stack n=ansible | 2022-05-11 13:33:24.118959 | 48d539a1-1679-6875-cb99-000000000084 | TASK | Get the ceph orchestrator status 2022-05-11 13:33:24,411 p=335214 u=stack n=ansible | 2022-05-11 13:33:24.409598 | 48d539a1-1679-6875-cb99-000000000084 | SKIPPED | Get the ceph orchestrator status | overcloud-controller-0 2022-05-11 13:33:24,480 p=335214 u=stack n=ansible | 2022-05-11 13:33:24.479063 | 48d539a1-1679-6875-cb99-000000000085 | TASK | Fail if ceph orchestrator is not available 2022-05-11 13:33:25,040 p=335214 u=stack n=ansible | 2022-05-11 13:33:25.038636 | 48d539a1-1679-6875-cb99-000000000085 | SKIPPED | Fail if ceph orchestrator is not available | overcloud-controller-0 2022-05-11 13:33:25,069 p=335214 u=stack n=ansible | 2022-05-11 13:33:25.068183 | 48d539a1-1679-6875-cb99-000000000086 | TASK | Set container images used by the applied services 2022-05-11 13:33:25,226 p=335214 u=stack n=ansible | 2022-05-11 13:33:25.224359 | 48d539a1-1679-6875-cb99-000000000086 | SKIPPED | Set container images used by the applied services | overcloud-controller-0 2022-05-11 13:33:25,258 p=335214 u=stack n=ansible | 2022-05-11 13:33:25.256812 | 48d539a1-1679-6875-cb99-000000000087 | TASK | Apply spec 2022-05-11 13:33:25,413 p=335214 u=stack n=ansible | 2022-05-11 13:33:25.412048 | 48d539a1-1679-6875-cb99-000000000087 | SKIPPED | Apply spec | overcloud-controller-0 2022-05-11 13:33:25,461 p=335214 u=stack n=ansible | 2022-05-11 13:33:25.460380 | 48d539a1-1679-6875-cb99-000000000088 | TASK | Show results of spec apply 2022-05-11 13:33:25,645 p=335214 u=stack n=ansible | 2022-05-11 13:33:25.643526 | 48d539a1-1679-6875-cb99-000000000088 | SKIPPED | Show results of spec apply | overcloud-controller-0 2022-05-11 13:33:25,721 p=335214 u=stack n=ansible | 2022-05-11 13:33:25.720127 | 48d539a1-1679-6875-cb99-000000000089 | TASK | Wait for the expected number of osds to be running 2022-05-11 13:33:25,929 p=335214 u=stack n=ansible | 2022-05-11 13:33:25.928387 | 48d539a1-1679-6875-cb99-000000000089 | SKIPPED | Wait for the expected number of osds to be running | overcloud-controller-0 2022-05-11 13:33:25,983 p=335214 u=stack n=ansible | 2022-05-11 13:33:25.982254 | 48d539a1-1679-6875-cb99-0000000000a2 | TASK | Get ceph_cli 2022-05-11 13:33:26,228 p=335214 u=stack n=ansible | 2022-05-11 13:33:26.226839 | baf8c0dd-4cdc-4b76-959a-33124e7d67fe | INCLUDED | /usr/share/ansible/roles/tripleo_cephadm/tasks/ceph_cli.yaml | overcloud-controller-0 2022-05-11 13:33:26,385 p=335214 u=stack n=ansible | 2022-05-11 13:33:26.384193 | 48d539a1-1679-6875-cb99-00000000021e | TASK | Set ceph CLI 2022-05-11 13:33:26,586 p=335214 u=stack n=ansible | 2022-05-11 13:33:26.584870 | 48d539a1-1679-6875-cb99-00000000021e | OK | Set ceph CLI | overcloud-controller-0 2022-05-11 13:33:26,621 p=335214 u=stack n=ansible | 2022-05-11 13:33:26.620666 | 48d539a1-1679-6875-cb99-0000000000a3 | TASK | create configured crush rules 2022-05-11 13:33:29,616 p=335214 u=stack n=ansible | 2022-05-11 13:33:29.614420 | 48d539a1-1679-6875-cb99-0000000000a3 | CHANGED | create configured crush rules | overcloud-controller-0 | item={'class': 'hdd', 'default': True, 'name': 'HDD', 'root': 'default', 'type': 'host'} 2022-05-11 13:33:31,677 p=335214 u=stack n=ansible | 2022-05-11 13:33:31.675830 | 48d539a1-1679-6875-cb99-0000000000a3 | CHANGED | create configured crush rules | overcloud-controller-0 | item={'class': 'ssd', 'default': False, 'name': 'SSD', 'root': 'default', 'type': 'host'} 2022-05-11 13:33:31,787 p=335214 u=stack n=ansible | 2022-05-11 13:33:31.786328 | 48d539a1-1679-6875-cb99-0000000000a4 | TASK | get id for new default crush rule 2022-05-11 13:33:33,049 p=335214 u=stack n=ansible | 2022-05-11 13:33:33.047431 | 48d539a1-1679-6875-cb99-0000000000a4 | OK | get id for new default crush rule | overcloud-controller-0 | item={'class': 'hdd', 'default': True, 'name': 'HDD', 'root': 'default', 'type': 'host'} 2022-05-11 13:33:33,054 p=335214 u=stack n=ansible | 2022-05-11 13:33:33.053856 | 48d539a1-1679-6875-cb99-0000000000a4 | SKIPPED | get id for new default crush rule | overcloud-controller-0 | item={'class': 'ssd', 'default': False, 'name': 'SSD', 'root': 'default', 'type': 'host'} 2022-05-11 13:33:33,151 p=335214 u=stack n=ansible | 2022-05-11 13:33:33.150174 | 48d539a1-1679-6875-cb99-0000000000a5 | TASK | set_fact info_ceph_default_crush_rule_yaml, ceph_osd_pool_default_crush_rule_name 2022-05-11 13:33:33,542 p=335214 u=stack n=ansible | 2022-05-11 13:33:33.540553 | 48d539a1-1679-6875-cb99-0000000000a5 | OK | set_fact info_ceph_default_crush_rule_yaml, ceph_osd_pool_default_crush_rule_name | overcloud-controller-0 | item={'cmd': ['podman', 'run', '--rm', '--net=host', '-v', '/etc/ceph:/etc/ceph:z', '-v', '/var/lib/ceph/:/var/lib/ceph/:z', '-v', '/var/log/ceph/:/var/log/ceph/:z', '--entrypoint=ceph', '172.25.201.68:8787/ceph/daemon:v6.0.6-stable-6.0-pacific-centos-8-x86_64', '-n', 'client.admin', '-k', '/etc/ceph/ceph.client.admin.keyring', '--cluster', 'ceph', 'osd', 'crush', 'rule', 'dump', 'HDD', '--format=json'], 'start': '2022-05-11 05:33:32.226812', 'end': '2022-05-11 05:33:32.969031', 'delta': '0:00:00.742219', 'rc': 0, 'stdout': '\n{"rule_id":1,"rule_name":"HDD","ruleset":1,"type":1,"min_size":1,"max_size":10,"steps":[{"op":"take","item":-2,"item_name":"default~hdd"},{"op":"chooseleaf_firstn","num":0,"type":"host"},{"op":"emit"}]}', 'stderr': '', 'changed': False, 'invocation': {'module_args': {'name': 'HDD', 'cluster': 'ceph', 'state': 'info', 'rule_type': None, 'bucket_root': None, 'bucket_type': None, 'device_class': None, 'profile': None}}, 'stdout_lines': ['', '{"rule_id":1,"rule_name":"HDD","ruleset":1,"type":1,"min_size":1,"max_size":10,"steps":[{"op":"take","item":-2,"item_name":"default~hdd"},{"op":"chooseleaf_firstn","num":0,"type":"host"},{"op":"emit"}]}'], 'stderr_lines': [], 'failed': False, 'item': {'class': 'hdd', 'default': True, 'name': 'HDD', 'root': 'default', 'type': 'host'}, 'ansible_loop_var': 'item'} 2022-05-11 13:33:33,561 p=335214 u=stack n=ansible | 2022-05-11 13:33:33.560444 | 48d539a1-1679-6875-cb99-0000000000a5 | SKIPPED | set_fact info_ceph_default_crush_rule_yaml, ceph_osd_pool_default_crush_rule_name | overcloud-controller-0 | item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': {'class': 'ssd', 'default': False, 'name': 'SSD', 'root': 'default', 'type': 'host'}, 'ansible_loop_var': 'item'} 2022-05-11 13:33:34,398 p=335214 u=stack n=ansible | 2022-05-11 13:33:34.377552 | 48d539a1-1679-6875-cb99-0000000000a6 | TASK | insert new default crush rule into daemon to prevent restart 2022-05-11 13:33:35,297 p=335214 u=stack n=ansible | 2022-05-11 13:33:35.294928 | 48d539a1-1679-6875-cb99-0000000000a6 | FATAL | insert new default crush rule into daemon to prevent restart | overcloud-controller-0 -> 172.25.201.98 | item=overcloud-controller-0 | error={"ansible_loop_var": "item", "changed": false, "cmd": ["podman", "run", "--rm", "--net=host", "--ipc=host", "--volume", "/etc/ceph:/etc/ceph:z", "--volume", "/home/ceph-admin/assimilate_ceph.conf:/home/assimilate_ceph.conf:z", "--volume", "/var/run/ceph/2dbb93bd-e7ae-4a34-94e5-231eb65202dd:/var/run/ceph:z", "--entrypoint", "ceph", "172.25.201.68:8787/ceph/daemon:v6.0.6-stable-6.0-pacific-centos-8-x86_64", "--admin-daemon", "/var/run/ceph/ceph-mon.overcloud-controller-0.bdxworld.com.asok", "config", "set", "osd_pool_default_crush_rule", "1"], "delta": "0:00:00.443544", "end": "2022-05-11 05:33:35.239401", "item": "overcloud-controller-0", "msg": "non-zero return code", "rc": 22, "start": "2022-05-11 05:33:34.795857", "stderr": "admin_socket: exception getting command descriptions: [Errno 2] No such file or directory", "stderr_lines": ["admin_socket: exception getting command descriptions: [Errno 2] No such file or directory"], "stdout": "", "stdout_lines": []} 2022-05-11 13:33:35,301 p=335214 u=stack n=ansible | [WARNING]: ('overcloud-controller-0 -> 172.25.201.98', '48d539a1-1679-6875-cb99-0000000000a6') missing from stats 2022-05-11 13:33:36,104 p=335214 u=stack n=ansible | 2022-05-11 13:33:36.102979 | 48d539a1-1679-6875-cb99-0000000000a6 | FATAL | insert new default crush rule into daemon to prevent restart | overcloud-controller-0 -> 172.25.201.103 | item=overcloud-controller-1 | error={"ansible_loop_var": "item", "changed": false, "cmd": ["podman", "run", "--rm", "--net=host", "--ipc=host", "--volume", "/etc/ceph:/etc/ceph:z", "--volume", "/home/ceph-admin/assimilate_ceph.conf:/home/assimilate_ceph.conf:z", "--volume", "/var/run/ceph/2dbb93bd-e7ae-4a34-94e5-231eb65202dd:/var/run/ceph:z", "--entrypoint", "ceph", "172.25.201.68:8787/ceph/daemon:v6.0.6-stable-6.0-pacific-centos-8-x86_64", "--admin-daemon", "/var/run/ceph/ceph-mon.overcloud-controller-1.bdxworld.com.asok", "config", "set", "osd_pool_default_crush_rule", "1"], "delta": "0:00:00.241118", "end": "2022-05-11 05:33:36.041206", "item": "overcloud-controller-1", "msg": "non-zero return code", "rc": 125, "start": "2022-05-11 05:33:35.800088", "stderr": "Trying to pull 172.25.201.68:8787/ceph/daemon:v6.0.6-stable-6.0-pacific-centos-8-x86_64...\nError: initializing source docker://172.25.201.68:8787/ceph/daemon:v6.0.6-stable-6.0-pacific-centos-8-x86_64: pinging container registry 172.25.201.68:8787: Get \"https://172.25.201.68:8787/v2/\": http: server gave HTTP response to HTTPS client", "stderr_lines": ["Trying to pull 172.25.201.68:8787/ceph/daemon:v6.0.6-stable-6.0-pacific-centos-8-x86_64...", "Error: initializing source docker://172.25.201.68:8787/ceph/daemon:v6.0.6-stable-6.0-pacific-centos-8-x86_64: pinging container registry 172.25.201.68:8787: Get \"https://172.25.201.68:8787/v2/\": http: server gave HTTP response to HTTPS client"], "stdout": "", "stdout_lines": []} 2022-05-11 13:33:36,107 p=335214 u=stack n=ansible | [WARNING]: ('overcloud-controller-0 -> 172.25.201.103', '48d539a1-1679-6875-cb99-0000000000a6') missing from stats 2022-05-11 13:33:36,623 p=335214 u=stack n=ansible | 2022-05-11 13:33:36.621807 | 48d539a1-1679-6875-cb99-0000000000a6 | FATAL | insert new default crush rule into daemon to prevent restart | overcloud-controller-0 -> 172.25.201.102 | item=overcloud-controller-2 | error={"ansible_loop_var": "item", "changed": false, "cmd": ["podman", "run", "--rm", "--net=host", "--ipc=host", "--volume", "/etc/ceph:/etc/ceph:z", "--volume", "/home/ceph-admin/assimilate_ceph.conf:/home/assimilate_ceph.conf:z", "--volume", "/var/run/ceph/2dbb93bd-e7ae-4a34-94e5-231eb65202dd:/var/run/ceph:z", "--entrypoint", "ceph", "172.25.201.68:8787/ceph/daemon:v6.0.6-stable-6.0-pacific-centos-8-x86_64", "--admin-daemon", "/var/run/ceph/ceph-mon.overcloud-controller-2.bdxworld.com.asok", "config", "set", "osd_pool_default_crush_rule", "1"], "delta": "0:00:00.148024", "end": "2022-05-11 05:33:36.576571", "item": "overcloud-controller-2", "msg": "non-zero return code", "rc": 125, "start": "2022-05-11 05:33:36.428547", "stderr": "Trying to pull 172.25.201.68:8787/ceph/daemon:v6.0.6-stable-6.0-pacific-centos-8-x86_64...\nError: initializing source docker://172.25.201.68:8787/ceph/daemon:v6.0.6-stable-6.0-pacific-centos-8-x86_64: pinging container registry 172.25.201.68:8787: Get \"https://172.25.201.68:8787/v2/\": http: server gave HTTP response to HTTPS client", "stderr_lines": ["Trying to pull 172.25.201.68:8787/ceph/daemon:v6.0.6-stable-6.0-pacific-centos-8-x86_64...", "Error: initializing source docker://172.25.201.68:8787/ceph/daemon:v6.0.6-stable-6.0-pacific-centos-8-x86_64: pinging container registry 172.25.201.68:8787: Get \"https://172.25.201.68:8787/v2/\": http: server gave HTTP response to HTTPS client"], "stdout": "", "stdout_lines": []} 2022-05-11 13:33:36,626 p=335214 u=stack n=ansible | [WARNING]: ('overcloud-controller-0 -> 172.25.201.102', '48d539a1-1679-6875-cb99-0000000000a6') missing from stats 2022-05-11 13:33:36,639 p=335214 u=stack n=ansible | NO MORE HOSTS LEFT ************************************************************* 2022-05-11 13:33:36,645 p=335214 u=stack n=ansible | PLAY RECAP ********************************************************************* 2022-05-11 13:33:36,646 p=335214 u=stack n=ansible | overcloud-controller-0 : ok=15 changed=4 unreachable=0 failed=1 skipped=25 rescued=0 ignored=0 2022-05-11 13:33:36,649 p=335214 u=stack n=ansible | 2022-05-11 13:33:36.648697 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Summary Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2022-05-11 13:33:36,650 p=335214 u=stack n=ansible | 2022-05-11 13:33:36.649606 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Total Tasks: 41 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2022-05-11 13:33:36,651 p=335214 u=stack n=ansible | 2022-05-11 13:33:36.650473 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Elapsed Time: 0:00:27.141784 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2022-05-11 13:33:36,651 p=335214 u=stack n=ansible | 2022-05-11 13:33:36.651256 | UUID | Info | Host | Task Name | Run Time 2022-05-11 13:33:36,652 p=335214 u=stack n=ansible | 2022-05-11 13:33:36.652025 | 48d539a1-1679-6875-cb99-0000000000a3 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : create configured crush rules | 5.06s 2022-05-11 13:33:36,653 p=335214 u=stack n=ansible | 2022-05-11 13:33:36.652796 | 48d539a1-1679-6875-cb99-000000000026 | SUMMARY | overcloud-controller-0 | push tripleo_cephadm_spec to bootstrap node if spec file exsits | 2.29s 2022-05-11 13:33:36,654 p=335214 u=stack n=ansible | 2022-05-11 13:33:36.653583 | 48d539a1-1679-6875-cb99-0000000000a6 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : insert new default crush rule into daemon to prevent restart | 2.23s 2022-05-11 13:33:36,655 p=335214 u=stack n=ansible | 2022-05-11 13:33:36.654574 | 48d539a1-1679-6875-cb99-00000000001b | SUMMARY | overcloud-controller-0 | tripleo_cephadm : List Ceph daemon instances on this host | 1.32s 2022-05-11 13:33:36,656 p=335214 u=stack n=ansible | 2022-05-11 13:33:36.655719 | 48d539a1-1679-6875-cb99-0000000000a4 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : get id for new default crush rule | 1.27s 2022-05-11 13:33:36,657 p=335214 u=stack n=ansible | 2022-05-11 13:33:36.656560 | 48d539a1-1679-6875-cb99-000000000076 | SUMMARY | overcloud-controller-0 | Assimilate configuration from tripleo_cephadm_assimilate_conf | 1.15s 2022-05-11 13:33:36,657 p=335214 u=stack n=ansible | 2022-05-11 13:33:36.657381 | 48d539a1-1679-6875-cb99-000000000019 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Stat cephadm file | 0.82s 2022-05-11 13:33:36,658 p=335214 u=stack n=ansible | 2022-05-11 13:33:36.658133 | 48d539a1-1679-6875-cb99-000000000023 | SUMMARY | overcloud-controller-0 | Ensure tripleo_cephadm_config_home (e.g. /etc/ceph) exists | 0.81s 2022-05-11 13:33:36,659 p=335214 u=stack n=ansible | 2022-05-11 13:33:36.658898 | 48d539a1-1679-6875-cb99-00000000002b | SUMMARY | overcloud-controller-0 | generate tripleo_cephadm_assimilate_conf on bootstrap node | 0.80s 2022-05-11 13:33:36,660 p=335214 u=stack n=ansible | 2022-05-11 13:33:36.659641 | 48d539a1-1679-6875-cb99-000000000085 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Fail if ceph orchestrator is not available | 0.56s 2022-05-11 13:33:36,661 p=335214 u=stack n=ansible | 2022-05-11 13:33:36.660626 | 48d539a1-1679-6875-cb99-00000000005b | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Run cephadm bootstrap | 0.54s 2022-05-11 13:33:36,662 p=335214 u=stack n=ansible | 2022-05-11 13:33:36.661448 | 48d539a1-1679-6875-cb99-000000000059 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Stat pre ceph conf file in case we should bootrap with it | 0.46s 2022-05-11 13:33:36,662 p=335214 u=stack n=ansible | 2022-05-11 13:33:36.662216 | 48d539a1-1679-6875-cb99-0000000000a5 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : set_fact info_ceph_default_crush_rule_yaml, ceph_osd_pool_default_crush_rule_name | 0.43s 2022-05-11 13:33:36,663 p=335214 u=stack n=ansible | 2022-05-11 13:33:36.662999 | 48d539a1-1679-6875-cb99-000000000024 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Ensure specs directory exists | 0.36s 2022-05-11 13:33:36,664 p=335214 u=stack n=ansible | 2022-05-11 13:33:36.663765 | 48d539a1-1679-6875-cb99-0000000001f1 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Set ceph CLI | 0.34s 2022-05-11 13:33:36,665 p=335214 u=stack n=ansible | 2022-05-11 13:33:36.664523 | 48d539a1-1679-6875-cb99-000000000082 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Fail if spec file is missing | 0.33s 2022-05-11 13:33:36,666 p=335214 u=stack n=ansible | 2022-05-11 13:33:36.665286 | 48d539a1-1679-6875-cb99-000000000081 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Stat spec file on bootstrap node | 0.32s 2022-05-11 13:33:36,667 p=335214 u=stack n=ansible | 2022-05-11 13:33:36.666343 | 48d539a1-1679-6875-cb99-000000000083 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Get ceph_cli | 0.30s 2022-05-11 13:33:36,667 p=335214 u=stack n=ansible | 2022-05-11 13:33:36.667362 | 48d539a1-1679-6875-cb99-000000000084 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Get the ceph orchestrator status | 0.29s 2022-05-11 13:33:36,668 p=335214 u=stack n=ansible | 2022-05-11 13:33:36.668119 | 48d539a1-1679-6875-cb99-00000000001a | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Fail if cephadm is not available | 0.21s 2022-05-11 13:33:36,669 p=335214 u=stack n=ansible | 2022-05-11 13:33:36.668900 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ End Summary Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2022-05-11 13:33:36,670 p=335214 u=stack n=ansible | 2022-05-11 13:33:36.669737 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ State Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2022-05-11 13:33:36,671 p=335214 u=stack n=ansible | 2022-05-11 13:33:36.670641 | ~~~~~~~~~~~~~~~~~~ Number of nodes which did not deploy successfully: 1 ~~~~~~~~~~~~~~~~~ 2022-05-11 13:33:36,672 p=335214 u=stack n=ansible | 2022-05-11 13:33:36.671452 | The following node(s) had failures: overcloud-controller-0 2022-05-11 13:33:36,672 p=335214 u=stack n=ansible | 2022-05-11 13:33:36.672294 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Running /home/stack/overcloud-deploy/overcloud/config-download/overcloud/cephadm/cephadm_command.sh 2022-05-11 15:34:33,157 p=628194 u=stack n=ansible | PLAY [Deploy Ceph with cephadm] ************************************************ 2022-05-11 15:34:33,238 p=628194 u=stack n=ansible | 2022-05-11 15:34:33.236984 | 48d539a1-1679-f04f-5ac6-000000000018 | TASK | Install cephadm package 2022-05-11 15:34:33,382 p=628194 u=stack n=ansible | 2022-05-11 15:34:33.380999 | 48d539a1-1679-f04f-5ac6-000000000018 | SKIPPED | Install cephadm package | overcloud-controller-0 2022-05-11 15:34:33,412 p=628194 u=stack n=ansible | 2022-05-11 15:34:33.411192 | 48d539a1-1679-f04f-5ac6-000000000019 | TASK | Stat cephadm file 2022-05-11 15:34:34,233 p=628194 u=stack n=ansible | 2022-05-11 15:34:34.232023 | 48d539a1-1679-f04f-5ac6-000000000019 | OK | Stat cephadm file | overcloud-controller-0 2022-05-11 15:34:34,267 p=628194 u=stack n=ansible | 2022-05-11 15:34:34.266564 | 48d539a1-1679-f04f-5ac6-00000000001a | TASK | Fail if cephadm is not available 2022-05-11 15:34:34,412 p=628194 u=stack n=ansible | 2022-05-11 15:34:34.410210 | 48d539a1-1679-f04f-5ac6-00000000001a | SKIPPED | Fail if cephadm is not available | overcloud-controller-0 2022-05-11 15:34:34,443 p=628194 u=stack n=ansible | 2022-05-11 15:34:34.442904 | 48d539a1-1679-f04f-5ac6-00000000001b | TASK | List Ceph daemon instances on this host 2022-05-11 15:34:35,644 p=628194 u=stack n=ansible | 2022-05-11 15:34:35.643382 | 48d539a1-1679-f04f-5ac6-00000000001b | CHANGED | List Ceph daemon instances on this host | overcloud-controller-0 2022-05-11 15:34:35,677 p=628194 u=stack n=ansible | 2022-05-11 15:34:35.676157 | 48d539a1-1679-f04f-5ac6-00000000001d | TASK | Set list of found FSIDs 2022-05-11 15:34:35,832 p=628194 u=stack n=ansible | 2022-05-11 15:34:35.831119 | 48d539a1-1679-f04f-5ac6-00000000001d | SKIPPED | Set list of found FSIDs | overcloud-controller-0 2022-05-11 15:34:35,871 p=628194 u=stack n=ansible | 2022-05-11 15:34:35.870809 | 48d539a1-1679-f04f-5ac6-00000000001e | TASK | Fail if >1 FSID was discovered 2022-05-11 15:34:36,018 p=628194 u=stack n=ansible | 2022-05-11 15:34:36.017381 | 48d539a1-1679-f04f-5ac6-00000000001e | SKIPPED | Fail if >1 FSID was discovered | overcloud-controller-0 2022-05-11 15:34:36,051 p=628194 u=stack n=ansible | 2022-05-11 15:34:36.050497 | 48d539a1-1679-f04f-5ac6-00000000001f | TASK | Set FSID to the discovered value 2022-05-11 15:34:36,197 p=628194 u=stack n=ansible | 2022-05-11 15:34:36.193197 | 48d539a1-1679-f04f-5ac6-00000000001f | SKIPPED | Set FSID to the discovered value | overcloud-controller-0 2022-05-11 15:34:36,231 p=628194 u=stack n=ansible | 2022-05-11 15:34:36.230676 | 48d539a1-1679-f04f-5ac6-000000000020 | TASK | Set random tripleo_cephadm_fsid if no running ceph containers were found 2022-05-11 15:34:36,397 p=628194 u=stack n=ansible | 2022-05-11 15:34:36.396372 | 48d539a1-1679-f04f-5ac6-000000000020 | SKIPPED | Set random tripleo_cephadm_fsid if no running ceph containers were found | overcloud-controller-0 2022-05-11 15:34:36,430 p=628194 u=stack n=ansible | 2022-05-11 15:34:36.429218 | 48d539a1-1679-f04f-5ac6-000000000022 | TASK | Set first monitor IP if it was not passed 2022-05-11 15:34:36,584 p=628194 u=stack n=ansible | 2022-05-11 15:34:36.583186 | 48d539a1-1679-f04f-5ac6-000000000022 | SKIPPED | Set first monitor IP if it was not passed | overcloud-controller-0 2022-05-11 15:34:36,615 p=628194 u=stack n=ansible | 2022-05-11 15:34:36.614611 | 48d539a1-1679-f04f-5ac6-000000000023 | TASK | Ensure tripleo_cephadm_config_home (e.g. /etc/ceph) exists 2022-05-11 15:34:37,442 p=628194 u=stack n=ansible | 2022-05-11 15:34:37.441199 | 48d539a1-1679-f04f-5ac6-000000000023 | OK | Ensure tripleo_cephadm_config_home (e.g. /etc/ceph) exists | overcloud-controller-0 2022-05-11 15:34:37,474 p=628194 u=stack n=ansible | 2022-05-11 15:34:37.473232 | 48d539a1-1679-f04f-5ac6-000000000024 | TASK | Ensure specs directory exists 2022-05-11 15:34:37,854 p=628194 u=stack n=ansible | 2022-05-11 15:34:37.852627 | 48d539a1-1679-f04f-5ac6-000000000024 | OK | Ensure specs directory exists | overcloud-controller-0 2022-05-11 15:34:37,985 p=628194 u=stack n=ansible | 2022-05-11 15:34:37.984858 | 48d539a1-1679-f04f-5ac6-000000000025 | TASK | Stat spec file on ansible host 2022-05-11 15:34:38,894 p=628194 u=stack n=ansible | 2022-05-11 15:34:38.893152 | 48d539a1-1679-f04f-5ac6-000000000025 | OK | Stat spec file on ansible host | overcloud-controller-0 -> localhost 2022-05-11 15:34:38,897 p=628194 u=stack n=ansible | [WARNING]: ('overcloud-controller-0 -> localhost', '48d539a1-1679-f04f-5ac6-000000000025') missing from stats 2022-05-11 15:34:38,984 p=628194 u=stack n=ansible | 2022-05-11 15:34:38.983373 | 48d539a1-1679-f04f-5ac6-000000000026 | TASK | push tripleo_cephadm_spec to bootstrap node if spec file exsits 2022-05-11 15:34:41,473 p=628194 u=stack n=ansible | 2022-05-11 15:34:41.471720 | 48d539a1-1679-f04f-5ac6-000000000026 | CHANGED | push tripleo_cephadm_spec to bootstrap node if spec file exsits | overcloud-controller-0 2022-05-11 15:34:41,554 p=628194 u=stack n=ansible | 2022-05-11 15:34:41.553552 | 48d539a1-1679-f04f-5ac6-000000000028 | TASK | stat tripleo_cephadm_bootstrap_conf file 2022-05-11 15:34:41,734 p=628194 u=stack n=ansible | 2022-05-11 15:34:41.733056 | 48d539a1-1679-f04f-5ac6-000000000028 | SKIPPED | stat tripleo_cephadm_bootstrap_conf file | overcloud-controller-0 2022-05-11 15:34:41,776 p=628194 u=stack n=ansible | 2022-05-11 15:34:41.775858 | 48d539a1-1679-f04f-5ac6-000000000029 | TASK | push it to tripleo_cephadm_assimilate_conf on bootstrap node 2022-05-11 15:34:41,982 p=628194 u=stack n=ansible | 2022-05-11 15:34:41.980806 | 48d539a1-1679-f04f-5ac6-000000000029 | SKIPPED | push it to tripleo_cephadm_assimilate_conf on bootstrap node | overcloud-controller-0 2022-05-11 15:34:42,018 p=628194 u=stack n=ansible | 2022-05-11 15:34:42.017111 | 48d539a1-1679-f04f-5ac6-00000000002b | TASK | generate tripleo_cephadm_assimilate_conf on bootstrap node 2022-05-11 15:34:42,815 p=628194 u=stack n=ansible | 2022-05-11 15:34:42.813572 | 48d539a1-1679-f04f-5ac6-00000000002b | CHANGED | generate tripleo_cephadm_assimilate_conf on bootstrap node | overcloud-controller-0 2022-05-11 15:34:42,849 p=628194 u=stack n=ansible | 2022-05-11 15:34:42.848745 | 48d539a1-1679-f04f-5ac6-000000000056 | TASK | Add spec to necessary file list when using spec to bootstrap 2022-05-11 15:34:43,002 p=628194 u=stack n=ansible | 2022-05-11 15:34:43.001247 | 48d539a1-1679-f04f-5ac6-000000000056 | SKIPPED | Add spec to necessary file list when using spec to bootstrap | overcloud-controller-0 2022-05-11 15:34:43,035 p=628194 u=stack n=ansible | 2022-05-11 15:34:43.034173 | 48d539a1-1679-f04f-5ac6-000000000057 | TASK | Stat necessary files to bootstrap with cephadm 2022-05-11 15:34:43,213 p=628194 u=stack n=ansible | 2022-05-11 15:34:43.211355 | 48d539a1-1679-f04f-5ac6-000000000057 | SKIPPED | Stat necessary files to bootstrap with cephadm | overcloud-controller-0 | item=/home/ceph-admin/.ssh/id_rsa 2022-05-11 15:34:43,226 p=628194 u=stack n=ansible | 2022-05-11 15:34:43.225335 | 48d539a1-1679-f04f-5ac6-000000000057 | SKIPPED | Stat necessary files to bootstrap with cephadm | overcloud-controller-0 | item=/home/ceph-admin/.ssh/id_rsa.pub 2022-05-11 15:34:43,258 p=628194 u=stack n=ansible | 2022-05-11 15:34:43.257771 | 48d539a1-1679-f04f-5ac6-000000000058 | TASK | Fail if necessary files are missing 2022-05-11 15:34:43,436 p=628194 u=stack n=ansible | 2022-05-11 15:34:43.434618 | 48d539a1-1679-f04f-5ac6-000000000058 | SKIPPED | Fail if necessary files are missing | overcloud-controller-0 | item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': '/home/ceph-admin/.ssh/id_rsa', 'ansible_loop_var': 'item'} 2022-05-11 15:34:43,439 p=628194 u=stack n=ansible | 2022-05-11 15:34:43.438023 | 48d539a1-1679-f04f-5ac6-000000000058 | SKIPPED | Fail if necessary files are missing | overcloud-controller-0 | item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': '/home/ceph-admin/.ssh/id_rsa.pub', 'ansible_loop_var': 'item'} 2022-05-11 15:34:43,558 p=628194 u=stack n=ansible | 2022-05-11 15:34:43.505323 | 48d539a1-1679-f04f-5ac6-000000000059 | TASK | Stat pre ceph conf file in case we should bootrap with it 2022-05-11 15:34:43,706 p=628194 u=stack n=ansible | 2022-05-11 15:34:43.704799 | 48d539a1-1679-f04f-5ac6-000000000059 | SKIPPED | Stat pre ceph conf file in case we should bootrap with it | overcloud-controller-0 2022-05-11 15:34:43,738 p=628194 u=stack n=ansible | 2022-05-11 15:34:43.737221 | 48d539a1-1679-f04f-5ac6-00000000005b | TASK | Run cephadm bootstrap 2022-05-11 15:34:43,888 p=628194 u=stack n=ansible | 2022-05-11 15:34:43.886886 | 48d539a1-1679-f04f-5ac6-00000000005b | SKIPPED | Run cephadm bootstrap | overcloud-controller-0 2022-05-11 15:34:43,919 p=628194 u=stack n=ansible | 2022-05-11 15:34:43.917982 | 48d539a1-1679-f04f-5ac6-00000000005c | TASK | Show results of bootstrap 2022-05-11 15:34:44,089 p=628194 u=stack n=ansible | 2022-05-11 15:34:44.087994 | 48d539a1-1679-f04f-5ac6-00000000005c | SKIPPED | Show results of bootstrap | overcloud-controller-0 2022-05-11 15:34:44,123 p=628194 u=stack n=ansible | 2022-05-11 15:34:44.122056 | 48d539a1-1679-f04f-5ac6-00000000005e | TASK | If cephadm bootstrap was not run report the reason 2022-05-11 15:34:44,551 p=628194 u=stack n=ansible | 2022-05-11 15:34:44.549484 | 48d539a1-1679-f04f-5ac6-00000000005e | SKIPPED | If cephadm bootstrap was not run report the reason | overcloud-controller-0 2022-05-11 15:34:44,589 p=628194 u=stack n=ansible | 2022-05-11 15:34:44.587672 | 48d539a1-1679-f04f-5ac6-000000000075 | TASK | Ensure tripleo_cephadm_ceph_cli is set 2022-05-11 15:34:44,797 p=628194 u=stack n=ansible | 2022-05-11 15:34:44.795637 | f3d28f90-39f2-41ae-9613-d54251fece1c | INCLUDED | /usr/share/ansible/roles/tripleo_cephadm/tasks/ceph_cli.yaml | overcloud-controller-0 2022-05-11 15:34:44,831 p=628194 u=stack n=ansible | 2022-05-11 15:34:44.830313 | 48d539a1-1679-f04f-5ac6-0000000001f1 | TASK | Set ceph CLI 2022-05-11 15:34:45,030 p=628194 u=stack n=ansible | 2022-05-11 15:34:45.028299 | 48d539a1-1679-f04f-5ac6-0000000001f1 | OK | Set ceph CLI | overcloud-controller-0 2022-05-11 15:34:45,220 p=628194 u=stack n=ansible | 2022-05-11 15:34:45.164438 | 48d539a1-1679-f04f-5ac6-000000000076 | TASK | Assimilate configuration from tripleo_cephadm_assimilate_conf 2022-05-11 15:34:46,295 p=628194 u=stack n=ansible | 2022-05-11 15:34:46.293772 | 48d539a1-1679-f04f-5ac6-000000000076 | OK | Assimilate configuration from tripleo_cephadm_assimilate_conf | overcloud-controller-0 2022-05-11 15:34:46,399 p=628194 u=stack n=ansible | 2022-05-11 15:34:46.398126 | 48d539a1-1679-f04f-5ac6-000000000081 | TASK | Stat spec file on bootstrap node 2022-05-11 15:34:46,773 p=628194 u=stack n=ansible | 2022-05-11 15:34:46.771602 | 48d539a1-1679-f04f-5ac6-000000000081 | SKIPPED | Stat spec file on bootstrap node | overcloud-controller-0 | item=/home/ceph-admin/specs/ceph_spec.yaml 2022-05-11 15:34:46,869 p=628194 u=stack n=ansible | 2022-05-11 15:34:46.867841 | 48d539a1-1679-f04f-5ac6-000000000082 | TASK | Fail if spec file is missing 2022-05-11 15:34:47,236 p=628194 u=stack n=ansible | 2022-05-11 15:34:47.235012 | 48d539a1-1679-f04f-5ac6-000000000082 | SKIPPED | Fail if spec file is missing | overcloud-controller-0 | item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': '/home/ceph-admin/specs/ceph_spec.yaml', 'ansible_loop_var': 'item'} 2022-05-11 15:34:47,333 p=628194 u=stack n=ansible | 2022-05-11 15:34:47.331884 | 48d539a1-1679-f04f-5ac6-000000000083 | TASK | Get ceph_cli 2022-05-11 15:34:47,620 p=628194 u=stack n=ansible | 2022-05-11 15:34:47.618516 | 48d539a1-1679-f04f-5ac6-000000000083 | SKIPPED | Get ceph_cli | overcloud-controller-0 2022-05-11 15:34:47,655 p=628194 u=stack n=ansible | 2022-05-11 15:34:47.653897 | 48d539a1-1679-f04f-5ac6-000000000084 | TASK | Get the ceph orchestrator status 2022-05-11 15:34:47,974 p=628194 u=stack n=ansible | 2022-05-11 15:34:47.972373 | 48d539a1-1679-f04f-5ac6-000000000084 | SKIPPED | Get the ceph orchestrator status | overcloud-controller-0 2022-05-11 15:34:48,007 p=628194 u=stack n=ansible | 2022-05-11 15:34:48.006149 | 48d539a1-1679-f04f-5ac6-000000000085 | TASK | Fail if ceph orchestrator is not available 2022-05-11 15:34:48,329 p=628194 u=stack n=ansible | 2022-05-11 15:34:48.327356 | 48d539a1-1679-f04f-5ac6-000000000085 | SKIPPED | Fail if ceph orchestrator is not available | overcloud-controller-0 2022-05-11 15:34:48,363 p=628194 u=stack n=ansible | 2022-05-11 15:34:48.362048 | 48d539a1-1679-f04f-5ac6-000000000086 | TASK | Set container images used by the applied services 2022-05-11 15:34:48,663 p=628194 u=stack n=ansible | 2022-05-11 15:34:48.661439 | 48d539a1-1679-f04f-5ac6-000000000086 | SKIPPED | Set container images used by the applied services | overcloud-controller-0 2022-05-11 15:34:48,695 p=628194 u=stack n=ansible | 2022-05-11 15:34:48.694510 | 48d539a1-1679-f04f-5ac6-000000000087 | TASK | Apply spec 2022-05-11 15:34:49,346 p=628194 u=stack n=ansible | 2022-05-11 15:34:49.344363 | 48d539a1-1679-f04f-5ac6-000000000087 | SKIPPED | Apply spec | overcloud-controller-0 2022-05-11 15:34:49,384 p=628194 u=stack n=ansible | 2022-05-11 15:34:49.382888 | 48d539a1-1679-f04f-5ac6-000000000088 | TASK | Show results of spec apply 2022-05-11 15:34:49,798 p=628194 u=stack n=ansible | 2022-05-11 15:34:49.796736 | 48d539a1-1679-f04f-5ac6-000000000088 | SKIPPED | Show results of spec apply | overcloud-controller-0 2022-05-11 15:34:49,828 p=628194 u=stack n=ansible | 2022-05-11 15:34:49.827090 | 48d539a1-1679-f04f-5ac6-000000000089 | TASK | Wait for the expected number of osds to be running 2022-05-11 15:34:50,003 p=628194 u=stack n=ansible | 2022-05-11 15:34:50.001452 | 48d539a1-1679-f04f-5ac6-000000000089 | SKIPPED | Wait for the expected number of osds to be running | overcloud-controller-0 2022-05-11 15:34:50,035 p=628194 u=stack n=ansible | 2022-05-11 15:34:50.034281 | 48d539a1-1679-f04f-5ac6-0000000000a2 | TASK | Get ceph_cli 2022-05-11 15:34:50,212 p=628194 u=stack n=ansible | 2022-05-11 15:34:50.210896 | b6c96bd0-4ce3-484a-9009-9ffad1c1d6a5 | INCLUDED | /usr/share/ansible/roles/tripleo_cephadm/tasks/ceph_cli.yaml | overcloud-controller-0 2022-05-11 15:34:50,248 p=628194 u=stack n=ansible | 2022-05-11 15:34:50.247797 | 48d539a1-1679-f04f-5ac6-00000000021e | TASK | Set ceph CLI 2022-05-11 15:34:50,441 p=628194 u=stack n=ansible | 2022-05-11 15:34:50.439480 | 48d539a1-1679-f04f-5ac6-00000000021e | OK | Set ceph CLI | overcloud-controller-0 2022-05-11 15:34:50,475 p=628194 u=stack n=ansible | 2022-05-11 15:34:50.474258 | 48d539a1-1679-f04f-5ac6-0000000000a3 | TASK | create configured crush rules 2022-05-11 15:34:51,915 p=628194 u=stack n=ansible | 2022-05-11 15:34:51.913874 | 48d539a1-1679-f04f-5ac6-0000000000a3 | OK | create configured crush rules | overcloud-controller-0 | item={'class': 'hdd', 'default': True, 'name': 'HDD', 'root': 'default', 'type': 'host'} 2022-05-11 15:34:52,803 p=628194 u=stack n=ansible | 2022-05-11 15:34:52.801393 | 48d539a1-1679-f04f-5ac6-0000000000a3 | OK | create configured crush rules | overcloud-controller-0 | item={'class': 'ssd', 'default': False, 'name': 'SSD', 'root': 'default', 'type': 'host'} 2022-05-11 15:34:52,845 p=628194 u=stack n=ansible | 2022-05-11 15:34:52.844147 | 48d539a1-1679-f04f-5ac6-0000000000a4 | TASK | get id for new default crush rule 2022-05-11 15:34:54,087 p=628194 u=stack n=ansible | 2022-05-11 15:34:54.084902 | 48d539a1-1679-f04f-5ac6-0000000000a4 | OK | get id for new default crush rule | overcloud-controller-0 | item={'class': 'hdd', 'default': True, 'name': 'HDD', 'root': 'default', 'type': 'host'} 2022-05-11 15:34:54,103 p=628194 u=stack n=ansible | 2022-05-11 15:34:54.102668 | 48d539a1-1679-f04f-5ac6-0000000000a4 | SKIPPED | get id for new default crush rule | overcloud-controller-0 | item={'class': 'ssd', 'default': False, 'name': 'SSD', 'root': 'default', 'type': 'host'} 2022-05-11 15:34:54,140 p=628194 u=stack n=ansible | 2022-05-11 15:34:54.139641 | 48d539a1-1679-f04f-5ac6-0000000000a5 | TASK | set_fact info_ceph_default_crush_rule_yaml, ceph_osd_pool_default_crush_rule_name 2022-05-11 15:34:54,418 p=628194 u=stack n=ansible | 2022-05-11 15:34:54.416520 | 48d539a1-1679-f04f-5ac6-0000000000a5 | OK | set_fact info_ceph_default_crush_rule_yaml, ceph_osd_pool_default_crush_rule_name | overcloud-controller-0 | item={'cmd': ['podman', 'run', '--rm', '--net=host', '-v', '/etc/ceph:/etc/ceph:z', '-v', '/var/lib/ceph/:/var/lib/ceph/:z', '-v', '/var/log/ceph/:/var/log/ceph/:z', '--entrypoint=ceph', '172.25.201.68:8787/ceph/daemon:v6.0.6-stable-6.0-pacific-centos-8-x86_64', '-n', 'client.admin', '-k', '/etc/ceph/ceph.client.admin.keyring', '--cluster', 'ceph', 'osd', 'crush', 'rule', 'dump', 'HDD', '--format=json'], 'start': '2022-05-11 07:34:53.241728', 'end': '2022-05-11 07:34:53.983565', 'delta': '0:00:00.741837', 'rc': 0, 'stdout': '\n{"rule_id":1,"rule_name":"HDD","ruleset":1,"type":1,"min_size":1,"max_size":10,"steps":[{"op":"take","item":-2,"item_name":"default~hdd"},{"op":"chooseleaf_firstn","num":0,"type":"host"},{"op":"emit"}]}', 'stderr': '', 'changed': False, 'invocation': {'module_args': {'name': 'HDD', 'cluster': 'ceph', 'state': 'info', 'rule_type': None, 'bucket_root': None, 'bucket_type': None, 'device_class': None, 'profile': None}}, 'stdout_lines': ['', '{"rule_id":1,"rule_name":"HDD","ruleset":1,"type":1,"min_size":1,"max_size":10,"steps":[{"op":"take","item":-2,"item_name":"default~hdd"},{"op":"chooseleaf_firstn","num":0,"type":"host"},{"op":"emit"}]}'], 'stderr_lines': [], 'failed': False, 'item': {'class': 'hdd', 'default': True, 'name': 'HDD', 'root': 'default', 'type': 'host'}, 'ansible_loop_var': 'item'} 2022-05-11 15:34:54,427 p=628194 u=stack n=ansible | 2022-05-11 15:34:54.426636 | 48d539a1-1679-f04f-5ac6-0000000000a5 | SKIPPED | set_fact info_ceph_default_crush_rule_yaml, ceph_osd_pool_default_crush_rule_name | overcloud-controller-0 | item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': {'class': 'ssd', 'default': False, 'name': 'SSD', 'root': 'default', 'type': 'host'}, 'ansible_loop_var': 'item'} 2022-05-11 15:34:55,372 p=628194 u=stack n=ansible | 2022-05-11 15:34:55.369475 | 48d539a1-1679-f04f-5ac6-0000000000a6 | TASK | insert new default crush rule into daemon to prevent restart 2022-05-11 15:34:56,164 p=628194 u=stack n=ansible | 2022-05-11 15:34:56.162023 | 48d539a1-1679-f04f-5ac6-0000000000a6 | FATAL | insert new default crush rule into daemon to prevent restart | overcloud-controller-0 -> 172.25.201.98 | item=overcloud-controller-0 | error={"ansible_loop_var": "item", "changed": false, "cmd": ["podman", "run", "--rm", "--net=host", "--ipc=host", "--volume", "/etc/ceph:/etc/ceph:z", "--volume", "/home/ceph-admin/assimilate_ceph.conf:/home/assimilate_ceph.conf:z", "--volume", "/var/run/ceph/2dbb93bd-e7ae-4a34-94e5-231eb65202dd:/var/run/ceph:z", "--entrypoint", "ceph", "172.25.201.68:8787/ceph/daemon:v6.0.6-stable-6.0-pacific-centos-8-x86_64", "--admin-daemon", "/var/run/ceph/ceph-mon.overcloud-controller-0.bdxworld.com.asok", "config", "set", "osd_pool_default_crush_rule", "1"], "delta": "0:00:00.439903", "end": "2022-05-11 07:34:56.102055", "item": "overcloud-controller-0", "msg": "non-zero return code", "rc": 22, "start": "2022-05-11 07:34:55.662152", "stderr": "admin_socket: exception getting command descriptions: [Errno 2] No such file or directory", "stderr_lines": ["admin_socket: exception getting command descriptions: [Errno 2] No such file or directory"], "stdout": "", "stdout_lines": []} 2022-05-11 15:34:56,167 p=628194 u=stack n=ansible | [WARNING]: ('overcloud-controller-0 -> 172.25.201.98', '48d539a1-1679-f04f-5ac6-0000000000a6') missing from stats 2022-05-11 15:34:56,993 p=628194 u=stack n=ansible | 2022-05-11 15:34:56.992625 | 48d539a1-1679-f04f-5ac6-0000000000a6 | FATAL | insert new default crush rule into daemon to prevent restart | overcloud-controller-0 -> 172.25.201.103 | item=overcloud-controller-1 | error={"ansible_loop_var": "item", "changed": false, "cmd": ["podman", "run", "--rm", "--net=host", "--ipc=host", "--volume", "/etc/ceph:/etc/ceph:z", "--volume", "/home/ceph-admin/assimilate_ceph.conf:/home/assimilate_ceph.conf:z", "--volume", "/var/run/ceph/2dbb93bd-e7ae-4a34-94e5-231eb65202dd:/var/run/ceph:z", "--entrypoint", "ceph", "172.25.201.68:8787/ceph/daemon:v6.0.6-stable-6.0-pacific-centos-8-x86_64", "--admin-daemon", "/var/run/ceph/ceph-mon.overcloud-controller-1.bdxworld.com.asok", "config", "set", "osd_pool_default_crush_rule", "1"], "delta": "0:00:00.237641", "end": "2022-05-11 07:34:56.940091", "item": "overcloud-controller-1", "msg": "non-zero return code", "rc": 125, "start": "2022-05-11 07:34:56.702450", "stderr": "Trying to pull 172.25.201.68:8787/ceph/daemon:v6.0.6-stable-6.0-pacific-centos-8-x86_64...\nError: initializing source docker://172.25.201.68:8787/ceph/daemon:v6.0.6-stable-6.0-pacific-centos-8-x86_64: pinging container registry 172.25.201.68:8787: Get \"https://172.25.201.68:8787/v2/\": http: server gave HTTP response to HTTPS client", "stderr_lines": ["Trying to pull 172.25.201.68:8787/ceph/daemon:v6.0.6-stable-6.0-pacific-centos-8-x86_64...", "Error: initializing source docker://172.25.201.68:8787/ceph/daemon:v6.0.6-stable-6.0-pacific-centos-8-x86_64: pinging container registry 172.25.201.68:8787: Get \"https://172.25.201.68:8787/v2/\": http: server gave HTTP response to HTTPS client"], "stdout": "", "stdout_lines": []} 2022-05-11 15:34:56,997 p=628194 u=stack n=ansible | [WARNING]: ('overcloud-controller-0 -> 172.25.201.103', '48d539a1-1679-f04f-5ac6-0000000000a6') missing from stats 2022-05-11 15:34:57,503 p=628194 u=stack n=ansible | 2022-05-11 15:34:57.502128 | 48d539a1-1679-f04f-5ac6-0000000000a6 | FATAL | insert new default crush rule into daemon to prevent restart | overcloud-controller-0 -> 172.25.201.102 | item=overcloud-controller-2 | error={"ansible_loop_var": "item", "changed": false, "cmd": ["podman", "run", "--rm", "--net=host", "--ipc=host", "--volume", "/etc/ceph:/etc/ceph:z", "--volume", "/home/ceph-admin/assimilate_ceph.conf:/home/assimilate_ceph.conf:z", "--volume", "/var/run/ceph/2dbb93bd-e7ae-4a34-94e5-231eb65202dd:/var/run/ceph:z", "--entrypoint", "ceph", "172.25.201.68:8787/ceph/daemon:v6.0.6-stable-6.0-pacific-centos-8-x86_64", "--admin-daemon", "/var/run/ceph/ceph-mon.overcloud-controller-2.bdxworld.com.asok", "config", "set", "osd_pool_default_crush_rule", "1"], "delta": "0:00:00.152384", "end": "2022-05-11 07:34:57.452370", "item": "overcloud-controller-2", "msg": "non-zero return code", "rc": 125, "start": "2022-05-11 07:34:57.299986", "stderr": "Trying to pull 172.25.201.68:8787/ceph/daemon:v6.0.6-stable-6.0-pacific-centos-8-x86_64...\nError: initializing source docker://172.25.201.68:8787/ceph/daemon:v6.0.6-stable-6.0-pacific-centos-8-x86_64: pinging container registry 172.25.201.68:8787: Get \"https://172.25.201.68:8787/v2/\": http: server gave HTTP response to HTTPS client", "stderr_lines": ["Trying to pull 172.25.201.68:8787/ceph/daemon:v6.0.6-stable-6.0-pacific-centos-8-x86_64...", "Error: initializing source docker://172.25.201.68:8787/ceph/daemon:v6.0.6-stable-6.0-pacific-centos-8-x86_64: pinging container registry 172.25.201.68:8787: Get \"https://172.25.201.68:8787/v2/\": http: server gave HTTP response to HTTPS client"], "stdout": "", "stdout_lines": []} 2022-05-11 15:34:57,506 p=628194 u=stack n=ansible | [WARNING]: ('overcloud-controller-0 -> 172.25.201.102', '48d539a1-1679-f04f-5ac6-0000000000a6') missing from stats 2022-05-11 15:34:57,519 p=628194 u=stack n=ansible | NO MORE HOSTS LEFT ************************************************************* 2022-05-11 15:34:57,523 p=628194 u=stack n=ansible | PLAY RECAP ********************************************************************* 2022-05-11 15:34:57,525 p=628194 u=stack n=ansible | overcloud-controller-0 : ok=15 changed=3 unreachable=0 failed=1 skipped=25 rescued=0 ignored=0 2022-05-11 15:34:57,527 p=628194 u=stack n=ansible | 2022-05-11 15:34:57.527194 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Summary Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2022-05-11 15:34:57,528 p=628194 u=stack n=ansible | 2022-05-11 15:34:57.528269 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Total Tasks: 41 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2022-05-11 15:34:57,529 p=628194 u=stack n=ansible | 2022-05-11 15:34:57.529178 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Elapsed Time: 0:00:24.380145 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2022-05-11 15:34:57,530 p=628194 u=stack n=ansible | 2022-05-11 15:34:57.530100 | UUID | Info | Host | Task Name | Run Time 2022-05-11 15:34:57,531 p=628194 u=stack n=ansible | 2022-05-11 15:34:57.531023 | 48d539a1-1679-f04f-5ac6-000000000026 | SUMMARY | overcloud-controller-0 | push tripleo_cephadm_spec to bootstrap node if spec file exsits | 2.49s 2022-05-11 15:34:57,534 p=628194 u=stack n=ansible | 2022-05-11 15:34:57.532136 | 48d539a1-1679-f04f-5ac6-0000000000a3 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : create configured crush rules | 2.34s 2022-05-11 15:34:57,536 p=628194 u=stack n=ansible | 2022-05-11 15:34:57.534973 | 48d539a1-1679-f04f-5ac6-0000000000a6 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : insert new default crush rule into daemon to prevent restart | 2.14s 2022-05-11 15:34:57,537 p=628194 u=stack n=ansible | 2022-05-11 15:34:57.536420 | 48d539a1-1679-f04f-5ac6-0000000000a4 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : get id for new default crush rule | 1.26s 2022-05-11 15:34:57,538 p=628194 u=stack n=ansible | 2022-05-11 15:34:57.537563 | 48d539a1-1679-f04f-5ac6-00000000001b | SUMMARY | overcloud-controller-0 | tripleo_cephadm : List Ceph daemon instances on this host | 1.20s 2022-05-11 15:34:57,539 p=628194 u=stack n=ansible | 2022-05-11 15:34:57.538561 | 48d539a1-1679-f04f-5ac6-000000000076 | SUMMARY | overcloud-controller-0 | Assimilate configuration from tripleo_cephadm_assimilate_conf | 1.04s 2022-05-11 15:34:57,540 p=628194 u=stack n=ansible | 2022-05-11 15:34:57.539610 | 48d539a1-1679-f04f-5ac6-000000000023 | SUMMARY | overcloud-controller-0 | Ensure tripleo_cephadm_config_home (e.g. /etc/ceph) exists | 0.83s 2022-05-11 15:34:57,541 p=628194 u=stack n=ansible | 2022-05-11 15:34:57.540570 | 48d539a1-1679-f04f-5ac6-000000000019 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Stat cephadm file | 0.82s 2022-05-11 15:34:57,542 p=628194 u=stack n=ansible | 2022-05-11 15:34:57.541519 | 48d539a1-1679-f04f-5ac6-00000000002b | SUMMARY | overcloud-controller-0 | generate tripleo_cephadm_assimilate_conf on bootstrap node | 0.80s 2022-05-11 15:34:57,543 p=628194 u=stack n=ansible | 2022-05-11 15:34:57.542411 | 48d539a1-1679-f04f-5ac6-000000000087 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Apply spec | 0.65s 2022-05-11 15:34:57,544 p=628194 u=stack n=ansible | 2022-05-11 15:34:57.543295 | 48d539a1-1679-f04f-5ac6-00000000005e | SUMMARY | overcloud-controller-0 | tripleo_cephadm : If cephadm bootstrap was not run report the reason | 0.43s 2022-05-11 15:34:57,545 p=628194 u=stack n=ansible | 2022-05-11 15:34:57.544526 | 48d539a1-1679-f04f-5ac6-000000000088 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Show results of spec apply | 0.41s 2022-05-11 15:34:57,546 p=628194 u=stack n=ansible | 2022-05-11 15:34:57.545500 | 48d539a1-1679-f04f-5ac6-000000000024 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Ensure specs directory exists | 0.38s 2022-05-11 15:34:57,547 p=628194 u=stack n=ansible | 2022-05-11 15:34:57.546384 | 48d539a1-1679-f04f-5ac6-000000000081 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Stat spec file on bootstrap node | 0.37s 2022-05-11 15:34:57,547 p=628194 u=stack n=ansible | 2022-05-11 15:34:57.547263 | 48d539a1-1679-f04f-5ac6-000000000082 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Fail if spec file is missing | 0.37s 2022-05-11 15:34:57,548 p=628194 u=stack n=ansible | 2022-05-11 15:34:57.548126 | 48d539a1-1679-f04f-5ac6-000000000085 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Fail if ceph orchestrator is not available | 0.32s 2022-05-11 15:34:57,549 p=628194 u=stack n=ansible | 2022-05-11 15:34:57.548938 | 48d539a1-1679-f04f-5ac6-000000000084 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Get the ceph orchestrator status | 0.32s 2022-05-11 15:34:57,550 p=628194 u=stack n=ansible | 2022-05-11 15:34:57.549767 | 48d539a1-1679-f04f-5ac6-000000000086 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Set container images used by the applied services | 0.30s 2022-05-11 15:34:57,551 p=628194 u=stack n=ansible | 2022-05-11 15:34:57.550626 | 48d539a1-1679-f04f-5ac6-0000000000a5 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : set_fact info_ceph_default_crush_rule_yaml, ceph_osd_pool_default_crush_rule_name | 0.29s 2022-05-11 15:34:57,552 p=628194 u=stack n=ansible | 2022-05-11 15:34:57.551565 | 48d539a1-1679-f04f-5ac6-000000000083 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Get ceph_cli | 0.29s 2022-05-11 15:34:57,553 p=628194 u=stack n=ansible | 2022-05-11 15:34:57.552424 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ End Summary Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2022-05-11 15:34:57,554 p=628194 u=stack n=ansible | 2022-05-11 15:34:57.553346 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ State Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2022-05-11 15:34:57,555 p=628194 u=stack n=ansible | 2022-05-11 15:34:57.554239 | ~~~~~~~~~~~~~~~~~~ Number of nodes which did not deploy successfully: 1 ~~~~~~~~~~~~~~~~~ 2022-05-11 15:34:57,556 p=628194 u=stack n=ansible | 2022-05-11 15:34:57.555334 | The following node(s) had failures: overcloud-controller-0 2022-05-11 15:34:57,557 p=628194 u=stack n=ansible | 2022-05-11 15:34:57.556490 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Running /home/stack/overcloud-deploy/overcloud/config-download/overcloud/cephadm/cephadm_command.sh 2022-05-11 19:28:20,660 p=112379 u=stack n=ansible | PLAY [Deploy Ceph with cephadm] ************************************************ 2022-05-11 19:28:20,745 p=112379 u=stack n=ansible | 2022-05-11 19:28:20.744377 | 48d539a1-1679-13d1-7c5a-000000000018 | TASK | Install cephadm package 2022-05-11 19:28:20,900 p=112379 u=stack n=ansible | 2022-05-11 19:28:20.894428 | 48d539a1-1679-13d1-7c5a-000000000018 | SKIPPED | Install cephadm package | overcloud-controller-0 2022-05-11 19:28:20,942 p=112379 u=stack n=ansible | 2022-05-11 19:28:20.941098 | 48d539a1-1679-13d1-7c5a-000000000019 | TASK | Stat cephadm file 2022-05-11 19:28:21,723 p=112379 u=stack n=ansible | 2022-05-11 19:28:21.722380 | 48d539a1-1679-13d1-7c5a-000000000019 | OK | Stat cephadm file | overcloud-controller-0 2022-05-11 19:28:21,797 p=112379 u=stack n=ansible | 2022-05-11 19:28:21.796485 | 48d539a1-1679-13d1-7c5a-00000000001a | TASK | Fail if cephadm is not available 2022-05-11 19:28:21,955 p=112379 u=stack n=ansible | 2022-05-11 19:28:21.954010 | 48d539a1-1679-13d1-7c5a-00000000001a | SKIPPED | Fail if cephadm is not available | overcloud-controller-0 2022-05-11 19:28:21,995 p=112379 u=stack n=ansible | 2022-05-11 19:28:21.994430 | 48d539a1-1679-13d1-7c5a-00000000001b | TASK | List Ceph daemon instances on this host 2022-05-11 19:28:23,204 p=112379 u=stack n=ansible | 2022-05-11 19:28:23.202829 | 48d539a1-1679-13d1-7c5a-00000000001b | CHANGED | List Ceph daemon instances on this host | overcloud-controller-0 2022-05-11 19:28:23,266 p=112379 u=stack n=ansible | 2022-05-11 19:28:23.265557 | 48d539a1-1679-13d1-7c5a-00000000001d | TASK | Set list of found FSIDs 2022-05-11 19:28:23,405 p=112379 u=stack n=ansible | 2022-05-11 19:28:23.404136 | 48d539a1-1679-13d1-7c5a-00000000001d | SKIPPED | Set list of found FSIDs | overcloud-controller-0 2022-05-11 19:28:23,455 p=112379 u=stack n=ansible | 2022-05-11 19:28:23.454200 | 48d539a1-1679-13d1-7c5a-00000000001e | TASK | Fail if >1 FSID was discovered 2022-05-11 19:28:23,605 p=112379 u=stack n=ansible | 2022-05-11 19:28:23.603825 | 48d539a1-1679-13d1-7c5a-00000000001e | SKIPPED | Fail if >1 FSID was discovered | overcloud-controller-0 2022-05-11 19:28:23,660 p=112379 u=stack n=ansible | 2022-05-11 19:28:23.659624 | 48d539a1-1679-13d1-7c5a-00000000001f | TASK | Set FSID to the discovered value 2022-05-11 19:28:23,797 p=112379 u=stack n=ansible | 2022-05-11 19:28:23.795889 | 48d539a1-1679-13d1-7c5a-00000000001f | SKIPPED | Set FSID to the discovered value | overcloud-controller-0 2022-05-11 19:28:23,834 p=112379 u=stack n=ansible | 2022-05-11 19:28:23.833198 | 48d539a1-1679-13d1-7c5a-000000000020 | TASK | Set random tripleo_cephadm_fsid if no running ceph containers were found 2022-05-11 19:28:23,999 p=112379 u=stack n=ansible | 2022-05-11 19:28:23.998011 | 48d539a1-1679-13d1-7c5a-000000000020 | SKIPPED | Set random tripleo_cephadm_fsid if no running ceph containers were found | overcloud-controller-0 2022-05-11 19:28:24,070 p=112379 u=stack n=ansible | 2022-05-11 19:28:24.069579 | 48d539a1-1679-13d1-7c5a-000000000022 | TASK | Set first monitor IP if it was not passed 2022-05-11 19:28:24,223 p=112379 u=stack n=ansible | 2022-05-11 19:28:24.221869 | 48d539a1-1679-13d1-7c5a-000000000022 | SKIPPED | Set first monitor IP if it was not passed | overcloud-controller-0 2022-05-11 19:28:24,292 p=112379 u=stack n=ansible | 2022-05-11 19:28:24.291226 | 48d539a1-1679-13d1-7c5a-000000000023 | TASK | Ensure tripleo_cephadm_config_home (e.g. /etc/ceph) exists 2022-05-11 19:28:25,082 p=112379 u=stack n=ansible | 2022-05-11 19:28:25.080380 | 48d539a1-1679-13d1-7c5a-000000000023 | OK | Ensure tripleo_cephadm_config_home (e.g. /etc/ceph) exists | overcloud-controller-0 2022-05-11 19:28:25,131 p=112379 u=stack n=ansible | 2022-05-11 19:28:25.111245 | 48d539a1-1679-13d1-7c5a-000000000024 | TASK | Ensure specs directory exists 2022-05-11 19:28:25,527 p=112379 u=stack n=ansible | 2022-05-11 19:28:25.526422 | 48d539a1-1679-13d1-7c5a-000000000024 | OK | Ensure specs directory exists | overcloud-controller-0 2022-05-11 19:28:25,628 p=112379 u=stack n=ansible | 2022-05-11 19:28:25.627376 | 48d539a1-1679-13d1-7c5a-000000000025 | TASK | Stat spec file on ansible host 2022-05-11 19:28:26,637 p=112379 u=stack n=ansible | 2022-05-11 19:28:26.636251 | 48d539a1-1679-13d1-7c5a-000000000025 | OK | Stat spec file on ansible host | overcloud-controller-0 -> localhost 2022-05-11 19:28:26,641 p=112379 u=stack n=ansible | [WARNING]: ('overcloud-controller-0 -> localhost', '48d539a1-1679-13d1-7c5a-000000000025') missing from stats 2022-05-11 19:28:26,672 p=112379 u=stack n=ansible | 2022-05-11 19:28:26.671719 | 48d539a1-1679-13d1-7c5a-000000000026 | TASK | push tripleo_cephadm_spec to bootstrap node if spec file exsits 2022-05-11 19:28:29,113 p=112379 u=stack n=ansible | 2022-05-11 19:28:29.111590 | 48d539a1-1679-13d1-7c5a-000000000026 | CHANGED | push tripleo_cephadm_spec to bootstrap node if spec file exsits | overcloud-controller-0 2022-05-11 19:28:29,266 p=112379 u=stack n=ansible | 2022-05-11 19:28:29.265480 | 48d539a1-1679-13d1-7c5a-000000000028 | TASK | stat tripleo_cephadm_bootstrap_conf file 2022-05-11 19:28:29,415 p=112379 u=stack n=ansible | 2022-05-11 19:28:29.414244 | 48d539a1-1679-13d1-7c5a-000000000028 | SKIPPED | stat tripleo_cephadm_bootstrap_conf file | overcloud-controller-0 2022-05-11 19:28:29,469 p=112379 u=stack n=ansible | 2022-05-11 19:28:29.468268 | 48d539a1-1679-13d1-7c5a-000000000029 | TASK | push it to tripleo_cephadm_assimilate_conf on bootstrap node 2022-05-11 19:28:29,619 p=112379 u=stack n=ansible | 2022-05-11 19:28:29.618267 | 48d539a1-1679-13d1-7c5a-000000000029 | SKIPPED | push it to tripleo_cephadm_assimilate_conf on bootstrap node | overcloud-controller-0 2022-05-11 19:28:29,680 p=112379 u=stack n=ansible | 2022-05-11 19:28:29.679693 | 48d539a1-1679-13d1-7c5a-00000000002b | TASK | generate tripleo_cephadm_assimilate_conf on bootstrap node 2022-05-11 19:28:30,469 p=112379 u=stack n=ansible | 2022-05-11 19:28:30.467604 | 48d539a1-1679-13d1-7c5a-00000000002b | CHANGED | generate tripleo_cephadm_assimilate_conf on bootstrap node | overcloud-controller-0 2022-05-11 19:28:30,528 p=112379 u=stack n=ansible | 2022-05-11 19:28:30.507126 | 48d539a1-1679-13d1-7c5a-000000000056 | TASK | Add spec to necessary file list when using spec to bootstrap 2022-05-11 19:28:30,688 p=112379 u=stack n=ansible | 2022-05-11 19:28:30.687115 | 48d539a1-1679-13d1-7c5a-000000000056 | SKIPPED | Add spec to necessary file list when using spec to bootstrap | overcloud-controller-0 2022-05-11 19:28:30,767 p=112379 u=stack n=ansible | 2022-05-11 19:28:30.727708 | 48d539a1-1679-13d1-7c5a-000000000057 | TASK | Stat necessary files to bootstrap with cephadm 2022-05-11 19:28:30,962 p=112379 u=stack n=ansible | 2022-05-11 19:28:30.960678 | 48d539a1-1679-13d1-7c5a-000000000057 | SKIPPED | Stat necessary files to bootstrap with cephadm | overcloud-controller-0 | item=/home/ceph-admin/.ssh/id_rsa 2022-05-11 19:28:30,965 p=112379 u=stack n=ansible | 2022-05-11 19:28:30.964216 | 48d539a1-1679-13d1-7c5a-000000000057 | SKIPPED | Stat necessary files to bootstrap with cephadm | overcloud-controller-0 | item=/home/ceph-admin/.ssh/id_rsa.pub 2022-05-11 19:28:31,006 p=112379 u=stack n=ansible | 2022-05-11 19:28:31.004418 | 48d539a1-1679-13d1-7c5a-000000000058 | TASK | Fail if necessary files are missing 2022-05-11 19:28:31,504 p=112379 u=stack n=ansible | 2022-05-11 19:28:31.502685 | 48d539a1-1679-13d1-7c5a-000000000058 | SKIPPED | Fail if necessary files are missing | overcloud-controller-0 | item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': '/home/ceph-admin/.ssh/id_rsa', 'ansible_loop_var': 'item'} 2022-05-11 19:28:31,507 p=112379 u=stack n=ansible | 2022-05-11 19:28:31.506341 | 48d539a1-1679-13d1-7c5a-000000000058 | SKIPPED | Fail if necessary files are missing | overcloud-controller-0 | item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': '/home/ceph-admin/.ssh/id_rsa.pub', 'ansible_loop_var': 'item'} 2022-05-11 19:28:31,543 p=112379 u=stack n=ansible | 2022-05-11 19:28:31.542247 | 48d539a1-1679-13d1-7c5a-000000000059 | TASK | Stat pre ceph conf file in case we should bootrap with it 2022-05-11 19:28:31,883 p=112379 u=stack n=ansible | 2022-05-11 19:28:31.881951 | 48d539a1-1679-13d1-7c5a-000000000059 | SKIPPED | Stat pre ceph conf file in case we should bootrap with it | overcloud-controller-0 2022-05-11 19:28:31,917 p=112379 u=stack n=ansible | 2022-05-11 19:28:31.915534 | 48d539a1-1679-13d1-7c5a-00000000005b | TASK | Run cephadm bootstrap 2022-05-11 19:28:32,086 p=112379 u=stack n=ansible | 2022-05-11 19:28:32.085183 | 48d539a1-1679-13d1-7c5a-00000000005b | SKIPPED | Run cephadm bootstrap | overcloud-controller-0 2022-05-11 19:28:32,119 p=112379 u=stack n=ansible | 2022-05-11 19:28:32.118417 | 48d539a1-1679-13d1-7c5a-00000000005c | TASK | Show results of bootstrap 2022-05-11 19:28:32,297 p=112379 u=stack n=ansible | 2022-05-11 19:28:32.296263 | 48d539a1-1679-13d1-7c5a-00000000005c | SKIPPED | Show results of bootstrap | overcloud-controller-0 2022-05-11 19:28:32,355 p=112379 u=stack n=ansible | 2022-05-11 19:28:32.353670 | 48d539a1-1679-13d1-7c5a-00000000005e | TASK | If cephadm bootstrap was not run report the reason 2022-05-11 19:28:32,519 p=112379 u=stack n=ansible | 2022-05-11 19:28:32.517272 | 48d539a1-1679-13d1-7c5a-00000000005e | SKIPPED | If cephadm bootstrap was not run report the reason | overcloud-controller-0 2022-05-11 19:28:32,569 p=112379 u=stack n=ansible | 2022-05-11 19:28:32.567961 | 48d539a1-1679-13d1-7c5a-000000000075 | TASK | Ensure tripleo_cephadm_ceph_cli is set 2022-05-11 19:28:32,824 p=112379 u=stack n=ansible | 2022-05-11 19:28:32.815639 | cf451169-ccd0-42d5-88d5-12fd8ae550d5 | INCLUDED | /usr/share/ansible/roles/tripleo_cephadm/tasks/ceph_cli.yaml | overcloud-controller-0 2022-05-11 19:28:33,002 p=112379 u=stack n=ansible | 2022-05-11 19:28:33.000738 | 48d539a1-1679-13d1-7c5a-0000000001f1 | TASK | Set ceph CLI 2022-05-11 19:28:33,209 p=112379 u=stack n=ansible | 2022-05-11 19:28:33.207901 | 48d539a1-1679-13d1-7c5a-0000000001f1 | OK | Set ceph CLI | overcloud-controller-0 2022-05-11 19:28:33,266 p=112379 u=stack n=ansible | 2022-05-11 19:28:33.265254 | 48d539a1-1679-13d1-7c5a-000000000076 | TASK | Assimilate configuration from tripleo_cephadm_assimilate_conf 2022-05-11 19:28:34,349 p=112379 u=stack n=ansible | 2022-05-11 19:28:34.346425 | 48d539a1-1679-13d1-7c5a-000000000076 | OK | Assimilate configuration from tripleo_cephadm_assimilate_conf | overcloud-controller-0 2022-05-11 19:28:34,476 p=112379 u=stack n=ansible | 2022-05-11 19:28:34.475102 | 48d539a1-1679-13d1-7c5a-000000000081 | TASK | Stat spec file on bootstrap node 2022-05-11 19:28:34,646 p=112379 u=stack n=ansible | 2022-05-11 19:28:34.644341 | 48d539a1-1679-13d1-7c5a-000000000081 | SKIPPED | Stat spec file on bootstrap node | overcloud-controller-0 | item=/home/ceph-admin/specs/ceph_spec.yaml 2022-05-11 19:28:34,733 p=112379 u=stack n=ansible | 2022-05-11 19:28:34.700919 | 48d539a1-1679-13d1-7c5a-000000000082 | TASK | Fail if spec file is missing 2022-05-11 19:28:35,058 p=112379 u=stack n=ansible | 2022-05-11 19:28:35.056983 | 48d539a1-1679-13d1-7c5a-000000000082 | SKIPPED | Fail if spec file is missing | overcloud-controller-0 | item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': '/home/ceph-admin/specs/ceph_spec.yaml', 'ansible_loop_var': 'item'} 2022-05-11 19:28:35,129 p=112379 u=stack n=ansible | 2022-05-11 19:28:35.128183 | 48d539a1-1679-13d1-7c5a-000000000083 | TASK | Get ceph_cli 2022-05-11 19:28:35,433 p=112379 u=stack n=ansible | 2022-05-11 19:28:35.431766 | 48d539a1-1679-13d1-7c5a-000000000083 | SKIPPED | Get ceph_cli | overcloud-controller-0 2022-05-11 19:28:35,462 p=112379 u=stack n=ansible | 2022-05-11 19:28:35.461781 | 48d539a1-1679-13d1-7c5a-000000000084 | TASK | Get the ceph orchestrator status 2022-05-11 19:28:35,756 p=112379 u=stack n=ansible | 2022-05-11 19:28:35.755364 | 48d539a1-1679-13d1-7c5a-000000000084 | SKIPPED | Get the ceph orchestrator status | overcloud-controller-0 2022-05-11 19:28:35,787 p=112379 u=stack n=ansible | 2022-05-11 19:28:35.786138 | 48d539a1-1679-13d1-7c5a-000000000085 | TASK | Fail if ceph orchestrator is not available 2022-05-11 19:28:36,073 p=112379 u=stack n=ansible | 2022-05-11 19:28:36.072120 | 48d539a1-1679-13d1-7c5a-000000000085 | SKIPPED | Fail if ceph orchestrator is not available | overcloud-controller-0 2022-05-11 19:28:36,104 p=112379 u=stack n=ansible | 2022-05-11 19:28:36.103503 | 48d539a1-1679-13d1-7c5a-000000000086 | TASK | Set container images used by the applied services 2022-05-11 19:28:36,398 p=112379 u=stack n=ansible | 2022-05-11 19:28:36.397347 | 48d539a1-1679-13d1-7c5a-000000000086 | SKIPPED | Set container images used by the applied services | overcloud-controller-0 2022-05-11 19:28:36,453 p=112379 u=stack n=ansible | 2022-05-11 19:28:36.452512 | 48d539a1-1679-13d1-7c5a-000000000087 | TASK | Apply spec 2022-05-11 19:28:36,876 p=112379 u=stack n=ansible | 2022-05-11 19:28:36.874943 | 48d539a1-1679-13d1-7c5a-000000000087 | SKIPPED | Apply spec | overcloud-controller-0 2022-05-11 19:28:36,932 p=112379 u=stack n=ansible | 2022-05-11 19:28:36.931683 | 48d539a1-1679-13d1-7c5a-000000000088 | TASK | Show results of spec apply 2022-05-11 19:28:37,230 p=112379 u=stack n=ansible | 2022-05-11 19:28:37.229116 | 48d539a1-1679-13d1-7c5a-000000000088 | SKIPPED | Show results of spec apply | overcloud-controller-0 2022-05-11 19:28:37,303 p=112379 u=stack n=ansible | 2022-05-11 19:28:37.302453 | 48d539a1-1679-13d1-7c5a-000000000089 | TASK | Wait for the expected number of osds to be running 2022-05-11 19:28:37,592 p=112379 u=stack n=ansible | 2022-05-11 19:28:37.591212 | 48d539a1-1679-13d1-7c5a-000000000089 | SKIPPED | Wait for the expected number of osds to be running | overcloud-controller-0 2022-05-11 19:28:37,669 p=112379 u=stack n=ansible | 2022-05-11 19:28:37.667739 | 48d539a1-1679-13d1-7c5a-0000000000a2 | TASK | Get ceph_cli 2022-05-11 19:28:38,021 p=112379 u=stack n=ansible | 2022-05-11 19:28:38.019871 | 2fb5f3b7-49a1-4725-83dd-054cb7ad4bfe | INCLUDED | /usr/share/ansible/roles/tripleo_cephadm/tasks/ceph_cli.yaml | overcloud-controller-0 2022-05-11 19:28:38,572 p=112379 u=stack n=ansible | 2022-05-11 19:28:38.570582 | 48d539a1-1679-13d1-7c5a-00000000021e | TASK | Set ceph CLI 2022-05-11 19:28:38,771 p=112379 u=stack n=ansible | 2022-05-11 19:28:38.769654 | 48d539a1-1679-13d1-7c5a-00000000021e | OK | Set ceph CLI | overcloud-controller-0 2022-05-11 19:28:38,840 p=112379 u=stack n=ansible | 2022-05-11 19:28:38.839153 | 48d539a1-1679-13d1-7c5a-0000000000a3 | TASK | create configured crush rules 2022-05-11 19:28:40,278 p=112379 u=stack n=ansible | 2022-05-11 19:28:40.277215 | 48d539a1-1679-13d1-7c5a-0000000000a3 | OK | create configured crush rules | overcloud-controller-0 | item={'class': 'hdd', 'default': True, 'name': 'HDD', 'root': 'default', 'type': 'host'} 2022-05-11 19:28:41,265 p=112379 u=stack n=ansible | 2022-05-11 19:28:41.264247 | 48d539a1-1679-13d1-7c5a-0000000000a3 | OK | create configured crush rules | overcloud-controller-0 | item={'class': 'ssd', 'default': False, 'name': 'SSD', 'root': 'default', 'type': 'host'} 2022-05-11 19:28:41,304 p=112379 u=stack n=ansible | 2022-05-11 19:28:41.302957 | 48d539a1-1679-13d1-7c5a-0000000000a4 | TASK | get id for new default crush rule 2022-05-11 19:28:42,463 p=112379 u=stack n=ansible | 2022-05-11 19:28:42.461982 | 48d539a1-1679-13d1-7c5a-0000000000a4 | OK | get id for new default crush rule | overcloud-controller-0 | item={'class': 'hdd', 'default': True, 'name': 'HDD', 'root': 'default', 'type': 'host'} 2022-05-11 19:28:42,491 p=112379 u=stack n=ansible | 2022-05-11 19:28:42.490448 | 48d539a1-1679-13d1-7c5a-0000000000a4 | SKIPPED | get id for new default crush rule | overcloud-controller-0 | item={'class': 'ssd', 'default': False, 'name': 'SSD', 'root': 'default', 'type': 'host'} 2022-05-11 19:28:42,526 p=112379 u=stack n=ansible | 2022-05-11 19:28:42.525416 | 48d539a1-1679-13d1-7c5a-0000000000a5 | TASK | set_fact info_ceph_default_crush_rule_yaml, ceph_osd_pool_default_crush_rule_name 2022-05-11 19:28:42,817 p=112379 u=stack n=ansible | 2022-05-11 19:28:42.816165 | 48d539a1-1679-13d1-7c5a-0000000000a5 | OK | set_fact info_ceph_default_crush_rule_yaml, ceph_osd_pool_default_crush_rule_name | overcloud-controller-0 | item={'cmd': ['podman', 'run', '--rm', '--net=host', '-v', '/etc/ceph:/etc/ceph:z', '-v', '/var/lib/ceph/:/var/lib/ceph/:z', '-v', '/var/log/ceph/:/var/log/ceph/:z', '--entrypoint=ceph', '172.25.201.68:8787/ceph/daemon:v6.0.6-stable-6.0-pacific-centos-8-x86_64', '-n', 'client.admin', '-k', '/etc/ceph/ceph.client.admin.keyring', '--cluster', 'ceph', 'osd', 'crush', 'rule', 'dump', 'HDD', '--format=json'], 'start': '2022-05-11 11:28:41.676080', 'end': '2022-05-11 11:28:42.419171', 'delta': '0:00:00.743091', 'rc': 0, 'stdout': '\n{"rule_id":1,"rule_name":"HDD","ruleset":1,"type":1,"min_size":1,"max_size":10,"steps":[{"op":"take","item":-2,"item_name":"default~hdd"},{"op":"chooseleaf_firstn","num":0,"type":"host"},{"op":"emit"}]}', 'stderr': '', 'changed': False, 'invocation': {'module_args': {'name': 'HDD', 'cluster': 'ceph', 'state': 'info', 'rule_type': None, 'bucket_root': None, 'bucket_type': None, 'device_class': None, 'profile': None}}, 'stdout_lines': ['', '{"rule_id":1,"rule_name":"HDD","ruleset":1,"type":1,"min_size":1,"max_size":10,"steps":[{"op":"take","item":-2,"item_name":"default~hdd"},{"op":"chooseleaf_firstn","num":0,"type":"host"},{"op":"emit"}]}'], 'stderr_lines': [], 'failed': False, 'item': {'class': 'hdd', 'default': True, 'name': 'HDD', 'root': 'default', 'type': 'host'}, 'ansible_loop_var': 'item'} 2022-05-11 19:28:42,837 p=112379 u=stack n=ansible | 2022-05-11 19:28:42.836210 | 48d539a1-1679-13d1-7c5a-0000000000a5 | SKIPPED | set_fact info_ceph_default_crush_rule_yaml, ceph_osd_pool_default_crush_rule_name | overcloud-controller-0 | item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': {'class': 'ssd', 'default': False, 'name': 'SSD', 'root': 'default', 'type': 'host'}, 'ansible_loop_var': 'item'} 2022-05-11 19:28:43,091 p=112379 u=stack n=ansible | 2022-05-11 19:28:43.090053 | 48d539a1-1679-13d1-7c5a-0000000000a6 | TASK | insert new default crush rule into daemon to prevent restart 2022-05-11 19:28:44,006 p=112379 u=stack n=ansible | 2022-05-11 19:28:44.004692 | 48d539a1-1679-13d1-7c5a-0000000000a6 | OK | insert new default crush rule into daemon to prevent restart | overcloud-controller-0 -> 172.25.201.98 | item=overcloud-controller-0 2022-05-11 19:28:44,021 p=112379 u=stack n=ansible | [WARNING]: ('overcloud-controller-0 -> 172.25.201.98', '48d539a1-1679-13d1-7c5a-0000000000a6') missing from stats 2022-05-11 19:28:45,886 p=112379 u=stack n=ansible | 2022-05-11 19:28:45.884913 | 48d539a1-1679-13d1-7c5a-0000000000a6 | OK | insert new default crush rule into daemon to prevent restart | overcloud-controller-0 -> 172.25.201.103 | item=overcloud-controller-1 2022-05-11 19:28:45,890 p=112379 u=stack n=ansible | [WARNING]: ('overcloud-controller-0 -> 172.25.201.103', '48d539a1-1679-13d1-7c5a-0000000000a6') missing from stats 2022-05-11 19:28:46,890 p=112379 u=stack n=ansible | 2022-05-11 19:28:46.888773 | 48d539a1-1679-13d1-7c5a-0000000000a6 | OK | insert new default crush rule into daemon to prevent restart | overcloud-controller-0 -> 172.25.201.102 | item=overcloud-controller-2 2022-05-11 19:28:46,893 p=112379 u=stack n=ansible | [WARNING]: ('overcloud-controller-0 -> 172.25.201.102', '48d539a1-1679-13d1-7c5a-0000000000a6') missing from stats 2022-05-11 19:28:47,031 p=112379 u=stack n=ansible | 2022-05-11 19:28:47.030427 | 48d539a1-1679-13d1-7c5a-0000000000b7 | TASK | Create pool(s) 2022-05-11 19:28:50,622 p=112379 u=stack n=ansible | 2022-05-11 19:28:50.620262 | 48d539a1-1679-13d1-7c5a-0000000000b7 | CHANGED | Create pool(s) | overcloud-controller-0 | item={'name': 'vms', 'pg_num': '16', 'rule_name': 'replicated_rule', 'application': 'rbd'} 2022-05-11 19:28:53,987 p=112379 u=stack n=ansible | 2022-05-11 19:28:53.985646 | 48d539a1-1679-13d1-7c5a-0000000000b7 | CHANGED | Create pool(s) | overcloud-controller-0 | item={'name': 'volumes', 'pg_num': '16', 'rule_name': 'replicated_rule', 'application': 'rbd'} 2022-05-11 19:28:57,265 p=112379 u=stack n=ansible | 2022-05-11 19:28:57.264314 | 48d539a1-1679-13d1-7c5a-0000000000b7 | CHANGED | Create pool(s) | overcloud-controller-0 | item={'name': 'images', 'pg_num': '16', 'rule_name': 'replicated_rule', 'application': 'rbd'} 2022-05-11 19:29:00,586 p=112379 u=stack n=ansible | 2022-05-11 19:29:00.585419 | 48d539a1-1679-13d1-7c5a-0000000000b7 | CHANGED | Create pool(s) | overcloud-controller-0 | item={'name': 'metrics', 'pg_num': '16', 'rule_name': 'replicated_rule', 'application': 'openstack_gnocchi'} 2022-05-11 19:29:04,359 p=112379 u=stack n=ansible | 2022-05-11 19:29:04.358684 | 48d539a1-1679-13d1-7c5a-0000000000b7 | CHANGED | Create pool(s) | overcloud-controller-0 | item={'name': 'backups', 'pg_num': '16', 'rule_name': 'replicated_rule', 'application': 'rbd'} 2022-05-11 19:29:07,781 p=112379 u=stack n=ansible | 2022-05-11 19:29:07.780709 | 48d539a1-1679-13d1-7c5a-0000000000b7 | CHANGED | Create pool(s) | overcloud-controller-0 | item={'application': 'rbd', 'name': 'slow_pool', 'rule_name': 'HDD'} 2022-05-11 19:29:10,931 p=112379 u=stack n=ansible | 2022-05-11 19:29:10.930043 | 48d539a1-1679-13d1-7c5a-0000000000b7 | CHANGED | Create pool(s) | overcloud-controller-0 | item={'application': 'rbd', 'name': 'fast_pool', 'rule_name': 'SSD'} 2022-05-11 19:29:11,101 p=112379 u=stack n=ansible | 2022-05-11 19:29:11.099889 | 48d539a1-1679-13d1-7c5a-0000000000b9 | TASK | Get the RBD ceph_cli 2022-05-11 19:29:11,275 p=112379 u=stack n=ansible | 2022-05-11 19:29:11.274125 | 0a1b37c7-cedb-4284-8264-5991e0d0405b | INCLUDED | /usr/share/ansible/roles/tripleo_cephadm/tasks/ceph_cli.yaml | overcloud-controller-0 2022-05-11 19:29:11,448 p=112379 u=stack n=ansible | 2022-05-11 19:29:11.446537 | 48d539a1-1679-13d1-7c5a-00000000025b | TASK | Set ceph CLI 2022-05-11 19:29:11,636 p=112379 u=stack n=ansible | 2022-05-11 19:29:11.634725 | 48d539a1-1679-13d1-7c5a-00000000025b | OK | Set ceph CLI | overcloud-controller-0 2022-05-11 19:29:11,748 p=112379 u=stack n=ansible | 2022-05-11 19:29:11.747223 | 48d539a1-1679-13d1-7c5a-0000000000ba | TASK | Set trash interval 2022-05-11 19:29:12,506 p=112379 u=stack n=ansible | 2022-05-11 19:29:12.504460 | 48d539a1-1679-13d1-7c5a-0000000000ba | OK | Set trash interval | overcloud-controller-0 | item=volumes 2022-05-11 19:29:12,627 p=112379 u=stack n=ansible | 2022-05-11 19:29:12.626393 | 48d539a1-1679-13d1-7c5a-0000000000c8 | TASK | Collect the host and build the resulting host list 2022-05-11 19:29:12,732 p=112379 u=stack n=ansible | 2022-05-11 19:29:12.730919 | 48d539a1-1679-13d1-7c5a-0000000000c8 | SKIPPED | Collect the host and build the resulting host list | overcloud-controller-0 | item=overcloud-controller-0 2022-05-11 19:29:12,746 p=112379 u=stack n=ansible | 2022-05-11 19:29:12.744868 | 48d539a1-1679-13d1-7c5a-0000000000c8 | SKIPPED | Collect the host and build the resulting host list | overcloud-controller-0 | item=overcloud-controller-1 2022-05-11 19:29:12,759 p=112379 u=stack n=ansible | 2022-05-11 19:29:12.757877 | 48d539a1-1679-13d1-7c5a-0000000000c8 | SKIPPED | Collect the host and build the resulting host list | overcloud-controller-0 | item=overcloud-controller-2 2022-05-11 19:29:12,814 p=112379 u=stack n=ansible | 2022-05-11 19:29:12.813081 | 48d539a1-1679-13d1-7c5a-0000000000c9 | TASK | Collect the host and build the resulting host list 2022-05-11 19:29:13,004 p=112379 u=stack n=ansible | 2022-05-11 19:29:13.002640 | 48d539a1-1679-13d1-7c5a-0000000000c9 | OK | Collect the host and build the resulting host list | overcloud-controller-0 | item=overcloud-controller-0 2022-05-11 19:29:13,060 p=112379 u=stack n=ansible | 2022-05-11 19:29:13.059677 | 48d539a1-1679-13d1-7c5a-0000000000c9 | OK | Collect the host and build the resulting host list | overcloud-controller-0 | item=overcloud-controller-1 2022-05-11 19:29:13,101 p=112379 u=stack n=ansible | 2022-05-11 19:29:13.100686 | 48d539a1-1679-13d1-7c5a-0000000000c9 | OK | Collect the host and build the resulting host list | overcloud-controller-0 | item=overcloud-controller-2 2022-05-11 19:29:13,184 p=112379 u=stack n=ansible | 2022-05-11 19:29:13.182876 | 48d539a1-1679-13d1-7c5a-0000000000ca | TASK | Get certificate from remote node if needed 2022-05-11 19:29:13,268 p=112379 u=stack n=ansible | 2022-05-11 19:29:13.266538 | 48d539a1-1679-13d1-7c5a-0000000000ca | SKIPPED | Get certificate from remote node if needed | overcloud-controller-0 2022-05-11 19:29:13,346 p=112379 u=stack n=ansible | 2022-05-11 19:29:13.345108 | 48d539a1-1679-13d1-7c5a-0000000000cb | TASK | Create the RGW Daemon spec definition 2022-05-11 19:29:14,193 p=112379 u=stack n=ansible | An exception occurred during task execution. To see the full traceback, use -vvv. The error was: ModuleNotFoundError: No module named 'tripleo_common' 2022-05-11 19:29:14,195 p=112379 u=stack n=ansible | 2022-05-11 19:29:14.194335 | 48d539a1-1679-13d1-7c5a-0000000000cb | FATAL | Create the RGW Daemon spec definition | overcloud-controller-0 | error={"changed": false, "module_stderr": "Traceback (most recent call last):\n File \"\", line 102, in \n File \"\", line 94, in _ansiballz_main\n File \"\", line 40, in invoke_module\n File \"/usr/lib64/python3.6/runpy.py\", line 205, in run_module\n return _run_module_code(code, init_globals, run_name, mod_spec)\n File \"/usr/lib64/python3.6/runpy.py\", line 96, in _run_module_code\n mod_name, mod_spec, pkg_name, script_name)\n File \"/usr/lib64/python3.6/runpy.py\", line 85, in _run_code\n exec(code, run_globals)\n File \"/tmp/ansible_ceph_mkspec_payload_iiu3rx9h/ansible_ceph_mkspec_payload.zip/ansible/modules/ceph_mkspec.py\", line 24, in \nModuleNotFoundError: No module named 'tripleo_common'\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1} 2022-05-11 19:29:14,201 p=112379 u=stack n=ansible | PLAY RECAP ********************************************************************* 2022-05-11 19:29:14,203 p=112379 u=stack n=ansible | overcloud-controller-0 : ok=21 changed=4 unreachable=0 failed=1 skipped=27 rescued=0 ignored=0 2022-05-11 19:29:14,205 p=112379 u=stack n=ansible | 2022-05-11 19:29:14.205155 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Summary Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2022-05-11 19:29:14,206 p=112379 u=stack n=ansible | 2022-05-11 19:29:14.206225 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Total Tasks: 49 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2022-05-11 19:29:14,207 p=112379 u=stack n=ansible | 2022-05-11 19:29:14.207186 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Elapsed Time: 0:00:53.555235 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2022-05-11 19:29:14,208 p=112379 u=stack n=ansible | 2022-05-11 19:29:14.208111 | UUID | Info | Host | Task Name | Run Time 2022-05-11 19:29:14,209 p=112379 u=stack n=ansible | 2022-05-11 19:29:14.209035 | 48d539a1-1679-13d1-7c5a-0000000000b7 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Create pool(s) | 23.93s 2022-05-11 19:29:14,210 p=112379 u=stack n=ansible | 2022-05-11 19:29:14.209959 | 48d539a1-1679-13d1-7c5a-0000000000a6 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : insert new default crush rule into daemon to prevent restart | 3.81s 2022-05-11 19:29:14,211 p=112379 u=stack n=ansible | 2022-05-11 19:29:14.210921 | 48d539a1-1679-13d1-7c5a-000000000026 | SUMMARY | overcloud-controller-0 | push tripleo_cephadm_spec to bootstrap node if spec file exsits | 2.44s 2022-05-11 19:29:14,212 p=112379 u=stack n=ansible | 2022-05-11 19:29:14.211839 | 48d539a1-1679-13d1-7c5a-0000000000a3 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : create configured crush rules | 2.43s 2022-05-11 19:29:14,213 p=112379 u=stack n=ansible | 2022-05-11 19:29:14.212659 | 48d539a1-1679-13d1-7c5a-00000000001b | SUMMARY | overcloud-controller-0 | tripleo_cephadm : List Ceph daemon instances on this host | 1.21s 2022-05-11 19:29:14,214 p=112379 u=stack n=ansible | 2022-05-11 19:29:14.213501 | 48d539a1-1679-13d1-7c5a-0000000000a4 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : get id for new default crush rule | 1.19s 2022-05-11 19:29:14,215 p=112379 u=stack n=ansible | 2022-05-11 19:29:14.214381 | 48d539a1-1679-13d1-7c5a-000000000076 | SUMMARY | overcloud-controller-0 | Assimilate configuration from tripleo_cephadm_assimilate_conf | 1.08s 2022-05-11 19:29:14,215 p=112379 u=stack n=ansible | 2022-05-11 19:29:14.215213 | 48d539a1-1679-13d1-7c5a-0000000000cb | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Create the RGW Daemon spec definition | 0.85s 2022-05-11 19:29:14,216 p=112379 u=stack n=ansible | 2022-05-11 19:29:14.216047 | 48d539a1-1679-13d1-7c5a-000000000023 | SUMMARY | overcloud-controller-0 | Ensure tripleo_cephadm_config_home (e.g. /etc/ceph) exists | 0.79s 2022-05-11 19:29:14,217 p=112379 u=stack n=ansible | 2022-05-11 19:29:14.216873 | 48d539a1-1679-13d1-7c5a-00000000002b | SUMMARY | overcloud-controller-0 | generate tripleo_cephadm_assimilate_conf on bootstrap node | 0.79s 2022-05-11 19:29:14,218 p=112379 u=stack n=ansible | 2022-05-11 19:29:14.217837 | 48d539a1-1679-13d1-7c5a-000000000019 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Stat cephadm file | 0.78s 2022-05-11 19:29:14,219 p=112379 u=stack n=ansible | 2022-05-11 19:29:14.218667 | 48d539a1-1679-13d1-7c5a-0000000000ba | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Set trash interval | 0.76s 2022-05-11 19:29:14,220 p=112379 u=stack n=ansible | 2022-05-11 19:29:14.219536 | 48d539a1-1679-13d1-7c5a-000000000058 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Fail if necessary files are missing | 0.50s 2022-05-11 19:29:14,221 p=112379 u=stack n=ansible | 2022-05-11 19:29:14.220427 | 48d539a1-1679-13d1-7c5a-000000000087 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Apply spec | 0.42s 2022-05-11 19:29:14,222 p=112379 u=stack n=ansible | 2022-05-11 19:29:14.221298 | 48d539a1-1679-13d1-7c5a-000000000024 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Ensure specs directory exists | 0.40s 2022-05-11 19:29:14,222 p=112379 u=stack n=ansible | 2022-05-11 19:29:14.222297 | 48d539a1-1679-13d1-7c5a-000000000059 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Stat pre ceph conf file in case we should bootrap with it | 0.34s 2022-05-11 19:29:14,223 p=112379 u=stack n=ansible | 2022-05-11 19:29:14.223184 | 48d539a1-1679-13d1-7c5a-000000000082 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Fail if spec file is missing | 0.33s 2022-05-11 19:29:14,224 p=112379 u=stack n=ansible | 2022-05-11 19:29:14.224054 | 48d539a1-1679-13d1-7c5a-0000000000a5 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : set_fact info_ceph_default_crush_rule_yaml, ceph_osd_pool_default_crush_rule_name | 0.32s 2022-05-11 19:29:14,225 p=112379 u=stack n=ansible | 2022-05-11 19:29:14.225075 | 48d539a1-1679-13d1-7c5a-000000000083 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Get ceph_cli | 0.30s 2022-05-11 19:29:14,226 p=112379 u=stack n=ansible | 2022-05-11 19:29:14.225931 | 48d539a1-1679-13d1-7c5a-000000000088 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Show results of spec apply | 0.30s 2022-05-11 19:29:14,227 p=112379 u=stack n=ansible | 2022-05-11 19:29:14.226789 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ End Summary Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2022-05-11 19:29:14,228 p=112379 u=stack n=ansible | 2022-05-11 19:29:14.227751 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ State Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2022-05-11 19:29:14,229 p=112379 u=stack n=ansible | 2022-05-11 19:29:14.228578 | ~~~~~~~~~~~~~~~~~~ Number of nodes which did not deploy successfully: 1 ~~~~~~~~~~~~~~~~~ 2022-05-11 19:29:14,230 p=112379 u=stack n=ansible | 2022-05-11 19:29:14.229415 | The following node(s) had failures: overcloud-controller-0 2022-05-11 19:29:14,230 p=112379 u=stack n=ansible | 2022-05-11 19:29:14.230241 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Running /home/stack/overcloud-deploy/overcloud/config-download/overcloud/cephadm/cephadm_command.sh 2022-05-13 12:25:59,222 p=331839 u=stack n=ansible | PLAY [Deploy Ceph with cephadm] ************************************************ 2022-05-13 12:25:59,309 p=331839 u=stack n=ansible | 2022-05-13 12:25:59.306311 | 48d539a1-1679-f37d-2794-000000000018 | TASK | Install cephadm package 2022-05-13 12:25:59,597 p=331839 u=stack n=ansible | 2022-05-13 12:25:59.595240 | 48d539a1-1679-f37d-2794-000000000018 | SKIPPED | Install cephadm package | overcloud-controller-0 2022-05-13 12:25:59,626 p=331839 u=stack n=ansible | 2022-05-13 12:25:59.625474 | 48d539a1-1679-f37d-2794-000000000019 | TASK | Stat cephadm file 2022-05-13 12:26:00,454 p=331839 u=stack n=ansible | 2022-05-13 12:26:00.452442 | 48d539a1-1679-f37d-2794-000000000019 | OK | Stat cephadm file | overcloud-controller-0 2022-05-13 12:26:00,549 p=331839 u=stack n=ansible | 2022-05-13 12:26:00.548476 | 48d539a1-1679-f37d-2794-00000000001a | TASK | Fail if cephadm is not available 2022-05-13 12:26:00,697 p=331839 u=stack n=ansible | 2022-05-13 12:26:00.695704 | 48d539a1-1679-f37d-2794-00000000001a | SKIPPED | Fail if cephadm is not available | overcloud-controller-0 2022-05-13 12:26:00,778 p=331839 u=stack n=ansible | 2022-05-13 12:26:00.777242 | 48d539a1-1679-f37d-2794-00000000001b | TASK | List Ceph daemon instances on this host 2022-05-13 12:26:01,981 p=331839 u=stack n=ansible | 2022-05-13 12:26:01.978494 | 48d539a1-1679-f37d-2794-00000000001b | CHANGED | List Ceph daemon instances on this host | overcloud-controller-0 2022-05-13 12:26:02,116 p=331839 u=stack n=ansible | 2022-05-13 12:26:02.115041 | 48d539a1-1679-f37d-2794-00000000001d | TASK | Set list of found FSIDs 2022-05-13 12:26:02,319 p=331839 u=stack n=ansible | 2022-05-13 12:26:02.317648 | 48d539a1-1679-f37d-2794-00000000001d | SKIPPED | Set list of found FSIDs | overcloud-controller-0 2022-05-13 12:26:02,400 p=331839 u=stack n=ansible | 2022-05-13 12:26:02.359070 | 48d539a1-1679-f37d-2794-00000000001e | TASK | Fail if >1 FSID was discovered 2022-05-13 12:26:02,606 p=331839 u=stack n=ansible | 2022-05-13 12:26:02.605309 | 48d539a1-1679-f37d-2794-00000000001e | SKIPPED | Fail if >1 FSID was discovered | overcloud-controller-0 2022-05-13 12:26:02,665 p=331839 u=stack n=ansible | 2022-05-13 12:26:02.664362 | 48d539a1-1679-f37d-2794-00000000001f | TASK | Set FSID to the discovered value 2022-05-13 12:26:02,838 p=331839 u=stack n=ansible | 2022-05-13 12:26:02.837025 | 48d539a1-1679-f37d-2794-00000000001f | SKIPPED | Set FSID to the discovered value | overcloud-controller-0 2022-05-13 12:26:02,922 p=331839 u=stack n=ansible | 2022-05-13 12:26:02.920862 | 48d539a1-1679-f37d-2794-000000000020 | TASK | Set random tripleo_cephadm_fsid if no running ceph containers were found 2022-05-13 12:26:03,332 p=331839 u=stack n=ansible | 2022-05-13 12:26:03.330839 | 48d539a1-1679-f37d-2794-000000000020 | SKIPPED | Set random tripleo_cephadm_fsid if no running ceph containers were found | overcloud-controller-0 2022-05-13 12:26:03,404 p=331839 u=stack n=ansible | 2022-05-13 12:26:03.383214 | 48d539a1-1679-f37d-2794-000000000022 | TASK | Set first monitor IP if it was not passed 2022-05-13 12:26:03,544 p=331839 u=stack n=ansible | 2022-05-13 12:26:03.542617 | 48d539a1-1679-f37d-2794-000000000022 | SKIPPED | Set first monitor IP if it was not passed | overcloud-controller-0 2022-05-13 12:26:03,622 p=331839 u=stack n=ansible | 2022-05-13 12:26:03.591902 | 48d539a1-1679-f37d-2794-000000000023 | TASK | Ensure tripleo_cephadm_config_home (e.g. /etc/ceph) exists 2022-05-13 12:26:04,417 p=331839 u=stack n=ansible | 2022-05-13 12:26:04.415798 | 48d539a1-1679-f37d-2794-000000000023 | OK | Ensure tripleo_cephadm_config_home (e.g. /etc/ceph) exists | overcloud-controller-0 2022-05-13 12:26:04,499 p=331839 u=stack n=ansible | 2022-05-13 12:26:04.497891 | 48d539a1-1679-f37d-2794-000000000024 | TASK | Ensure specs directory exists 2022-05-13 12:26:04,871 p=331839 u=stack n=ansible | 2022-05-13 12:26:04.870057 | 48d539a1-1679-f37d-2794-000000000024 | OK | Ensure specs directory exists | overcloud-controller-0 2022-05-13 12:26:05,020 p=331839 u=stack n=ansible | 2022-05-13 12:26:05.018905 | 48d539a1-1679-f37d-2794-000000000025 | TASK | Stat spec file on ansible host 2022-05-13 12:26:05,859 p=331839 u=stack n=ansible | 2022-05-13 12:26:05.857410 | 48d539a1-1679-f37d-2794-000000000025 | OK | Stat spec file on ansible host | overcloud-controller-0 -> localhost 2022-05-13 12:26:05,862 p=331839 u=stack n=ansible | [WARNING]: ('overcloud-controller-0 -> localhost', '48d539a1-1679-f37d-2794-000000000025') missing from stats 2022-05-13 12:26:05,947 p=331839 u=stack n=ansible | 2022-05-13 12:26:05.946203 | 48d539a1-1679-f37d-2794-000000000026 | TASK | push tripleo_cephadm_spec to bootstrap node if spec file exsits 2022-05-13 12:26:07,260 p=331839 u=stack n=ansible | 2022-05-13 12:26:07.251165 | 48d539a1-1679-f37d-2794-000000000026 | CHANGED | push tripleo_cephadm_spec to bootstrap node if spec file exsits | overcloud-controller-0 2022-05-13 12:26:07,657 p=331839 u=stack n=ansible | 2022-05-13 12:26:07.655837 | 48d539a1-1679-f37d-2794-000000000028 | TASK | stat tripleo_cephadm_bootstrap_conf file 2022-05-13 12:26:07,791 p=331839 u=stack n=ansible | 2022-05-13 12:26:07.790240 | 48d539a1-1679-f37d-2794-000000000028 | SKIPPED | stat tripleo_cephadm_bootstrap_conf file | overcloud-controller-0 2022-05-13 12:26:07,885 p=331839 u=stack n=ansible | 2022-05-13 12:26:07.848536 | 48d539a1-1679-f37d-2794-000000000029 | TASK | push it to tripleo_cephadm_assimilate_conf on bootstrap node 2022-05-13 12:26:08,029 p=331839 u=stack n=ansible | 2022-05-13 12:26:08.027879 | 48d539a1-1679-f37d-2794-000000000029 | SKIPPED | push it to tripleo_cephadm_assimilate_conf on bootstrap node | overcloud-controller-0 2022-05-13 12:26:08,145 p=331839 u=stack n=ansible | 2022-05-13 12:26:08.103833 | 48d539a1-1679-f37d-2794-00000000002b | TASK | generate tripleo_cephadm_assimilate_conf on bootstrap node 2022-05-13 12:26:08,814 p=331839 u=stack n=ansible | 2022-05-13 12:26:08.812780 | 48d539a1-1679-f37d-2794-00000000002b | CHANGED | generate tripleo_cephadm_assimilate_conf on bootstrap node | overcloud-controller-0 2022-05-13 12:26:08,887 p=331839 u=stack n=ansible | 2022-05-13 12:26:08.863669 | 48d539a1-1679-f37d-2794-000000000056 | TASK | Add spec to necessary file list when using spec to bootstrap 2022-05-13 12:26:09,092 p=331839 u=stack n=ansible | 2022-05-13 12:26:09.090408 | 48d539a1-1679-f37d-2794-000000000056 | SKIPPED | Add spec to necessary file list when using spec to bootstrap | overcloud-controller-0 2022-05-13 12:26:09,185 p=331839 u=stack n=ansible | 2022-05-13 12:26:09.163464 | 48d539a1-1679-f37d-2794-000000000057 | TASK | Stat necessary files to bootstrap with cephadm 2022-05-13 12:26:09,340 p=331839 u=stack n=ansible | 2022-05-13 12:26:09.339137 | 48d539a1-1679-f37d-2794-000000000057 | SKIPPED | Stat necessary files to bootstrap with cephadm | overcloud-controller-0 | item=/home/ceph-admin/.ssh/id_rsa 2022-05-13 12:26:09,354 p=331839 u=stack n=ansible | 2022-05-13 12:26:09.353007 | 48d539a1-1679-f37d-2794-000000000057 | SKIPPED | Stat necessary files to bootstrap with cephadm | overcloud-controller-0 | item=/home/ceph-admin/.ssh/id_rsa.pub 2022-05-13 12:26:09,386 p=331839 u=stack n=ansible | 2022-05-13 12:26:09.384862 | 48d539a1-1679-f37d-2794-000000000058 | TASK | Fail if necessary files are missing 2022-05-13 12:26:09,551 p=331839 u=stack n=ansible | 2022-05-13 12:26:09.549777 | 48d539a1-1679-f37d-2794-000000000058 | SKIPPED | Fail if necessary files are missing | overcloud-controller-0 | item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': '/home/ceph-admin/.ssh/id_rsa', 'ansible_loop_var': 'item'} 2022-05-13 12:26:09,565 p=331839 u=stack n=ansible | 2022-05-13 12:26:09.564103 | 48d539a1-1679-f37d-2794-000000000058 | SKIPPED | Fail if necessary files are missing | overcloud-controller-0 | item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': '/home/ceph-admin/.ssh/id_rsa.pub', 'ansible_loop_var': 'item'} 2022-05-13 12:26:09,684 p=331839 u=stack n=ansible | 2022-05-13 12:26:09.683045 | 48d539a1-1679-f37d-2794-000000000059 | TASK | Stat pre ceph conf file in case we should bootrap with it 2022-05-13 12:26:09,843 p=331839 u=stack n=ansible | 2022-05-13 12:26:09.841989 | 48d539a1-1679-f37d-2794-000000000059 | SKIPPED | Stat pre ceph conf file in case we should bootrap with it | overcloud-controller-0 2022-05-13 12:26:09,924 p=331839 u=stack n=ansible | 2022-05-13 12:26:09.922736 | 48d539a1-1679-f37d-2794-00000000005b | TASK | Run cephadm bootstrap 2022-05-13 12:26:10,056 p=331839 u=stack n=ansible | 2022-05-13 12:26:10.055173 | 48d539a1-1679-f37d-2794-00000000005b | SKIPPED | Run cephadm bootstrap | overcloud-controller-0 2022-05-13 12:26:10,157 p=331839 u=stack n=ansible | 2022-05-13 12:26:10.156039 | 48d539a1-1679-f37d-2794-00000000005c | TASK | Show results of bootstrap 2022-05-13 12:26:10,307 p=331839 u=stack n=ansible | 2022-05-13 12:26:10.305647 | 48d539a1-1679-f37d-2794-00000000005c | SKIPPED | Show results of bootstrap | overcloud-controller-0 2022-05-13 12:26:10,380 p=331839 u=stack n=ansible | 2022-05-13 12:26:10.379230 | 48d539a1-1679-f37d-2794-00000000005e | TASK | If cephadm bootstrap was not run report the reason 2022-05-13 12:26:10,522 p=331839 u=stack n=ansible | 2022-05-13 12:26:10.520637 | 48d539a1-1679-f37d-2794-00000000005e | SKIPPED | If cephadm bootstrap was not run report the reason | overcloud-controller-0 2022-05-13 12:26:10,588 p=331839 u=stack n=ansible | 2022-05-13 12:26:10.587272 | 48d539a1-1679-f37d-2794-000000000075 | TASK | Ensure tripleo_cephadm_ceph_cli is set 2022-05-13 12:26:10,813 p=331839 u=stack n=ansible | 2022-05-13 12:26:10.811956 | 98e7bf94-7971-4028-aa0c-9bc36efe0fe0 | INCLUDED | /usr/share/ansible/roles/tripleo_cephadm/tasks/ceph_cli.yaml | overcloud-controller-0 2022-05-13 12:26:10,960 p=331839 u=stack n=ansible | 2022-05-13 12:26:10.959229 | 48d539a1-1679-f37d-2794-0000000001f1 | TASK | Set ceph CLI 2022-05-13 12:26:11,139 p=331839 u=stack n=ansible | 2022-05-13 12:26:11.138104 | 48d539a1-1679-f37d-2794-0000000001f1 | OK | Set ceph CLI | overcloud-controller-0 2022-05-13 12:26:11,220 p=331839 u=stack n=ansible | 2022-05-13 12:26:11.218977 | 48d539a1-1679-f37d-2794-000000000076 | TASK | Assimilate configuration from tripleo_cephadm_assimilate_conf 2022-05-13 12:26:12,580 p=331839 u=stack n=ansible | 2022-05-13 12:26:12.578819 | 48d539a1-1679-f37d-2794-000000000076 | OK | Assimilate configuration from tripleo_cephadm_assimilate_conf | overcloud-controller-0 2022-05-13 12:26:12,645 p=331839 u=stack n=ansible | 2022-05-13 12:26:12.643792 | 48d539a1-1679-f37d-2794-000000000081 | TASK | Stat spec file on bootstrap node 2022-05-13 12:26:12,799 p=331839 u=stack n=ansible | 2022-05-13 12:26:12.797816 | 48d539a1-1679-f37d-2794-000000000081 | SKIPPED | Stat spec file on bootstrap node | overcloud-controller-0 | item=/home/ceph-admin/specs/ceph_spec.yaml 2022-05-13 12:26:12,867 p=331839 u=stack n=ansible | 2022-05-13 12:26:12.866077 | 48d539a1-1679-f37d-2794-000000000082 | TASK | Fail if spec file is missing 2022-05-13 12:26:13,047 p=331839 u=stack n=ansible | 2022-05-13 12:26:13.045910 | 48d539a1-1679-f37d-2794-000000000082 | SKIPPED | Fail if spec file is missing | overcloud-controller-0 | item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': '/home/ceph-admin/specs/ceph_spec.yaml', 'ansible_loop_var': 'item'} 2022-05-13 12:26:13,143 p=331839 u=stack n=ansible | 2022-05-13 12:26:13.142530 | 48d539a1-1679-f37d-2794-000000000083 | TASK | Get ceph_cli 2022-05-13 12:26:13,335 p=331839 u=stack n=ansible | 2022-05-13 12:26:13.333071 | 48d539a1-1679-f37d-2794-000000000083 | SKIPPED | Get ceph_cli | overcloud-controller-0 2022-05-13 12:26:13,381 p=331839 u=stack n=ansible | 2022-05-13 12:26:13.380444 | 48d539a1-1679-f37d-2794-000000000084 | TASK | Get the ceph orchestrator status 2022-05-13 12:26:13,889 p=331839 u=stack n=ansible | 2022-05-13 12:26:13.887669 | 48d539a1-1679-f37d-2794-000000000084 | SKIPPED | Get the ceph orchestrator status | overcloud-controller-0 2022-05-13 12:26:13,981 p=331839 u=stack n=ansible | 2022-05-13 12:26:13.934415 | 48d539a1-1679-f37d-2794-000000000085 | TASK | Fail if ceph orchestrator is not available 2022-05-13 12:26:14,128 p=331839 u=stack n=ansible | 2022-05-13 12:26:14.126689 | 48d539a1-1679-f37d-2794-000000000085 | SKIPPED | Fail if ceph orchestrator is not available | overcloud-controller-0 2022-05-13 12:26:14,201 p=331839 u=stack n=ansible | 2022-05-13 12:26:14.200118 | 48d539a1-1679-f37d-2794-000000000086 | TASK | Set container images used by the applied services 2022-05-13 12:26:14,369 p=331839 u=stack n=ansible | 2022-05-13 12:26:14.368011 | 48d539a1-1679-f37d-2794-000000000086 | SKIPPED | Set container images used by the applied services | overcloud-controller-0 2022-05-13 12:26:14,465 p=331839 u=stack n=ansible | 2022-05-13 12:26:14.464581 | 48d539a1-1679-f37d-2794-000000000087 | TASK | Apply spec 2022-05-13 12:26:14,631 p=331839 u=stack n=ansible | 2022-05-13 12:26:14.629438 | 48d539a1-1679-f37d-2794-000000000087 | SKIPPED | Apply spec | overcloud-controller-0 2022-05-13 12:26:14,690 p=331839 u=stack n=ansible | 2022-05-13 12:26:14.688886 | 48d539a1-1679-f37d-2794-000000000088 | TASK | Show results of spec apply 2022-05-13 12:26:14,827 p=331839 u=stack n=ansible | 2022-05-13 12:26:14.826256 | 48d539a1-1679-f37d-2794-000000000088 | SKIPPED | Show results of spec apply | overcloud-controller-0 2022-05-13 12:26:14,889 p=331839 u=stack n=ansible | 2022-05-13 12:26:14.888027 | 48d539a1-1679-f37d-2794-000000000089 | TASK | Wait for the expected number of osds to be running 2022-05-13 12:26:15,184 p=331839 u=stack n=ansible | 2022-05-13 12:26:15.183354 | 48d539a1-1679-f37d-2794-000000000089 | SKIPPED | Wait for the expected number of osds to be running | overcloud-controller-0 2022-05-13 12:26:15,259 p=331839 u=stack n=ansible | 2022-05-13 12:26:15.258568 | 48d539a1-1679-f37d-2794-0000000000a2 | TASK | Get ceph_cli 2022-05-13 12:26:15,621 p=331839 u=stack n=ansible | 2022-05-13 12:26:15.620207 | d26309ae-a210-4e75-a60a-382b0e66b637 | INCLUDED | /usr/share/ansible/roles/tripleo_cephadm/tasks/ceph_cli.yaml | overcloud-controller-0 2022-05-13 12:26:15,744 p=331839 u=stack n=ansible | 2022-05-13 12:26:15.743534 | 48d539a1-1679-f37d-2794-00000000021e | TASK | Set ceph CLI 2022-05-13 12:26:16,061 p=331839 u=stack n=ansible | 2022-05-13 12:26:16.059683 | 48d539a1-1679-f37d-2794-00000000021e | OK | Set ceph CLI | overcloud-controller-0 2022-05-13 12:26:16,090 p=331839 u=stack n=ansible | 2022-05-13 12:26:16.089782 | 48d539a1-1679-f37d-2794-0000000000a3 | TASK | create configured crush rules 2022-05-13 12:26:18,786 p=331839 u=stack n=ansible | 2022-05-13 12:26:18.785115 | 48d539a1-1679-f37d-2794-0000000000a3 | CHANGED | create configured crush rules | overcloud-controller-0 | item={'class': 'hdd', 'default': True, 'name': 'HDD', 'root': 'default', 'type': 'host'} 2022-05-13 12:26:21,213 p=331839 u=stack n=ansible | 2022-05-13 12:26:21.212037 | 48d539a1-1679-f37d-2794-0000000000a3 | CHANGED | create configured crush rules | overcloud-controller-0 | item={'class': 'ssd', 'default': False, 'name': 'SSD', 'root': 'default', 'type': 'host'} 2022-05-13 12:26:21,250 p=331839 u=stack n=ansible | 2022-05-13 12:26:21.249696 | 48d539a1-1679-f37d-2794-0000000000a4 | TASK | get id for new default crush rule 2022-05-13 12:26:22,316 p=331839 u=stack n=ansible | 2022-05-13 12:26:22.314470 | 48d539a1-1679-f37d-2794-0000000000a4 | OK | get id for new default crush rule | overcloud-controller-0 | item={'class': 'hdd', 'default': True, 'name': 'HDD', 'root': 'default', 'type': 'host'} 2022-05-13 12:26:22,332 p=331839 u=stack n=ansible | 2022-05-13 12:26:22.331699 | 48d539a1-1679-f37d-2794-0000000000a4 | SKIPPED | get id for new default crush rule | overcloud-controller-0 | item={'class': 'ssd', 'default': False, 'name': 'SSD', 'root': 'default', 'type': 'host'} 2022-05-13 12:26:22,368 p=331839 u=stack n=ansible | 2022-05-13 12:26:22.367052 | 48d539a1-1679-f37d-2794-0000000000a5 | TASK | set_fact info_ceph_default_crush_rule_yaml, ceph_osd_pool_default_crush_rule_name 2022-05-13 12:26:22,664 p=331839 u=stack n=ansible | 2022-05-13 12:26:22.662789 | 48d539a1-1679-f37d-2794-0000000000a5 | OK | set_fact info_ceph_default_crush_rule_yaml, ceph_osd_pool_default_crush_rule_name | overcloud-controller-0 | item={'cmd': ['podman', 'run', '--rm', '--net=host', '-v', '/etc/ceph:/etc/ceph:z', '-v', '/var/lib/ceph/:/var/lib/ceph/:z', '-v', '/var/log/ceph/:/var/log/ceph/:z', '--entrypoint=ceph', '172.25.201.68:8787/ceph/daemon:v6.0.6-stable-6.0-pacific-centos-8-x86_64', '-n', 'client.admin', '-k', '/etc/ceph/ceph.client.admin.keyring', '--cluster', 'ceph', 'osd', 'crush', 'rule', 'dump', 'HDD', '--format=json'], 'start': '2022-05-13 04:26:21.604812', 'end': '2022-05-13 04:26:22.246815', 'delta': '0:00:00.642003', 'rc': 0, 'stdout': '\n{"rule_id":1,"rule_name":"HDD","ruleset":1,"type":1,"min_size":1,"max_size":10,"steps":[{"op":"take","item":-2,"item_name":"default~hdd"},{"op":"chooseleaf_firstn","num":0,"type":"host"},{"op":"emit"}]}', 'stderr': '', 'changed': False, 'invocation': {'module_args': {'name': 'HDD', 'cluster': 'ceph', 'state': 'info', 'rule_type': None, 'bucket_root': None, 'bucket_type': None, 'device_class': None, 'profile': None}}, 'stdout_lines': ['', '{"rule_id":1,"rule_name":"HDD","ruleset":1,"type":1,"min_size":1,"max_size":10,"steps":[{"op":"take","item":-2,"item_name":"default~hdd"},{"op":"chooseleaf_firstn","num":0,"type":"host"},{"op":"emit"}]}'], 'stderr_lines': [], 'failed': False, 'item': {'class': 'hdd', 'default': True, 'name': 'HDD', 'root': 'default', 'type': 'host'}, 'ansible_loop_var': 'item'} 2022-05-13 12:26:22,684 p=331839 u=stack n=ansible | 2022-05-13 12:26:22.683064 | 48d539a1-1679-f37d-2794-0000000000a5 | SKIPPED | set_fact info_ceph_default_crush_rule_yaml, ceph_osd_pool_default_crush_rule_name | overcloud-controller-0 | item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': {'class': 'ssd', 'default': False, 'name': 'SSD', 'root': 'default', 'type': 'host'}, 'ansible_loop_var': 'item'} 2022-05-13 12:26:22,935 p=331839 u=stack n=ansible | 2022-05-13 12:26:22.934152 | 48d539a1-1679-f37d-2794-0000000000a6 | TASK | insert new default crush rule into daemon to prevent restart 2022-05-13 12:26:23,676 p=331839 u=stack n=ansible | 2022-05-13 12:26:23.674823 | 48d539a1-1679-f37d-2794-0000000000a6 | FATAL | insert new default crush rule into daemon to prevent restart | overcloud-controller-0 -> 172.25.201.106 | item=overcloud-controller-0 | error={"ansible_loop_var": "item", "changed": false, "cmd": ["podman", "run", "--rm", "--net=host", "--ipc=host", "--volume", "/etc/ceph:/etc/ceph:z", "--volume", "/home/ceph-admin/assimilate_ceph.conf:/home/assimilate_ceph.conf:z", "--volume", "/var/run/ceph/80c8599f-1698-50bf-8d07-fe3fc94b554a:/var/run/ceph:z", "--entrypoint", "ceph", "172.25.201.68:8787/ceph/daemon:v6.0.6-stable-6.0-pacific-centos-8-x86_64", "--admin-daemon", "/var/run/ceph/ceph-mon.overcloud-controller-0.bdxworld.com.asok", "config", "set", "osd_pool_default_crush_rule", "1"], "delta": "0:00:00.440441", "end": "2022-05-13 04:26:23.624180", "item": "overcloud-controller-0", "msg": "non-zero return code", "rc": 22, "start": "2022-05-13 04:26:23.183739", "stderr": "admin_socket: exception getting command descriptions: [Errno 2] No such file or directory", "stderr_lines": ["admin_socket: exception getting command descriptions: [Errno 2] No such file or directory"], "stdout": "", "stdout_lines": []} 2022-05-13 12:26:23,679 p=331839 u=stack n=ansible | [WARNING]: ('overcloud-controller-0 -> 172.25.201.106', '48d539a1-1679-f37d-2794-0000000000a6') missing from stats 2022-05-13 12:26:25,814 p=331839 u=stack n=ansible | 2022-05-13 12:26:25.813271 | 48d539a1-1679-f37d-2794-0000000000a6 | FATAL | insert new default crush rule into daemon to prevent restart | overcloud-controller-0 -> 172.25.201.105 | item=overcloud-controller-1 | error={"ansible_loop_var": "item", "changed": false, "cmd": ["podman", "run", "--rm", "--net=host", "--ipc=host", "--volume", "/etc/ceph:/etc/ceph:z", "--volume", "/home/ceph-admin/assimilate_ceph.conf:/home/assimilate_ceph.conf:z", "--volume", "/var/run/ceph/80c8599f-1698-50bf-8d07-fe3fc94b554a:/var/run/ceph:z", "--entrypoint", "ceph", "172.25.201.68:8787/ceph/daemon:v6.0.6-stable-6.0-pacific-centos-8-x86_64", "--admin-daemon", "/var/run/ceph/ceph-mon.overcloud-controller-1.bdxworld.com.asok", "config", "set", "osd_pool_default_crush_rule", "1"], "delta": "0:00:01.607169", "end": "2022-05-13 04:26:25.759348", "item": "overcloud-controller-1", "msg": "non-zero return code", "rc": 22, "start": "2022-05-13 04:26:24.152179", "stderr": "admin_socket: exception getting command descriptions: [Errno 2] No such file or directory", "stderr_lines": ["admin_socket: exception getting command descriptions: [Errno 2] No such file or directory"], "stdout": "", "stdout_lines": []} 2022-05-13 12:26:25,817 p=331839 u=stack n=ansible | [WARNING]: ('overcloud-controller-0 -> 172.25.201.105', '48d539a1-1679-f37d-2794-0000000000a6') missing from stats 2022-05-13 12:26:26,817 p=331839 u=stack n=ansible | 2022-05-13 12:26:26.816330 | 48d539a1-1679-f37d-2794-0000000000a6 | FATAL | insert new default crush rule into daemon to prevent restart | overcloud-controller-0 -> 172.25.201.97 | item=overcloud-controller-2 | error={"ansible_loop_var": "item", "changed": false, "cmd": ["podman", "run", "--rm", "--net=host", "--ipc=host", "--volume", "/etc/ceph:/etc/ceph:z", "--volume", "/home/ceph-admin/assimilate_ceph.conf:/home/assimilate_ceph.conf:z", "--volume", "/var/run/ceph/80c8599f-1698-50bf-8d07-fe3fc94b554a:/var/run/ceph:z", "--entrypoint", "ceph", "172.25.201.68:8787/ceph/daemon:v6.0.6-stable-6.0-pacific-centos-8-x86_64", "--admin-daemon", "/var/run/ceph/ceph-mon.overcloud-controller-2.bdxworld.com.asok", "config", "set", "osd_pool_default_crush_rule", "1"], "delta": "0:00:00.660536", "end": "2022-05-13 04:26:26.780280", "item": "overcloud-controller-2", "msg": "non-zero return code", "rc": 22, "start": "2022-05-13 04:26:26.119744", "stderr": "admin_socket: exception getting command descriptions: [Errno 2] No such file or directory", "stderr_lines": ["admin_socket: exception getting command descriptions: [Errno 2] No such file or directory"], "stdout": "", "stdout_lines": []} 2022-05-13 12:26:26,828 p=331839 u=stack n=ansible | [WARNING]: ('overcloud-controller-0 -> 172.25.201.97', '48d539a1-1679-f37d-2794-0000000000a6') missing from stats 2022-05-13 12:26:26,840 p=331839 u=stack n=ansible | NO MORE HOSTS LEFT ************************************************************* 2022-05-13 12:26:26,845 p=331839 u=stack n=ansible | PLAY RECAP ********************************************************************* 2022-05-13 12:26:26,846 p=331839 u=stack n=ansible | overcloud-controller-0 : ok=15 changed=4 unreachable=0 failed=1 skipped=25 rescued=0 ignored=0 2022-05-13 12:26:26,848 p=331839 u=stack n=ansible | 2022-05-13 12:26:26.847958 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Summary Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2022-05-13 12:26:26,849 p=331839 u=stack n=ansible | 2022-05-13 12:26:26.848779 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Total Tasks: 41 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2022-05-13 12:26:26,850 p=331839 u=stack n=ansible | 2022-05-13 12:26:26.849484 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Elapsed Time: 0:00:27.638064 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2022-05-13 12:26:26,850 p=331839 u=stack n=ansible | 2022-05-13 12:26:26.850180 | UUID | Info | Host | Task Name | Run Time 2022-05-13 12:26:26,851 p=331839 u=stack n=ansible | 2022-05-13 12:26:26.850909 | 48d539a1-1679-f37d-2794-0000000000a3 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : create configured crush rules | 5.13s 2022-05-13 12:26:26,852 p=331839 u=stack n=ansible | 2022-05-13 12:26:26.851647 | 48d539a1-1679-f37d-2794-0000000000a6 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : insert new default crush rule into daemon to prevent restart | 3.90s 2022-05-13 12:26:26,852 p=331839 u=stack n=ansible | 2022-05-13 12:26:26.852402 | 48d539a1-1679-f37d-2794-000000000076 | SUMMARY | overcloud-controller-0 | Assimilate configuration from tripleo_cephadm_assimilate_conf | 1.36s 2022-05-13 12:26:26,853 p=331839 u=stack n=ansible | 2022-05-13 12:26:26.853108 | 48d539a1-1679-f37d-2794-000000000026 | SUMMARY | overcloud-controller-0 | push tripleo_cephadm_spec to bootstrap node if spec file exsits | 1.32s 2022-05-13 12:26:26,854 p=331839 u=stack n=ansible | 2022-05-13 12:26:26.853834 | 48d539a1-1679-f37d-2794-00000000001b | SUMMARY | overcloud-controller-0 | tripleo_cephadm : List Ceph daemon instances on this host | 1.21s 2022-05-13 12:26:26,855 p=331839 u=stack n=ansible | 2022-05-13 12:26:26.854785 | 48d539a1-1679-f37d-2794-0000000000a4 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : get id for new default crush rule | 1.09s 2022-05-13 12:26:26,856 p=331839 u=stack n=ansible | 2022-05-13 12:26:26.855787 | 48d539a1-1679-f37d-2794-000000000019 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Stat cephadm file | 0.83s 2022-05-13 12:26:26,857 p=331839 u=stack n=ansible | 2022-05-13 12:26:26.856704 | 48d539a1-1679-f37d-2794-000000000023 | SUMMARY | overcloud-controller-0 | Ensure tripleo_cephadm_config_home (e.g. /etc/ceph) exists | 0.80s 2022-05-13 12:26:26,858 p=331839 u=stack n=ansible | 2022-05-13 12:26:26.857523 | 48d539a1-1679-f37d-2794-00000000002b | SUMMARY | overcloud-controller-0 | generate tripleo_cephadm_assimilate_conf on bootstrap node | 0.67s 2022-05-13 12:26:26,859 p=331839 u=stack n=ansible | 2022-05-13 12:26:26.858258 | 48d539a1-1679-f37d-2794-000000000084 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Get the ceph orchestrator status | 0.45s 2022-05-13 12:26:26,859 p=331839 u=stack n=ansible | 2022-05-13 12:26:26.859194 | 48d539a1-1679-f37d-2794-000000000020 | SUMMARY | overcloud-controller-0 | Set random tripleo_cephadm_fsid if no running ceph containers were found | 0.41s 2022-05-13 12:26:26,860 p=331839 u=stack n=ansible | 2022-05-13 12:26:26.860035 | 48d539a1-1679-f37d-2794-000000000024 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Ensure specs directory exists | 0.37s 2022-05-13 12:26:26,861 p=331839 u=stack n=ansible | 2022-05-13 12:26:26.860884 | 48d539a1-1679-f37d-2794-0000000000a5 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : set_fact info_ceph_default_crush_rule_yaml, ceph_osd_pool_default_crush_rule_name | 0.32s 2022-05-13 12:26:26,862 p=331839 u=stack n=ansible | 2022-05-13 12:26:26.861676 | 48d539a1-1679-f37d-2794-00000000021e | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Set ceph CLI | 0.32s 2022-05-13 12:26:26,862 p=331839 u=stack n=ansible | 2022-05-13 12:26:26.862343 | 48d539a1-1679-f37d-2794-0000000000a2 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Get ceph_cli | 0.31s 2022-05-13 12:26:26,863 p=331839 u=stack n=ansible | 2022-05-13 12:26:26.863043 | 48d539a1-1679-f37d-2794-000000000089 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Wait for the expected number of osds to be running | 0.30s 2022-05-13 12:26:26,864 p=331839 u=stack n=ansible | 2022-05-13 12:26:26.863759 | 48d539a1-1679-f37d-2794-000000000018 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Install cephadm package | 0.29s 2022-05-13 12:26:26,864 p=331839 u=stack n=ansible | 2022-05-13 12:26:26.864456 | 48d539a1-1679-f37d-2794-00000000001e | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Fail if >1 FSID was discovered | 0.21s 2022-05-13 12:26:26,865 p=331839 u=stack n=ansible | 2022-05-13 12:26:26.865157 | 48d539a1-1679-f37d-2794-000000000056 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Add spec to necessary file list when using spec to bootstrap | 0.20s 2022-05-13 12:26:26,866 p=331839 u=stack n=ansible | 2022-05-13 12:26:26.865878 | 48d539a1-1679-f37d-2794-00000000001d | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Set list of found FSIDs | 0.20s 2022-05-13 12:26:26,867 p=331839 u=stack n=ansible | 2022-05-13 12:26:26.866580 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ End Summary Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2022-05-13 12:26:26,867 p=331839 u=stack n=ansible | 2022-05-13 12:26:26.867351 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ State Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2022-05-13 12:26:26,868 p=331839 u=stack n=ansible | 2022-05-13 12:26:26.868118 | ~~~~~~~~~~~~~~~~~~ Number of nodes which did not deploy successfully: 1 ~~~~~~~~~~~~~~~~~ 2022-05-13 12:26:26,869 p=331839 u=stack n=ansible | 2022-05-13 12:26:26.868810 | The following node(s) had failures: overcloud-controller-0 2022-05-13 12:26:26,870 p=331839 u=stack n=ansible | 2022-05-13 12:26:26.869801 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Running /home/stack/overcloud-deploy/overcloud/config-download/overcloud/cephadm/cephadm_command.sh 2022-05-13 12:54:02,259 p=410772 u=stack n=ansible | PLAY [Deploy Ceph with cephadm] ************************************************ 2022-05-13 12:54:02,502 p=410772 u=stack n=ansible | 2022-05-13 12:54:02.501518 | 48d539a1-1679-3280-4b7d-000000000018 | TASK | Install cephadm package 2022-05-13 12:54:02,634 p=410772 u=stack n=ansible | 2022-05-13 12:54:02.633241 | 48d539a1-1679-3280-4b7d-000000000018 | SKIPPED | Install cephadm package | overcloud-controller-0 2022-05-13 12:54:02,664 p=410772 u=stack n=ansible | 2022-05-13 12:54:02.663105 | 48d539a1-1679-3280-4b7d-000000000019 | TASK | Stat cephadm file 2022-05-13 12:54:03,413 p=410772 u=stack n=ansible | 2022-05-13 12:54:03.411707 | 48d539a1-1679-3280-4b7d-000000000019 | OK | Stat cephadm file | overcloud-controller-0 2022-05-13 12:54:03,446 p=410772 u=stack n=ansible | 2022-05-13 12:54:03.445791 | 48d539a1-1679-3280-4b7d-00000000001a | TASK | Fail if cephadm is not available 2022-05-13 12:54:03,586 p=410772 u=stack n=ansible | 2022-05-13 12:54:03.584860 | 48d539a1-1679-3280-4b7d-00000000001a | SKIPPED | Fail if cephadm is not available | overcloud-controller-0 2022-05-13 12:54:03,617 p=410772 u=stack n=ansible | 2022-05-13 12:54:03.616519 | 48d539a1-1679-3280-4b7d-00000000001b | TASK | List Ceph daemon instances on this host 2022-05-13 12:54:04,797 p=410772 u=stack n=ansible | 2022-05-13 12:54:04.795894 | 48d539a1-1679-3280-4b7d-00000000001b | CHANGED | List Ceph daemon instances on this host | overcloud-controller-0 2022-05-13 12:54:04,829 p=410772 u=stack n=ansible | 2022-05-13 12:54:04.827892 | 48d539a1-1679-3280-4b7d-00000000001d | TASK | Set list of found FSIDs 2022-05-13 12:54:04,960 p=410772 u=stack n=ansible | 2022-05-13 12:54:04.958812 | 48d539a1-1679-3280-4b7d-00000000001d | SKIPPED | Set list of found FSIDs | overcloud-controller-0 2022-05-13 12:54:04,990 p=410772 u=stack n=ansible | 2022-05-13 12:54:04.988793 | 48d539a1-1679-3280-4b7d-00000000001e | TASK | Fail if >1 FSID was discovered 2022-05-13 12:54:05,130 p=410772 u=stack n=ansible | 2022-05-13 12:54:05.128868 | 48d539a1-1679-3280-4b7d-00000000001e | SKIPPED | Fail if >1 FSID was discovered | overcloud-controller-0 2022-05-13 12:54:05,159 p=410772 u=stack n=ansible | 2022-05-13 12:54:05.158742 | 48d539a1-1679-3280-4b7d-00000000001f | TASK | Set FSID to the discovered value 2022-05-13 12:54:05,299 p=410772 u=stack n=ansible | 2022-05-13 12:54:05.297772 | 48d539a1-1679-3280-4b7d-00000000001f | SKIPPED | Set FSID to the discovered value | overcloud-controller-0 2022-05-13 12:54:05,330 p=410772 u=stack n=ansible | 2022-05-13 12:54:05.329084 | 48d539a1-1679-3280-4b7d-000000000020 | TASK | Set random tripleo_cephadm_fsid if no running ceph containers were found 2022-05-13 12:54:05,496 p=410772 u=stack n=ansible | 2022-05-13 12:54:05.494381 | 48d539a1-1679-3280-4b7d-000000000020 | SKIPPED | Set random tripleo_cephadm_fsid if no running ceph containers were found | overcloud-controller-0 2022-05-13 12:54:05,527 p=410772 u=stack n=ansible | 2022-05-13 12:54:05.526061 | 48d539a1-1679-3280-4b7d-000000000022 | TASK | Set first monitor IP if it was not passed 2022-05-13 12:54:05,744 p=410772 u=stack n=ansible | 2022-05-13 12:54:05.742859 | 48d539a1-1679-3280-4b7d-000000000022 | SKIPPED | Set first monitor IP if it was not passed | overcloud-controller-0 2022-05-13 12:54:05,775 p=410772 u=stack n=ansible | 2022-05-13 12:54:05.774284 | 48d539a1-1679-3280-4b7d-000000000023 | TASK | Ensure tripleo_cephadm_config_home (e.g. /etc/ceph) exists 2022-05-13 12:54:06,636 p=410772 u=stack n=ansible | 2022-05-13 12:54:06.634329 | 48d539a1-1679-3280-4b7d-000000000023 | OK | Ensure tripleo_cephadm_config_home (e.g. /etc/ceph) exists | overcloud-controller-0 2022-05-13 12:54:06,674 p=410772 u=stack n=ansible | 2022-05-13 12:54:06.668991 | 48d539a1-1679-3280-4b7d-000000000024 | TASK | Ensure specs directory exists 2022-05-13 12:54:07,013 p=410772 u=stack n=ansible | 2022-05-13 12:54:07.011993 | 48d539a1-1679-3280-4b7d-000000000024 | OK | Ensure specs directory exists | overcloud-controller-0 2022-05-13 12:54:07,053 p=410772 u=stack n=ansible | 2022-05-13 12:54:07.052272 | 48d539a1-1679-3280-4b7d-000000000025 | TASK | Stat spec file on ansible host 2022-05-13 12:54:07,888 p=410772 u=stack n=ansible | 2022-05-13 12:54:07.886889 | 48d539a1-1679-3280-4b7d-000000000025 | OK | Stat spec file on ansible host | overcloud-controller-0 -> localhost 2022-05-13 12:54:07,891 p=410772 u=stack n=ansible | [WARNING]: ('overcloud-controller-0 -> localhost', '48d539a1-1679-3280-4b7d-000000000025') missing from stats 2022-05-13 12:54:07,921 p=410772 u=stack n=ansible | 2022-05-13 12:54:07.920383 | 48d539a1-1679-3280-4b7d-000000000026 | TASK | push tripleo_cephadm_spec to bootstrap node if spec file exsits 2022-05-13 12:54:08,637 p=410772 u=stack n=ansible | 2022-05-13 12:54:08.635539 | 48d539a1-1679-3280-4b7d-000000000026 | OK | push tripleo_cephadm_spec to bootstrap node if spec file exsits | overcloud-controller-0 2022-05-13 12:54:08,678 p=410772 u=stack n=ansible | 2022-05-13 12:54:08.676986 | 48d539a1-1679-3280-4b7d-000000000028 | TASK | stat tripleo_cephadm_bootstrap_conf file 2022-05-13 12:54:08,810 p=410772 u=stack n=ansible | 2022-05-13 12:54:08.809017 | 48d539a1-1679-3280-4b7d-000000000028 | SKIPPED | stat tripleo_cephadm_bootstrap_conf file | overcloud-controller-0 2022-05-13 12:54:08,840 p=410772 u=stack n=ansible | 2022-05-13 12:54:08.838897 | 48d539a1-1679-3280-4b7d-000000000029 | TASK | push it to tripleo_cephadm_assimilate_conf on bootstrap node 2022-05-13 12:54:08,971 p=410772 u=stack n=ansible | 2022-05-13 12:54:08.969913 | 48d539a1-1679-3280-4b7d-000000000029 | SKIPPED | push it to tripleo_cephadm_assimilate_conf on bootstrap node | overcloud-controller-0 2022-05-13 12:54:09,004 p=410772 u=stack n=ansible | 2022-05-13 12:54:09.002878 | 48d539a1-1679-3280-4b7d-00000000002b | TASK | generate tripleo_cephadm_assimilate_conf on bootstrap node 2022-05-13 12:54:10,024 p=410772 u=stack n=ansible | 2022-05-13 12:54:10.022581 | 48d539a1-1679-3280-4b7d-00000000002b | OK | generate tripleo_cephadm_assimilate_conf on bootstrap node | overcloud-controller-0 2022-05-13 12:54:10,054 p=410772 u=stack n=ansible | 2022-05-13 12:54:10.053353 | 48d539a1-1679-3280-4b7d-000000000056 | TASK | Add spec to necessary file list when using spec to bootstrap 2022-05-13 12:54:10,186 p=410772 u=stack n=ansible | 2022-05-13 12:54:10.185431 | 48d539a1-1679-3280-4b7d-000000000056 | SKIPPED | Add spec to necessary file list when using spec to bootstrap | overcloud-controller-0 2022-05-13 12:54:10,217 p=410772 u=stack n=ansible | 2022-05-13 12:54:10.216767 | 48d539a1-1679-3280-4b7d-000000000057 | TASK | Stat necessary files to bootstrap with cephadm 2022-05-13 12:54:10,417 p=410772 u=stack n=ansible | 2022-05-13 12:54:10.416071 | 48d539a1-1679-3280-4b7d-000000000057 | SKIPPED | Stat necessary files to bootstrap with cephadm | overcloud-controller-0 | item=/home/ceph-admin/.ssh/id_rsa 2022-05-13 12:54:10,420 p=410772 u=stack n=ansible | 2022-05-13 12:54:10.419173 | 48d539a1-1679-3280-4b7d-000000000057 | SKIPPED | Stat necessary files to bootstrap with cephadm | overcloud-controller-0 | item=/home/ceph-admin/.ssh/id_rsa.pub 2022-05-13 12:54:10,451 p=410772 u=stack n=ansible | 2022-05-13 12:54:10.449880 | 48d539a1-1679-3280-4b7d-000000000058 | TASK | Fail if necessary files are missing 2022-05-13 12:54:10,638 p=410772 u=stack n=ansible | 2022-05-13 12:54:10.636499 | 48d539a1-1679-3280-4b7d-000000000058 | SKIPPED | Fail if necessary files are missing | overcloud-controller-0 | item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': '/home/ceph-admin/.ssh/id_rsa', 'ansible_loop_var': 'item'} 2022-05-13 12:54:10,654 p=410772 u=stack n=ansible | 2022-05-13 12:54:10.653130 | 48d539a1-1679-3280-4b7d-000000000058 | SKIPPED | Fail if necessary files are missing | overcloud-controller-0 | item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': '/home/ceph-admin/.ssh/id_rsa.pub', 'ansible_loop_var': 'item'} 2022-05-13 12:54:10,686 p=410772 u=stack n=ansible | 2022-05-13 12:54:10.685447 | 48d539a1-1679-3280-4b7d-000000000059 | TASK | Stat pre ceph conf file in case we should bootrap with it 2022-05-13 12:54:10,817 p=410772 u=stack n=ansible | 2022-05-13 12:54:10.815857 | 48d539a1-1679-3280-4b7d-000000000059 | SKIPPED | Stat pre ceph conf file in case we should bootrap with it | overcloud-controller-0 2022-05-13 12:54:10,848 p=410772 u=stack n=ansible | 2022-05-13 12:54:10.847262 | 48d539a1-1679-3280-4b7d-00000000005b | TASK | Run cephadm bootstrap 2022-05-13 12:54:10,990 p=410772 u=stack n=ansible | 2022-05-13 12:54:10.988877 | 48d539a1-1679-3280-4b7d-00000000005b | SKIPPED | Run cephadm bootstrap | overcloud-controller-0 2022-05-13 12:54:11,021 p=410772 u=stack n=ansible | 2022-05-13 12:54:11.020537 | 48d539a1-1679-3280-4b7d-00000000005c | TASK | Show results of bootstrap 2022-05-13 12:54:11,163 p=410772 u=stack n=ansible | 2022-05-13 12:54:11.161835 | 48d539a1-1679-3280-4b7d-00000000005c | SKIPPED | Show results of bootstrap | overcloud-controller-0 2022-05-13 12:54:11,194 p=410772 u=stack n=ansible | 2022-05-13 12:54:11.193157 | 48d539a1-1679-3280-4b7d-00000000005e | TASK | If cephadm bootstrap was not run report the reason 2022-05-13 12:54:11,348 p=410772 u=stack n=ansible | 2022-05-13 12:54:11.346677 | 48d539a1-1679-3280-4b7d-00000000005e | SKIPPED | If cephadm bootstrap was not run report the reason | overcloud-controller-0 2022-05-13 12:54:11,379 p=410772 u=stack n=ansible | 2022-05-13 12:54:11.378845 | 48d539a1-1679-3280-4b7d-000000000075 | TASK | Ensure tripleo_cephadm_ceph_cli is set 2022-05-13 12:54:11,545 p=410772 u=stack n=ansible | 2022-05-13 12:54:11.544690 | 203a13b6-a319-47b5-91fd-e3146c5df92e | INCLUDED | /usr/share/ansible/roles/tripleo_cephadm/tasks/ceph_cli.yaml | overcloud-controller-0 2022-05-13 12:54:11,578 p=410772 u=stack n=ansible | 2022-05-13 12:54:11.577207 | 48d539a1-1679-3280-4b7d-0000000001f1 | TASK | Set ceph CLI 2022-05-13 12:54:11,753 p=410772 u=stack n=ansible | 2022-05-13 12:54:11.751955 | 48d539a1-1679-3280-4b7d-0000000001f1 | OK | Set ceph CLI | overcloud-controller-0 2022-05-13 12:54:11,783 p=410772 u=stack n=ansible | 2022-05-13 12:54:11.782293 | 48d539a1-1679-3280-4b7d-000000000076 | TASK | Assimilate configuration from tripleo_cephadm_assimilate_conf 2022-05-13 12:54:12,793 p=410772 u=stack n=ansible | 2022-05-13 12:54:12.792153 | 48d539a1-1679-3280-4b7d-000000000076 | OK | Assimilate configuration from tripleo_cephadm_assimilate_conf | overcloud-controller-0 2022-05-13 12:54:12,873 p=410772 u=stack n=ansible | 2022-05-13 12:54:12.868981 | 48d539a1-1679-3280-4b7d-000000000081 | TASK | Stat spec file on bootstrap node 2022-05-13 12:54:13,053 p=410772 u=stack n=ansible | 2022-05-13 12:54:13.050201 | 48d539a1-1679-3280-4b7d-000000000081 | SKIPPED | Stat spec file on bootstrap node | overcloud-controller-0 | item=/home/ceph-admin/specs/ceph_spec.yaml 2022-05-13 12:54:13,090 p=410772 u=stack n=ansible | 2022-05-13 12:54:13.089787 | 48d539a1-1679-3280-4b7d-000000000082 | TASK | Fail if spec file is missing 2022-05-13 12:54:13,246 p=410772 u=stack n=ansible | 2022-05-13 12:54:13.245251 | 48d539a1-1679-3280-4b7d-000000000082 | SKIPPED | Fail if spec file is missing | overcloud-controller-0 | item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': '/home/ceph-admin/specs/ceph_spec.yaml', 'ansible_loop_var': 'item'} 2022-05-13 12:54:13,281 p=410772 u=stack n=ansible | 2022-05-13 12:54:13.279688 | 48d539a1-1679-3280-4b7d-000000000083 | TASK | Get ceph_cli 2022-05-13 12:54:13,425 p=410772 u=stack n=ansible | 2022-05-13 12:54:13.423897 | 48d539a1-1679-3280-4b7d-000000000083 | SKIPPED | Get ceph_cli | overcloud-controller-0 2022-05-13 12:54:13,456 p=410772 u=stack n=ansible | 2022-05-13 12:54:13.455505 | 48d539a1-1679-3280-4b7d-000000000084 | TASK | Get the ceph orchestrator status 2022-05-13 12:54:13,749 p=410772 u=stack n=ansible | 2022-05-13 12:54:13.747980 | 48d539a1-1679-3280-4b7d-000000000084 | SKIPPED | Get the ceph orchestrator status | overcloud-controller-0 2022-05-13 12:54:13,780 p=410772 u=stack n=ansible | 2022-05-13 12:54:13.779687 | 48d539a1-1679-3280-4b7d-000000000085 | TASK | Fail if ceph orchestrator is not available 2022-05-13 12:54:14,097 p=410772 u=stack n=ansible | 2022-05-13 12:54:14.096056 | 48d539a1-1679-3280-4b7d-000000000085 | SKIPPED | Fail if ceph orchestrator is not available | overcloud-controller-0 2022-05-13 12:54:14,135 p=410772 u=stack n=ansible | 2022-05-13 12:54:14.131605 | 48d539a1-1679-3280-4b7d-000000000086 | TASK | Set container images used by the applied services 2022-05-13 12:54:14,819 p=410772 u=stack n=ansible | 2022-05-13 12:54:14.817877 | 48d539a1-1679-3280-4b7d-000000000086 | SKIPPED | Set container images used by the applied services | overcloud-controller-0 2022-05-13 12:54:14,905 p=410772 u=stack n=ansible | 2022-05-13 12:54:14.895988 | 48d539a1-1679-3280-4b7d-000000000087 | TASK | Apply spec 2022-05-13 12:54:15,196 p=410772 u=stack n=ansible | 2022-05-13 12:54:15.194726 | 48d539a1-1679-3280-4b7d-000000000087 | SKIPPED | Apply spec | overcloud-controller-0 2022-05-13 12:54:15,227 p=410772 u=stack n=ansible | 2022-05-13 12:54:15.226102 | 48d539a1-1679-3280-4b7d-000000000088 | TASK | Show results of spec apply 2022-05-13 12:54:15,527 p=410772 u=stack n=ansible | 2022-05-13 12:54:15.526231 | 48d539a1-1679-3280-4b7d-000000000088 | SKIPPED | Show results of spec apply | overcloud-controller-0 2022-05-13 12:54:15,564 p=410772 u=stack n=ansible | 2022-05-13 12:54:15.562873 | 48d539a1-1679-3280-4b7d-000000000089 | TASK | Wait for the expected number of osds to be running 2022-05-13 12:54:15,845 p=410772 u=stack n=ansible | 2022-05-13 12:54:15.843796 | 48d539a1-1679-3280-4b7d-000000000089 | SKIPPED | Wait for the expected number of osds to be running | overcloud-controller-0 2022-05-13 12:54:15,875 p=410772 u=stack n=ansible | 2022-05-13 12:54:15.874719 | 48d539a1-1679-3280-4b7d-0000000000a2 | TASK | Get ceph_cli 2022-05-13 12:54:16,191 p=410772 u=stack n=ansible | 2022-05-13 12:54:16.190351 | 66e7ed25-2057-472e-8440-e386f33b7d4d | INCLUDED | /usr/share/ansible/roles/tripleo_cephadm/tasks/ceph_cli.yaml | overcloud-controller-0 2022-05-13 12:54:16,231 p=410772 u=stack n=ansible | 2022-05-13 12:54:16.230624 | 48d539a1-1679-3280-4b7d-00000000021e | TASK | Set ceph CLI 2022-05-13 12:54:16,545 p=410772 u=stack n=ansible | 2022-05-13 12:54:16.543786 | 48d539a1-1679-3280-4b7d-00000000021e | OK | Set ceph CLI | overcloud-controller-0 2022-05-13 12:54:16,575 p=410772 u=stack n=ansible | 2022-05-13 12:54:16.574542 | 48d539a1-1679-3280-4b7d-0000000000a3 | TASK | create configured crush rules 2022-05-13 12:54:18,004 p=410772 u=stack n=ansible | 2022-05-13 12:54:18.002346 | 48d539a1-1679-3280-4b7d-0000000000a3 | OK | create configured crush rules | overcloud-controller-0 | item={'class': 'hdd', 'default': True, 'name': 'HDD', 'root': 'default', 'type': 'host'} 2022-05-13 12:54:18,970 p=410772 u=stack n=ansible | 2022-05-13 12:54:18.969066 | 48d539a1-1679-3280-4b7d-0000000000a3 | OK | create configured crush rules | overcloud-controller-0 | item={'class': 'ssd', 'default': False, 'name': 'SSD', 'root': 'default', 'type': 'host'} 2022-05-13 12:54:19,011 p=410772 u=stack n=ansible | 2022-05-13 12:54:19.010202 | 48d539a1-1679-3280-4b7d-0000000000a4 | TASK | get id for new default crush rule 2022-05-13 12:54:20,149 p=410772 u=stack n=ansible | 2022-05-13 12:54:20.148103 | 48d539a1-1679-3280-4b7d-0000000000a4 | OK | get id for new default crush rule | overcloud-controller-0 | item={'class': 'hdd', 'default': True, 'name': 'HDD', 'root': 'default', 'type': 'host'} 2022-05-13 12:54:20,166 p=410772 u=stack n=ansible | 2022-05-13 12:54:20.165286 | 48d539a1-1679-3280-4b7d-0000000000a4 | SKIPPED | get id for new default crush rule | overcloud-controller-0 | item={'class': 'ssd', 'default': False, 'name': 'SSD', 'root': 'default', 'type': 'host'} 2022-05-13 12:54:20,204 p=410772 u=stack n=ansible | 2022-05-13 12:54:20.203192 | 48d539a1-1679-3280-4b7d-0000000000a5 | TASK | set_fact info_ceph_default_crush_rule_yaml, ceph_osd_pool_default_crush_rule_name 2022-05-13 12:54:20,423 p=410772 u=stack n=ansible | 2022-05-13 12:54:20.421962 | 48d539a1-1679-3280-4b7d-0000000000a5 | OK | set_fact info_ceph_default_crush_rule_yaml, ceph_osd_pool_default_crush_rule_name | overcloud-controller-0 | item={'cmd': ['podman', 'run', '--rm', '--net=host', '-v', '/etc/ceph:/etc/ceph:z', '-v', '/var/lib/ceph/:/var/lib/ceph/:z', '-v', '/var/log/ceph/:/var/log/ceph/:z', '--entrypoint=ceph', '172.25.201.68:8787/ceph/daemon:v6.0.6-stable-6.0-pacific-centos-8-x86_64', '-n', 'client.admin', '-k', '/etc/ceph/ceph.client.admin.keyring', '--cluster', 'ceph', 'osd', 'crush', 'rule', 'dump', 'HDD', '--format=json'], 'start': '2022-05-13 04:54:19.369834', 'end': '2022-05-13 04:54:20.111638', 'delta': '0:00:00.741804', 'rc': 0, 'stdout': '\n{"rule_id":1,"rule_name":"HDD","ruleset":1,"type":1,"min_size":1,"max_size":10,"steps":[{"op":"take","item":-2,"item_name":"default~hdd"},{"op":"chooseleaf_firstn","num":0,"type":"host"},{"op":"emit"}]}', 'stderr': '', 'changed': False, 'invocation': {'module_args': {'name': 'HDD', 'cluster': 'ceph', 'state': 'info', 'rule_type': None, 'bucket_root': None, 'bucket_type': None, 'device_class': None, 'profile': None}}, 'stdout_lines': ['', '{"rule_id":1,"rule_name":"HDD","ruleset":1,"type":1,"min_size":1,"max_size":10,"steps":[{"op":"take","item":-2,"item_name":"default~hdd"},{"op":"chooseleaf_firstn","num":0,"type":"host"},{"op":"emit"}]}'], 'stderr_lines': [], 'failed': False, 'item': {'class': 'hdd', 'default': True, 'name': 'HDD', 'root': 'default', 'type': 'host'}, 'ansible_loop_var': 'item'} 2022-05-13 12:54:20,432 p=410772 u=stack n=ansible | 2022-05-13 12:54:20.430949 | 48d539a1-1679-3280-4b7d-0000000000a5 | SKIPPED | set_fact info_ceph_default_crush_rule_yaml, ceph_osd_pool_default_crush_rule_name | overcloud-controller-0 | item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': {'class': 'ssd', 'default': False, 'name': 'SSD', 'root': 'default', 'type': 'host'}, 'ansible_loop_var': 'item'} 2022-05-13 12:54:20,708 p=410772 u=stack n=ansible | 2022-05-13 12:54:20.705512 | 48d539a1-1679-3280-4b7d-0000000000a6 | TASK | insert new default crush rule into daemon to prevent restart 2022-05-13 12:54:21,481 p=410772 u=stack n=ansible | 2022-05-13 12:54:21.479579 | 48d539a1-1679-3280-4b7d-0000000000a6 | FATAL | insert new default crush rule into daemon to prevent restart | overcloud-controller-0 -> 172.25.201.106 | item=overcloud-controller-0 | error={"ansible_loop_var": "item", "changed": false, "cmd": ["podman", "run", "--rm", "--net=host", "--ipc=host", "--volume", "/etc/ceph:/etc/ceph:z", "--volume", "/home/ceph-admin/assimilate_ceph.conf:/home/assimilate_ceph.conf:z", "--volume", "/var/run/ceph/80c8599f-1698-50bf-8d07-fe3fc94b554a:/var/run/ceph:z", "--entrypoint", "ceph", "172.25.201.68:8787/ceph/daemon:v6.0.6-stable-6.0-pacific-centos-8-x86_64", "--admin-daemon", "/var/run/ceph/ceph-mon.overcloud-controller-0.bdxworld.com.asok", "config", "set", "osd_pool_default_crush_rule", "1"], "delta": "0:00:00.442243", "end": "2022-05-13 04:54:21.412176", "item": "overcloud-controller-0", "msg": "non-zero return code", "rc": 22, "start": "2022-05-13 04:54:20.969933", "stderr": "admin_socket: exception getting command descriptions: [Errno 2] No such file or directory", "stderr_lines": ["admin_socket: exception getting command descriptions: [Errno 2] No such file or directory"], "stdout": "", "stdout_lines": []} 2022-05-13 12:54:21,484 p=410772 u=stack n=ansible | [WARNING]: ('overcloud-controller-0 -> 172.25.201.106', '48d539a1-1679-3280-4b7d-0000000000a6') missing from stats 2022-05-13 12:54:23,090 p=410772 u=stack n=ansible | 2022-05-13 12:54:23.089195 | 48d539a1-1679-3280-4b7d-0000000000a6 | FATAL | insert new default crush rule into daemon to prevent restart | overcloud-controller-0 -> 172.25.201.105 | item=overcloud-controller-1 | error={"ansible_loop_var": "item", "changed": false, "cmd": ["podman", "run", "--rm", "--net=host", "--ipc=host", "--volume", "/etc/ceph:/etc/ceph:z", "--volume", "/home/ceph-admin/assimilate_ceph.conf:/home/assimilate_ceph.conf:z", "--volume", "/var/run/ceph/80c8599f-1698-50bf-8d07-fe3fc94b554a:/var/run/ceph:z", "--entrypoint", "ceph", "172.25.201.68:8787/ceph/daemon:v6.0.6-stable-6.0-pacific-centos-8-x86_64", "--admin-daemon", "/var/run/ceph/ceph-mon.overcloud-controller-1.bdxworld.com.asok", "config", "set", "osd_pool_default_crush_rule", "1"], "delta": "0:00:01.114824", "end": "2022-05-13 04:54:23.040914", "item": "overcloud-controller-1", "msg": "non-zero return code", "rc": 22, "start": "2022-05-13 04:54:21.926090", "stderr": "admin_socket: exception getting command descriptions: [Errno 2] No such file or directory", "stderr_lines": ["admin_socket: exception getting command descriptions: [Errno 2] No such file or directory"], "stdout": "", "stdout_lines": []} 2022-05-13 12:54:23,093 p=410772 u=stack n=ansible | [WARNING]: ('overcloud-controller-0 -> 172.25.201.105', '48d539a1-1679-3280-4b7d-0000000000a6') missing from stats 2022-05-13 12:54:24,087 p=410772 u=stack n=ansible | 2022-05-13 12:54:24.086303 | 48d539a1-1679-3280-4b7d-0000000000a6 | FATAL | insert new default crush rule into daemon to prevent restart | overcloud-controller-0 -> 172.25.201.97 | item=overcloud-controller-2 | error={"ansible_loop_var": "item", "changed": false, "cmd": ["podman", "run", "--rm", "--net=host", "--ipc=host", "--volume", "/etc/ceph:/etc/ceph:z", "--volume", "/home/ceph-admin/assimilate_ceph.conf:/home/assimilate_ceph.conf:z", "--volume", "/var/run/ceph/80c8599f-1698-50bf-8d07-fe3fc94b554a:/var/run/ceph:z", "--entrypoint", "ceph", "172.25.201.68:8787/ceph/daemon:v6.0.6-stable-6.0-pacific-centos-8-x86_64", "--admin-daemon", "/var/run/ceph/ceph-mon.overcloud-controller-2.bdxworld.com.asok", "config", "set", "osd_pool_default_crush_rule", "1"], "delta": "0:00:00.661832", "end": "2022-05-13 04:54:24.045620", "item": "overcloud-controller-2", "msg": "non-zero return code", "rc": 22, "start": "2022-05-13 04:54:23.383788", "stderr": "admin_socket: exception getting command descriptions: [Errno 2] No such file or directory", "stderr_lines": ["admin_socket: exception getting command descriptions: [Errno 2] No such file or directory"], "stdout": "", "stdout_lines": []} 2022-05-13 12:54:24,090 p=410772 u=stack n=ansible | [WARNING]: ('overcloud-controller-0 -> 172.25.201.97', '48d539a1-1679-3280-4b7d-0000000000a6') missing from stats 2022-05-13 12:54:24,103 p=410772 u=stack n=ansible | NO MORE HOSTS LEFT ************************************************************* 2022-05-13 12:54:24,110 p=410772 u=stack n=ansible | PLAY RECAP ********************************************************************* 2022-05-13 12:54:24,111 p=410772 u=stack n=ansible | overcloud-controller-0 : ok=15 changed=1 unreachable=0 failed=1 skipped=25 rescued=0 ignored=0 2022-05-13 12:54:24,113 p=410772 u=stack n=ansible | 2022-05-13 12:54:24.112844 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Summary Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2022-05-13 12:54:24,114 p=410772 u=stack n=ansible | 2022-05-13 12:54:24.113624 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Total Tasks: 41 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2022-05-13 12:54:24,114 p=410772 u=stack n=ansible | 2022-05-13 12:54:24.114345 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Elapsed Time: 0:00:21.863949 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2022-05-13 12:54:24,115 p=410772 u=stack n=ansible | 2022-05-13 12:54:24.115070 | UUID | Info | Host | Task Name | Run Time 2022-05-13 12:54:24,116 p=410772 u=stack n=ansible | 2022-05-13 12:54:24.116205 | 48d539a1-1679-3280-4b7d-0000000000a6 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : insert new default crush rule into daemon to prevent restart | 3.39s 2022-05-13 12:54:24,117 p=410772 u=stack n=ansible | 2022-05-13 12:54:24.116990 | 48d539a1-1679-3280-4b7d-0000000000a3 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : create configured crush rules | 2.40s 2022-05-13 12:54:24,118 p=410772 u=stack n=ansible | 2022-05-13 12:54:24.117756 | 48d539a1-1679-3280-4b7d-00000000001b | SUMMARY | overcloud-controller-0 | tripleo_cephadm : List Ceph daemon instances on this host | 1.18s 2022-05-13 12:54:24,119 p=410772 u=stack n=ansible | 2022-05-13 12:54:24.118483 | 48d539a1-1679-3280-4b7d-0000000000a4 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : get id for new default crush rule | 1.16s 2022-05-13 12:54:24,119 p=410772 u=stack n=ansible | 2022-05-13 12:54:24.119201 | 48d539a1-1679-3280-4b7d-00000000002b | SUMMARY | overcloud-controller-0 | generate tripleo_cephadm_assimilate_conf on bootstrap node | 1.02s 2022-05-13 12:54:24,120 p=410772 u=stack n=ansible | 2022-05-13 12:54:24.119927 | 48d539a1-1679-3280-4b7d-000000000076 | SUMMARY | overcloud-controller-0 | Assimilate configuration from tripleo_cephadm_assimilate_conf | 1.01s 2022-05-13 12:54:24,121 p=410772 u=stack n=ansible | 2022-05-13 12:54:24.120631 | 48d539a1-1679-3280-4b7d-000000000023 | SUMMARY | overcloud-controller-0 | Ensure tripleo_cephadm_config_home (e.g. /etc/ceph) exists | 0.86s 2022-05-13 12:54:24,121 p=410772 u=stack n=ansible | 2022-05-13 12:54:24.121335 | 48d539a1-1679-3280-4b7d-000000000019 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Stat cephadm file | 0.75s 2022-05-13 12:54:24,122 p=410772 u=stack n=ansible | 2022-05-13 12:54:24.122043 | 48d539a1-1679-3280-4b7d-000000000026 | SUMMARY | overcloud-controller-0 | push tripleo_cephadm_spec to bootstrap node if spec file exsits | 0.72s 2022-05-13 12:54:24,123 p=410772 u=stack n=ansible | 2022-05-13 12:54:24.122752 | 48d539a1-1679-3280-4b7d-000000000086 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Set container images used by the applied services | 0.68s 2022-05-13 12:54:24,124 p=410772 u=stack n=ansible | 2022-05-13 12:54:24.123590 | 48d539a1-1679-3280-4b7d-000000000024 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Ensure specs directory exists | 0.34s 2022-05-13 12:54:24,124 p=410772 u=stack n=ansible | 2022-05-13 12:54:24.124251 | 48d539a1-1679-3280-4b7d-000000000085 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Fail if ceph orchestrator is not available | 0.32s 2022-05-13 12:54:24,125 p=410772 u=stack n=ansible | 2022-05-13 12:54:24.124989 | 48d539a1-1679-3280-4b7d-00000000021e | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Set ceph CLI | 0.31s 2022-05-13 12:54:24,126 p=410772 u=stack n=ansible | 2022-05-13 12:54:24.125937 | 48d539a1-1679-3280-4b7d-000000000088 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Show results of spec apply | 0.30s 2022-05-13 12:54:24,127 p=410772 u=stack n=ansible | 2022-05-13 12:54:24.126801 | 48d539a1-1679-3280-4b7d-000000000084 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Get the ceph orchestrator status | 0.29s 2022-05-13 12:54:24,128 p=410772 u=stack n=ansible | 2022-05-13 12:54:24.127759 | 48d539a1-1679-3280-4b7d-000000000087 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Apply spec | 0.29s 2022-05-13 12:54:24,129 p=410772 u=stack n=ansible | 2022-05-13 12:54:24.128589 | 48d539a1-1679-3280-4b7d-0000000000a2 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Get ceph_cli | 0.28s 2022-05-13 12:54:24,130 p=410772 u=stack n=ansible | 2022-05-13 12:54:24.129361 | 48d539a1-1679-3280-4b7d-000000000089 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Wait for the expected number of osds to be running | 0.28s 2022-05-13 12:54:24,130 p=410772 u=stack n=ansible | 2022-05-13 12:54:24.130253 | 48d539a1-1679-3280-4b7d-0000000000a5 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : set_fact info_ceph_default_crush_rule_yaml, ceph_osd_pool_default_crush_rule_name | 0.23s 2022-05-13 12:54:24,131 p=410772 u=stack n=ansible | 2022-05-13 12:54:24.131134 | 48d539a1-1679-3280-4b7d-000000000022 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Set first monitor IP if it was not passed | 0.22s 2022-05-13 12:54:24,132 p=410772 u=stack n=ansible | 2022-05-13 12:54:24.131853 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ End Summary Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2022-05-13 12:54:24,133 p=410772 u=stack n=ansible | 2022-05-13 12:54:24.132639 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ State Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2022-05-13 12:54:24,133 p=410772 u=stack n=ansible | 2022-05-13 12:54:24.133338 | ~~~~~~~~~~~~~~~~~~ Number of nodes which did not deploy successfully: 1 ~~~~~~~~~~~~~~~~~ 2022-05-13 12:54:24,134 p=410772 u=stack n=ansible | 2022-05-13 12:54:24.134074 | The following node(s) had failures: overcloud-controller-0 2022-05-13 12:54:24,135 p=410772 u=stack n=ansible | 2022-05-13 12:54:24.134816 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Running /home/stack/overcloud-deploy/overcloud/config-download/overcloud/cephadm/cephadm_command.sh 2022-05-13 13:33:55,792 p=515717 u=stack n=ansible | PLAY [Deploy Ceph with cephadm] ************************************************ 2022-05-13 13:33:55,877 p=515717 u=stack n=ansible | 2022-05-13 13:33:55.875946 | 48d539a1-1679-ae16-8bd7-000000000018 | TASK | Install cephadm package 2022-05-13 13:33:56,009 p=515717 u=stack n=ansible | 2022-05-13 13:33:56.007856 | 48d539a1-1679-ae16-8bd7-000000000018 | SKIPPED | Install cephadm package | overcloud-controller-0 2022-05-13 13:33:56,041 p=515717 u=stack n=ansible | 2022-05-13 13:33:56.039926 | 48d539a1-1679-ae16-8bd7-000000000019 | TASK | Stat cephadm file 2022-05-13 13:33:56,891 p=515717 u=stack n=ansible | 2022-05-13 13:33:56.890430 | 48d539a1-1679-ae16-8bd7-000000000019 | OK | Stat cephadm file | overcloud-controller-0 2022-05-13 13:33:56,926 p=515717 u=stack n=ansible | 2022-05-13 13:33:56.924878 | 48d539a1-1679-ae16-8bd7-00000000001a | TASK | Fail if cephadm is not available 2022-05-13 13:33:57,055 p=515717 u=stack n=ansible | 2022-05-13 13:33:57.053678 | 48d539a1-1679-ae16-8bd7-00000000001a | SKIPPED | Fail if cephadm is not available | overcloud-controller-0 2022-05-13 13:33:57,085 p=515717 u=stack n=ansible | 2022-05-13 13:33:57.084138 | 48d539a1-1679-ae16-8bd7-00000000001b | TASK | List Ceph daemon instances on this host 2022-05-13 13:33:58,447 p=515717 u=stack n=ansible | 2022-05-13 13:33:58.446209 | 48d539a1-1679-ae16-8bd7-00000000001b | CHANGED | List Ceph daemon instances on this host | overcloud-controller-0 2022-05-13 13:33:58,480 p=515717 u=stack n=ansible | 2022-05-13 13:33:58.479062 | 48d539a1-1679-ae16-8bd7-00000000001d | TASK | Set list of found FSIDs 2022-05-13 13:33:58,778 p=515717 u=stack n=ansible | 2022-05-13 13:33:58.775893 | 48d539a1-1679-ae16-8bd7-00000000001d | SKIPPED | Set list of found FSIDs | overcloud-controller-0 2022-05-13 13:33:58,808 p=515717 u=stack n=ansible | 2022-05-13 13:33:58.807320 | 48d539a1-1679-ae16-8bd7-00000000001e | TASK | Fail if >1 FSID was discovered 2022-05-13 13:33:58,950 p=515717 u=stack n=ansible | 2022-05-13 13:33:58.949431 | 48d539a1-1679-ae16-8bd7-00000000001e | SKIPPED | Fail if >1 FSID was discovered | overcloud-controller-0 2022-05-13 13:33:58,984 p=515717 u=stack n=ansible | 2022-05-13 13:33:58.983144 | 48d539a1-1679-ae16-8bd7-00000000001f | TASK | Set FSID to the discovered value 2022-05-13 13:33:59,139 p=515717 u=stack n=ansible | 2022-05-13 13:33:59.138188 | 48d539a1-1679-ae16-8bd7-00000000001f | SKIPPED | Set FSID to the discovered value | overcloud-controller-0 2022-05-13 13:33:59,170 p=515717 u=stack n=ansible | 2022-05-13 13:33:59.169476 | 48d539a1-1679-ae16-8bd7-000000000020 | TASK | Set random tripleo_cephadm_fsid if no running ceph containers were found 2022-05-13 13:33:59,748 p=515717 u=stack n=ansible | 2022-05-13 13:33:59.747101 | 48d539a1-1679-ae16-8bd7-000000000020 | SKIPPED | Set random tripleo_cephadm_fsid if no running ceph containers were found | overcloud-controller-0 2022-05-13 13:33:59,778 p=515717 u=stack n=ansible | 2022-05-13 13:33:59.777601 | 48d539a1-1679-ae16-8bd7-000000000022 | TASK | Set first monitor IP if it was not passed 2022-05-13 13:33:59,950 p=515717 u=stack n=ansible | 2022-05-13 13:33:59.948927 | 48d539a1-1679-ae16-8bd7-000000000022 | SKIPPED | Set first monitor IP if it was not passed | overcloud-controller-0 2022-05-13 13:33:59,979 p=515717 u=stack n=ansible | 2022-05-13 13:33:59.978064 | 48d539a1-1679-ae16-8bd7-000000000023 | TASK | Ensure tripleo_cephadm_config_home (e.g. /etc/ceph) exists 2022-05-13 13:34:00,754 p=515717 u=stack n=ansible | 2022-05-13 13:34:00.752508 | 48d539a1-1679-ae16-8bd7-000000000023 | OK | Ensure tripleo_cephadm_config_home (e.g. /etc/ceph) exists | overcloud-controller-0 2022-05-13 13:34:00,785 p=515717 u=stack n=ansible | 2022-05-13 13:34:00.784333 | 48d539a1-1679-ae16-8bd7-000000000024 | TASK | Ensure specs directory exists 2022-05-13 13:34:01,126 p=515717 u=stack n=ansible | 2022-05-13 13:34:01.125003 | 48d539a1-1679-ae16-8bd7-000000000024 | OK | Ensure specs directory exists | overcloud-controller-0 2022-05-13 13:34:01,167 p=515717 u=stack n=ansible | 2022-05-13 13:34:01.166607 | 48d539a1-1679-ae16-8bd7-000000000025 | TASK | Stat spec file on ansible host 2022-05-13 13:34:02,029 p=515717 u=stack n=ansible | 2022-05-13 13:34:02.028317 | 48d539a1-1679-ae16-8bd7-000000000025 | OK | Stat spec file on ansible host | overcloud-controller-0 -> localhost 2022-05-13 13:34:02,033 p=515717 u=stack n=ansible | [WARNING]: ('overcloud-controller-0 -> localhost', '48d539a1-1679-ae16-8bd7-000000000025') missing from stats 2022-05-13 13:34:02,064 p=515717 u=stack n=ansible | 2022-05-13 13:34:02.063639 | 48d539a1-1679-ae16-8bd7-000000000026 | TASK | push tripleo_cephadm_spec to bootstrap node if spec file exsits 2022-05-13 13:34:03,364 p=515717 u=stack n=ansible | 2022-05-13 13:34:03.363042 | 48d539a1-1679-ae16-8bd7-000000000026 | CHANGED | push tripleo_cephadm_spec to bootstrap node if spec file exsits | overcloud-controller-0 2022-05-13 13:34:03,404 p=515717 u=stack n=ansible | 2022-05-13 13:34:03.403785 | 48d539a1-1679-ae16-8bd7-000000000028 | TASK | stat tripleo_cephadm_bootstrap_conf file 2022-05-13 13:34:03,535 p=515717 u=stack n=ansible | 2022-05-13 13:34:03.533568 | 48d539a1-1679-ae16-8bd7-000000000028 | SKIPPED | stat tripleo_cephadm_bootstrap_conf file | overcloud-controller-0 2022-05-13 13:34:03,565 p=515717 u=stack n=ansible | 2022-05-13 13:34:03.562948 | 48d539a1-1679-ae16-8bd7-000000000029 | TASK | push it to tripleo_cephadm_assimilate_conf on bootstrap node 2022-05-13 13:34:03,774 p=515717 u=stack n=ansible | 2022-05-13 13:34:03.773429 | 48d539a1-1679-ae16-8bd7-000000000029 | SKIPPED | push it to tripleo_cephadm_assimilate_conf on bootstrap node | overcloud-controller-0 2022-05-13 13:34:03,806 p=515717 u=stack n=ansible | 2022-05-13 13:34:03.805810 | 48d539a1-1679-ae16-8bd7-00000000002b | TASK | generate tripleo_cephadm_assimilate_conf on bootstrap node 2022-05-13 13:34:04,536 p=515717 u=stack n=ansible | 2022-05-13 13:34:04.534838 | 48d539a1-1679-ae16-8bd7-00000000002b | CHANGED | generate tripleo_cephadm_assimilate_conf on bootstrap node | overcloud-controller-0 2022-05-13 13:34:04,567 p=515717 u=stack n=ansible | 2022-05-13 13:34:04.565371 | 48d539a1-1679-ae16-8bd7-000000000056 | TASK | Add spec to necessary file list when using spec to bootstrap 2022-05-13 13:34:04,758 p=515717 u=stack n=ansible | 2022-05-13 13:34:04.756634 | 48d539a1-1679-ae16-8bd7-000000000056 | SKIPPED | Add spec to necessary file list when using spec to bootstrap | overcloud-controller-0 2022-05-13 13:34:04,787 p=515717 u=stack n=ansible | 2022-05-13 13:34:04.786490 | 48d539a1-1679-ae16-8bd7-000000000057 | TASK | Stat necessary files to bootstrap with cephadm 2022-05-13 13:34:04,951 p=515717 u=stack n=ansible | 2022-05-13 13:34:04.950115 | 48d539a1-1679-ae16-8bd7-000000000057 | SKIPPED | Stat necessary files to bootstrap with cephadm | overcloud-controller-0 | item=/home/ceph-admin/.ssh/id_rsa 2022-05-13 13:34:04,953 p=515717 u=stack n=ansible | 2022-05-13 13:34:04.953029 | 48d539a1-1679-ae16-8bd7-000000000057 | SKIPPED | Stat necessary files to bootstrap with cephadm | overcloud-controller-0 | item=/home/ceph-admin/.ssh/id_rsa.pub 2022-05-13 13:34:04,984 p=515717 u=stack n=ansible | 2022-05-13 13:34:04.983583 | 48d539a1-1679-ae16-8bd7-000000000058 | TASK | Fail if necessary files are missing 2022-05-13 13:34:05,152 p=515717 u=stack n=ansible | 2022-05-13 13:34:05.150784 | 48d539a1-1679-ae16-8bd7-000000000058 | SKIPPED | Fail if necessary files are missing | overcloud-controller-0 | item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': '/home/ceph-admin/.ssh/id_rsa', 'ansible_loop_var': 'item'} 2022-05-13 13:34:05,154 p=515717 u=stack n=ansible | 2022-05-13 13:34:05.154039 | 48d539a1-1679-ae16-8bd7-000000000058 | SKIPPED | Fail if necessary files are missing | overcloud-controller-0 | item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': '/home/ceph-admin/.ssh/id_rsa.pub', 'ansible_loop_var': 'item'} 2022-05-13 13:34:05,189 p=515717 u=stack n=ansible | 2022-05-13 13:34:05.186786 | 48d539a1-1679-ae16-8bd7-000000000059 | TASK | Stat pre ceph conf file in case we should bootrap with it 2022-05-13 13:34:05,321 p=515717 u=stack n=ansible | 2022-05-13 13:34:05.319444 | 48d539a1-1679-ae16-8bd7-000000000059 | SKIPPED | Stat pre ceph conf file in case we should bootrap with it | overcloud-controller-0 2022-05-13 13:34:05,351 p=515717 u=stack n=ansible | 2022-05-13 13:34:05.350241 | 48d539a1-1679-ae16-8bd7-00000000005b | TASK | Run cephadm bootstrap 2022-05-13 13:34:05,482 p=515717 u=stack n=ansible | 2022-05-13 13:34:05.481166 | 48d539a1-1679-ae16-8bd7-00000000005b | SKIPPED | Run cephadm bootstrap | overcloud-controller-0 2022-05-13 13:34:05,514 p=515717 u=stack n=ansible | 2022-05-13 13:34:05.512838 | 48d539a1-1679-ae16-8bd7-00000000005c | TASK | Show results of bootstrap 2022-05-13 13:34:05,657 p=515717 u=stack n=ansible | 2022-05-13 13:34:05.655654 | 48d539a1-1679-ae16-8bd7-00000000005c | SKIPPED | Show results of bootstrap | overcloud-controller-0 2022-05-13 13:34:05,688 p=515717 u=stack n=ansible | 2022-05-13 13:34:05.687165 | 48d539a1-1679-ae16-8bd7-00000000005e | TASK | If cephadm bootstrap was not run report the reason 2022-05-13 13:34:05,829 p=515717 u=stack n=ansible | 2022-05-13 13:34:05.827928 | 48d539a1-1679-ae16-8bd7-00000000005e | SKIPPED | If cephadm bootstrap was not run report the reason | overcloud-controller-0 2022-05-13 13:34:05,860 p=515717 u=stack n=ansible | 2022-05-13 13:34:05.858662 | 48d539a1-1679-ae16-8bd7-000000000075 | TASK | Ensure tripleo_cephadm_ceph_cli is set 2022-05-13 13:34:06,035 p=515717 u=stack n=ansible | 2022-05-13 13:34:06.034329 | 16a2c759-cc26-4340-b39e-ded6b8398bba | INCLUDED | /usr/share/ansible/roles/tripleo_cephadm/tasks/ceph_cli.yaml | overcloud-controller-0 2022-05-13 13:34:06,068 p=515717 u=stack n=ansible | 2022-05-13 13:34:06.067261 | 48d539a1-1679-ae16-8bd7-0000000001f1 | TASK | Set ceph CLI 2022-05-13 13:34:06,318 p=515717 u=stack n=ansible | 2022-05-13 13:34:06.316694 | 48d539a1-1679-ae16-8bd7-0000000001f1 | OK | Set ceph CLI | overcloud-controller-0 2022-05-13 13:34:06,349 p=515717 u=stack n=ansible | 2022-05-13 13:34:06.348079 | 48d539a1-1679-ae16-8bd7-000000000076 | TASK | Assimilate configuration from tripleo_cephadm_assimilate_conf 2022-05-13 13:34:07,330 p=515717 u=stack n=ansible | 2022-05-13 13:34:07.329234 | 48d539a1-1679-ae16-8bd7-000000000076 | OK | Assimilate configuration from tripleo_cephadm_assimilate_conf | overcloud-controller-0 2022-05-13 13:34:07,361 p=515717 u=stack n=ansible | 2022-05-13 13:34:07.360334 | 48d539a1-1679-ae16-8bd7-000000000081 | TASK | Stat spec file on bootstrap node 2022-05-13 13:34:07,652 p=515717 u=stack n=ansible | 2022-05-13 13:34:07.651281 | 48d539a1-1679-ae16-8bd7-000000000081 | SKIPPED | Stat spec file on bootstrap node | overcloud-controller-0 | item=/home/ceph-admin/specs/ceph_spec.yaml 2022-05-13 13:34:07,690 p=515717 u=stack n=ansible | 2022-05-13 13:34:07.689016 | 48d539a1-1679-ae16-8bd7-000000000082 | TASK | Fail if spec file is missing 2022-05-13 13:34:07,978 p=515717 u=stack n=ansible | 2022-05-13 13:34:07.976866 | 48d539a1-1679-ae16-8bd7-000000000082 | SKIPPED | Fail if spec file is missing | overcloud-controller-0 | item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': '/home/ceph-admin/specs/ceph_spec.yaml', 'ansible_loop_var': 'item'} 2022-05-13 13:34:08,020 p=515717 u=stack n=ansible | 2022-05-13 13:34:08.019580 | 48d539a1-1679-ae16-8bd7-000000000083 | TASK | Get ceph_cli 2022-05-13 13:34:08,291 p=515717 u=stack n=ansible | 2022-05-13 13:34:08.290370 | 48d539a1-1679-ae16-8bd7-000000000083 | SKIPPED | Get ceph_cli | overcloud-controller-0 2022-05-13 13:34:08,321 p=515717 u=stack n=ansible | 2022-05-13 13:34:08.320659 | 48d539a1-1679-ae16-8bd7-000000000084 | TASK | Get the ceph orchestrator status 2022-05-13 13:34:08,681 p=515717 u=stack n=ansible | 2022-05-13 13:34:08.679864 | 48d539a1-1679-ae16-8bd7-000000000084 | SKIPPED | Get the ceph orchestrator status | overcloud-controller-0 2022-05-13 13:34:08,712 p=515717 u=stack n=ansible | 2022-05-13 13:34:08.711342 | 48d539a1-1679-ae16-8bd7-000000000085 | TASK | Fail if ceph orchestrator is not available 2022-05-13 13:34:09,117 p=515717 u=stack n=ansible | 2022-05-13 13:34:09.115898 | 48d539a1-1679-ae16-8bd7-000000000085 | SKIPPED | Fail if ceph orchestrator is not available | overcloud-controller-0 2022-05-13 13:34:09,163 p=515717 u=stack n=ansible | 2022-05-13 13:34:09.154684 | 48d539a1-1679-ae16-8bd7-000000000086 | TASK | Set container images used by the applied services 2022-05-13 13:34:09,467 p=515717 u=stack n=ansible | 2022-05-13 13:34:09.466508 | 48d539a1-1679-ae16-8bd7-000000000086 | SKIPPED | Set container images used by the applied services | overcloud-controller-0 2022-05-13 13:34:09,496 p=515717 u=stack n=ansible | 2022-05-13 13:34:09.495597 | 48d539a1-1679-ae16-8bd7-000000000087 | TASK | Apply spec 2022-05-13 13:34:09,775 p=515717 u=stack n=ansible | 2022-05-13 13:34:09.773917 | 48d539a1-1679-ae16-8bd7-000000000087 | SKIPPED | Apply spec | overcloud-controller-0 2022-05-13 13:34:09,830 p=515717 u=stack n=ansible | 2022-05-13 13:34:09.810914 | 48d539a1-1679-ae16-8bd7-000000000088 | TASK | Show results of spec apply 2022-05-13 13:34:10,120 p=515717 u=stack n=ansible | 2022-05-13 13:34:10.119381 | 48d539a1-1679-ae16-8bd7-000000000088 | SKIPPED | Show results of spec apply | overcloud-controller-0 2022-05-13 13:34:10,562 p=515717 u=stack n=ansible | 2022-05-13 13:34:10.536384 | 48d539a1-1679-ae16-8bd7-000000000089 | TASK | Wait for the expected number of osds to be running 2022-05-13 13:34:10,944 p=515717 u=stack n=ansible | 2022-05-13 13:34:10.943168 | 48d539a1-1679-ae16-8bd7-000000000089 | SKIPPED | Wait for the expected number of osds to be running | overcloud-controller-0 2022-05-13 13:34:10,984 p=515717 u=stack n=ansible | 2022-05-13 13:34:10.983440 | 48d539a1-1679-ae16-8bd7-0000000000a2 | TASK | Get ceph_cli 2022-05-13 13:34:11,189 p=515717 u=stack n=ansible | 2022-05-13 13:34:11.188854 | 9b441ad8-1f67-47f7-a7a2-222980bbdf02 | INCLUDED | /usr/share/ansible/roles/tripleo_cephadm/tasks/ceph_cli.yaml | overcloud-controller-0 2022-05-13 13:34:11,224 p=515717 u=stack n=ansible | 2022-05-13 13:34:11.223679 | 48d539a1-1679-ae16-8bd7-00000000021e | TASK | Set ceph CLI 2022-05-13 13:34:11,429 p=515717 u=stack n=ansible | 2022-05-13 13:34:11.427761 | 48d539a1-1679-ae16-8bd7-00000000021e | OK | Set ceph CLI | overcloud-controller-0 2022-05-13 13:34:11,464 p=515717 u=stack n=ansible | 2022-05-13 13:34:11.463832 | 48d539a1-1679-ae16-8bd7-0000000000a3 | TASK | create configured crush rules 2022-05-13 13:34:13,413 p=515717 u=stack n=ansible | 2022-05-13 13:34:13.411058 | 48d539a1-1679-ae16-8bd7-0000000000a3 | OK | create configured crush rules | overcloud-controller-0 | item={'class': 'hdd', 'default': True, 'name': 'HDD', 'root': 'default', 'type': 'host'} 2022-05-13 13:34:14,305 p=515717 u=stack n=ansible | 2022-05-13 13:34:14.303806 | 48d539a1-1679-ae16-8bd7-0000000000a3 | OK | create configured crush rules | overcloud-controller-0 | item={'class': 'ssd', 'default': False, 'name': 'SSD', 'root': 'default', 'type': 'host'} 2022-05-13 13:34:14,435 p=515717 u=stack n=ansible | 2022-05-13 13:34:14.434575 | 48d539a1-1679-ae16-8bd7-0000000000a4 | TASK | get id for new default crush rule 2022-05-13 13:34:15,600 p=515717 u=stack n=ansible | 2022-05-13 13:34:15.598044 | 48d539a1-1679-ae16-8bd7-0000000000a4 | OK | get id for new default crush rule | overcloud-controller-0 | item={'class': 'hdd', 'default': True, 'name': 'HDD', 'root': 'default', 'type': 'host'} 2022-05-13 13:34:15,624 p=515717 u=stack n=ansible | 2022-05-13 13:34:15.622884 | 48d539a1-1679-ae16-8bd7-0000000000a4 | SKIPPED | get id for new default crush rule | overcloud-controller-0 | item={'class': 'ssd', 'default': False, 'name': 'SSD', 'root': 'default', 'type': 'host'} 2022-05-13 13:34:15,744 p=515717 u=stack n=ansible | 2022-05-13 13:34:15.743024 | 48d539a1-1679-ae16-8bd7-0000000000a5 | TASK | set_fact info_ceph_default_crush_rule_yaml, ceph_osd_pool_default_crush_rule_name 2022-05-13 13:34:15,956 p=515717 u=stack n=ansible | 2022-05-13 13:34:15.954444 | 48d539a1-1679-ae16-8bd7-0000000000a5 | OK | set_fact info_ceph_default_crush_rule_yaml, ceph_osd_pool_default_crush_rule_name | overcloud-controller-0 | item={'cmd': ['podman', 'run', '--rm', '--net=host', '-v', '/etc/ceph:/etc/ceph:z', '-v', '/var/lib/ceph/:/var/lib/ceph/:z', '-v', '/var/log/ceph/:/var/log/ceph/:z', '--entrypoint=ceph', '172.25.201.68:8787/ceph/daemon:v6.0.6-stable-6.0-pacific-centos-8-x86_64', '-n', 'client.admin', '-k', '/etc/ceph/ceph.client.admin.keyring', '--cluster', 'ceph', 'osd', 'crush', 'rule', 'dump', 'HDD', '--format=json'], 'start': '2022-05-13 05:34:14.801711', 'end': '2022-05-13 05:34:15.548471', 'delta': '0:00:00.746760', 'rc': 0, 'stdout': '\n{"rule_id":1,"rule_name":"HDD","ruleset":1,"type":1,"min_size":1,"max_size":10,"steps":[{"op":"take","item":-2,"item_name":"default~hdd"},{"op":"chooseleaf_firstn","num":0,"type":"host"},{"op":"emit"}]}', 'stderr': '', 'changed': False, 'invocation': {'module_args': {'name': 'HDD', 'cluster': 'ceph', 'state': 'info', 'rule_type': None, 'bucket_root': None, 'bucket_type': None, 'device_class': None, 'profile': None}}, 'stdout_lines': ['', '{"rule_id":1,"rule_name":"HDD","ruleset":1,"type":1,"min_size":1,"max_size":10,"steps":[{"op":"take","item":-2,"item_name":"default~hdd"},{"op":"chooseleaf_firstn","num":0,"type":"host"},{"op":"emit"}]}'], 'stderr_lines': [], 'failed': False, 'item': {'class': 'hdd', 'default': True, 'name': 'HDD', 'root': 'default', 'type': 'host'}, 'ansible_loop_var': 'item'} 2022-05-13 13:34:15,964 p=515717 u=stack n=ansible | 2022-05-13 13:34:15.963471 | 48d539a1-1679-ae16-8bd7-0000000000a5 | SKIPPED | set_fact info_ceph_default_crush_rule_yaml, ceph_osd_pool_default_crush_rule_name | overcloud-controller-0 | item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': {'class': 'ssd', 'default': False, 'name': 'SSD', 'root': 'default', 'type': 'host'}, 'ansible_loop_var': 'item'} 2022-05-13 13:34:16,532 p=515717 u=stack n=ansible | 2022-05-13 13:34:16.529334 | 48d539a1-1679-ae16-8bd7-0000000000a6 | TASK | insert new default crush rule into daemon to prevent restart 2022-05-13 13:34:17,382 p=515717 u=stack n=ansible | 2022-05-13 13:34:17.380311 | 48d539a1-1679-ae16-8bd7-0000000000a6 | OK | insert new default crush rule into daemon to prevent restart | overcloud-controller-0 -> 172.25.201.106 | item=overcloud-controller-0 2022-05-13 13:34:17,385 p=515717 u=stack n=ansible | [WARNING]: ('overcloud-controller-0 -> 172.25.201.106', '48d539a1-1679-ae16-8bd7-0000000000a6') missing from stats 2022-05-13 13:34:19,355 p=515717 u=stack n=ansible | 2022-05-13 13:34:19.354728 | 48d539a1-1679-ae16-8bd7-0000000000a6 | OK | insert new default crush rule into daemon to prevent restart | overcloud-controller-0 -> 172.25.201.105 | item=overcloud-controller-1 2022-05-13 13:34:19,358 p=515717 u=stack n=ansible | [WARNING]: ('overcloud-controller-0 -> 172.25.201.105', '48d539a1-1679-ae16-8bd7-0000000000a6') missing from stats 2022-05-13 13:34:20,356 p=515717 u=stack n=ansible | 2022-05-13 13:34:20.355429 | 48d539a1-1679-ae16-8bd7-0000000000a6 | OK | insert new default crush rule into daemon to prevent restart | overcloud-controller-0 -> 172.25.201.97 | item=overcloud-controller-2 2022-05-13 13:34:20,359 p=515717 u=stack n=ansible | [WARNING]: ('overcloud-controller-0 -> 172.25.201.97', '48d539a1-1679-ae16-8bd7-0000000000a6') missing from stats 2022-05-13 13:34:20,397 p=515717 u=stack n=ansible | 2022-05-13 13:34:20.395851 | 48d539a1-1679-ae16-8bd7-0000000000b7 | TASK | Create pool(s) 2022-05-13 13:34:24,041 p=515717 u=stack n=ansible | 2022-05-13 13:34:24.039228 | 48d539a1-1679-ae16-8bd7-0000000000b7 | CHANGED | Create pool(s) | overcloud-controller-0 | item={'name': 'vms', 'pg_num': '16', 'rule_name': 'replicated_rule', 'application': 'rbd'} 2022-05-13 13:34:27,298 p=515717 u=stack n=ansible | 2022-05-13 13:34:27.297199 | 48d539a1-1679-ae16-8bd7-0000000000b7 | CHANGED | Create pool(s) | overcloud-controller-0 | item={'name': 'volumes', 'pg_num': '16', 'rule_name': 'replicated_rule', 'application': 'rbd'} 2022-05-13 13:34:30,565 p=515717 u=stack n=ansible | 2022-05-13 13:34:30.564493 | 48d539a1-1679-ae16-8bd7-0000000000b7 | CHANGED | Create pool(s) | overcloud-controller-0 | item={'name': 'images', 'pg_num': '16', 'rule_name': 'replicated_rule', 'application': 'rbd'} 2022-05-13 13:34:33,769 p=515717 u=stack n=ansible | 2022-05-13 13:34:33.768416 | 48d539a1-1679-ae16-8bd7-0000000000b7 | CHANGED | Create pool(s) | overcloud-controller-0 | item={'name': 'metrics', 'pg_num': '16', 'rule_name': 'replicated_rule', 'application': 'openstack_gnocchi'} 2022-05-13 13:34:36,989 p=515717 u=stack n=ansible | 2022-05-13 13:34:36.988204 | 48d539a1-1679-ae16-8bd7-0000000000b7 | CHANGED | Create pool(s) | overcloud-controller-0 | item={'name': 'backups', 'pg_num': '16', 'rule_name': 'replicated_rule', 'application': 'rbd'} 2022-05-13 13:34:40,319 p=515717 u=stack n=ansible | 2022-05-13 13:34:40.317473 | 48d539a1-1679-ae16-8bd7-0000000000b7 | CHANGED | Create pool(s) | overcloud-controller-0 | item={'application': 'rbd', 'name': 'slow_pool', 'rule_name': 'HDD'} 2022-05-13 13:34:43,439 p=515717 u=stack n=ansible | 2022-05-13 13:34:43.437753 | 48d539a1-1679-ae16-8bd7-0000000000b7 | CHANGED | Create pool(s) | overcloud-controller-0 | item={'application': 'rbd', 'name': 'fast_pool', 'rule_name': 'SSD'} 2022-05-13 13:34:43,508 p=515717 u=stack n=ansible | 2022-05-13 13:34:43.507236 | 48d539a1-1679-ae16-8bd7-0000000000b9 | TASK | Get the RBD ceph_cli 2022-05-13 13:34:43,584 p=515717 u=stack n=ansible | 2022-05-13 13:34:43.583070 | f3413a45-8e8f-4ee9-b3ea-169571e37a1d | INCLUDED | /usr/share/ansible/roles/tripleo_cephadm/tasks/ceph_cli.yaml | overcloud-controller-0 2022-05-13 13:34:43,618 p=515717 u=stack n=ansible | 2022-05-13 13:34:43.617349 | 48d539a1-1679-ae16-8bd7-00000000025b | TASK | Set ceph CLI 2022-05-13 13:34:43,696 p=515717 u=stack n=ansible | 2022-05-13 13:34:43.694358 | 48d539a1-1679-ae16-8bd7-00000000025b | OK | Set ceph CLI | overcloud-controller-0 2022-05-13 13:34:43,749 p=515717 u=stack n=ansible | 2022-05-13 13:34:43.748183 | 48d539a1-1679-ae16-8bd7-0000000000ba | TASK | Set trash interval 2022-05-13 13:34:44,573 p=515717 u=stack n=ansible | 2022-05-13 13:34:44.571302 | 48d539a1-1679-ae16-8bd7-0000000000ba | OK | Set trash interval | overcloud-controller-0 | item=volumes 2022-05-13 13:34:44,648 p=515717 u=stack n=ansible | 2022-05-13 13:34:44.645801 | 48d539a1-1679-ae16-8bd7-0000000000c8 | TASK | Collect the host and build the resulting host list 2022-05-13 13:34:44,739 p=515717 u=stack n=ansible | 2022-05-13 13:34:44.737659 | 48d539a1-1679-ae16-8bd7-0000000000c8 | SKIPPED | Collect the host and build the resulting host list | overcloud-controller-0 | item=overcloud-controller-0 2022-05-13 13:34:44,743 p=515717 u=stack n=ansible | 2022-05-13 13:34:44.741474 | 48d539a1-1679-ae16-8bd7-0000000000c8 | SKIPPED | Collect the host and build the resulting host list | overcloud-controller-0 | item=overcloud-controller-1 2022-05-13 13:34:44,788 p=515717 u=stack n=ansible | 2022-05-13 13:34:44.787814 | 48d539a1-1679-ae16-8bd7-0000000000c8 | SKIPPED | Collect the host and build the resulting host list | overcloud-controller-0 | item=overcloud-controller-2 2022-05-13 13:34:44,825 p=515717 u=stack n=ansible | 2022-05-13 13:34:44.824547 | 48d539a1-1679-ae16-8bd7-0000000000c9 | TASK | Collect the host and build the resulting host list 2022-05-13 13:34:44,937 p=515717 u=stack n=ansible | 2022-05-13 13:34:44.935334 | 48d539a1-1679-ae16-8bd7-0000000000c9 | OK | Collect the host and build the resulting host list | overcloud-controller-0 | item=overcloud-controller-0 2022-05-13 13:34:44,994 p=515717 u=stack n=ansible | 2022-05-13 13:34:44.993206 | 48d539a1-1679-ae16-8bd7-0000000000c9 | OK | Collect the host and build the resulting host list | overcloud-controller-0 | item=overcloud-controller-1 2022-05-13 13:34:45,040 p=515717 u=stack n=ansible | 2022-05-13 13:34:45.039453 | 48d539a1-1679-ae16-8bd7-0000000000c9 | OK | Collect the host and build the resulting host list | overcloud-controller-0 | item=overcloud-controller-2 2022-05-13 13:34:45,073 p=515717 u=stack n=ansible | 2022-05-13 13:34:45.072863 | 48d539a1-1679-ae16-8bd7-0000000000ca | TASK | Get certificate from remote node if needed 2022-05-13 13:34:45,129 p=515717 u=stack n=ansible | 2022-05-13 13:34:45.127894 | 48d539a1-1679-ae16-8bd7-0000000000ca | SKIPPED | Get certificate from remote node if needed | overcloud-controller-0 2022-05-13 13:34:45,159 p=515717 u=stack n=ansible | 2022-05-13 13:34:45.158519 | 48d539a1-1679-ae16-8bd7-0000000000cb | TASK | Create the RGW Daemon spec definition 2022-05-13 13:34:46,245 p=515717 u=stack n=ansible | An exception occurred during task execution. To see the full traceback, use -vvv. The error was: ImportError: cannot import name 'ceph_spec' 2022-05-13 13:34:46,246 p=515717 u=stack n=ansible | 2022-05-13 13:34:46.245781 | 48d539a1-1679-ae16-8bd7-0000000000cb | FATAL | Create the RGW Daemon spec definition | overcloud-controller-0 | error={"changed": false, "module_stderr": "Traceback (most recent call last):\n File \"\", line 102, in \n File \"\", line 94, in _ansiballz_main\n File \"\", line 40, in invoke_module\n File \"/usr/lib64/python3.6/runpy.py\", line 205, in run_module\n return _run_module_code(code, init_globals, run_name, mod_spec)\n File \"/usr/lib64/python3.6/runpy.py\", line 96, in _run_module_code\n mod_name, mod_spec, pkg_name, script_name)\n File \"/usr/lib64/python3.6/runpy.py\", line 85, in _run_code\n exec(code, run_globals)\n File \"/tmp/ansible_ceph_mkspec_payload_6kta0bhr/ansible_ceph_mkspec_payload.zip/ansible/modules/ceph_mkspec.py\", line 24, in \nImportError: cannot import name 'ceph_spec'\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1} 2022-05-13 13:34:46,253 p=515717 u=stack n=ansible | PLAY RECAP ********************************************************************* 2022-05-13 13:34:46,255 p=515717 u=stack n=ansible | overcloud-controller-0 : ok=21 changed=4 unreachable=0 failed=1 skipped=27 rescued=0 ignored=0 2022-05-13 13:34:46,257 p=515717 u=stack n=ansible | 2022-05-13 13:34:46.256837 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Summary Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2022-05-13 13:34:46,258 p=515717 u=stack n=ansible | 2022-05-13 13:34:46.257728 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Total Tasks: 49 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2022-05-13 13:34:46,259 p=515717 u=stack n=ansible | 2022-05-13 13:34:46.258602 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Elapsed Time: 0:00:50.477104 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2022-05-13 13:34:46,259 p=515717 u=stack n=ansible | 2022-05-13 13:34:46.259307 | UUID | Info | Host | Task Name | Run Time 2022-05-13 13:34:46,260 p=515717 u=stack n=ansible | 2022-05-13 13:34:46.260065 | 48d539a1-1679-ae16-8bd7-0000000000b7 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Create pool(s) | 23.06s 2022-05-13 13:34:46,261 p=515717 u=stack n=ansible | 2022-05-13 13:34:46.260844 | 48d539a1-1679-ae16-8bd7-0000000000a6 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : insert new default crush rule into daemon to prevent restart | 3.83s 2022-05-13 13:34:46,262 p=515717 u=stack n=ansible | 2022-05-13 13:34:46.261704 | 48d539a1-1679-ae16-8bd7-0000000000a3 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : create configured crush rules | 2.85s 2022-05-13 13:34:46,263 p=515717 u=stack n=ansible | 2022-05-13 13:34:46.262459 | 48d539a1-1679-ae16-8bd7-00000000001b | SUMMARY | overcloud-controller-0 | tripleo_cephadm : List Ceph daemon instances on this host | 1.36s 2022-05-13 13:34:46,264 p=515717 u=stack n=ansible | 2022-05-13 13:34:46.263214 | 48d539a1-1679-ae16-8bd7-000000000026 | SUMMARY | overcloud-controller-0 | push tripleo_cephadm_spec to bootstrap node if spec file exsits | 1.30s 2022-05-13 13:34:46,265 p=515717 u=stack n=ansible | 2022-05-13 13:34:46.264909 | 48d539a1-1679-ae16-8bd7-0000000000a4 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : get id for new default crush rule | 1.19s 2022-05-13 13:34:46,266 p=515717 u=stack n=ansible | 2022-05-13 13:34:46.265716 | 48d539a1-1679-ae16-8bd7-0000000000cb | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Create the RGW Daemon spec definition | 1.09s 2022-05-13 13:34:46,267 p=515717 u=stack n=ansible | 2022-05-13 13:34:46.266456 | 48d539a1-1679-ae16-8bd7-000000000076 | SUMMARY | overcloud-controller-0 | Assimilate configuration from tripleo_cephadm_assimilate_conf | 0.98s 2022-05-13 13:34:46,267 p=515717 u=stack n=ansible | 2022-05-13 13:34:46.267216 | 48d539a1-1679-ae16-8bd7-000000000019 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Stat cephadm file | 0.85s 2022-05-13 13:34:46,268 p=515717 u=stack n=ansible | 2022-05-13 13:34:46.267953 | 48d539a1-1679-ae16-8bd7-0000000000ba | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Set trash interval | 0.83s 2022-05-13 13:34:46,269 p=515717 u=stack n=ansible | 2022-05-13 13:34:46.268687 | 48d539a1-1679-ae16-8bd7-000000000023 | SUMMARY | overcloud-controller-0 | Ensure tripleo_cephadm_config_home (e.g. /etc/ceph) exists | 0.78s 2022-05-13 13:34:46,269 p=515717 u=stack n=ansible | 2022-05-13 13:34:46.269409 | 48d539a1-1679-ae16-8bd7-00000000002b | SUMMARY | overcloud-controller-0 | generate tripleo_cephadm_assimilate_conf on bootstrap node | 0.73s 2022-05-13 13:34:46,271 p=515717 u=stack n=ansible | 2022-05-13 13:34:46.270133 | 48d539a1-1679-ae16-8bd7-000000000020 | SUMMARY | overcloud-controller-0 | Set random tripleo_cephadm_fsid if no running ceph containers were found | 0.58s 2022-05-13 13:34:46,271 p=515717 u=stack n=ansible | 2022-05-13 13:34:46.271293 | 48d539a1-1679-ae16-8bd7-000000000085 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Fail if ceph orchestrator is not available | 0.40s 2022-05-13 13:34:46,272 p=515717 u=stack n=ansible | 2022-05-13 13:34:46.272125 | 48d539a1-1679-ae16-8bd7-000000000089 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Wait for the expected number of osds to be running | 0.38s 2022-05-13 13:34:46,273 p=515717 u=stack n=ansible | 2022-05-13 13:34:46.272833 | 48d539a1-1679-ae16-8bd7-000000000084 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Get the ceph orchestrator status | 0.36s 2022-05-13 13:34:46,274 p=515717 u=stack n=ansible | 2022-05-13 13:34:46.273571 | 48d539a1-1679-ae16-8bd7-000000000024 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Ensure specs directory exists | 0.34s 2022-05-13 13:34:46,275 p=515717 u=stack n=ansible | 2022-05-13 13:34:46.274320 | 48d539a1-1679-ae16-8bd7-000000000086 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Set container images used by the applied services | 0.30s 2022-05-13 13:34:46,275 p=515717 u=stack n=ansible | 2022-05-13 13:34:46.275401 | 48d539a1-1679-ae16-8bd7-000000000082 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Fail if spec file is missing | 0.30s 2022-05-13 13:34:46,276 p=515717 u=stack n=ansible | 2022-05-13 13:34:46.276139 | 48d539a1-1679-ae16-8bd7-00000000001d | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Set list of found FSIDs | 0.30s 2022-05-13 13:34:46,277 p=515717 u=stack n=ansible | 2022-05-13 13:34:46.276886 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ End Summary Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2022-05-13 13:34:46,278 p=515717 u=stack n=ansible | 2022-05-13 13:34:46.277719 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ State Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2022-05-13 13:34:46,279 p=515717 u=stack n=ansible | 2022-05-13 13:34:46.278537 | ~~~~~~~~~~~~~~~~~~ Number of nodes which did not deploy successfully: 1 ~~~~~~~~~~~~~~~~~ 2022-05-13 13:34:46,279 p=515717 u=stack n=ansible | 2022-05-13 13:34:46.279248 | The following node(s) had failures: overcloud-controller-0 2022-05-13 13:34:46,280 p=515717 u=stack n=ansible | 2022-05-13 13:34:46.279974 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Running /home/stack/overcloud-deploy/overcloud/config-download/overcloud/cephadm/cephadm_command.sh 2022-05-13 13:59:06,075 p=583630 u=stack n=ansible | PLAY [Deploy Ceph with cephadm] ************************************************ 2022-05-13 13:59:06,143 p=583630 u=stack n=ansible | 2022-05-13 13:59:06.142758 | 48d539a1-1679-97b4-7607-000000000197 | TASK | Gathering Facts 2022-05-13 13:59:07,804 p=583630 u=stack n=ansible | 2022-05-13 13:59:07.804543 | 48d539a1-1679-97b4-7607-000000000197 | OK | Gathering Facts | overcloud-controller-0 2022-05-13 13:59:07,844 p=583630 u=stack n=ansible | 2022-05-13 13:59:07.843923 | 48d539a1-1679-97b4-7607-000000000018 | TASK | Install cephadm package 2022-05-13 13:59:08,016 p=583630 u=stack n=ansible | 2022-05-13 13:59:08.016370 | 48d539a1-1679-97b4-7607-000000000018 | SKIPPED | Install cephadm package | overcloud-controller-0 2022-05-13 13:59:08,049 p=583630 u=stack n=ansible | 2022-05-13 13:59:08.049200 | 48d539a1-1679-97b4-7607-000000000019 | TASK | Stat cephadm file 2022-05-13 13:59:08,759 p=583630 u=stack n=ansible | 2022-05-13 13:59:08.759231 | 48d539a1-1679-97b4-7607-000000000019 | OK | Stat cephadm file | overcloud-controller-0 2022-05-13 13:59:08,789 p=583630 u=stack n=ansible | 2022-05-13 13:59:08.788799 | 48d539a1-1679-97b4-7607-00000000001a | TASK | Fail if cephadm is not available 2022-05-13 13:59:08,934 p=583630 u=stack n=ansible | 2022-05-13 13:59:08.933757 | 48d539a1-1679-97b4-7607-00000000001a | SKIPPED | Fail if cephadm is not available | overcloud-controller-0 2022-05-13 13:59:08,976 p=583630 u=stack n=ansible | 2022-05-13 13:59:08.976573 | 48d539a1-1679-97b4-7607-00000000001b | TASK | List Ceph daemon instances on this host 2022-05-13 13:59:10,132 p=583630 u=stack n=ansible | 2022-05-13 13:59:10.132325 | 48d539a1-1679-97b4-7607-00000000001b | CHANGED | List Ceph daemon instances on this host | overcloud-controller-0 2022-05-13 13:59:10,161 p=583630 u=stack n=ansible | 2022-05-13 13:59:10.161506 | 48d539a1-1679-97b4-7607-00000000001d | TASK | Set list of found FSIDs 2022-05-13 13:59:10,355 p=583630 u=stack n=ansible | 2022-05-13 13:59:10.355453 | 48d539a1-1679-97b4-7607-00000000001d | SKIPPED | Set list of found FSIDs | overcloud-controller-0 2022-05-13 13:59:10,376 p=583630 u=stack n=ansible | 2022-05-13 13:59:10.376371 | 48d539a1-1679-97b4-7607-00000000001e | TASK | Fail if >1 FSID was discovered 2022-05-13 13:59:10,534 p=583630 u=stack n=ansible | 2022-05-13 13:59:10.533998 | 48d539a1-1679-97b4-7607-00000000001e | SKIPPED | Fail if >1 FSID was discovered | overcloud-controller-0 2022-05-13 13:59:10,564 p=583630 u=stack n=ansible | 2022-05-13 13:59:10.563920 | 48d539a1-1679-97b4-7607-00000000001f | TASK | Set FSID to the discovered value 2022-05-13 13:59:10,747 p=583630 u=stack n=ansible | 2022-05-13 13:59:10.746826 | 48d539a1-1679-97b4-7607-00000000001f | SKIPPED | Set FSID to the discovered value | overcloud-controller-0 2022-05-13 13:59:10,773 p=583630 u=stack n=ansible | 2022-05-13 13:59:10.772985 | 48d539a1-1679-97b4-7607-000000000020 | TASK | Set random tripleo_cephadm_fsid if no running ceph containers were found 2022-05-13 13:59:10,920 p=583630 u=stack n=ansible | 2022-05-13 13:59:10.920096 | 48d539a1-1679-97b4-7607-000000000020 | SKIPPED | Set random tripleo_cephadm_fsid if no running ceph containers were found | overcloud-controller-0 2022-05-13 13:59:10,956 p=583630 u=stack n=ansible | 2022-05-13 13:59:10.955991 | 48d539a1-1679-97b4-7607-000000000022 | TASK | Set first monitor IP if it was not passed 2022-05-13 13:59:11,096 p=583630 u=stack n=ansible | 2022-05-13 13:59:11.095984 | 48d539a1-1679-97b4-7607-000000000022 | SKIPPED | Set first monitor IP if it was not passed | overcloud-controller-0 2022-05-13 13:59:11,131 p=583630 u=stack n=ansible | 2022-05-13 13:59:11.130958 | 48d539a1-1679-97b4-7607-000000000023 | TASK | Ensure tripleo_cephadm_config_home (e.g. /etc/ceph) exists 2022-05-13 13:59:11,864 p=583630 u=stack n=ansible | 2022-05-13 13:59:11.864154 | 48d539a1-1679-97b4-7607-000000000023 | OK | Ensure tripleo_cephadm_config_home (e.g. /etc/ceph) exists | overcloud-controller-0 2022-05-13 13:59:11,900 p=583630 u=stack n=ansible | 2022-05-13 13:59:11.900351 | 48d539a1-1679-97b4-7607-000000000024 | TASK | Ensure specs directory exists 2022-05-13 13:59:12,296 p=583630 u=stack n=ansible | 2022-05-13 13:59:12.295727 | 48d539a1-1679-97b4-7607-000000000024 | OK | Ensure specs directory exists | overcloud-controller-0 2022-05-13 13:59:12,336 p=583630 u=stack n=ansible | 2022-05-13 13:59:12.335906 | 48d539a1-1679-97b4-7607-000000000025 | TASK | Stat spec file on ansible host 2022-05-13 13:59:13,250 p=583630 u=stack n=ansible | 2022-05-13 13:59:13.249831 | 48d539a1-1679-97b4-7607-000000000025 | OK | Stat spec file on ansible host | overcloud-controller-0 -> localhost 2022-05-13 13:59:13,252 p=583630 u=stack n=ansible | [WARNING]: ('overcloud-controller-0 -> localhost', '48d539a1-1679-97b4-7607-000000000025') missing from stats 2022-05-13 13:59:13,288 p=583630 u=stack n=ansible | 2022-05-13 13:59:13.288535 | 48d539a1-1679-97b4-7607-000000000026 | TASK | push tripleo_cephadm_spec to bootstrap node if spec file exsits 2022-05-13 13:59:14,165 p=583630 u=stack n=ansible | 2022-05-13 13:59:14.165134 | 48d539a1-1679-97b4-7607-000000000026 | OK | push tripleo_cephadm_spec to bootstrap node if spec file exsits | overcloud-controller-0 2022-05-13 13:59:14,207 p=583630 u=stack n=ansible | 2022-05-13 13:59:14.207233 | 48d539a1-1679-97b4-7607-000000000028 | TASK | stat tripleo_cephadm_bootstrap_conf file 2022-05-13 13:59:14,358 p=583630 u=stack n=ansible | 2022-05-13 13:59:14.357794 | 48d539a1-1679-97b4-7607-000000000028 | SKIPPED | stat tripleo_cephadm_bootstrap_conf file | overcloud-controller-0 2022-05-13 13:59:14,397 p=583630 u=stack n=ansible | 2022-05-13 13:59:14.397344 | 48d539a1-1679-97b4-7607-000000000029 | TASK | push it to tripleo_cephadm_assimilate_conf on bootstrap node 2022-05-13 13:59:14,550 p=583630 u=stack n=ansible | 2022-05-13 13:59:14.549913 | 48d539a1-1679-97b4-7607-000000000029 | SKIPPED | push it to tripleo_cephadm_assimilate_conf on bootstrap node | overcloud-controller-0 2022-05-13 13:59:14,583 p=583630 u=stack n=ansible | 2022-05-13 13:59:14.583289 | 48d539a1-1679-97b4-7607-00000000002b | TASK | generate tripleo_cephadm_assimilate_conf on bootstrap node 2022-05-13 13:59:16,816 p=583630 u=stack n=ansible | 2022-05-13 13:59:16.816125 | 48d539a1-1679-97b4-7607-00000000002b | OK | generate tripleo_cephadm_assimilate_conf on bootstrap node | overcloud-controller-0 2022-05-13 13:59:16,852 p=583630 u=stack n=ansible | 2022-05-13 13:59:16.852601 | 48d539a1-1679-97b4-7607-000000000056 | TASK | Add spec to necessary file list when using spec to bootstrap 2022-05-13 13:59:17,005 p=583630 u=stack n=ansible | 2022-05-13 13:59:17.004780 | 48d539a1-1679-97b4-7607-000000000056 | SKIPPED | Add spec to necessary file list when using spec to bootstrap | overcloud-controller-0 2022-05-13 13:59:17,041 p=583630 u=stack n=ansible | 2022-05-13 13:59:17.040899 | 48d539a1-1679-97b4-7607-000000000057 | TASK | Stat necessary files to bootstrap with cephadm 2022-05-13 13:59:17,204 p=583630 u=stack n=ansible | 2022-05-13 13:59:17.203551 | 48d539a1-1679-97b4-7607-000000000057 | SKIPPED | Stat necessary files to bootstrap with cephadm | overcloud-controller-0 | item=/home/ceph-admin/.ssh/id_rsa 2022-05-13 13:59:17,205 p=583630 u=stack n=ansible | 2022-05-13 13:59:17.205201 | 48d539a1-1679-97b4-7607-000000000057 | SKIPPED | Stat necessary files to bootstrap with cephadm | overcloud-controller-0 | item=/home/ceph-admin/.ssh/id_rsa.pub 2022-05-13 13:59:17,242 p=583630 u=stack n=ansible | 2022-05-13 13:59:17.241811 | 48d539a1-1679-97b4-7607-000000000058 | TASK | Fail if necessary files are missing 2022-05-13 13:59:17,426 p=583630 u=stack n=ansible | 2022-05-13 13:59:17.425988 | 48d539a1-1679-97b4-7607-000000000058 | SKIPPED | Fail if necessary files are missing | overcloud-controller-0 | item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': '/home/ceph-admin/.ssh/id_rsa', 'ansible_loop_var': 'item'} 2022-05-13 13:59:17,457 p=583630 u=stack n=ansible | 2022-05-13 13:59:17.457607 | 48d539a1-1679-97b4-7607-000000000058 | SKIPPED | Fail if necessary files are missing | overcloud-controller-0 | item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': '/home/ceph-admin/.ssh/id_rsa.pub', 'ansible_loop_var': 'item'} 2022-05-13 13:59:17,513 p=583630 u=stack n=ansible | 2022-05-13 13:59:17.512992 | 48d539a1-1679-97b4-7607-000000000059 | TASK | Stat pre ceph conf file in case we should bootrap with it 2022-05-13 13:59:17,671 p=583630 u=stack n=ansible | 2022-05-13 13:59:17.671112 | 48d539a1-1679-97b4-7607-000000000059 | SKIPPED | Stat pre ceph conf file in case we should bootrap with it | overcloud-controller-0 2022-05-13 13:59:17,695 p=583630 u=stack n=ansible | 2022-05-13 13:59:17.694763 | 48d539a1-1679-97b4-7607-00000000005b | TASK | Run cephadm bootstrap 2022-05-13 13:59:17,850 p=583630 u=stack n=ansible | 2022-05-13 13:59:17.850211 | 48d539a1-1679-97b4-7607-00000000005b | SKIPPED | Run cephadm bootstrap | overcloud-controller-0 2022-05-13 13:59:17,872 p=583630 u=stack n=ansible | 2022-05-13 13:59:17.872117 | 48d539a1-1679-97b4-7607-00000000005c | TASK | Show results of bootstrap 2022-05-13 13:59:18,052 p=583630 u=stack n=ansible | 2022-05-13 13:59:18.052405 | 48d539a1-1679-97b4-7607-00000000005c | SKIPPED | Show results of bootstrap | overcloud-controller-0 2022-05-13 13:59:18,097 p=583630 u=stack n=ansible | 2022-05-13 13:59:18.097179 | 48d539a1-1679-97b4-7607-00000000005e | TASK | If cephadm bootstrap was not run report the reason 2022-05-13 13:59:18,246 p=583630 u=stack n=ansible | 2022-05-13 13:59:18.245823 | 48d539a1-1679-97b4-7607-00000000005e | SKIPPED | If cephadm bootstrap was not run report the reason | overcloud-controller-0 2022-05-13 13:59:18,276 p=583630 u=stack n=ansible | 2022-05-13 13:59:18.276077 | 48d539a1-1679-97b4-7607-000000000075 | TASK | Ensure tripleo_cephadm_ceph_cli is set 2022-05-13 13:59:18,503 p=583630 u=stack n=ansible | 2022-05-13 13:59:18.503493 | 23f677ae-1114-4cf0-9b08-3aea0099d52d | INCLUDED | /usr/share/ansible/roles/tripleo_cephadm/tasks/ceph_cli.yaml | overcloud-controller-0 2022-05-13 13:59:18,652 p=583630 u=stack n=ansible | 2022-05-13 13:59:18.651830 | 48d539a1-1679-97b4-7607-0000000001f3 | TASK | Set ceph CLI 2022-05-13 13:59:18,842 p=583630 u=stack n=ansible | 2022-05-13 13:59:18.842064 | 48d539a1-1679-97b4-7607-0000000001f3 | OK | Set ceph CLI | overcloud-controller-0 2022-05-13 13:59:18,884 p=583630 u=stack n=ansible | 2022-05-13 13:59:18.884520 | 48d539a1-1679-97b4-7607-000000000076 | TASK | Assimilate configuration from tripleo_cephadm_assimilate_conf 2022-05-13 13:59:19,898 p=583630 u=stack n=ansible | 2022-05-13 13:59:19.897774 | 48d539a1-1679-97b4-7607-000000000076 | OK | Assimilate configuration from tripleo_cephadm_assimilate_conf | overcloud-controller-0 2022-05-13 13:59:19,928 p=583630 u=stack n=ansible | 2022-05-13 13:59:19.928173 | 48d539a1-1679-97b4-7607-000000000081 | TASK | Stat spec file on bootstrap node 2022-05-13 13:59:20,122 p=583630 u=stack n=ansible | 2022-05-13 13:59:20.122495 | 48d539a1-1679-97b4-7607-000000000081 | SKIPPED | Stat spec file on bootstrap node | overcloud-controller-0 | item=/home/ceph-admin/specs/ceph_spec.yaml 2022-05-13 13:59:20,152 p=583630 u=stack n=ansible | 2022-05-13 13:59:20.152731 | 48d539a1-1679-97b4-7607-000000000082 | TASK | Fail if spec file is missing 2022-05-13 13:59:20,319 p=583630 u=stack n=ansible | 2022-05-13 13:59:20.318773 | 48d539a1-1679-97b4-7607-000000000082 | SKIPPED | Fail if spec file is missing | overcloud-controller-0 | item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': '/home/ceph-admin/specs/ceph_spec.yaml', 'ansible_loop_var': 'item'} 2022-05-13 13:59:20,358 p=583630 u=stack n=ansible | 2022-05-13 13:59:20.357755 | 48d539a1-1679-97b4-7607-000000000083 | TASK | Get ceph_cli 2022-05-13 13:59:20,498 p=583630 u=stack n=ansible | 2022-05-13 13:59:20.498199 | 48d539a1-1679-97b4-7607-000000000083 | SKIPPED | Get ceph_cli | overcloud-controller-0 2022-05-13 13:59:20,547 p=583630 u=stack n=ansible | 2022-05-13 13:59:20.546790 | 48d539a1-1679-97b4-7607-000000000084 | TASK | Get the ceph orchestrator status 2022-05-13 13:59:20,707 p=583630 u=stack n=ansible | 2022-05-13 13:59:20.707579 | 48d539a1-1679-97b4-7607-000000000084 | SKIPPED | Get the ceph orchestrator status | overcloud-controller-0 2022-05-13 13:59:20,735 p=583630 u=stack n=ansible | 2022-05-13 13:59:20.734800 | 48d539a1-1679-97b4-7607-000000000085 | TASK | Fail if ceph orchestrator is not available 2022-05-13 13:59:20,892 p=583630 u=stack n=ansible | 2022-05-13 13:59:20.891834 | 48d539a1-1679-97b4-7607-000000000085 | SKIPPED | Fail if ceph orchestrator is not available | overcloud-controller-0 2022-05-13 13:59:20,920 p=583630 u=stack n=ansible | 2022-05-13 13:59:20.920152 | 48d539a1-1679-97b4-7607-000000000086 | TASK | Set container images used by the applied services 2022-05-13 13:59:21,060 p=583630 u=stack n=ansible | 2022-05-13 13:59:21.060295 | 48d539a1-1679-97b4-7607-000000000086 | SKIPPED | Set container images used by the applied services | overcloud-controller-0 2022-05-13 13:59:21,082 p=583630 u=stack n=ansible | 2022-05-13 13:59:21.081633 | 48d539a1-1679-97b4-7607-000000000087 | TASK | Apply spec 2022-05-13 13:59:21,268 p=583630 u=stack n=ansible | 2022-05-13 13:59:21.268046 | 48d539a1-1679-97b4-7607-000000000087 | SKIPPED | Apply spec | overcloud-controller-0 2022-05-13 13:59:21,289 p=583630 u=stack n=ansible | 2022-05-13 13:59:21.289430 | 48d539a1-1679-97b4-7607-000000000088 | TASK | Show results of spec apply 2022-05-13 13:59:21,444 p=583630 u=stack n=ansible | 2022-05-13 13:59:21.443642 | 48d539a1-1679-97b4-7607-000000000088 | SKIPPED | Show results of spec apply | overcloud-controller-0 2022-05-13 13:59:21,465 p=583630 u=stack n=ansible | 2022-05-13 13:59:21.464720 | 48d539a1-1679-97b4-7607-000000000089 | TASK | Wait for the expected number of osds to be running 2022-05-13 13:59:21,624 p=583630 u=stack n=ansible | 2022-05-13 13:59:21.624197 | 48d539a1-1679-97b4-7607-000000000089 | SKIPPED | Wait for the expected number of osds to be running | overcloud-controller-0 2022-05-13 13:59:21,659 p=583630 u=stack n=ansible | 2022-05-13 13:59:21.659427 | 48d539a1-1679-97b4-7607-0000000000a2 | TASK | Get ceph_cli 2022-05-13 13:59:21,845 p=583630 u=stack n=ansible | 2022-05-13 13:59:21.845617 | 75e007c3-d44a-4d1c-9aec-0b9c467257d4 | INCLUDED | /usr/share/ansible/roles/tripleo_cephadm/tasks/ceph_cli.yaml | overcloud-controller-0 2022-05-13 13:59:21,874 p=583630 u=stack n=ansible | 2022-05-13 13:59:21.874064 | 48d539a1-1679-97b4-7607-000000000220 | TASK | Set ceph CLI 2022-05-13 13:59:22,055 p=583630 u=stack n=ansible | 2022-05-13 13:59:22.054718 | 48d539a1-1679-97b4-7607-000000000220 | OK | Set ceph CLI | overcloud-controller-0 2022-05-13 13:59:22,089 p=583630 u=stack n=ansible | 2022-05-13 13:59:22.089171 | 48d539a1-1679-97b4-7607-0000000000a3 | TASK | create configured crush rules 2022-05-13 13:59:23,646 p=583630 u=stack n=ansible | 2022-05-13 13:59:23.646332 | 48d539a1-1679-97b4-7607-0000000000a3 | OK | create configured crush rules | overcloud-controller-0 | item={'class': 'hdd', 'default': True, 'name': 'HDD', 'root': 'default', 'type': 'host'} 2022-05-13 13:59:24,657 p=583630 u=stack n=ansible | 2022-05-13 13:59:24.657474 | 48d539a1-1679-97b4-7607-0000000000a3 | OK | create configured crush rules | overcloud-controller-0 | item={'class': 'ssd', 'default': False, 'name': 'SSD', 'root': 'default', 'type': 'host'} 2022-05-13 13:59:24,688 p=583630 u=stack n=ansible | 2022-05-13 13:59:24.688084 | 48d539a1-1679-97b4-7607-0000000000a4 | TASK | get id for new default crush rule 2022-05-13 13:59:26,043 p=583630 u=stack n=ansible | 2022-05-13 13:59:26.043324 | 48d539a1-1679-97b4-7607-0000000000a4 | OK | get id for new default crush rule | overcloud-controller-0 | item={'class': 'hdd', 'default': True, 'name': 'HDD', 'root': 'default', 'type': 'host'} 2022-05-13 13:59:26,071 p=583630 u=stack n=ansible | 2022-05-13 13:59:26.071149 | 48d539a1-1679-97b4-7607-0000000000a4 | SKIPPED | get id for new default crush rule | overcloud-controller-0 | item={'class': 'ssd', 'default': False, 'name': 'SSD', 'root': 'default', 'type': 'host'} 2022-05-13 13:59:26,125 p=583630 u=stack n=ansible | 2022-05-13 13:59:26.124718 | 48d539a1-1679-97b4-7607-0000000000a5 | TASK | set_fact info_ceph_default_crush_rule_yaml, ceph_osd_pool_default_crush_rule_name 2022-05-13 13:59:26,447 p=583630 u=stack n=ansible | 2022-05-13 13:59:26.447257 | 48d539a1-1679-97b4-7607-0000000000a5 | OK | set_fact info_ceph_default_crush_rule_yaml, ceph_osd_pool_default_crush_rule_name | overcloud-controller-0 | item={'cmd': ['podman', 'run', '--rm', '--net=host', '-v', '/etc/ceph:/etc/ceph:z', '-v', '/var/lib/ceph/:/var/lib/ceph/:z', '-v', '/var/log/ceph/:/var/log/ceph/:z', '--entrypoint=ceph', '172.25.201.68:8787/ceph/daemon:v6.0.6-stable-6.0-pacific-centos-8-x86_64', '-n', 'client.admin', '-k', '/etc/ceph/ceph.client.admin.keyring', '--cluster', 'ceph', 'osd', 'crush', 'rule', 'dump', 'HDD', '--format=json'], 'start': '2022-05-13 05:59:25.229286', 'end': '2022-05-13 05:59:25.972809', 'delta': '0:00:00.743523', 'rc': 0, 'stdout': '\n{"rule_id":1,"rule_name":"HDD","ruleset":1,"type":1,"min_size":1,"max_size":10,"steps":[{"op":"take","item":-2,"item_name":"default~hdd"},{"op":"chooseleaf_firstn","num":0,"type":"host"},{"op":"emit"}]}', 'stderr': '', 'changed': False, 'invocation': {'module_args': {'name': 'HDD', 'cluster': 'ceph', 'state': 'info', 'rule_type': None, 'bucket_root': None, 'bucket_type': None, 'device_class': None, 'profile': None}}, 'stdout_lines': ['', '{"rule_id":1,"rule_name":"HDD","ruleset":1,"type":1,"min_size":1,"max_size":10,"steps":[{"op":"take","item":-2,"item_name":"default~hdd"},{"op":"chooseleaf_firstn","num":0,"type":"host"},{"op":"emit"}]}'], 'stderr_lines': [], 'failed': False, 'item': {'class': 'hdd', 'default': True, 'name': 'HDD', 'root': 'default', 'type': 'host'}, 'ansible_loop_var': 'item'} 2022-05-13 13:59:26,453 p=583630 u=stack n=ansible | 2022-05-13 13:59:26.453061 | 48d539a1-1679-97b4-7607-0000000000a5 | SKIPPED | set_fact info_ceph_default_crush_rule_yaml, ceph_osd_pool_default_crush_rule_name | overcloud-controller-0 | item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': {'class': 'ssd', 'default': False, 'name': 'SSD', 'root': 'default', 'type': 'host'}, 'ansible_loop_var': 'item'} 2022-05-13 13:59:27,540 p=583630 u=stack n=ansible | 2022-05-13 13:59:27.540306 | 48d539a1-1679-97b4-7607-0000000000a6 | TASK | insert new default crush rule into daemon to prevent restart 2022-05-13 13:59:28,444 p=583630 u=stack n=ansible | 2022-05-13 13:59:28.443913 | 48d539a1-1679-97b4-7607-0000000000a6 | OK | insert new default crush rule into daemon to prevent restart | overcloud-controller-0 -> 172.25.201.106 | item=overcloud-controller-0 2022-05-13 13:59:28,446 p=583630 u=stack n=ansible | [WARNING]: ('overcloud-controller-0 -> 172.25.201.106', '48d539a1-1679-97b4-7607-0000000000a6') missing from stats 2022-05-13 13:59:30,319 p=583630 u=stack n=ansible | 2022-05-13 13:59:30.319650 | 48d539a1-1679-97b4-7607-0000000000a6 | OK | insert new default crush rule into daemon to prevent restart | overcloud-controller-0 -> 172.25.201.105 | item=overcloud-controller-1 2022-05-13 13:59:30,321 p=583630 u=stack n=ansible | [WARNING]: ('overcloud-controller-0 -> 172.25.201.105', '48d539a1-1679-97b4-7607-0000000000a6') missing from stats 2022-05-13 13:59:31,406 p=583630 u=stack n=ansible | 2022-05-13 13:59:31.406359 | 48d539a1-1679-97b4-7607-0000000000a6 | OK | insert new default crush rule into daemon to prevent restart | overcloud-controller-0 -> 172.25.201.97 | item=overcloud-controller-2 2022-05-13 13:59:31,408 p=583630 u=stack n=ansible | [WARNING]: ('overcloud-controller-0 -> 172.25.201.97', '48d539a1-1679-97b4-7607-0000000000a6') missing from stats 2022-05-13 13:59:31,558 p=583630 u=stack n=ansible | 2022-05-13 13:59:31.552710 | 48d539a1-1679-97b4-7607-0000000000b7 | TASK | Create pool(s) 2022-05-13 13:59:34,301 p=583630 u=stack n=ansible | 2022-05-13 13:59:34.301500 | 48d539a1-1679-97b4-7607-0000000000b7 | OK | Create pool(s) | overcloud-controller-0 | item={'name': 'vms', 'pg_num': '16', 'rule_name': 'replicated_rule', 'application': 'rbd'} 2022-05-13 13:59:36,741 p=583630 u=stack n=ansible | 2022-05-13 13:59:36.740821 | 48d539a1-1679-97b4-7607-0000000000b7 | OK | Create pool(s) | overcloud-controller-0 | item={'name': 'volumes', 'pg_num': '16', 'rule_name': 'replicated_rule', 'application': 'rbd'} 2022-05-13 13:59:39,119 p=583630 u=stack n=ansible | 2022-05-13 13:59:39.119206 | 48d539a1-1679-97b4-7607-0000000000b7 | OK | Create pool(s) | overcloud-controller-0 | item={'name': 'images', 'pg_num': '16', 'rule_name': 'replicated_rule', 'application': 'rbd'} 2022-05-13 13:59:41,752 p=583630 u=stack n=ansible | 2022-05-13 13:59:41.751869 | 48d539a1-1679-97b4-7607-0000000000b7 | OK | Create pool(s) | overcloud-controller-0 | item={'name': 'metrics', 'pg_num': '16', 'rule_name': 'replicated_rule', 'application': 'openstack_gnocchi'} 2022-05-13 13:59:44,333 p=583630 u=stack n=ansible | 2022-05-13 13:59:44.331552 | 48d539a1-1679-97b4-7607-0000000000b7 | OK | Create pool(s) | overcloud-controller-0 | item={'name': 'backups', 'pg_num': '16', 'rule_name': 'replicated_rule', 'application': 'rbd'} 2022-05-13 13:59:46,834 p=583630 u=stack n=ansible | 2022-05-13 13:59:46.834119 | 48d539a1-1679-97b4-7607-0000000000b7 | OK | Create pool(s) | overcloud-controller-0 | item={'application': 'rbd', 'name': 'slow_pool', 'rule_name': 'HDD'} 2022-05-13 13:59:49,264 p=583630 u=stack n=ansible | 2022-05-13 13:59:49.264124 | 48d539a1-1679-97b4-7607-0000000000b7 | OK | Create pool(s) | overcloud-controller-0 | item={'application': 'rbd', 'name': 'fast_pool', 'rule_name': 'SSD'} 2022-05-13 13:59:49,320 p=583630 u=stack n=ansible | 2022-05-13 13:59:49.320022 | 48d539a1-1679-97b4-7607-0000000000b9 | TASK | Get the RBD ceph_cli 2022-05-13 13:59:49,430 p=583630 u=stack n=ansible | 2022-05-13 13:59:49.430340 | 3487fc50-a8b8-4153-9d6a-a30067154d48 | INCLUDED | /usr/share/ansible/roles/tripleo_cephadm/tasks/ceph_cli.yaml | overcloud-controller-0 2022-05-13 13:59:49,457 p=583630 u=stack n=ansible | 2022-05-13 13:59:49.457333 | 48d539a1-1679-97b4-7607-00000000025d | TASK | Set ceph CLI 2022-05-13 13:59:49,553 p=583630 u=stack n=ansible | 2022-05-13 13:59:49.553266 | 48d539a1-1679-97b4-7607-00000000025d | OK | Set ceph CLI | overcloud-controller-0 2022-05-13 13:59:49,578 p=583630 u=stack n=ansible | 2022-05-13 13:59:49.577800 | 48d539a1-1679-97b4-7607-0000000000ba | TASK | Set trash interval 2022-05-13 13:59:50,318 p=583630 u=stack n=ansible | 2022-05-13 13:59:50.317987 | 48d539a1-1679-97b4-7607-0000000000ba | OK | Set trash interval | overcloud-controller-0 | item=volumes 2022-05-13 13:59:50,364 p=583630 u=stack n=ansible | 2022-05-13 13:59:50.363843 | 48d539a1-1679-97b4-7607-0000000000c8 | TASK | Collect the host and build the resulting host list 2022-05-13 13:59:50,465 p=583630 u=stack n=ansible | 2022-05-13 13:59:50.465328 | 48d539a1-1679-97b4-7607-0000000000c8 | SKIPPED | Collect the host and build the resulting host list | overcloud-controller-0 | item=overcloud-controller-0 2022-05-13 13:59:50,489 p=583630 u=stack n=ansible | 2022-05-13 13:59:50.489347 | 48d539a1-1679-97b4-7607-0000000000c8 | SKIPPED | Collect the host and build the resulting host list | overcloud-controller-0 | item=overcloud-controller-1 2022-05-13 13:59:50,490 p=583630 u=stack n=ansible | 2022-05-13 13:59:50.490554 | 48d539a1-1679-97b4-7607-0000000000c8 | SKIPPED | Collect the host and build the resulting host list | overcloud-controller-0 | item=overcloud-controller-2 2022-05-13 13:59:50,527 p=583630 u=stack n=ansible | 2022-05-13 13:59:50.527276 | 48d539a1-1679-97b4-7607-0000000000c9 | TASK | Collect the host and build the resulting host list 2022-05-13 13:59:50,657 p=583630 u=stack n=ansible | 2022-05-13 13:59:50.657009 | 48d539a1-1679-97b4-7607-0000000000c9 | OK | Collect the host and build the resulting host list | overcloud-controller-0 | item=overcloud-controller-0 2022-05-13 13:59:50,748 p=583630 u=stack n=ansible | 2022-05-13 13:59:50.748511 | 48d539a1-1679-97b4-7607-0000000000c9 | OK | Collect the host and build the resulting host list | overcloud-controller-0 | item=overcloud-controller-1 2022-05-13 13:59:50,750 p=583630 u=stack n=ansible | 2022-05-13 13:59:50.750358 | 48d539a1-1679-97b4-7607-0000000000c9 | OK | Collect the host and build the resulting host list | overcloud-controller-0 | item=overcloud-controller-2 2022-05-13 13:59:50,792 p=583630 u=stack n=ansible | 2022-05-13 13:59:50.792060 | 48d539a1-1679-97b4-7607-0000000000ca | TASK | Get certificate from remote node if needed 2022-05-13 13:59:50,857 p=583630 u=stack n=ansible | 2022-05-13 13:59:50.856992 | 48d539a1-1679-97b4-7607-0000000000ca | SKIPPED | Get certificate from remote node if needed | overcloud-controller-0 2022-05-13 13:59:50,893 p=583630 u=stack n=ansible | 2022-05-13 13:59:50.893221 | 48d539a1-1679-97b4-7607-0000000000cb | TASK | Create the RGW Daemon spec definition 2022-05-13 13:59:51,549 p=583630 u=stack n=ansible | An exception occurred during task execution. To see the full traceback, use -vvv. The error was: ImportError: cannot import name 'ceph_spec' 2022-05-13 13:59:51,550 p=583630 u=stack n=ansible | 2022-05-13 13:59:51.550405 | 48d539a1-1679-97b4-7607-0000000000cb | FATAL | Create the RGW Daemon spec definition | overcloud-controller-0 | error={"changed": false, "module_stderr": "Traceback (most recent call last):\n File \"\", line 102, in \n File \"\", line 94, in _ansiballz_main\n File \"\", line 40, in invoke_module\n File \"/usr/lib64/python3.6/runpy.py\", line 205, in run_module\n return _run_module_code(code, init_globals, run_name, mod_spec)\n File \"/usr/lib64/python3.6/runpy.py\", line 96, in _run_module_code\n mod_name, mod_spec, pkg_name, script_name)\n File \"/usr/lib64/python3.6/runpy.py\", line 85, in _run_code\n exec(code, run_globals)\n File \"/tmp/ansible_ceph_mkspec_payload_5cc4ln2_/ansible_ceph_mkspec_payload.zip/ansible/modules/ceph_mkspec.py\", line 24, in \nImportError: cannot import name 'ceph_spec'\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1} 2022-05-13 13:59:51,554 p=583630 u=stack n=ansible | PLAY RECAP ********************************************************************* 2022-05-13 13:59:51,555 p=583630 u=stack n=ansible | overcloud-controller-0 : ok=22 changed=1 unreachable=0 failed=1 skipped=27 rescued=0 ignored=0 2022-05-13 13:59:51,556 p=583630 u=stack n=ansible | 2022-05-13 13:59:51.555886 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Summary Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2022-05-13 13:59:51,556 p=583630 u=stack n=ansible | 2022-05-13 13:59:51.556142 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Total Tasks: 50 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2022-05-13 13:59:51,556 p=583630 u=stack n=ansible | 2022-05-13 13:59:51.556328 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Elapsed Time: 0:00:45.487079 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2022-05-13 13:59:51,556 p=583630 u=stack n=ansible | 2022-05-13 13:59:51.556516 | UUID | Info | Host | Task Name | Run Time 2022-05-13 13:59:51,556 p=583630 u=stack n=ansible | 2022-05-13 13:59:51.556712 | 48d539a1-1679-97b4-7607-0000000000b7 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Create pool(s) | 17.73s 2022-05-13 13:59:51,556 p=583630 u=stack n=ansible | 2022-05-13 13:59:51.556892 | 48d539a1-1679-97b4-7607-0000000000a6 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : insert new default crush rule into daemon to prevent restart | 3.87s 2022-05-13 13:59:51,557 p=583630 u=stack n=ansible | 2022-05-13 13:59:51.557070 | 48d539a1-1679-97b4-7607-0000000000a3 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : create configured crush rules | 2.57s 2022-05-13 13:59:51,557 p=583630 u=stack n=ansible | 2022-05-13 13:59:51.557246 | 48d539a1-1679-97b4-7607-00000000002b | SUMMARY | overcloud-controller-0 | generate tripleo_cephadm_assimilate_conf on bootstrap node | 2.23s 2022-05-13 13:59:51,557 p=583630 u=stack n=ansible | 2022-05-13 13:59:51.557441 | 48d539a1-1679-97b4-7607-000000000197 | SUMMARY | overcloud-controller-0 | Gathering Facts | 1.66s 2022-05-13 13:59:51,557 p=583630 u=stack n=ansible | 2022-05-13 13:59:51.557616 | 48d539a1-1679-97b4-7607-0000000000a4 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : get id for new default crush rule | 1.39s 2022-05-13 13:59:51,557 p=583630 u=stack n=ansible | 2022-05-13 13:59:51.557795 | 48d539a1-1679-97b4-7607-00000000001b | SUMMARY | overcloud-controller-0 | tripleo_cephadm : List Ceph daemon instances on this host | 1.16s 2022-05-13 13:59:51,558 p=583630 u=stack n=ansible | 2022-05-13 13:59:51.557971 | 48d539a1-1679-97b4-7607-000000000076 | SUMMARY | overcloud-controller-0 | Assimilate configuration from tripleo_cephadm_assimilate_conf | 1.01s 2022-05-13 13:59:51,558 p=583630 u=stack n=ansible | 2022-05-13 13:59:51.558147 | 48d539a1-1679-97b4-7607-000000000026 | SUMMARY | overcloud-controller-0 | push tripleo_cephadm_spec to bootstrap node if spec file exsits | 0.88s 2022-05-13 13:59:51,558 p=583630 u=stack n=ansible | 2022-05-13 13:59:51.558332 | 48d539a1-1679-97b4-7607-0000000000ba | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Set trash interval | 0.74s 2022-05-13 13:59:51,558 p=583630 u=stack n=ansible | 2022-05-13 13:59:51.558515 | 48d539a1-1679-97b4-7607-000000000023 | SUMMARY | overcloud-controller-0 | Ensure tripleo_cephadm_config_home (e.g. /etc/ceph) exists | 0.73s 2022-05-13 13:59:51,558 p=583630 u=stack n=ansible | 2022-05-13 13:59:51.558692 | 48d539a1-1679-97b4-7607-000000000019 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Stat cephadm file | 0.71s 2022-05-13 13:59:51,558 p=583630 u=stack n=ansible | 2022-05-13 13:59:51.558860 | 48d539a1-1679-97b4-7607-0000000000cb | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Create the RGW Daemon spec definition | 0.66s 2022-05-13 13:59:51,559 p=583630 u=stack n=ansible | 2022-05-13 13:59:51.559027 | 48d539a1-1679-97b4-7607-000000000024 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Ensure specs directory exists | 0.40s 2022-05-13 13:59:51,559 p=583630 u=stack n=ansible | 2022-05-13 13:59:51.559202 | 48d539a1-1679-97b4-7607-0000000000a5 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : set_fact info_ceph_default_crush_rule_yaml, ceph_osd_pool_default_crush_rule_name | 0.34s 2022-05-13 13:59:51,559 p=583630 u=stack n=ansible | 2022-05-13 13:59:51.559376 | 48d539a1-1679-97b4-7607-0000000000c9 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Collect the host and build the resulting host list | 0.22s 2022-05-13 13:59:51,559 p=583630 u=stack n=ansible | 2022-05-13 13:59:51.559570 | 48d539a1-1679-97b4-7607-000000000058 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Fail if necessary files are missing | 0.22s 2022-05-13 13:59:51,559 p=583630 u=stack n=ansible | 2022-05-13 13:59:51.559746 | 48d539a1-1679-97b4-7607-000000000075 | SUMMARY | overcloud-controller-0 | Ensure tripleo_cephadm_ceph_cli is set | 0.20s 2022-05-13 13:59:51,559 p=583630 u=stack n=ansible | 2022-05-13 13:59:51.559912 | 48d539a1-1679-97b4-7607-000000000081 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Stat spec file on bootstrap node | 0.20s 2022-05-13 13:59:51,560 p=583630 u=stack n=ansible | 2022-05-13 13:59:51.560081 | 48d539a1-1679-97b4-7607-00000000001d | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Set list of found FSIDs | 0.19s 2022-05-13 13:59:51,560 p=583630 u=stack n=ansible | 2022-05-13 13:59:51.560263 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ End Summary Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2022-05-13 13:59:51,560 p=583630 u=stack n=ansible | 2022-05-13 13:59:51.560509 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ State Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2022-05-13 13:59:51,560 p=583630 u=stack n=ansible | 2022-05-13 13:59:51.560735 | ~~~~~~~~~~~~~~~~~~ Number of nodes which did not deploy successfully: 1 ~~~~~~~~~~~~~~~~~ 2022-05-13 13:59:51,561 p=583630 u=stack n=ansible | 2022-05-13 13:59:51.560972 | The following node(s) had failures: overcloud-controller-0 2022-05-13 13:59:51,561 p=583630 u=stack n=ansible | 2022-05-13 13:59:51.561133 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Running /home/stack/overcloud-deploy/overcloud/config-download/overcloud/cephadm/cephadm_command.sh 2022-05-13 14:17:06,126 p=636835 u=stack n=ansible | PLAY [Deploy Ceph with cephadm] ************************************************ 2022-05-13 14:17:06,193 p=636835 u=stack n=ansible | 2022-05-13 14:17:06.193021 | 48d539a1-1679-730f-272a-000000000197 | TASK | Gathering Facts 2022-05-13 14:17:07,138 p=636835 u=stack n=ansible | 2022-05-13 14:17:07.137884 | 48d539a1-1679-730f-272a-000000000197 | OK | Gathering Facts | overcloud-controller-0 2022-05-13 14:17:07,177 p=636835 u=stack n=ansible | 2022-05-13 14:17:07.176846 | 48d539a1-1679-730f-272a-000000000018 | TASK | Install cephadm package 2022-05-13 14:17:07,317 p=636835 u=stack n=ansible | 2022-05-13 14:17:07.316788 | 48d539a1-1679-730f-272a-000000000018 | SKIPPED | Install cephadm package | overcloud-controller-0 2022-05-13 14:17:07,359 p=636835 u=stack n=ansible | 2022-05-13 14:17:07.358832 | 48d539a1-1679-730f-272a-000000000019 | TASK | Stat cephadm file 2022-05-13 14:17:08,073 p=636835 u=stack n=ansible | 2022-05-13 14:17:08.072950 | 48d539a1-1679-730f-272a-000000000019 | OK | Stat cephadm file | overcloud-controller-0 2022-05-13 14:17:08,110 p=636835 u=stack n=ansible | 2022-05-13 14:17:08.110203 | 48d539a1-1679-730f-272a-00000000001a | TASK | Fail if cephadm is not available 2022-05-13 14:17:08,261 p=636835 u=stack n=ansible | 2022-05-13 14:17:08.260814 | 48d539a1-1679-730f-272a-00000000001a | SKIPPED | Fail if cephadm is not available | overcloud-controller-0 2022-05-13 14:17:08,297 p=636835 u=stack n=ansible | 2022-05-13 14:17:08.296904 | 48d539a1-1679-730f-272a-00000000001b | TASK | List Ceph daemon instances on this host 2022-05-13 14:17:09,540 p=636835 u=stack n=ansible | 2022-05-13 14:17:09.540235 | 48d539a1-1679-730f-272a-00000000001b | CHANGED | List Ceph daemon instances on this host | overcloud-controller-0 2022-05-13 14:17:09,577 p=636835 u=stack n=ansible | 2022-05-13 14:17:09.577021 | 48d539a1-1679-730f-272a-00000000001d | TASK | Set list of found FSIDs 2022-05-13 14:17:09,718 p=636835 u=stack n=ansible | 2022-05-13 14:17:09.717851 | 48d539a1-1679-730f-272a-00000000001d | SKIPPED | Set list of found FSIDs | overcloud-controller-0 2022-05-13 14:17:09,746 p=636835 u=stack n=ansible | 2022-05-13 14:17:09.745791 | 48d539a1-1679-730f-272a-00000000001e | TASK | Fail if >1 FSID was discovered 2022-05-13 14:17:09,886 p=636835 u=stack n=ansible | 2022-05-13 14:17:09.885732 | 48d539a1-1679-730f-272a-00000000001e | SKIPPED | Fail if >1 FSID was discovered | overcloud-controller-0 2022-05-13 14:17:09,910 p=636835 u=stack n=ansible | 2022-05-13 14:17:09.910125 | 48d539a1-1679-730f-272a-00000000001f | TASK | Set FSID to the discovered value 2022-05-13 14:17:10,073 p=636835 u=stack n=ansible | 2022-05-13 14:17:10.072824 | 48d539a1-1679-730f-272a-00000000001f | SKIPPED | Set FSID to the discovered value | overcloud-controller-0 2022-05-13 14:17:10,108 p=636835 u=stack n=ansible | 2022-05-13 14:17:10.108376 | 48d539a1-1679-730f-272a-000000000020 | TASK | Set random tripleo_cephadm_fsid if no running ceph containers were found 2022-05-13 14:17:10,315 p=636835 u=stack n=ansible | 2022-05-13 14:17:10.314635 | 48d539a1-1679-730f-272a-000000000020 | SKIPPED | Set random tripleo_cephadm_fsid if no running ceph containers were found | overcloud-controller-0 2022-05-13 14:17:10,349 p=636835 u=stack n=ansible | 2022-05-13 14:17:10.349258 | 48d539a1-1679-730f-272a-000000000022 | TASK | Set first monitor IP if it was not passed 2022-05-13 14:17:10,483 p=636835 u=stack n=ansible | 2022-05-13 14:17:10.482652 | 48d539a1-1679-730f-272a-000000000022 | SKIPPED | Set first monitor IP if it was not passed | overcloud-controller-0 2022-05-13 14:17:10,574 p=636835 u=stack n=ansible | 2022-05-13 14:17:10.574277 | 48d539a1-1679-730f-272a-000000000023 | TASK | Ensure tripleo_cephadm_config_home (e.g. /etc/ceph) exists 2022-05-13 14:17:11,296 p=636835 u=stack n=ansible | 2022-05-13 14:17:11.296597 | 48d539a1-1679-730f-272a-000000000023 | OK | Ensure tripleo_cephadm_config_home (e.g. /etc/ceph) exists | overcloud-controller-0 2022-05-13 14:17:11,321 p=636835 u=stack n=ansible | 2022-05-13 14:17:11.320989 | 48d539a1-1679-730f-272a-000000000024 | TASK | Ensure specs directory exists 2022-05-13 14:17:11,712 p=636835 u=stack n=ansible | 2022-05-13 14:17:11.712071 | 48d539a1-1679-730f-272a-000000000024 | OK | Ensure specs directory exists | overcloud-controller-0 2022-05-13 14:17:11,751 p=636835 u=stack n=ansible | 2022-05-13 14:17:11.751490 | 48d539a1-1679-730f-272a-000000000025 | TASK | Stat spec file on ansible host 2022-05-13 14:17:12,709 p=636835 u=stack n=ansible | 2022-05-13 14:17:12.708634 | 48d539a1-1679-730f-272a-000000000025 | OK | Stat spec file on ansible host | overcloud-controller-0 -> localhost 2022-05-13 14:17:12,711 p=636835 u=stack n=ansible | [WARNING]: ('overcloud-controller-0 -> localhost', '48d539a1-1679-730f-272a-000000000025') missing from stats 2022-05-13 14:17:12,748 p=636835 u=stack n=ansible | 2022-05-13 14:17:12.747853 | 48d539a1-1679-730f-272a-000000000026 | TASK | push tripleo_cephadm_spec to bootstrap node if spec file exsits 2022-05-13 14:17:13,625 p=636835 u=stack n=ansible | 2022-05-13 14:17:13.624781 | 48d539a1-1679-730f-272a-000000000026 | OK | push tripleo_cephadm_spec to bootstrap node if spec file exsits | overcloud-controller-0 2022-05-13 14:17:13,700 p=636835 u=stack n=ansible | 2022-05-13 14:17:13.700051 | 48d539a1-1679-730f-272a-000000000028 | TASK | stat tripleo_cephadm_bootstrap_conf file 2022-05-13 14:17:13,857 p=636835 u=stack n=ansible | 2022-05-13 14:17:13.857043 | 48d539a1-1679-730f-272a-000000000028 | SKIPPED | stat tripleo_cephadm_bootstrap_conf file | overcloud-controller-0 2022-05-13 14:17:13,900 p=636835 u=stack n=ansible | 2022-05-13 14:17:13.882239 | 48d539a1-1679-730f-272a-000000000029 | TASK | push it to tripleo_cephadm_assimilate_conf on bootstrap node 2022-05-13 14:17:14,055 p=636835 u=stack n=ansible | 2022-05-13 14:17:14.055504 | 48d539a1-1679-730f-272a-000000000029 | SKIPPED | push it to tripleo_cephadm_assimilate_conf on bootstrap node | overcloud-controller-0 2022-05-13 14:17:14,085 p=636835 u=stack n=ansible | 2022-05-13 14:17:14.085563 | 48d539a1-1679-730f-272a-00000000002b | TASK | generate tripleo_cephadm_assimilate_conf on bootstrap node 2022-05-13 14:17:16,649 p=636835 u=stack n=ansible | 2022-05-13 14:17:16.649578 | 48d539a1-1679-730f-272a-00000000002b | OK | generate tripleo_cephadm_assimilate_conf on bootstrap node | overcloud-controller-0 2022-05-13 14:17:16,679 p=636835 u=stack n=ansible | 2022-05-13 14:17:16.678580 | 48d539a1-1679-730f-272a-000000000056 | TASK | Add spec to necessary file list when using spec to bootstrap 2022-05-13 14:17:16,819 p=636835 u=stack n=ansible | 2022-05-13 14:17:16.818726 | 48d539a1-1679-730f-272a-000000000056 | SKIPPED | Add spec to necessary file list when using spec to bootstrap | overcloud-controller-0 2022-05-13 14:17:16,846 p=636835 u=stack n=ansible | 2022-05-13 14:17:16.845876 | 48d539a1-1679-730f-272a-000000000057 | TASK | Stat necessary files to bootstrap with cephadm 2022-05-13 14:17:17,011 p=636835 u=stack n=ansible | 2022-05-13 14:17:17.011173 | 48d539a1-1679-730f-272a-000000000057 | SKIPPED | Stat necessary files to bootstrap with cephadm | overcloud-controller-0 | item=/home/ceph-admin/.ssh/id_rsa 2022-05-13 14:17:17,027 p=636835 u=stack n=ansible | 2022-05-13 14:17:17.027244 | 48d539a1-1679-730f-272a-000000000057 | SKIPPED | Stat necessary files to bootstrap with cephadm | overcloud-controller-0 | item=/home/ceph-admin/.ssh/id_rsa.pub 2022-05-13 14:17:17,057 p=636835 u=stack n=ansible | 2022-05-13 14:17:17.056666 | 48d539a1-1679-730f-272a-000000000058 | TASK | Fail if necessary files are missing 2022-05-13 14:17:17,230 p=636835 u=stack n=ansible | 2022-05-13 14:17:17.230116 | 48d539a1-1679-730f-272a-000000000058 | SKIPPED | Fail if necessary files are missing | overcloud-controller-0 | item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': '/home/ceph-admin/.ssh/id_rsa', 'ansible_loop_var': 'item'} 2022-05-13 14:17:17,232 p=636835 u=stack n=ansible | 2022-05-13 14:17:17.232051 | 48d539a1-1679-730f-272a-000000000058 | SKIPPED | Fail if necessary files are missing | overcloud-controller-0 | item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': '/home/ceph-admin/.ssh/id_rsa.pub', 'ansible_loop_var': 'item'} 2022-05-13 14:17:17,308 p=636835 u=stack n=ansible | 2022-05-13 14:17:17.296297 | 48d539a1-1679-730f-272a-000000000059 | TASK | Stat pre ceph conf file in case we should bootrap with it 2022-05-13 14:17:17,456 p=636835 u=stack n=ansible | 2022-05-13 14:17:17.456418 | 48d539a1-1679-730f-272a-000000000059 | SKIPPED | Stat pre ceph conf file in case we should bootrap with it | overcloud-controller-0 2022-05-13 14:17:17,484 p=636835 u=stack n=ansible | 2022-05-13 14:17:17.484100 | 48d539a1-1679-730f-272a-00000000005b | TASK | Run cephadm bootstrap 2022-05-13 14:17:17,711 p=636835 u=stack n=ansible | 2022-05-13 14:17:17.711595 | 48d539a1-1679-730f-272a-00000000005b | SKIPPED | Run cephadm bootstrap | overcloud-controller-0 2022-05-13 14:17:17,737 p=636835 u=stack n=ansible | 2022-05-13 14:17:17.737724 | 48d539a1-1679-730f-272a-00000000005c | TASK | Show results of bootstrap 2022-05-13 14:17:17,908 p=636835 u=stack n=ansible | 2022-05-13 14:17:17.907663 | 48d539a1-1679-730f-272a-00000000005c | SKIPPED | Show results of bootstrap | overcloud-controller-0 2022-05-13 14:17:17,936 p=636835 u=stack n=ansible | 2022-05-13 14:17:17.936550 | 48d539a1-1679-730f-272a-00000000005e | TASK | If cephadm bootstrap was not run report the reason 2022-05-13 14:17:18,109 p=636835 u=stack n=ansible | 2022-05-13 14:17:18.109135 | 48d539a1-1679-730f-272a-00000000005e | SKIPPED | If cephadm bootstrap was not run report the reason | overcloud-controller-0 2022-05-13 14:17:18,148 p=636835 u=stack n=ansible | 2022-05-13 14:17:18.148005 | 48d539a1-1679-730f-272a-000000000075 | TASK | Ensure tripleo_cephadm_ceph_cli is set 2022-05-13 14:17:18,365 p=636835 u=stack n=ansible | 2022-05-13 14:17:18.365302 | e1dc05e5-27d8-4838-b81d-787edb95de0c | INCLUDED | /usr/share/ansible/roles/tripleo_cephadm/tasks/ceph_cli.yaml | overcloud-controller-0 2022-05-13 14:17:18,446 p=636835 u=stack n=ansible | 2022-05-13 14:17:18.445773 | 48d539a1-1679-730f-272a-0000000001f3 | TASK | Set ceph CLI 2022-05-13 14:17:18,982 p=636835 u=stack n=ansible | 2022-05-13 14:17:18.981890 | 48d539a1-1679-730f-272a-0000000001f3 | OK | Set ceph CLI | overcloud-controller-0 2022-05-13 14:17:19,018 p=636835 u=stack n=ansible | 2022-05-13 14:17:19.018010 | 48d539a1-1679-730f-272a-000000000076 | TASK | Assimilate configuration from tripleo_cephadm_assimilate_conf 2022-05-13 14:17:20,057 p=636835 u=stack n=ansible | 2022-05-13 14:17:20.057577 | 48d539a1-1679-730f-272a-000000000076 | OK | Assimilate configuration from tripleo_cephadm_assimilate_conf | overcloud-controller-0 2022-05-13 14:17:20,090 p=636835 u=stack n=ansible | 2022-05-13 14:17:20.090511 | 48d539a1-1679-730f-272a-000000000081 | TASK | Stat spec file on bootstrap node 2022-05-13 14:17:20,270 p=636835 u=stack n=ansible | 2022-05-13 14:17:20.270165 | 48d539a1-1679-730f-272a-000000000081 | SKIPPED | Stat spec file on bootstrap node | overcloud-controller-0 | item=/home/ceph-admin/specs/ceph_spec.yaml 2022-05-13 14:17:20,327 p=636835 u=stack n=ansible | 2022-05-13 14:17:20.327364 | 48d539a1-1679-730f-272a-000000000082 | TASK | Fail if spec file is missing 2022-05-13 14:17:20,487 p=636835 u=stack n=ansible | 2022-05-13 14:17:20.487355 | 48d539a1-1679-730f-272a-000000000082 | SKIPPED | Fail if spec file is missing | overcloud-controller-0 | item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': '/home/ceph-admin/specs/ceph_spec.yaml', 'ansible_loop_var': 'item'} 2022-05-13 14:17:20,513 p=636835 u=stack n=ansible | 2022-05-13 14:17:20.513554 | 48d539a1-1679-730f-272a-000000000083 | TASK | Get ceph_cli 2022-05-13 14:17:20,683 p=636835 u=stack n=ansible | 2022-05-13 14:17:20.683225 | 48d539a1-1679-730f-272a-000000000083 | SKIPPED | Get ceph_cli | overcloud-controller-0 2022-05-13 14:17:20,711 p=636835 u=stack n=ansible | 2022-05-13 14:17:20.710973 | 48d539a1-1679-730f-272a-000000000084 | TASK | Get the ceph orchestrator status 2022-05-13 14:17:20,871 p=636835 u=stack n=ansible | 2022-05-13 14:17:20.871429 | 48d539a1-1679-730f-272a-000000000084 | SKIPPED | Get the ceph orchestrator status | overcloud-controller-0 2022-05-13 14:17:20,893 p=636835 u=stack n=ansible | 2022-05-13 14:17:20.892849 | 48d539a1-1679-730f-272a-000000000085 | TASK | Fail if ceph orchestrator is not available 2022-05-13 14:17:21,059 p=636835 u=stack n=ansible | 2022-05-13 14:17:21.058812 | 48d539a1-1679-730f-272a-000000000085 | SKIPPED | Fail if ceph orchestrator is not available | overcloud-controller-0 2022-05-13 14:17:21,103 p=636835 u=stack n=ansible | 2022-05-13 14:17:21.102942 | 48d539a1-1679-730f-272a-000000000086 | TASK | Set container images used by the applied services 2022-05-13 14:17:21,245 p=636835 u=stack n=ansible | 2022-05-13 14:17:21.245166 | 48d539a1-1679-730f-272a-000000000086 | SKIPPED | Set container images used by the applied services | overcloud-controller-0 2022-05-13 14:17:21,285 p=636835 u=stack n=ansible | 2022-05-13 14:17:21.285026 | 48d539a1-1679-730f-272a-000000000087 | TASK | Apply spec 2022-05-13 14:17:21,431 p=636835 u=stack n=ansible | 2022-05-13 14:17:21.431333 | 48d539a1-1679-730f-272a-000000000087 | SKIPPED | Apply spec | overcloud-controller-0 2022-05-13 14:17:21,460 p=636835 u=stack n=ansible | 2022-05-13 14:17:21.459966 | 48d539a1-1679-730f-272a-000000000088 | TASK | Show results of spec apply 2022-05-13 14:17:21,642 p=636835 u=stack n=ansible | 2022-05-13 14:17:21.641811 | 48d539a1-1679-730f-272a-000000000088 | SKIPPED | Show results of spec apply | overcloud-controller-0 2022-05-13 14:17:21,685 p=636835 u=stack n=ansible | 2022-05-13 14:17:21.685407 | 48d539a1-1679-730f-272a-000000000089 | TASK | Wait for the expected number of osds to be running 2022-05-13 14:17:21,855 p=636835 u=stack n=ansible | 2022-05-13 14:17:21.855117 | 48d539a1-1679-730f-272a-000000000089 | SKIPPED | Wait for the expected number of osds to be running | overcloud-controller-0 2022-05-13 14:17:21,877 p=636835 u=stack n=ansible | 2022-05-13 14:17:21.876692 | 48d539a1-1679-730f-272a-0000000000a2 | TASK | Get ceph_cli 2022-05-13 14:17:22,067 p=636835 u=stack n=ansible | 2022-05-13 14:17:22.066339 | d915d3cf-6169-4140-b2c6-6573e6f2c082 | INCLUDED | /usr/share/ansible/roles/tripleo_cephadm/tasks/ceph_cli.yaml | overcloud-controller-0 2022-05-13 14:17:22,191 p=636835 u=stack n=ansible | 2022-05-13 14:17:22.191397 | 48d539a1-1679-730f-272a-000000000220 | TASK | Set ceph CLI 2022-05-13 14:17:22,387 p=636835 u=stack n=ansible | 2022-05-13 14:17:22.386720 | 48d539a1-1679-730f-272a-000000000220 | OK | Set ceph CLI | overcloud-controller-0 2022-05-13 14:17:22,408 p=636835 u=stack n=ansible | 2022-05-13 14:17:22.408257 | 48d539a1-1679-730f-272a-0000000000a3 | TASK | create configured crush rules 2022-05-13 14:17:24,674 p=636835 u=stack n=ansible | 2022-05-13 14:17:24.673744 | 48d539a1-1679-730f-272a-0000000000a3 | OK | create configured crush rules | overcloud-controller-0 | item={'class': 'hdd', 'default': True, 'name': 'HDD', 'root': 'default', 'type': 'host'} 2022-05-13 14:17:25,685 p=636835 u=stack n=ansible | 2022-05-13 14:17:25.685134 | 48d539a1-1679-730f-272a-0000000000a3 | OK | create configured crush rules | overcloud-controller-0 | item={'class': 'ssd', 'default': False, 'name': 'SSD', 'root': 'default', 'type': 'host'} 2022-05-13 14:17:25,728 p=636835 u=stack n=ansible | 2022-05-13 14:17:25.727942 | 48d539a1-1679-730f-272a-0000000000a4 | TASK | get id for new default crush rule 2022-05-13 14:17:27,113 p=636835 u=stack n=ansible | 2022-05-13 14:17:27.113350 | 48d539a1-1679-730f-272a-0000000000a4 | OK | get id for new default crush rule | overcloud-controller-0 | item={'class': 'hdd', 'default': True, 'name': 'HDD', 'root': 'default', 'type': 'host'} 2022-05-13 14:17:27,117 p=636835 u=stack n=ansible | 2022-05-13 14:17:27.116926 | 48d539a1-1679-730f-272a-0000000000a4 | SKIPPED | get id for new default crush rule | overcloud-controller-0 | item={'class': 'ssd', 'default': False, 'name': 'SSD', 'root': 'default', 'type': 'host'} 2022-05-13 14:17:27,171 p=636835 u=stack n=ansible | 2022-05-13 14:17:27.148200 | 48d539a1-1679-730f-272a-0000000000a5 | TASK | set_fact info_ceph_default_crush_rule_yaml, ceph_osd_pool_default_crush_rule_name 2022-05-13 14:17:27,498 p=636835 u=stack n=ansible | 2022-05-13 14:17:27.497778 | 48d539a1-1679-730f-272a-0000000000a5 | OK | set_fact info_ceph_default_crush_rule_yaml, ceph_osd_pool_default_crush_rule_name | overcloud-controller-0 | item={'cmd': ['podman', 'run', '--rm', '--net=host', '-v', '/etc/ceph:/etc/ceph:z', '-v', '/var/lib/ceph/:/var/lib/ceph/:z', '-v', '/var/log/ceph/:/var/log/ceph/:z', '--entrypoint=ceph', '172.25.201.68:8787/ceph/daemon:v6.0.6-stable-6.0-pacific-centos-8-x86_64', '-n', 'client.admin', '-k', '/etc/ceph/ceph.client.admin.keyring', '--cluster', 'ceph', 'osd', 'crush', 'rule', 'dump', 'HDD', '--format=json'], 'start': '2022-05-13 06:17:26.297905', 'end': '2022-05-13 06:17:27.042420', 'delta': '0:00:00.744515', 'rc': 0, 'stdout': '\n{"rule_id":1,"rule_name":"HDD","ruleset":1,"type":1,"min_size":1,"max_size":10,"steps":[{"op":"take","item":-2,"item_name":"default~hdd"},{"op":"chooseleaf_firstn","num":0,"type":"host"},{"op":"emit"}]}', 'stderr': '', 'changed': False, 'invocation': {'module_args': {'name': 'HDD', 'cluster': 'ceph', 'state': 'info', 'rule_type': None, 'bucket_root': None, 'bucket_type': None, 'device_class': None, 'profile': None}}, 'stdout_lines': ['', '{"rule_id":1,"rule_name":"HDD","ruleset":1,"type":1,"min_size":1,"max_size":10,"steps":[{"op":"take","item":-2,"item_name":"default~hdd"},{"op":"chooseleaf_firstn","num":0,"type":"host"},{"op":"emit"}]}'], 'stderr_lines': [], 'failed': False, 'item': {'class': 'hdd', 'default': True, 'name': 'HDD', 'root': 'default', 'type': 'host'}, 'ansible_loop_var': 'item'} 2022-05-13 14:17:27,514 p=636835 u=stack n=ansible | 2022-05-13 14:17:27.514297 | 48d539a1-1679-730f-272a-0000000000a5 | SKIPPED | set_fact info_ceph_default_crush_rule_yaml, ceph_osd_pool_default_crush_rule_name | overcloud-controller-0 | item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': {'class': 'ssd', 'default': False, 'name': 'SSD', 'root': 'default', 'type': 'host'}, 'ansible_loop_var': 'item'} 2022-05-13 14:17:28,547 p=636835 u=stack n=ansible | 2022-05-13 14:17:28.534136 | 48d539a1-1679-730f-272a-0000000000a6 | TASK | insert new default crush rule into daemon to prevent restart 2022-05-13 14:17:29,371 p=636835 u=stack n=ansible | 2022-05-13 14:17:29.371419 | 48d539a1-1679-730f-272a-0000000000a6 | OK | insert new default crush rule into daemon to prevent restart | overcloud-controller-0 -> 172.25.201.106 | item=overcloud-controller-0 2022-05-13 14:17:29,373 p=636835 u=stack n=ansible | [WARNING]: ('overcloud-controller-0 -> 172.25.201.106', '48d539a1-1679-730f-272a-0000000000a6') missing from stats 2022-05-13 14:17:31,163 p=636835 u=stack n=ansible | 2022-05-13 14:17:31.162826 | 48d539a1-1679-730f-272a-0000000000a6 | OK | insert new default crush rule into daemon to prevent restart | overcloud-controller-0 -> 172.25.201.105 | item=overcloud-controller-1 2022-05-13 14:17:31,164 p=636835 u=stack n=ansible | [WARNING]: ('overcloud-controller-0 -> 172.25.201.105', '48d539a1-1679-730f-272a-0000000000a6') missing from stats 2022-05-13 14:17:32,108 p=636835 u=stack n=ansible | 2022-05-13 14:17:32.108129 | 48d539a1-1679-730f-272a-0000000000a6 | OK | insert new default crush rule into daemon to prevent restart | overcloud-controller-0 -> 172.25.201.97 | item=overcloud-controller-2 2022-05-13 14:17:32,110 p=636835 u=stack n=ansible | [WARNING]: ('overcloud-controller-0 -> 172.25.201.97', '48d539a1-1679-730f-272a-0000000000a6') missing from stats 2022-05-13 14:17:32,171 p=636835 u=stack n=ansible | 2022-05-13 14:17:32.171216 | 48d539a1-1679-730f-272a-0000000000b7 | TASK | Create pool(s) 2022-05-13 14:17:34,963 p=636835 u=stack n=ansible | 2022-05-13 14:17:34.962747 | 48d539a1-1679-730f-272a-0000000000b7 | OK | Create pool(s) | overcloud-controller-0 | item={'name': 'vms', 'pg_num': '16', 'rule_name': 'replicated_rule', 'application': 'rbd'} 2022-05-13 14:17:37,379 p=636835 u=stack n=ansible | 2022-05-13 14:17:37.378998 | 48d539a1-1679-730f-272a-0000000000b7 | OK | Create pool(s) | overcloud-controller-0 | item={'name': 'volumes', 'pg_num': '16', 'rule_name': 'replicated_rule', 'application': 'rbd'} 2022-05-13 14:17:39,756 p=636835 u=stack n=ansible | 2022-05-13 14:17:39.756549 | 48d539a1-1679-730f-272a-0000000000b7 | OK | Create pool(s) | overcloud-controller-0 | item={'name': 'images', 'pg_num': '16', 'rule_name': 'replicated_rule', 'application': 'rbd'} 2022-05-13 14:17:42,131 p=636835 u=stack n=ansible | 2022-05-13 14:17:42.131568 | 48d539a1-1679-730f-272a-0000000000b7 | OK | Create pool(s) | overcloud-controller-0 | item={'name': 'metrics', 'pg_num': '16', 'rule_name': 'replicated_rule', 'application': 'openstack_gnocchi'} 2022-05-13 14:17:44,583 p=636835 u=stack n=ansible | 2022-05-13 14:17:44.583069 | 48d539a1-1679-730f-272a-0000000000b7 | OK | Create pool(s) | overcloud-controller-0 | item={'name': 'backups', 'pg_num': '16', 'rule_name': 'replicated_rule', 'application': 'rbd'} 2022-05-13 14:17:46,815 p=636835 u=stack n=ansible | 2022-05-13 14:17:46.814659 | 48d539a1-1679-730f-272a-0000000000b7 | OK | Create pool(s) | overcloud-controller-0 | item={'application': 'rbd', 'name': 'slow_pool', 'rule_name': 'HDD'} 2022-05-13 14:17:49,239 p=636835 u=stack n=ansible | 2022-05-13 14:17:49.239270 | 48d539a1-1679-730f-272a-0000000000b7 | OK | Create pool(s) | overcloud-controller-0 | item={'application': 'rbd', 'name': 'fast_pool', 'rule_name': 'SSD'} 2022-05-13 14:17:49,363 p=636835 u=stack n=ansible | 2022-05-13 14:17:49.363314 | 48d539a1-1679-730f-272a-0000000000b9 | TASK | Get the RBD ceph_cli 2022-05-13 14:17:49,497 p=636835 u=stack n=ansible | 2022-05-13 14:17:49.497005 | 2b344252-9047-4c6a-974c-5254fe2e6831 | INCLUDED | /usr/share/ansible/roles/tripleo_cephadm/tasks/ceph_cli.yaml | overcloud-controller-0 2022-05-13 14:17:49,615 p=636835 u=stack n=ansible | 2022-05-13 14:17:49.615185 | 48d539a1-1679-730f-272a-00000000025d | TASK | Set ceph CLI 2022-05-13 14:17:49,702 p=636835 u=stack n=ansible | 2022-05-13 14:17:49.701696 | 48d539a1-1679-730f-272a-00000000025d | OK | Set ceph CLI | overcloud-controller-0 2022-05-13 14:17:49,754 p=636835 u=stack n=ansible | 2022-05-13 14:17:49.754180 | 48d539a1-1679-730f-272a-0000000000ba | TASK | Set trash interval 2022-05-13 14:17:50,487 p=636835 u=stack n=ansible | 2022-05-13 14:17:50.486794 | 48d539a1-1679-730f-272a-0000000000ba | OK | Set trash interval | overcloud-controller-0 | item=volumes 2022-05-13 14:17:50,513 p=636835 u=stack n=ansible | 2022-05-13 14:17:50.512697 | 48d539a1-1679-730f-272a-0000000000c8 | TASK | Collect the host and build the resulting host list 2022-05-13 14:17:50,614 p=636835 u=stack n=ansible | 2022-05-13 14:17:50.614475 | 48d539a1-1679-730f-272a-0000000000c8 | SKIPPED | Collect the host and build the resulting host list | overcloud-controller-0 | item=overcloud-controller-0 2022-05-13 14:17:50,616 p=636835 u=stack n=ansible | 2022-05-13 14:17:50.616043 | 48d539a1-1679-730f-272a-0000000000c8 | SKIPPED | Collect the host and build the resulting host list | overcloud-controller-0 | item=overcloud-controller-1 2022-05-13 14:17:50,617 p=636835 u=stack n=ansible | 2022-05-13 14:17:50.617021 | 48d539a1-1679-730f-272a-0000000000c8 | SKIPPED | Collect the host and build the resulting host list | overcloud-controller-0 | item=overcloud-controller-2 2022-05-13 14:17:50,677 p=636835 u=stack n=ansible | 2022-05-13 14:17:50.677546 | 48d539a1-1679-730f-272a-0000000000c9 | TASK | Collect the host and build the resulting host list 2022-05-13 14:17:50,810 p=636835 u=stack n=ansible | 2022-05-13 14:17:50.809944 | 48d539a1-1679-730f-272a-0000000000c9 | OK | Collect the host and build the resulting host list | overcloud-controller-0 | item=overcloud-controller-0 2022-05-13 14:17:50,889 p=636835 u=stack n=ansible | 2022-05-13 14:17:50.888767 | 48d539a1-1679-730f-272a-0000000000c9 | OK | Collect the host and build the resulting host list | overcloud-controller-0 | item=overcloud-controller-1 2022-05-13 14:17:50,890 p=636835 u=stack n=ansible | 2022-05-13 14:17:50.890385 | 48d539a1-1679-730f-272a-0000000000c9 | OK | Collect the host and build the resulting host list | overcloud-controller-0 | item=overcloud-controller-2 2022-05-13 14:17:50,923 p=636835 u=stack n=ansible | 2022-05-13 14:17:50.922956 | 48d539a1-1679-730f-272a-0000000000ca | TASK | Get certificate from remote node if needed 2022-05-13 14:17:50,987 p=636835 u=stack n=ansible | 2022-05-13 14:17:50.986916 | 48d539a1-1679-730f-272a-0000000000ca | SKIPPED | Get certificate from remote node if needed | overcloud-controller-0 2022-05-13 14:17:51,022 p=636835 u=stack n=ansible | 2022-05-13 14:17:51.022353 | 48d539a1-1679-730f-272a-0000000000cb | TASK | Create the RGW Daemon spec definition 2022-05-13 14:17:51,668 p=636835 u=stack n=ansible | An exception occurred during task execution. To see the full traceback, use -vvv. The error was: ImportError: cannot import name 'ceph_spec' 2022-05-13 14:17:51,669 p=636835 u=stack n=ansible | 2022-05-13 14:17:51.669288 | 48d539a1-1679-730f-272a-0000000000cb | FATAL | Create the RGW Daemon spec definition | overcloud-controller-0 | error={"changed": false, "module_stderr": "Traceback (most recent call last):\n File \"\", line 102, in \n File \"\", line 94, in _ansiballz_main\n File \"\", line 40, in invoke_module\n File \"/usr/lib64/python3.6/runpy.py\", line 205, in run_module\n return _run_module_code(code, init_globals, run_name, mod_spec)\n File \"/usr/lib64/python3.6/runpy.py\", line 96, in _run_module_code\n mod_name, mod_spec, pkg_name, script_name)\n File \"/usr/lib64/python3.6/runpy.py\", line 85, in _run_code\n exec(code, run_globals)\n File \"/tmp/ansible_ceph_mkspec_payload_iwqyfs7g/ansible_ceph_mkspec_payload.zip/ansible/modules/ceph_mkspec.py\", line 24, in \nImportError: cannot import name 'ceph_spec'\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1} 2022-05-13 14:17:51,683 p=636835 u=stack n=ansible | PLAY RECAP ********************************************************************* 2022-05-13 14:17:51,683 p=636835 u=stack n=ansible | overcloud-controller-0 : ok=22 changed=1 unreachable=0 failed=1 skipped=27 rescued=0 ignored=0 2022-05-13 14:17:51,684 p=636835 u=stack n=ansible | 2022-05-13 14:17:51.684007 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Summary Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2022-05-13 14:17:51,684 p=636835 u=stack n=ansible | 2022-05-13 14:17:51.684240 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Total Tasks: 50 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2022-05-13 14:17:51,684 p=636835 u=stack n=ansible | 2022-05-13 14:17:51.684514 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Elapsed Time: 0:00:45.563925 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2022-05-13 14:17:51,684 p=636835 u=stack n=ansible | 2022-05-13 14:17:51.684678 | UUID | Info | Host | Task Name | Run Time 2022-05-13 14:17:51,684 p=636835 u=stack n=ansible | 2022-05-13 14:17:51.684849 | 48d539a1-1679-730f-272a-0000000000b7 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Create pool(s) | 17.08s 2022-05-13 14:17:51,685 p=636835 u=stack n=ansible | 2022-05-13 14:17:51.685012 | 48d539a1-1679-730f-272a-0000000000a6 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : insert new default crush rule into daemon to prevent restart | 3.57s 2022-05-13 14:17:51,685 p=636835 u=stack n=ansible | 2022-05-13 14:17:51.685177 | 48d539a1-1679-730f-272a-0000000000a3 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : create configured crush rules | 3.28s 2022-05-13 14:17:51,685 p=636835 u=stack n=ansible | 2022-05-13 14:17:51.685340 | 48d539a1-1679-730f-272a-00000000002b | SUMMARY | overcloud-controller-0 | generate tripleo_cephadm_assimilate_conf on bootstrap node | 2.56s 2022-05-13 14:17:51,685 p=636835 u=stack n=ansible | 2022-05-13 14:17:51.685518 | 48d539a1-1679-730f-272a-0000000000a4 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : get id for new default crush rule | 1.39s 2022-05-13 14:17:51,685 p=636835 u=stack n=ansible | 2022-05-13 14:17:51.685682 | 48d539a1-1679-730f-272a-00000000001b | SUMMARY | overcloud-controller-0 | tripleo_cephadm : List Ceph daemon instances on this host | 1.24s 2022-05-13 14:17:51,685 p=636835 u=stack n=ansible | 2022-05-13 14:17:51.685869 | 48d539a1-1679-730f-272a-000000000076 | SUMMARY | overcloud-controller-0 | Assimilate configuration from tripleo_cephadm_assimilate_conf | 1.04s 2022-05-13 14:17:51,686 p=636835 u=stack n=ansible | 2022-05-13 14:17:51.686032 | 48d539a1-1679-730f-272a-000000000197 | SUMMARY | overcloud-controller-0 | Gathering Facts | 0.95s 2022-05-13 14:17:51,686 p=636835 u=stack n=ansible | 2022-05-13 14:17:51.686195 | 48d539a1-1679-730f-272a-000000000026 | SUMMARY | overcloud-controller-0 | push tripleo_cephadm_spec to bootstrap node if spec file exsits | 0.88s 2022-05-13 14:17:51,686 p=636835 u=stack n=ansible | 2022-05-13 14:17:51.686354 | 48d539a1-1679-730f-272a-0000000000ba | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Set trash interval | 0.74s 2022-05-13 14:17:51,686 p=636835 u=stack n=ansible | 2022-05-13 14:17:51.686536 | 48d539a1-1679-730f-272a-000000000023 | SUMMARY | overcloud-controller-0 | Ensure tripleo_cephadm_config_home (e.g. /etc/ceph) exists | 0.72s 2022-05-13 14:17:51,686 p=636835 u=stack n=ansible | 2022-05-13 14:17:51.686703 | 48d539a1-1679-730f-272a-000000000019 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Stat cephadm file | 0.72s 2022-05-13 14:17:51,686 p=636835 u=stack n=ansible | 2022-05-13 14:17:51.686863 | 48d539a1-1679-730f-272a-0000000000cb | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Create the RGW Daemon spec definition | 0.65s 2022-05-13 14:17:51,687 p=636835 u=stack n=ansible | 2022-05-13 14:17:51.687025 | 48d539a1-1679-730f-272a-0000000001f3 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Set ceph CLI | 0.54s 2022-05-13 14:17:51,687 p=636835 u=stack n=ansible | 2022-05-13 14:17:51.687194 | 48d539a1-1679-730f-272a-000000000024 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Ensure specs directory exists | 0.39s 2022-05-13 14:17:51,687 p=636835 u=stack n=ansible | 2022-05-13 14:17:51.687359 | 48d539a1-1679-730f-272a-0000000000a5 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : set_fact info_ceph_default_crush_rule_yaml, ceph_osd_pool_default_crush_rule_name | 0.35s 2022-05-13 14:17:51,687 p=636835 u=stack n=ansible | 2022-05-13 14:17:51.687546 | 48d539a1-1679-730f-272a-00000000005b | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Run cephadm bootstrap | 0.23s 2022-05-13 14:17:51,687 p=636835 u=stack n=ansible | 2022-05-13 14:17:51.687715 | 48d539a1-1679-730f-272a-0000000000c9 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Collect the host and build the resulting host list | 0.21s 2022-05-13 14:17:51,687 p=636835 u=stack n=ansible | 2022-05-13 14:17:51.687889 | 48d539a1-1679-730f-272a-000000000020 | SUMMARY | overcloud-controller-0 | Set random tripleo_cephadm_fsid if no running ceph containers were found | 0.21s 2022-05-13 14:17:51,688 p=636835 u=stack n=ansible | 2022-05-13 14:17:51.688054 | 48d539a1-1679-730f-272a-000000000220 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Set ceph CLI | 0.20s 2022-05-13 14:17:51,688 p=636835 u=stack n=ansible | 2022-05-13 14:17:51.688209 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ End Summary Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2022-05-13 14:17:51,688 p=636835 u=stack n=ansible | 2022-05-13 14:17:51.688435 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ State Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2022-05-13 14:17:51,688 p=636835 u=stack n=ansible | 2022-05-13 14:17:51.688611 | ~~~~~~~~~~~~~~~~~~ Number of nodes which did not deploy successfully: 1 ~~~~~~~~~~~~~~~~~ 2022-05-13 14:17:51,688 p=636835 u=stack n=ansible | 2022-05-13 14:17:51.688772 | The following node(s) had failures: overcloud-controller-0 2022-05-13 14:17:51,689 p=636835 u=stack n=ansible | 2022-05-13 14:17:51.688943 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~